Upload folder using huggingface_hub

README.md

@@ -0,0 +1,206 @@
+---
+license: apache-2.0
+language:
+  - en
+base_model: Qwen/Qwen3-1.7B-Base
+tags:
+  - decision-model
+  - candidate-scorer
+  - calibration
+  - selective-prediction
+  - on-device
+  - mlx
+  - onnx
+---
+
+# Decily — calibrated candidate-scorer decision model
+
+**Decily** scores a runtime-provided set of candidates against a `state` and a
+`question`, and returns a **calibrated probability distribution** over those
+candidates. It does not generate text. Decily is built for decisions where you
+want a probability you can threshold, not a sentence you have to parse.
+
+```python
+Decily.decide(
+    state="Our app crashes on startup after the latest update.",
+    question="What is the customer's intent?",
+    options=["technical", "billing", "shipping", "returns"],
+)  # -> {"technical": 0.9999, "billing": 0.0001, "returns": 0.0, "shipping": 0.0}
+# measured with the released weights (bf16, T=0.45)
+```
+
+## This repository
+
+**Apple MLX bf16 bundle** (3.47 GB) — backbone and scoring head in one
+safetensors file, the layout the standalone reference implementation
+(`mlx/decision_mlx.py` in [arczhi/decily](https://github.com/arczhi/decily))
+expects.
+
+```bash
+python mlx/decision_mlx.py --model-dir <this repo> \
+  --state "Our app crashes on startup after the latest update." \
+  --question "What is the customer's intent?" \
+  --options "technical,billing,shipping,returns" --temperature 0.45
+```
+
+## Model family
+
+This card covers all four released artifacts; they are the same model in
+different containers.
+
+| Variant | Format | Size | Target |
+|---|---|---|---|
+| **Decily-1.7B** | PyTorch bf16 safetensors + `config.json` | ~3.4 GB | server / GPU |
+| **Decily-MLX** | MLX bf16 safetensors | ~3.4 GB | Apple Silicon |
+| **Decily-MLX-4bit** | MLX 4-bit (group-64 affine) | 968 MB | on-device / low memory |
+| **Decily-ONNX-int8** | ONNX int8 (single file) | 1.66 GB | CPU / Windows / edge |
+
+## Model details
+
+- **Model type:** cross-encoder candidate scorer. The backbone encodes
+  `state + question + candidate` jointly; an attention-pooling layer over all
+  tokens feeds a LayerNorm/GELU MLP that outputs one logit per candidate.
+  Probabilities come from the softmax over the candidates you pass in.
+- **Backbone:** `Qwen/Qwen3-1.7B-Base` (1.72B parameters, Apache-2.0), with an
+  attention-pooling scoring head (`decision_head` recorded in `config.json`;
+  pool/head weights live in the checkpoint).
+- **Input limits (training configuration):** `state` ≤256 tokens, `question`
+  ≤96 tokens, each candidate ≤64 tokens, 2–16 candidates per call. Longer
+  inputs are truncated — for long documents, chunk the state and score chunks
+  separately (the cross-encoder logits are candidate-set independent, so
+  chunked shortlisting then re-ranking is lossless; see the repository's
+  two-stage inference utility).
+- **Precision:** bf16 (PyTorch/MLX), 4-bit affine group-64 (on-device),
+  int8 (ONNX).
+- **Temperature:** raw logits are over-confident out of distribution. Fitted
+  temperatures observed: ≈1.1 (in-domain), ≈0.45 on unseen label sets, ≈1.35 on
+  the zero-shot fair suite. Always report/calibrate the temperature for your
+  data (see Evaluation).
+
+## Training summary
+
+Full, reproducible recipe: [TRAINING.md](https://github.com/arczhi/decily/blob/main/TRAINING.md)
+in the training repository [arczhi/decily](https://github.com/arczhi/decily)
+(design rationale in `DESIGN.md`, all experiments and ablations in
+`TRAIN-REPORT.md`).
+
+| Stage | What | Notes |
+|---|---|---|
+| Teacher | 45-task Qwen3.5-2B decision model (LoRA, frozen backbone) | reference labeler |
+| SFT | Qwen3-1.7B LoRA on 24 task families (hard labels) | 3000 steps ≈1 h |
+| RLCD members | v1 / v2 LoRA + one full-FT member | belief calibration, abstention |
+| Distillation | 4-model probability-space ensemble → **single full-FT student** | KD T=2, α=0.5, +15% belief rows |
+| v5 (this model) | full fine-tune, 3000 steps, effective batch 16, lr 1e-5, 8-bit AdamW | ≈1.5 h on one RTX 5090 32 GB |
+
+Key recipe properties:
+
+- **Route B (explicit candidate scorer)** beats the LM-head-letter-logits route
+  (Route A) by **+10.6 pt** held-out accuracy in a controlled same-backbone test.
+- **Ensemble distillation:** the 4-model ensemble reaches NLL 1.455; the
+  distilled single model reaches **1.451** at 1× inference cost.
+- **Consumer hardware:** the whole pipeline trains on a single RTX 5090 32 GB
+  in well under a day.
+
+## Evaluation
+
+Protocol: accuracy / NLL / ECE after **temperature fitting** (raw T=1 numbers
+are over-confident). "In-task" = the 24 trained task families; "held-out" =
+unseen label sets; "fair suite" = 8 tasks unseen by both this model and the
+external baseline.
+
+| Evaluation | Decily (24 tasks) | decider-2b (95 tasks) |
+|---|---|---|
+| In-task, 24 task families (shared training tasks) | **0.863 / 0.390 / 0.034** | 0.811 / 0.453 / 0.032 |
+| Fair suite, zero-shot (8 tasks × 300) | 0.654 / 0.86 / 0.092 | **0.700 / 0.71 / 0.047** |
+| Held-out label sets (60/77-class intents, 1200) | 0.581 / 1.451 / 0.068 | — (scores include tasks it was trained on) |
+
+Selective prediction (unseen label sets): taking only the most-confident **5%**
+of predictions gives **93.3%** accuracy (an SFT baseline with the same protocol
+gives 79%); abstention thresholds can be calibrated to a target error rate
+(empirically 4.7% achieved at a 5% target).
+
+Honest reading of the numbers:
+
+- Decily leads the larger, 95-task decider-2b by **+5.2 pt** on tasks both were
+  trained on, at 1.7B vs 2B parameters and ~1/4 of the task coverage.
+- On tasks **neither** model has seen, decider-2b leads by 4.6 pt; the gap is
+  concentrated in knowledge-heavy tasks (sciq / pubmedqa / quality).
+- The zero-shot gap is an **efficiency** result, not a ceiling: with this
+  recipe, task coverage — not parameter count — is the documented main lever.
+  Scaling the 24-task mixture toward 60–95 tasks with the same pipeline is
+  expected to close and exceed the baseline (see `TRAINING.md`).
+
+## Uses
+
+**Direct use**
+
+- Classification / ranking / selection when the label set is known at runtime
+  and may change per call (intents, topics, sentiment, NLI-style relations,
+  multiple-choice answers, routing, moderation labels, tool selection).
+- Calibration-sensitive automation: threshold the probability, auto-handle the
+  confident head and escalate the rest to a human or a larger model.
+- On-device / private inference: 968 MB 4-bit build, no GPU required.
+- Synthetic decision data generation and reward/verifier scoring for LLM
+  pipelines (it returns probabilities, not text).
+
+**Out of scope**
+
+- Text generation, chat, summarization.
+- Open-ended label spaces (the candidate set must be provided at call time).
+- High-stakes decisions without a calibrated threshold and human review.
+- Very long inputs beyond the 256-token state window without chunking.
+
+## Quick start
+
+Apple Silicon (MLX) — the standalone reference implementation ships with the
+training repository (`mlx/decision_mlx.py`, pure `mlx.core`, no torch):
+
+```bash
+python mlx/decision_mlx.py --model-dir <Decily-MLX dir> \
+  --state "The invoice was paid twice..." \
+  --question "What is the customer's intent?" \
+  --options "billing,technical,shipping,returns" \
+  --temperature 0.45
+```
+
+4-bit on-device: load the quantized tower with `mlx_lm.load` and apply the
+attention-pooling head from `decision_head.safetensors` (both files are in the
+repository; `config.json` records the head spec).
+
+PyTorch: load `model.safetensors` + `config.json` with the training repository's
+model class; the checkpoint keeps backbone and head under their training key
+names (`model.*`, `pool.*`, `head.*`).
+
+## Limitations
+
+- **Task coverage:** trained on 24 task families; zero-shot behavior on unseen
+  domains trails a 95-task baseline (see Evaluation). Validate on your domain
+  before trusting the probabilities.
+- **Calibration drift:** the model needs temperature scaling; the fitted value
+  depends on the data distribution (≈0.45–1.35 in our measurements).
+- **Input truncation:** `state` is truncated at 256 tokens; long documents must
+  be chunked (importance within a chunk, then combine).
+- **Language:** training data is predominantly English (some multilingual
+  intent data); other languages are untested.
+- **No safety layer:** outputs are probabilities over the candidates you
+  provide; the model does not refuse or filter inputs. Do not expose it as an
+  autonomous decision-maker in safety-critical settings.
+- **Inherited biases:** as a fine-tune of Qwen3-1.7B-Base on public datasets,
+  it can reproduce biases present in those datasets (toxicity, sentiment and
+  bias-classification tasks were part of the mixture, which reduces but does
+  not eliminate this).
+
+## Environmental impact
+
+Training used a single RTX 5090 32 GB: teacher ≈4 h, SFT ≈1 h, members ≈1 h,
+distillation ≈1.5 h (plus data conversion). Total well under 10 GPU-hours; no
+model was trained more than once per stage.
+
+## Citation and acknowledgements
+
+- Backbone: [Qwen3-1.7B-Base](https://huggingface.co/Qwen/Qwen3-1.7B-Base) (Apache-2.0).
+- The Route B formulation and the calibration/abstention pipeline build on the
+  Decily line of work; the external baseline in the tables is
+  [Mapika/decider](https://github.com/Mapika/decider).
+- If you use Decily, please cite this model card and link the training
+  repository (`TRAINING.md`).

config.json

@@ -0,0 +1,44 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936,
+  "decision_head": {
+    "pool": "attention_full_sequence",
+    "head_hidden_mult": 4,
+    "roles": [
+      "state",
+      "question",
+      "candidate"
+    ],
+    "concat_order": [
+      "state",
+      "question",
+      "candidate"
+    ]
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89aea0a2520db6c10194f1439d1c48d09b93a44f8c2d4afe3715270e9c4225ee
+size 3474788164

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set content = message.content %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in message.content %}\n                {%- set content = message.content.split('</think>')[-1].lstrip('\\n') %}\n                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}