ashxhart commited on
Commit
f8c3752
·
verified ·
1 Parent(s): 8d07d17

Add Mac memory chooser and reproducible demo prompt

Browse files
Files changed (1) hide show
  1. README.md +64 -0
README.md CHANGED
@@ -23,10 +23,12 @@ tags:
23
  - 2-bit
24
  ---
25
 
 
26
  <p align="center">
27
  <a href="https://qwen.ai/"><img src="qwen-logo.png" width="96" height="95" alt="Qwen"></a>
28
  </p>
29
 
 
30
  <p align="center">
31
  <img src="https://img.shields.io/badge/Qwen-Qwen3.8-615CED?style=for-the-badge&logo=qwen&logoColor=white" alt="Qwen Qwen3.8">
32
  <img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX">
@@ -34,12 +36,15 @@ tags:
34
  <img src="https://img.shields.io/badge/Vontra-oQ2-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oQ2">
35
  </p>
36
 
 
37
  <h1 align="center">Qwen3.8 Flash Next, MLX oQ2 with native MTP</h1>
38
 
 
39
  <p align="center">
40
  A sensitivity-aware mixed-precision MLX conversion of <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Qwen/Qwen3.8-Flash-Next</a>, quantised from the official BF16 checkpoint with its native MTP draft block preserved.
41
  </p>
42
 
 
43
  <p align="center">
44
  <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Original model</a> ·
45
  <a href="https://qwen.ai/blog?id=qwen3.8-flash-next">Qwen overview</a> ·
@@ -47,10 +52,13 @@ tags:
47
  <a href="LICENSE">Qwen Community License 1.0</a>
48
  </p>
49
 
 
50
  ## About this conversion
51
 
 
52
  This release uses a 2-bit affine base and assigns more precision to sensitive modules. The final allocation includes 3, 4, 5, 6, and 8-bit modules. The native Qwen4Exp MTP block uses the same mixed-precision approach and remains part of the checkpoint.
53
 
 
54
  | Item | Value |
55
  | --- | --- |
56
  | Repository | `Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP` |
@@ -68,41 +76,56 @@ This release uses a 2-bit affine base and assigns more precision to sensitive mo
68
  | Configured context | 262,144 tokens |
69
  | Architecture | `qwen4_exp` vision-language sparse MoE |
70
 
 
71
  The upstream tokenizer, chat template, image and video processor configuration, generation configuration, licence, and native MTP configuration are included.
72
 
 
73
  > [!IMPORTANT]
74
  > Use a runtime with explicit `qwen4_exp` and native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP entries during strict loading.
75
 
 
76
  ## Download and use
77
 
 
78
  ```bash
79
  python -m pip install --upgrade huggingface_hub
80
 
 
81
  hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP \
82
  --local-dir ./Qwen3.8-Flash-Next-MLX-oQ2-MTP
83
  ```
84
 
 
85
  Add the downloaded directory to a compatible oMLX model directory and refresh the model registry. Native MTP is optional. Keep it disabled by default for this release because the measured depth-three path was slightly slower than baseline.
86
 
 
87
  ## Apple M3 Studio performance
88
 
 
89
  The validation used three measured 512-token runs per mode. The table reports median decode throughput.
90
 
 
91
  | Runtime mode | Runs | Output per run | Median generation speed |
92
  | --- | ---: | ---: | ---: |
93
  | Native MTP disabled | 3 | 512 tokens | **26.1398 tokens/s** |
94
  | Native MTP enabled, depth 3 | 3 | 512 tokens | **25.9290 tokens/s** |
95
 
 
96
  Native MTP changed median throughput by **-0.81%** in this test. The telemetry sample accepted 33 of 57 draft proposals, an acceptance rate of **57.89%**. Each of the three paired MTP-off and MTP-on 512-token runs produced matching output hashes.
97
 
 
98
  Instruction following, factual recall, arithmetic, and coherent long-generation gates passed in both modes. MTP worked correctly, but it did not improve throughput on this checkpoint and test. The recommended default is therefore MTP disabled.
99
 
 
100
  The benchmark covers text generation. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions.
101
 
 
102
  ## Architecture
103
 
 
104
  Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.
105
 
 
106
  | Architecture detail | Upstream value |
107
  | --- | ---: |
108
  | Language-model parameters | 125B total / 6B active |
@@ -113,10 +136,13 @@ Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixtur
113
  | Routed / active experts | 512 / 10, plus 1 shared expert |
114
  | Native context | 262,144 tokens, extensible upstream to 1,000,000 |
115
 
 
116
  For upstream evaluations, intended use, safety guidance, and the full architecture discussion, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
117
 
 
118
  ## Conversion and validation
119
 
 
120
  - The converter read the official BF16 checkpoint directly.
121
  - The recipe used a 2-bit affine base with 746 mixed-precision module overrides.
122
  - Full checkpoint validation covered all 3,747 indexed tensors, all 15 shards, and all 76 native-MTP tensor entries.
@@ -124,18 +150,56 @@ For upstream evaluations, intended use, safety guidance, and the full architectu
124
  - Instruction, factual, arithmetic, and coherence checks passed in both modes.
125
  - Native-MTP telemetry captured the draft and acceptance counters.
126
 
 
127
  This is a community conversion, not an official Qwen release.
128
 
 
129
  ## Limitations
130
 
 
131
  - oQ2 is an extreme-compression format. Evaluate instruction following, reasoning, factual accuracy, and visual understanding on the intended workload before deployment.
132
  - Native MTP was 0.81% slower in the measured test. Acceptance alone does not guarantee a throughput gain.
133
  - The configured 262,144-token context does not guarantee that every Apple-silicon system can run that length within available unified memory.
134
  - This is an MLX checkpoint for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
135
  - Upstream model limitations and safety considerations still apply.
136
 
 
137
  ## Licence and attribution
138
 
 
139
  The upstream model is released under the **Qwen Community License 1.0**. The required licence text is included in this repository and should be reviewed before use or redistribution.
140
 
 
141
  Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by [Vontra](https://huggingface.co/Vontra).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  - 2-bit
24
  ---
25
 
26
+
27
  <p align="center">
28
  <a href="https://qwen.ai/"><img src="qwen-logo.png" width="96" height="95" alt="Qwen"></a>
29
  </p>
30
 
31
+
32
  <p align="center">
33
  <img src="https://img.shields.io/badge/Qwen-Qwen3.8-615CED?style=for-the-badge&logo=qwen&logoColor=white" alt="Qwen Qwen3.8">
34
  <img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX">
 
36
  <img src="https://img.shields.io/badge/Vontra-oQ2-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oQ2">
37
  </p>
38
 
39
+
40
  <h1 align="center">Qwen3.8 Flash Next, MLX oQ2 with native MTP</h1>
41
 
42
+
43
  <p align="center">
44
  A sensitivity-aware mixed-precision MLX conversion of <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Qwen/Qwen3.8-Flash-Next</a>, quantised from the official BF16 checkpoint with its native MTP draft block preserved.
45
  </p>
46
 
47
+
48
  <p align="center">
49
  <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Original model</a> ·
50
  <a href="https://qwen.ai/blog?id=qwen3.8-flash-next">Qwen overview</a> ·
 
52
  <a href="LICENSE">Qwen Community License 1.0</a>
53
  </p>
54
 
55
+
56
  ## About this conversion
57
 
58
+
59
  This release uses a 2-bit affine base and assigns more precision to sensitive modules. The final allocation includes 3, 4, 5, 6, and 8-bit modules. The native Qwen4Exp MTP block uses the same mixed-precision approach and remains part of the checkpoint.
60
 
61
+
62
  | Item | Value |
63
  | --- | --- |
64
  | Repository | `Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP` |
 
76
  | Configured context | 262,144 tokens |
77
  | Architecture | `qwen4_exp` vision-language sparse MoE |
78
 
79
+
80
  The upstream tokenizer, chat template, image and video processor configuration, generation configuration, licence, and native MTP configuration are included.
81
 
82
+
83
  > [!IMPORTANT]
84
  > Use a runtime with explicit `qwen4_exp` and native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP entries during strict loading.
85
 
86
+
87
  ## Download and use
88
 
89
+
90
  ```bash
91
  python -m pip install --upgrade huggingface_hub
92
 
93
+
94
  hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP \
95
  --local-dir ./Qwen3.8-Flash-Next-MLX-oQ2-MTP
96
  ```
97
 
98
+
99
  Add the downloaded directory to a compatible oMLX model directory and refresh the model registry. Native MTP is optional. Keep it disabled by default for this release because the measured depth-three path was slightly slower than baseline.
100
 
101
+
102
  ## Apple M3 Studio performance
103
 
104
+
105
  The validation used three measured 512-token runs per mode. The table reports median decode throughput.
106
 
107
+
108
  | Runtime mode | Runs | Output per run | Median generation speed |
109
  | --- | ---: | ---: | ---: |
110
  | Native MTP disabled | 3 | 512 tokens | **26.1398 tokens/s** |
111
  | Native MTP enabled, depth 3 | 3 | 512 tokens | **25.9290 tokens/s** |
112
 
113
+
114
  Native MTP changed median throughput by **-0.81%** in this test. The telemetry sample accepted 33 of 57 draft proposals, an acceptance rate of **57.89%**. Each of the three paired MTP-off and MTP-on 512-token runs produced matching output hashes.
115
 
116
+
117
  Instruction following, factual recall, arithmetic, and coherent long-generation gates passed in both modes. MTP worked correctly, but it did not improve throughput on this checkpoint and test. The recommended default is therefore MTP disabled.
118
 
119
+
120
  The benchmark covers text generation. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions.
121
 
122
+
123
  ## Architecture
124
 
125
+
126
  Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.
127
 
128
+
129
  | Architecture detail | Upstream value |
130
  | --- | ---: |
131
  | Language-model parameters | 125B total / 6B active |
 
136
  | Routed / active experts | 512 / 10, plus 1 shared expert |
137
  | Native context | 262,144 tokens, extensible upstream to 1,000,000 |
138
 
139
+
140
  For upstream evaluations, intended use, safety guidance, and the full architecture discussion, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
141
 
142
+
143
  ## Conversion and validation
144
 
145
+
146
  - The converter read the official BF16 checkpoint directly.
147
  - The recipe used a 2-bit affine base with 746 mixed-precision module overrides.
148
  - Full checkpoint validation covered all 3,747 indexed tensors, all 15 shards, and all 76 native-MTP tensor entries.
 
150
  - Instruction, factual, arithmetic, and coherence checks passed in both modes.
151
  - Native-MTP telemetry captured the draft and acceptance counters.
152
 
153
+
154
  This is a community conversion, not an official Qwen release.
155
 
156
+
157
  ## Limitations
158
 
159
+
160
  - oQ2 is an extreme-compression format. Evaluate instruction following, reasoning, factual accuracy, and visual understanding on the intended workload before deployment.
161
  - Native MTP was 0.81% slower in the measured test. Acceptance alone does not guarantee a throughput gain.
162
  - The configured 262,144-token context does not guarantee that every Apple-silicon system can run that length within available unified memory.
163
  - This is an MLX checkpoint for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
164
  - Upstream model limitations and safety considerations still apply.
165
 
166
+
167
  ## Licence and attribution
168
 
169
+
170
  The upstream model is released under the **Qwen Community License 1.0**. The required licence text is included in this repository and should be reviewed before use or redistribution.
171
 
172
+
173
  Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by [Vontra](https://huggingface.co/Vontra).
174
+
175
+
176
+
177
+ <!-- vontra-chooser-start -->
178
+ ## Choose for your Mac
179
+
180
+ [64GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-64gb-macs-6a9fefda17932216ec9ab457) · [128GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-128gb-macs-6a9ff0abd31bc9abbe7922d7) · [256GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-256gb-macs-6a9ff0ef9fed7c5bdca15e9b)
181
+
182
+ No measured memory tier is assigned here. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.
183
+
184
+ ### Runtime and evidence
185
+
186
+ The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.
187
+
188
+ ### Quick start and demo prompt
189
+
190
+ ```bash
191
+ hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP --local-dir ./models/Qwen3.8-Flash-Next-MLX-oQ2-MTP
192
+ ```
193
+
194
+ Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.
195
+
196
+ Try this in a new chat with a 128-token output limit:
197
+
198
+ ```text
199
+ Explain why the sky looks blue in three short sentences.
200
+ ```
201
+
202
+ This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.
203
+
204
+ [Follow Vontra for new Apple Silicon releases and fixes.](https://huggingface.co/Vontra)
205
+ <!-- vontra-chooser-end -->