Image-Text-to-Text
MLX
Safetensors
qwen4_exp
mlx-vlm
omlx
oq
mtp
speculative-decoding
qwen
qwen3.8
qwen4-exp
mixture-of-experts
vision-language
quantized
apple-silicon
2-bit
conversational
Instructions to use TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP") config = load_config("TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "TensorFold/Qwen3.8-Flash-Next-MLX-oQ2-MTP" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add Mac memory chooser and reproducible demo prompt
Browse files
README.md
CHANGED
|
@@ -23,10 +23,12 @@ tags:
|
|
| 23 |
- 2-bit
|
| 24 |
---
|
| 25 |
|
|
|
|
| 26 |
<p align="center">
|
| 27 |
<a href="https://qwen.ai/"><img src="qwen-logo.png" width="96" height="95" alt="Qwen"></a>
|
| 28 |
</p>
|
| 29 |
|
|
|
|
| 30 |
<p align="center">
|
| 31 |
<img src="https://img.shields.io/badge/Qwen-Qwen3.8-615CED?style=for-the-badge&logo=qwen&logoColor=white" alt="Qwen Qwen3.8">
|
| 32 |
<img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX">
|
|
@@ -34,12 +36,15 @@ tags:
|
|
| 34 |
<img src="https://img.shields.io/badge/Vontra-oQ2-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oQ2">
|
| 35 |
</p>
|
| 36 |
|
|
|
|
| 37 |
<h1 align="center">Qwen3.8 Flash Next, MLX oQ2 with native MTP</h1>
|
| 38 |
|
|
|
|
| 39 |
<p align="center">
|
| 40 |
A sensitivity-aware mixed-precision MLX conversion of <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Qwen/Qwen3.8-Flash-Next</a>, quantised from the official BF16 checkpoint with its native MTP draft block preserved.
|
| 41 |
</p>
|
| 42 |
|
|
|
|
| 43 |
<p align="center">
|
| 44 |
<a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Original model</a> ·
|
| 45 |
<a href="https://qwen.ai/blog?id=qwen3.8-flash-next">Qwen overview</a> ·
|
|
@@ -47,10 +52,13 @@ tags:
|
|
| 47 |
<a href="LICENSE">Qwen Community License 1.0</a>
|
| 48 |
</p>
|
| 49 |
|
|
|
|
| 50 |
## About this conversion
|
| 51 |
|
|
|
|
| 52 |
This release uses a 2-bit affine base and assigns more precision to sensitive modules. The final allocation includes 3, 4, 5, 6, and 8-bit modules. The native Qwen4Exp MTP block uses the same mixed-precision approach and remains part of the checkpoint.
|
| 53 |
|
|
|
|
| 54 |
| Item | Value |
|
| 55 |
| --- | --- |
|
| 56 |
| Repository | `Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP` |
|
|
@@ -68,41 +76,56 @@ This release uses a 2-bit affine base and assigns more precision to sensitive mo
|
|
| 68 |
| Configured context | 262,144 tokens |
|
| 69 |
| Architecture | `qwen4_exp` vision-language sparse MoE |
|
| 70 |
|
|
|
|
| 71 |
The upstream tokenizer, chat template, image and video processor configuration, generation configuration, licence, and native MTP configuration are included.
|
| 72 |
|
|
|
|
| 73 |
> [!IMPORTANT]
|
| 74 |
> Use a runtime with explicit `qwen4_exp` and native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP entries during strict loading.
|
| 75 |
|
|
|
|
| 76 |
## Download and use
|
| 77 |
|
|
|
|
| 78 |
```bash
|
| 79 |
python -m pip install --upgrade huggingface_hub
|
| 80 |
|
|
|
|
| 81 |
hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP \
|
| 82 |
--local-dir ./Qwen3.8-Flash-Next-MLX-oQ2-MTP
|
| 83 |
```
|
| 84 |
|
|
|
|
| 85 |
Add the downloaded directory to a compatible oMLX model directory and refresh the model registry. Native MTP is optional. Keep it disabled by default for this release because the measured depth-three path was slightly slower than baseline.
|
| 86 |
|
|
|
|
| 87 |
## Apple M3 Studio performance
|
| 88 |
|
|
|
|
| 89 |
The validation used three measured 512-token runs per mode. The table reports median decode throughput.
|
| 90 |
|
|
|
|
| 91 |
| Runtime mode | Runs | Output per run | Median generation speed |
|
| 92 |
| --- | ---: | ---: | ---: |
|
| 93 |
| Native MTP disabled | 3 | 512 tokens | **26.1398 tokens/s** |
|
| 94 |
| Native MTP enabled, depth 3 | 3 | 512 tokens | **25.9290 tokens/s** |
|
| 95 |
|
|
|
|
| 96 |
Native MTP changed median throughput by **-0.81%** in this test. The telemetry sample accepted 33 of 57 draft proposals, an acceptance rate of **57.89%**. Each of the three paired MTP-off and MTP-on 512-token runs produced matching output hashes.
|
| 97 |
|
|
|
|
| 98 |
Instruction following, factual recall, arithmetic, and coherent long-generation gates passed in both modes. MTP worked correctly, but it did not improve throughput on this checkpoint and test. The recommended default is therefore MTP disabled.
|
| 99 |
|
|
|
|
| 100 |
The benchmark covers text generation. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions.
|
| 101 |
|
|
|
|
| 102 |
## Architecture
|
| 103 |
|
|
|
|
| 104 |
Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.
|
| 105 |
|
|
|
|
| 106 |
| Architecture detail | Upstream value |
|
| 107 |
| --- | ---: |
|
| 108 |
| Language-model parameters | 125B total / 6B active |
|
|
@@ -113,10 +136,13 @@ Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixtur
|
|
| 113 |
| Routed / active experts | 512 / 10, plus 1 shared expert |
|
| 114 |
| Native context | 262,144 tokens, extensible upstream to 1,000,000 |
|
| 115 |
|
|
|
|
| 116 |
For upstream evaluations, intended use, safety guidance, and the full architecture discussion, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
| 117 |
|
|
|
|
| 118 |
## Conversion and validation
|
| 119 |
|
|
|
|
| 120 |
- The converter read the official BF16 checkpoint directly.
|
| 121 |
- The recipe used a 2-bit affine base with 746 mixed-precision module overrides.
|
| 122 |
- Full checkpoint validation covered all 3,747 indexed tensors, all 15 shards, and all 76 native-MTP tensor entries.
|
|
@@ -124,18 +150,56 @@ For upstream evaluations, intended use, safety guidance, and the full architectu
|
|
| 124 |
- Instruction, factual, arithmetic, and coherence checks passed in both modes.
|
| 125 |
- Native-MTP telemetry captured the draft and acceptance counters.
|
| 126 |
|
|
|
|
| 127 |
This is a community conversion, not an official Qwen release.
|
| 128 |
|
|
|
|
| 129 |
## Limitations
|
| 130 |
|
|
|
|
| 131 |
- oQ2 is an extreme-compression format. Evaluate instruction following, reasoning, factual accuracy, and visual understanding on the intended workload before deployment.
|
| 132 |
- Native MTP was 0.81% slower in the measured test. Acceptance alone does not guarantee a throughput gain.
|
| 133 |
- The configured 262,144-token context does not guarantee that every Apple-silicon system can run that length within available unified memory.
|
| 134 |
- This is an MLX checkpoint for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
|
| 135 |
- Upstream model limitations and safety considerations still apply.
|
| 136 |
|
|
|
|
| 137 |
## Licence and attribution
|
| 138 |
|
|
|
|
| 139 |
The upstream model is released under the **Qwen Community License 1.0**. The required licence text is included in this repository and should be reviewed before use or redistribution.
|
| 140 |
|
|
|
|
| 141 |
Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by [Vontra](https://huggingface.co/Vontra).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
- 2-bit
|
| 24 |
---
|
| 25 |
|
| 26 |
+
|
| 27 |
<p align="center">
|
| 28 |
<a href="https://qwen.ai/"><img src="qwen-logo.png" width="96" height="95" alt="Qwen"></a>
|
| 29 |
</p>
|
| 30 |
|
| 31 |
+
|
| 32 |
<p align="center">
|
| 33 |
<img src="https://img.shields.io/badge/Qwen-Qwen3.8-615CED?style=for-the-badge&logo=qwen&logoColor=white" alt="Qwen Qwen3.8">
|
| 34 |
<img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX">
|
|
|
|
| 36 |
<img src="https://img.shields.io/badge/Vontra-oQ2-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oQ2">
|
| 37 |
</p>
|
| 38 |
|
| 39 |
+
|
| 40 |
<h1 align="center">Qwen3.8 Flash Next, MLX oQ2 with native MTP</h1>
|
| 41 |
|
| 42 |
+
|
| 43 |
<p align="center">
|
| 44 |
A sensitivity-aware mixed-precision MLX conversion of <a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Qwen/Qwen3.8-Flash-Next</a>, quantised from the official BF16 checkpoint with its native MTP draft block preserved.
|
| 45 |
</p>
|
| 46 |
|
| 47 |
+
|
| 48 |
<p align="center">
|
| 49 |
<a href="https://huggingface.co/Qwen/Qwen3.8-Flash-Next">Original model</a> ·
|
| 50 |
<a href="https://qwen.ai/blog?id=qwen3.8-flash-next">Qwen overview</a> ·
|
|
|
|
| 52 |
<a href="LICENSE">Qwen Community License 1.0</a>
|
| 53 |
</p>
|
| 54 |
|
| 55 |
+
|
| 56 |
## About this conversion
|
| 57 |
|
| 58 |
+
|
| 59 |
This release uses a 2-bit affine base and assigns more precision to sensitive modules. The final allocation includes 3, 4, 5, 6, and 8-bit modules. The native Qwen4Exp MTP block uses the same mixed-precision approach and remains part of the checkpoint.
|
| 60 |
|
| 61 |
+
|
| 62 |
| Item | Value |
|
| 63 |
| --- | --- |
|
| 64 |
| Repository | `Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP` |
|
|
|
|
| 76 |
| Configured context | 262,144 tokens |
|
| 77 |
| Architecture | `qwen4_exp` vision-language sparse MoE |
|
| 78 |
|
| 79 |
+
|
| 80 |
The upstream tokenizer, chat template, image and video processor configuration, generation configuration, licence, and native MTP configuration are included.
|
| 81 |
|
| 82 |
+
|
| 83 |
> [!IMPORTANT]
|
| 84 |
> Use a runtime with explicit `qwen4_exp` and native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP entries during strict loading.
|
| 85 |
|
| 86 |
+
|
| 87 |
## Download and use
|
| 88 |
|
| 89 |
+
|
| 90 |
```bash
|
| 91 |
python -m pip install --upgrade huggingface_hub
|
| 92 |
|
| 93 |
+
|
| 94 |
hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP \
|
| 95 |
--local-dir ./Qwen3.8-Flash-Next-MLX-oQ2-MTP
|
| 96 |
```
|
| 97 |
|
| 98 |
+
|
| 99 |
Add the downloaded directory to a compatible oMLX model directory and refresh the model registry. Native MTP is optional. Keep it disabled by default for this release because the measured depth-three path was slightly slower than baseline.
|
| 100 |
|
| 101 |
+
|
| 102 |
## Apple M3 Studio performance
|
| 103 |
|
| 104 |
+
|
| 105 |
The validation used three measured 512-token runs per mode. The table reports median decode throughput.
|
| 106 |
|
| 107 |
+
|
| 108 |
| Runtime mode | Runs | Output per run | Median generation speed |
|
| 109 |
| --- | ---: | ---: | ---: |
|
| 110 |
| Native MTP disabled | 3 | 512 tokens | **26.1398 tokens/s** |
|
| 111 |
| Native MTP enabled, depth 3 | 3 | 512 tokens | **25.9290 tokens/s** |
|
| 112 |
|
| 113 |
+
|
| 114 |
Native MTP changed median throughput by **-0.81%** in this test. The telemetry sample accepted 33 of 57 draft proposals, an acceptance rate of **57.89%**. Each of the three paired MTP-off and MTP-on 512-token runs produced matching output hashes.
|
| 115 |
|
| 116 |
+
|
| 117 |
Instruction following, factual recall, arithmetic, and coherent long-generation gates passed in both modes. MTP worked correctly, but it did not improve throughput on this checkpoint and test. The recommended default is therefore MTP disabled.
|
| 118 |
|
| 119 |
+
|
| 120 |
The benchmark covers text generation. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions.
|
| 121 |
|
| 122 |
+
|
| 123 |
## Architecture
|
| 124 |
|
| 125 |
+
|
| 126 |
Qwen3.8 Flash Next combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.
|
| 127 |
|
| 128 |
+
|
| 129 |
| Architecture detail | Upstream value |
|
| 130 |
| --- | ---: |
|
| 131 |
| Language-model parameters | 125B total / 6B active |
|
|
|
|
| 136 |
| Routed / active experts | 512 / 10, plus 1 shared expert |
|
| 137 |
| Native context | 262,144 tokens, extensible upstream to 1,000,000 |
|
| 138 |
|
| 139 |
+
|
| 140 |
For upstream evaluations, intended use, safety guidance, and the full architecture discussion, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
|
| 141 |
|
| 142 |
+
|
| 143 |
## Conversion and validation
|
| 144 |
|
| 145 |
+
|
| 146 |
- The converter read the official BF16 checkpoint directly.
|
| 147 |
- The recipe used a 2-bit affine base with 746 mixed-precision module overrides.
|
| 148 |
- Full checkpoint validation covered all 3,747 indexed tensors, all 15 shards, and all 76 native-MTP tensor entries.
|
|
|
|
| 150 |
- Instruction, factual, arithmetic, and coherence checks passed in both modes.
|
| 151 |
- Native-MTP telemetry captured the draft and acceptance counters.
|
| 152 |
|
| 153 |
+
|
| 154 |
This is a community conversion, not an official Qwen release.
|
| 155 |
|
| 156 |
+
|
| 157 |
## Limitations
|
| 158 |
|
| 159 |
+
|
| 160 |
- oQ2 is an extreme-compression format. Evaluate instruction following, reasoning, factual accuracy, and visual understanding on the intended workload before deployment.
|
| 161 |
- Native MTP was 0.81% slower in the measured test. Acceptance alone does not guarantee a throughput gain.
|
| 162 |
- The configured 262,144-token context does not guarantee that every Apple-silicon system can run that length within available unified memory.
|
| 163 |
- This is an MLX checkpoint for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
|
| 164 |
- Upstream model limitations and safety considerations still apply.
|
| 165 |
|
| 166 |
+
|
| 167 |
## Licence and attribution
|
| 168 |
|
| 169 |
+
|
| 170 |
The upstream model is released under the **Qwen Community License 1.0**. The required licence text is included in this repository and should be reviewed before use or redistribution.
|
| 171 |
|
| 172 |
+
|
| 173 |
Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by [Vontra](https://huggingface.co/Vontra).
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
<!-- vontra-chooser-start -->
|
| 178 |
+
## Choose for your Mac
|
| 179 |
+
|
| 180 |
+
[64GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-64gb-macs-6a9fefda17932216ec9ab457) · [128GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-128gb-macs-6a9ff0abd31bc9abbe7922d7) · [256GB Macs](https://huggingface.co/collections/Vontra/mlx-models-for-256gb-macs-6a9ff0ef9fed7c5bdca15e9b)
|
| 181 |
+
|
| 182 |
+
No measured memory tier is assigned here. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.
|
| 183 |
+
|
| 184 |
+
### Runtime and evidence
|
| 185 |
+
|
| 186 |
+
The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.
|
| 187 |
+
|
| 188 |
+
### Quick start and demo prompt
|
| 189 |
+
|
| 190 |
+
```bash
|
| 191 |
+
hf download Vontra/Qwen3.8-Flash-Next-MLX-oQ2-MTP --local-dir ./models/Qwen3.8-Flash-Next-MLX-oQ2-MTP
|
| 192 |
+
```
|
| 193 |
+
|
| 194 |
+
Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.
|
| 195 |
+
|
| 196 |
+
Try this in a new chat with a 128-token output limit:
|
| 197 |
+
|
| 198 |
+
```text
|
| 199 |
+
Explain why the sky looks blue in three short sentences.
|
| 200 |
+
```
|
| 201 |
+
|
| 202 |
+
This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.
|
| 203 |
+
|
| 204 |
+
[Follow Vontra for new Apple Silicon releases and fixes.](https://huggingface.co/Vontra)
|
| 205 |
+
<!-- vontra-chooser-end -->
|