TokForge NPU Packs
Collection
Snapdragon NPU packs for TokForge: the phone's NPU reads the prompt while your downloaded model writes the reply. • 11 items • Updated
Qwen3.5 2B (QNN NPU pack)
An NPU pack for Qwen3.5 2B in TokForge: the NPU of a Snapdragon phone reads long prompts, and the CPU writes every reply with the Qwen3.5 2B download you already have.
Published by TokForge for on-device inference.
| What it is | NPU prompt pack (for Qwen3.5 2B) |
| Format | QNN context binaries for the Hexagon NPU plus a small MNN graph, built with QAIRT 2.40; one folder per chip, about 1.4 GB each |
| Runs on | Snapdragon phones (Hexagon NPU), with the paired MNN model on the CPU |
| Phone class | Snapdragon 8 Elite Gen 5 or Snapdragon 8 Gen 3, with room for about 2.9 GB more memory while the model is loaded |
| Download | 2.82 GB (82 files) |
| Base model | Qwen/Qwen3.5-2B, through the MNN conversion taobao-mnn/Qwen3.5-2B-MNN |
| License | Apache-2.0 |
TokForge
- Website: https://tokforge.ai
- Discord: https://discord.gg/Acv3CBtfVm
- Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
- iOS TestFlight: https://testflight.apple.com/join/jnufjzRr
Runs on-device in the TokForge app.
Use it in TokForge
- Install TokForge from Google Play.
- On Android, open Models > Recommended > Chat and download Qwen3.5 2B. TokForge lists it on phones with 6 GB RAM or more.
- Get the pack: open Models > Browse and tap Get next to Qwen3.5 2B under NPU packs for your phone, or turn on Faster long prompts (NPU) in the model's settings when TokForge offers it. TokForge downloads only the folder for your chip, checks every file by sha256, and compares the NPU with the CPU on your phone before it uses the NPU.
- Load the model and chat as usual. Long prompts are read on the NPU; replies are written on the CPU.
TokForge reads the list of packs from darkmaniac7/TokForge-NPU-packs and downloads this pack at the commit named there. The Qualcomm runtime libraries are not in this repo; TokForge ships them inside the app.
Collections: TokForge NPU Packs.
Files
| File | Size | What it is |
|---|---|---|
Qwen3.5-2B-MNN-catalogpair/ |
2.82 GB in 81 files | The pack: MANIFEST.json plus one folder per chip (v81/, v75/), each with 37 QNN context binaries, qnn/llm.mnn, qnn/llm_config.json and config_qnn.json |
SHA256SUMS |
9.20 KB | sha256 of every file |
A phone downloads only MANIFEST.json and the folder for its own chip.
What is in it
QNN context binaries (the model's layers compiled for the Hexagon NPU) for two chips, plus the small MNN graph and configs that drive them. A phone downloads only the folder for its own chip. The pack is exported from the MNN conversion taobao-mnn/Qwen3.5-2B-MNN and only changes the runtime format for the phone's NPU.
| Chip | NPU | Folder | Download |
|---|---|---|---|
| Snapdragon 8 Elite Gen 5 (SM8850) | Hexagon V81 | Qwen3.5-2B-MNN-catalogpair/v81/ |
1.41 GB (1,412,771,405 bytes) |
| Snapdragon 8 Gen 3 (SM8650) | Hexagon V75 | Qwen3.5-2B-MNN-catalogpair/v75/ |
1.40 GB (1,403,626,469 bytes) |
Each chip folder holds 37 context binaries (qnn/graph0.bin to qnn/graph36.bin), qnn/llm.mnn (the MNN graph
that drives them), qnn/llm_config.json and config_qnn.json. Qwen3.5-2B-MNN-catalogpair/MANIFEST.json lists every file per chip
with its sha256 and size, and the exact download the pack pairs with. SHA256SUMS covers every file in this repo; its README.md line is for an earlier version of this card.
Pairs with one exact download
The pack works only on top of darkmaniac7/Qwen3.5-2B-MNN at commit
207d8671eeb7, whose model files are byte-identical to
taobao-mnn/Qwen3.5-2B-MNN. TokForge matches the two by file hashes:
| File in the paired download | bytes | sha256 |
|---|---|---|
| llm.mnn | 2,148,136 | 23df98f8b341b277365e0bbca025c1d192939e3d32d7f79776352c6f32e77960 |
| llm.mnn.weight | 1,176,647,702 | c93f71a2dbecf9328782bd38861656d8faa82e95e7f99607350074768a482054 |
| tokenizer.txt | 6,465,727 | 7e75de1f279a10b65bd9dc1a5207205cb8993823861c4c42bbbd74e48e1c23a4 |
With any other download the pack is not used. The pack has no weight file of its own: the NPU weights are inside the compiled graphs, and the token embeddings are read from the paired download's llm.mnn.weight.
What runs where
- NPU: the 37 compiled graphs, which hold the projections, the MLP blocks and most of the norms of all 24 layers.
- CPU, in the same prompt pass: the 18 linear-attention cores and the 6 full-attention cores with their
key/value cache, a few small norms (their constants are inside
qnn/llm.mnn) and the token embedding lookup. - CPU, with your download: the end of the prompt that does not fill a whole 128-token chunk, and every token of the reply. The pack has no output layer, so the reply always comes from the model you downloaded.
- Short prompts stay on the CPU. If the NPU stops during a reply, the CPU finishes it.
How it was made
Exported from the paired download's own int4 weights: MNN's loader dequantizes them, and MNN llmexport (TokForge's
fork of MNN) writes 8-bit per-channel symmetric weights with fp16 activations, feeds
the rotary tables as fp32 inputs and cuts the graph before the output layer. compilefornpu splits the 24 layers into
37 NPU graphs at 18 linear-attention breaks and 30 scalar breaks, each with a 128-row chunk shape and a 1-row shape.
The Qualcomm AI Engine Direct SDK from QAIRT 2.40.0.251030 compiles every graph into one context binary per
chip with the strict HTP options (optimization level 3, fp16 relaxed precision, high-precision sigmoid, advanced
activation fusion off, 8 MB VTCM, 4 HVX threads, weight sharing, and extended UDMA on V81 only, since V75 does not accept it).
The build stops on any difference between a graph's declared inputs and outputs and what QNN created, and every
context binary is read back against its declaration, on each chip. The NPU reads the prompt in 128-token chunks.
Source training weights were not edited.
Limitations
- Snapdragon 8 Elite Gen 5 and Snapdragon 8 Gen 3 only. On other chips TokForge reads prompts on the CPU.
- The NPU part needs about 2.9 GB of memory while the model is loaded; TokForge turns it on only when the phone has room for it. Measured on phones: 2.93 GB on the Snapdragon 8 Elite Gen 5, 2.90 GB on the Snapdragon 8 Gen 3.
- Runtime files for TokForge and MNN, not a Transformers checkpoint.
- The model can still get facts wrong. Check important answers.
License
Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0), as the original model and the MNN conversion. All source authors keep their copyrights; no endorsement is claimed.
The context binaries were compiled with Qualcomm's QAIRT SDK. The Qualcomm runtime libraries that run them are not in this repo; TokForge ships them inside the app under the Qualcomm AI Stack License.
Credits
- Qwen3.5-2B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
- The MNN conversion taobao-mnn/Qwen3.5-2B-MNN by the taobao-mnn team (Alibaba), whose int4 weights this pack is exported from. Used under the Apache-2.0 license.
- Exported with llmexport and run by MNN by Alibaba. Apache-2.0.
- Compiled for the Hexagon NPU with the Qualcomm AI Engine Direct SDK from the Qualcomm AI Runtime (QAIRT) 2.40.0.251030 by Qualcomm. This pack contains compiled model graphs only, no Qualcomm runtime libraries.