Qwen3-0.6B-int4-sym-g128-ov

Qwen/Qwen3-0.6B (revision c1899de) as an OpenVINO IR with int4 symmetric weights (group size 128, ratio 1.0), a form the OpenVINO GenAI NPU guide gives for LLMs on the Intel NPU.

The OpenVINO GenAI NPU guide says LLMs must be exported with --sym, --weight-format int4 or nf4, --group-size -1 or 128 and --ratio 1.0; OpenVINO/Qwen3-0.6B-int4-ov is int4_asym, group size 128, ratio 0.8 (its rt_info, the quantization metadata in the IR's .xml), so this folder was exported in that form (int4).

Export

optimum-cli export openvino --model <Qwen/Qwen3-0.6B folder> --task text-generation-with-past --weight-format int4 --sym --ratio 1.0 --group-size 128 <out folder>

The --model folder held the files of Qwen/Qwen3-0.6B at revision c1899de289a04d12100db370d81485cdf75e47ca, each checked against the Hub's sha256 (git blob sha1 for small files) before the export. Versions: optimum-intel 2.2.0, optimum 2.3.0, nncf 3.4.0, openvino 2026.4.1, openvino-tokenizers 2026.4.1.0, transformers 5.5.4, torch 2.14.0, Python 3.12.10, on Windows 11 Home. The IR's rt_info: mode int4_sym, group_size 128, ratio 1.0, awq False, scale_estimation False, backup_mode int8_asym.

Measured on one laptop's NPU

OpenVINO GenAI 2026.4.1 with LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>), no other NPU property. One user turn (edge-llm-bench's short-chat prompt) in the folder's chat template, 19 tokens; greedy, at most 128 new tokens. Each run is a fresh process that answers twice: cold = its first generation, warm = its second. The numbers below come from one measuring session on the laptop (three runs). Run 1 is not counted (the rule of windows-llm-zoo for NPU and GPU runs); runs 2 and 3 are. CACHE_DIR was the same folder in every run of these records.

laptop Intel Core Ultra 9 386H, Intel NPU driver 32.0.100.5540, Windows 11 Home build 26200.9711, 31.5 GB RAM (as Windows reports it), MSI Center's Balanced mode; 2026-11-11
decode, cold 73.2 tokens/s (median of 2 runs, spread 0.2 %)
decode, warm 73.8 tokens/s
prefill, cold 74.5 tokens/s, 255 ms to the first token
peak working set 1,536 MiB (the process)
load 0.9 s in run 1, 0.9, 0.9 s in runs 2 and 3 (all three after the first load below, with the same CACHE_DIR)
output check 6/6 PASS (edge-llm-bench's text check); new tokens per answer: 128, 128

On this driver the first load of this folder, with CACHE_DIR empty, took 9.4 s. It was in the answer-agreement run below (2026-11-11; load_s in run1.json).

The answer begins after its <think> line: "Okay, so the user is asking for an explanation of on-device AI in simple terms. Let me start by breaking down the term. โ€ฆ". At 128 new tokens every answer here ends inside the think block (no </think> in any of them).

Records: the three runs and the laptop's environment in records/2026-11-11/; the script: pc/run_ovgenai.py.

Answer agreement with the original model

On 20 short questions (thinking off), the first line of this export's answer matched Qwen/Qwen3-0.6B's (fp32, transformers, CPU) on 9. The pass mark in windows-model-zoo is 18. This measures agreement with the original model, not correctness. One run on the same NPU, 2026-11-11, driver 32.0.100.5540: record.

An int8 per-channel export of the same model, mlboydaisuke/Qwen3-0.6B-int8-sym-ov, matched on 18 of these 20 on the same NPU.

Facts in long prompts: on the NPU, this export got the fact right in as many prompts as on the CPU in f32 (154 of 160 each); the original model got it right in all 160. The 160 prompts (8 short facts ร— 5 places ร— 4 lengths: 256, 512, 768 and 1,000 tokens) put one fact into a passage (as its first sentence, about a quarter, half or three quarters in, or near the end) and ask about it at the end (thinking off); decoding is greedy. The NPU ran LLMPipeline(<this folder>, "NPU", CACHE_DIR=<dir>) as above (a second run, from the cache, gave the same answer tokens); the CPU ran the same IR in f32 (its properties are in the record); the original model is Qwen/Qwen3-0.6B in fp32 (transformers, CPU). Both devices missed the same 5 prompts, all at 768 or 1,000 tokens and on the two facts whose answer is a two-digit number, "The reading room has 38 chairs." and "The garden behind the building has 27 apple trees." (their answers: 28, 18 or 14 for 38; 18 for 27). The NPU missed one more (at 1,000 tokens) and the CPU in f32 one other (at 768 tokens), both on the 38 chairs. The int8 export above answered all 160 on the NPU. On the same laptop, 2026-11-12, NPU driver 32.0.100.5540: record (the answers and the prompts).

Downloads last month
208
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlboydaisuke/Qwen3-0.6B-int4-sym-g128-ov

Finetuned
Qwen/Qwen3-0.6B
Quantized
(577)
this model