Hob-forge/Qwen-Drive-1.0-4B-GGUF

Qwen-Drive-1.0-4B GGUF

GGUF builds of the vision-language model in Qwen/Qwen-Drive-1.0-4B (Apache-2.0), for llama.cpp and anything built on it. There are eight sizes, from 8.4 GB down to 1.9 GB, plus the vision projector.

This is an independent conversion. It is not made or endorsed by the Qwen team or the llama.cpp project.

What is and isn't in these files. Qwen-Drive is a Qwen3.5-4B vision-language model with two extra modules: a BEV perception head (3D detection, occupancy, map segmentation) and a flow-matching Planning Expert that outputs driving trajectories. Only the vision-language model is here. It chats and answers questions about images, including driving scenes. The perception head and the planner are not llama.cpp models and are not included. This is not a driving system and must never control a vehicle.

Which file should I download?

The two numbers in the right-hand columns are measured against the BF16 GGUF over ~9k held-out tokens (see "How we checked"). Lower KL means closer to the original. Every file runs CPU-only. With a GPU, put as many layers on it as fit (-ngl), and llama.cpp keeps the rest in RAM.

File	Size	Fits in about (estimate)	Mean KL vs BF16	Same top token
Qwen-Drive-1.0-4B-BF16.gguf	8.4 GB	10 GB	0 (reference)	100%
Qwen-Drive-1.0-4B-Q8_0.gguf	4.5 GB	6 GB	0.0021	97.3%
Qwen-Drive-1.0-4B-Q6_K.gguf	3.5 GB	5 GB	0.0136	94.7%
Qwen-Drive-1.0-4B-Q5_K_M.gguf	3.1 GB	4.5 GB	0.0194	92.8%
Qwen-Drive-1.0-4B-Q4_K_M.gguf	2.7 GB	4 GB	0.0504	88.9%
Qwen-Drive-1.0-4B-IQ4_XS.gguf	2.5 GB	4 GB	0.0640	88.0%
Qwen-Drive-1.0-4B-Q3_K_M.gguf	2.3 GB	3.5 GB	0.1781	80.4%
Qwen-Drive-1.0-4B-Q2_K.gguf	1.9 GB	3 GB	0.4202	71.4%
mmproj-Qwen-Drive-1.0-4B-F16.gguf	0.67 GB	(add to any of the above for images)

Our pick:

• Q5_K_M, if you have 8 GB of RAM or more.
• Q4_K_M, on tighter machines.
• Q8_0, if you want it practically identical to the original.

Q3_K_M and Q2_K are for very small machines; expect visibly weaker answers. The "fits in about" column includes the vision projector and a modest context. Longer contexts need more.

Run it

llama.cpp (chat with images), current upstream; no patch needed:

llama-mtmd-cli -m Qwen-Drive-1.0-4B-Q5_K_M.gguf --mmproj mmproj-Qwen-Drive-1.0-4B-F16.gguf \
  --image street.jpg -p "Describe this driving scene. What should the driver watch out for?"

llama.cpp server (OpenAI-compatible, text + images):

llama-server -m Qwen-Drive-1.0-4B-Q5_K_M.gguf --mmproj mmproj-Qwen-Drive-1.0-4B-F16.gguf --jinja -c 16384

LM Studio: search Hob-forge/Qwen-Drive-1.0-4B-GGUF, pick a file from the table, and keep the mmproj next to it for images.

The model thinks before answering by default (Qwen3.5 template). To turn thinking off with llama-server, pass "chat_template_kwargs": {"enable_thinking": false} in the request.

How we made it

1. The upstream checkpoint stores the VLM under a vlm. prefix, with its config nested in vlm_config. We lifted both out unchanged, giving a plain Qwen3_5ForConditionalGeneration with 723 tensors. No weights were modified.
2. The config lists one MTP (multi-token prediction) layer, but the weights contain none, so we converted with --no-mtp.
3. We converted to BF16 with llama.cpp's convert_hf_to_gguf.py, then quantized from that BF16.
   • Every quant except Q8_0 uses an importance matrix computed on the BF16 over a general calibration text (129 × 512 tokens).
4. We built the vision projector with --mmproj, at F16.

How we checked

• Conversion vs the original. We ran the HF checkpoint in float32 on CPU over a 120-token chat and compared logits with the BF16 GGUF: top-1 agreement 119/120, mean KL 0.0002. The GGUF computes what the original computes.
• Quants vs BF16. We ran llama-perplexity --kl-divergence over ~9k tokens of held-out English technical text (not the calibration text). Those are the numbers in the table above. Same-top-token is the share of positions where the quant's most likely next token equals the BF16's.
• Vision. We ran a smoke test on Q4_K_M + mmproj: it correctly described a forest image (tree, hanging lanterns, leaf-shaped hammock).

Not tested:

• driving-benchmark accuracy;
• long contexts;
• GPU backends other than CPU;
• non-English prompts.

Please report anything odd in the Community tab.

Licence

Apache-2.0, same as the original. All credit for the model goes to the Qwen team: see the technical report and QwenLM/Qwen-Drive-1.0.

Made by Hob-forge: verified quants of models other people can't run yet.