maxwelhelp/llama.cpp-DFlash2-pascal6-optimized

llama.cpp DFlash2 P40 (Pascal) optimization

Speculative decoding (DFlash2 block-diffusion drafter) tuned specifically for NVIDIA Tesla P40 (Pascal, compute capability 6.1, no FP16 tensor cores).

Fork of ggml-org/llama.cpp (PR #27342 lineage) with DFlash2 support, optimized for the Pascal backend: int8 DP4A (not FP16), multi-stage quantization of the draft, and an adaptive margin that shortens the verify batch where the draft's selector is uncertain.

Mini-article:  Title placeholder -> 30.26 tok/s on Pascal P40 with 27B dense

---

What this repo is

A runnable, measured build of llama.cpp + DFlash2 speculative decoding on a single Tesla P40 (24 GB, sm_61). Everything in this repo's DFLASH2_RING_PORT.md is a real measurement on the P40; no other GPUs / cloud numbers are claimed.

Primary pairing tested:

• Target: Qwen3.8-27B-UD Qwen3.8-27B-UD-Q4_K_XL.gguf (qwen35 hybrid, Gated DeltaNet)
• Draft: DFlash2 Qwen3.8-27B-DFlash2-q4mix-self.gguf (our q4-mix quant)

---

Best command (hit it)

See run_best.sh. Summary:

./build-p40-ring/bin/llama-server \
  -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -md Qwen3.8-27B-DFlash2-q4mix-self.gguf \
  -ngl 999 -ngld 999 -c 8192 -b 512 -ub 512 -np 1 \
  --load-mode mlock --cache-ram 32768 --checkpoint-min-step 512 \
  -fa 1 -ctk q8_0 -ctv q8_0 -ctkd q8_0 -ctvd q8_0 --kv-unified \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 --spec-draft-n-min 1 --spec-draft-p-min 0.35 \
  --spec-draft-ctx 0 --temp 0 --jinja --reasoning off -bs \
  -lv 4 --host 0.0.0.0 --port 8080

Key flags:

• --spec-type draft-dflash - DFlash2 block-diffusion drafter
• --spec-draft-n-max 7 - sweep optimum (Test 26: 7=30.26 > 8=30.18 > 5=27.4 > 4=27.8)
• --spec-draft-p-min 0.35 - adaptive margin (top1-top2 on raw selector logits), cuts the chain where the selector is a coin flip. Shortens the verify batch.
• --spec-draft-n-min 1 - keep short chains alive
• -bs - backend (GPU) argmax sampling, no CPU logit readback
• -fa 1, -ctk/ctv q8_0, --kv-unified - flash attention + q8 KV
• --reasoning off - fastest mode; use on for real reasoning (see below)

To switch to the reasoning scenario replace --reasoning off with --reasoning on.

---

Measured results (Tesla P40, single GPU)

All numbers are our own runs, logged in DFLASH2_RING_PORT.md.

Code prompt, reasoning OFF, greedy

Draft quant	size	tok/s	acceptance	mean len
q4-mix (recommended)	1.2 GB	30.26	0.83	5.25
Q8 (lucebox)	2.0 GB	~30	0.85	5.27
Q2-hybrid (fc Q2, head)	0.96GB	29.0	0.80	5.17
Q2-all	0.87GB	27.1	0.76	4.69

Draft quant is q4-mix = backbone q4_0 + dflash.* heads q8_0 (protects the selector). q4-mix wins: same acceptance as Q8 with a smaller, faster draft; Q2 is slower because the 2-bit selector heads lose near-tie accuracy.

n_max sweep (q4-mix, code, reasoning OFF)

n_max	tok/s
4	27.8
5	27.4
7	30.26
8	30.18

n_max=7 is the optimum; adaptive margin already caps the useful chain at ~5.25.

Thinking scenario (code, reasoning ON, greedy)

• ours (DFlash2 q4-mix + margin): ~18 t/s with real reasoning tokens.
• Luce chain with real thinking: 13.3 t/s (their spec is killed by a budget hook), i.e. our port is faster than Luce on the reasoning scenario.

Reference (same 27B, code, reasoning OFF)

• our DFlash2 q4-mix + margin: 30.26; plain Luce chain: ~32; ngram-mod / ngram-map-k4v on top of DFlash2: 28.2 (drags us down, not used).

---

What was done / optimizations (P40 Pascal)

1. MMQ DP4A (int8), no FP16. Pascal has no FP16 tensor cores. ggml-cuda selects the MMQ DP4A kernel (__dp4a, cc >= 610) for quantized weights via (!fp16_mma_hardware_available(cc) || ...), compile arch 61. Draft and target both run the int8 matmul path. No FP16 path is used for quant weights.
2. Draft quantization pipeline (server/scripts/quantize_dflash_draft.py
   • llama-quantize):
   • q4-mix: backbone q4_0, dflash.* heads + conv q8_0 -> 1.2 GB, acceptance-neutral.
   • Q2 available only via llama-quantize (python-gguf has no K-quant write).
3. DFlash2 adaptive chain (margin) (common/speculative.cpp): raw selector logits top1-top2 margin vs a threshold from p_min; stops the chain when the pick is a coin flip. This is what shortens the verify batch and makes n_max=7 optimal.
4. DFlash2 is block-diffusion: builds one noise block (id_last + n_max masks) and decodes it in a single pass (draft_dflash::draft, one llama_decode) - same approach as upstream PR #27342.
5. Backend (GPU) argmax (-bs): no CPU-side logit readback during draft/verify.
6. Flash attention (-fa) + q8_0 KV for both models, unified KV.
7. Duplicate draft quant tooling and honest per-scenario comparison vs Luce (see DFLASH2_RING_PORT.md).

---

HF model

Our recommended draft (q4-mix, 1.2 GB):

• Filename: Qwen3.8-27B-DFlash2-q4mix-self.gguf
• HF repo: https://huggingface.co/maxwelhelp/llama.cpp-DFlash2-pascal6-optimized

Target: Qwen3.8-27B-UD-Q4_K_XL.gguf (Qwen3.8-27B-UD, qwen35 hybrid).

---

Building for P40

cmake -B build-p40-ring -DCMAKE_CUDA_ARCHITECTURES=61 -DLLAMA_CUDA=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build build-p40-ring --config Release -j

---

Notes / honesty

• No cloud numbers; all measurements are local on the P40 (see the log).
• ngram-mod / ngram-map-k4v did not help on top of DFlash2 and are not used.
• On the reasoning scenario our port already beats Luce; q4-mix + margin is the recommended release config.

---

License note: this is a private fork for experimental Pascal tuning; it wraps upstream llama.cpp (MIT) and DFlash2 draft weights (Apache-2.0 / z-lab).