garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream

Qwen3.8 Flash-Next NVFP4 - SSD Stream

Run Qwen3.8 Flash-Next with 48 GB less RAM. Keep the speed.

This is RadixArk/Qwen3.8-Flash-Next-NVFP4, prepared for sglang-ssd-stream. RadixArk produced the NVFP4 checkpoint from Qwen3.8 Flash-Next; this layout loads its 48 GB lookup table from a local SSD instead of occupying system RAM. The required data is fetched while the GPU is already working.

What this model is

Source checkpoint	RadixArk/Qwen3.8-Flash-Next-NVFP4
Exact source revision	7b719225242aacd3dbd3f9407468c2ee9a9d2594
Base architecture	Qwen/Qwen3.8-Flash-Next
Serving engine	SGLang with sglang-ssd-stream
Prepared download	About 140 GiB

The source checkpoint uses ModelOpt NVFP4 W4A4 for the routed experts. Its attention, shared experts, embeddings, LM head, vision components, and MTP weights remain BF16; the predictive lookup table is FP8.

This prepared checkpoint does not retrain, fine-tune, or requantize the model. It makes one serving-oriented layout change:

1. The 128 FP8 lookup-table shards are removed from the normal model load.
2. Their rows are stored in one read-only 51,200,245,760-byte SSD sidecar.
3. ssd-stream.json records its exact shape, source revision, and SHA-256.
4. The remaining model and native MTP weights are preserved from the source.

Sidecar SHA-256: b070f9644adf93794d8a1030584ab705809387e64396a9327a68fa3a3a6666b3

v0.3.0 runtime update

The v0.3.0 extension packages 17 runtime modules, including two required compatibility interfaces, in a hardware-selected, private SGLang runtime view, leaving the base SGLang installation untouched. This is a runtime update, not a model-weight change. On the RTX PRO 6000 Blackwell (96 GB), the default enables adaptive four/eight-token MTP, corrected sparse-index packing, ReplaySSM, shared scratch/capture-stream resources, and early draft-head initialization. Strict acceptance thresholds remain 1, SSM remains FP32, and KV remains FP8. Default context remains 131,072 tokens; DGX Spark and portable-profile arguments are unchanged.

Clean Ubuntu 24.04 installation and full-model checks passed on the RTX PRO 6000: SSD GPU smoke, five GPU packing tests, and 12 functional checks covering tools, images, reasoning, and retrieval at 120,081 and 255,681 prompt tokens. Both x86_64 and aarch64 wheels passed 65 CPU tests each; aarch64 hardware acceptance remains pending. The 131K run recorded 30 window switches in each direction and CUDA-graph replay for all 275 logged decode batches.

Measured v0.3.0 default profile

Same prepared model and RTX PRO 6000, 131,072-token context, strict acceptance, and 1,024-token completions. The control uses --no-rtx-optimizations:

Workload	Fixed-four tok/s	Default tok/s	Median change
List	148.40	151.71	+2.23%
Prose	166.52	155.89	-6.39%
Code	208.78	217.59	+4.22%
Reasoning	134.02	136.59	+1.92%

Results are mixed, not a universal speedup: prose regressed in this sample. Each arm/workload has six measured requests, excluding warmups. Outputs varied; these small synthetic samples are not a general quality evaluation. The final control ran in a separate supervised pause after a connection interruption. Full measurements, validation results, and artifact identity and the benchmark client are public. The older measurements below describe a different runtime.

Historical performance

September 2026: qualified RTX PRO adaptive-MTP results

A separate, versioned runtime source bundle contains the exact optimization overlay deployed on September 9, including corrected sparse-index packing, adaptive four/eight-token MTP windows, corrected ReplaySSM, shared scratch/capture resources, and early draft-head initialization. The model weights did not change.

Matched measurements on one RTX PRO 6000 Blackwell Workstation Edition (96 GB):

Workload	Fixed-four tok/s	Adaptive bundle tok/s	Median change
List	175.57	186.13	+6.01%
Prose	187.64	194.61	+3.71%
Code	238.76	282.12	+18.16%
Reasoning	163.35	163.97	+0.38%

Both arms used corrected packing, original weights, strict acceptance thresholds of 1, FP32 SSM, FP8 KV, CUDA graphs, and a 262,144-token context/pool. Each arm/workload has six measured 1,024-token completions: 48 total measurements, excluding 16 warmups. The source bundle includes file hashes, a readable patch, all timing records, input token IDs, synthetic prompts, and an offline verifier. The historical qualification passed 12 functional checks and five GPU tests.

This is a workload-specific code-generation gain, not a universal 18% speedup or a coding-quality claim. Outputs varied, and the small non-code differences remain sensitive to output and run order. One candidate code request measured 241.15 tok/s. The historical runtime used the original RadixArk checkpoint and legacy SSD integration, not a fresh installation of this prepared model page.

The historical production runtime also used FRSpec with a 65,536-entry vocabulary, draft-head NVFP4, and lowm; these are not in the v0.3.0 installer. v0.3.0 enables the ported runtime by default on the RTX PRO 6000, but does not reproduce the entire historical setup. 282.12 tok/s and +18.16% are historical bundle results, not claims for the new default. The published historical overlay must not be copied into arbitrary SGLang installations. No complete legacy base image is included. See the versioned publication for applicability, verification commands, and test limitations.

These results are a separate experiment and should not be compared directly with the earlier SSD-versus-RAM figures below.

Earlier RTX PRO 6000 Blackwell SSD-versus-RAM results

Matched tests on one RTX PRO 6000 Blackwell used native MTP, CUDA graphs, and a 1,024-token completion:

RTX PRO 6000 result	Value
GPU memory	96 GB
Normal lookup-table RAM	47.68 GiB
SSD Stream working RAM	About 64 MiB
RAM returned	About 47.6 GiB
Normal RAM-loaded speed	148.5-156.2 tok/s
SSD Stream speed	164.7 tok/s

The measured result is the point of SSD Stream: removing the table from RAM did not make it the decode bottleneck. Content-heavy requests dominated by unseen, random lookup rows measured 126-137 tok/s.

The earlier RTX acceptance suite (not v0.3.0 acceptance) also passed structured tool calls, unrelated images, a 120,043-token retrieval request, native MTP, CUDA graph replay, restart reuse, and a sustained alternating workload without swap, OOM, restart, or host-memory growth.

DGX Spark (experimental)

DGX Spark result	Value
Unified memory	128 GB
Normal lookup-table memory	47.68 GiB
SSD Stream working memory	About 64 MiB
Unified memory returned	About 47.6 GiB
Normal RAM-loaded speed	Pending hardware run
SSD Stream speed	Pending hardware run

The Linux aarch64 wheel and automatic GB10 profile are included. It pins SGLang's SM121 QSA implementation and starts with 262K context, BF16 KV, FP32 model state, native MTP 3/1/4, decode CUDA graphs, and one request at a time. The wheel and Rust io_uring reader pass the complete native ARM64 test suite. Full model startup, long-context behavior, and performance still need to be measured on the rented DGX Spark, so this profile remains experimental.

24-32 GB NVIDIA GPUs (experimental)

The same prepared checkpoint can run on a Linux x86_64 system with an RTX 3090, 4090, 5090, or another NVIDIA GPU with at least 24 GB of VRAM and compute capability 8.0 or newer.

Portable profile	Value
GPU memory	24 GB minimum
Host memory	128 GB recommended
SSD Stream working memory	About 64 MiB
Default context	16K
MTP and CUDA graphs	Disabled

The launcher keeps the non-MoE model core on the GPU and uses SGLang's grouped CPU offloader for the large expert blocks. It prefetches one expert block at a time while continuing to serve the separate 47.68 GiB lookup table from SSD. This is a compatibility profile and will be much slower than keeping the model on a 96 GB GPU because expert weights cross PCIe for every token.

A four-block offload simulation on the RTX PRO 6000 produced exact text and structured tool calls without OOM or restart. Complete 24 and 32 GB hardware acceptance and performance measurements remain pending.

Run it

Use Linux with one supported NVIDIA GPU, a local SSD, and a standard C++ compiler. The native SSD reader is prebuilt; no Rust toolchain is required. Install or update with:

curl -LsSf https://raw.githubusercontent.com/garnermccloud/sglang-ssd-stream/main/install.sh | sh
~/.local/bin/sglang-ssd-stream serve

The first launch downloads and verifies this snapshot, then starts SGLang's OpenAI-compatible API at http://127.0.0.1:30000/v1.

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next-NVFP4-SSD-Stream",
    "messages": [{"role": "user", "content": "Reply with: SSD Stream works"}],
    "max_tokens": 32
  }'

The CLI automatically selects the validated RTX PRO 6000 profile, experimental DGX Spark profile, or experimental grouped CPU-offload profile for a supported 24-32 GB x86_64 GPU.

To opt out of the RTX ports and use the old fixed-four runtime for rollback or comparison:

sglang-ssd-stream serve --no-rtx-optimizations

The RTX default context remains 131,072 tokens. An explicit sglang-ssd-stream serve --context 262144 override passed full-model retrieval with a 255,681-token prompt.

Updates

Rerun the installer above for software updates. The v0.3.0 installer is version-bound to the 0.3.0 wheel. It builds and checks a new virtual environment at its permanent path, then atomically promotes the launcher. Previous environments are preserved, and installation failures leave the old launcher in place. The C++ compiler prerequisite remains.

Software and model updates are separate. v0.3.0 does not require a model update. To explicitly download and validate a newer prepared snapshot:

sglang-ssd-stream update-model

The existing snapshot stays active until validation succeeds; old model files are not deleted automatically.

Model capabilities

• Text, image, and video input with text output
• Reasoning and structured tool calling
• Native MTP speculative decoding on the RTX PRO and DGX Spark profiles
• Up to 262K context in the source architecture
• OpenAI-compatible chat-completions API through SGLang

Why Hugging Face shows 68B parameters and 8-bit precision

Those automatic labels describe visible storage tensors, not this model's logical architecture or its primary weight precision.

• The 68B figure counts about 60.4B packed uint8 storage elements plus 8.0B BF16 elements in the Safetensors files. Each packed byte can hold two NVFP4 values.
• The 51.2B FP8 lookup-table elements live in the SSD sidecar, so Hugging Face's Safetensors scanner does not count them.
• The 8-bit label comes from those storage dtypes. The routed-expert weights are NVFP4 W4A4; selected model components remain BF16; the streamed table is FP8.

The source model describes the architecture as approximately 180B logical parameters in total. The 68B header is therefore not a comparable parameter count.

Technical details

The model performs 16 deterministic table lookups per generated token. Each FP8 row is 160 bytes. SSD Stream starts those reads before the model reaches the block that consumes them, allowing SSD I/O to overlap GPU computation.

The extension:

1. copies row IDs asynchronously into pinned host memory;
2. deduplicates the exact 4 KiB filesystem pages required by those rows;
3. submits concurrent reads through a Rust io_uring engine;
4. restores rows in their requested order;
5. converts staged FP8 data on a separate CUDA stream; and
6. synchronizes only when a read outlasts the overlapping GPU work.

It reserves a 32 MiB registered page pool and two 16 MiB pinned staging buffers. It does not reserve a multi-gigabyte private cache. Multimodal input and structured tools remain enabled on every profile. Native MTP and CUDA graph replay remain enabled when the model core fits the RTX PRO or DGX Spark memory; the grouped CPU-offload profile disables both while expert weights move between host memory and the GPU.

sglang-ssd-stream is an independent extension for SGLang. It is not an official SGLang component.

License and attribution

The model weights retain the source model's license and terms. See:

• Qwen/Qwen3.8-Flash-Next
• RadixArk/Qwen3.8-Flash-Next-NVFP4
• sglang-ssd-stream, Apache-2.0