Konthee/dots-mocr-w4a16-nvfp4

dots.mocr W4A16 NVFP4

This repository is a NVIDIA Model Optimizer quantized derivative of:

dots-studio/dots.mocr

The checkpoint is intended for NVIDIA GPU deployment with runtimes that understand the ModelOpt Unified Hugging Face quantization format, especially vLLM.

This is a post-training-quantized model. Validate OCR/layout accuracy on your own document set before production use.

---

Quantization profile

Component	Precision / behavior
Language-model weights	NVFP4
Language-model activations	BF16 / FP16 (A16)
Vision tower	High precision / unquantized
Other multimodal components	High precision / unquantized
KV cache	Unquantized
ModelOpt qformat	w4a16_nvfp4
Export quantization algorithm	W4A16_NVFP4
PTQ language attention implementation	eager
PTQ vision attention implementation	sdpa

Intended trade-off

This variant quantizes language-model weights to NVFP4 but keeps language-model activations at 16-bit precision. It is less aggressive than W4A4 and is a useful accuracy-oriented baseline for OCR while still reducing weight memory.

ModelOpt's plain PTQ path for VLMs is used here intentionally.

The language model is quantized while the vision encoder and non-language multimodal components are kept at high precision.

No vision-quantization recipe is used.

---

Calibration / conversion

W4A16_NVFP4 is weight-only and does not require an activation calibration forward pass. ModelOpt hf_ptq.py still constructs a dataloader and consumes one preview batch before quantization, so this build supplies one tiny local JSONL sample (calib_size=1, batch_size=1) only for that CLI plumbing. It is not activation calibration. The build intentionally uses ModelOpt plain PTQ qformat mode so the VLM language model is quantized while the vision branch remains high precision.

Build settings:

• Base model: dots-studio/dots.mocr
• Calibration size: 256
• Calibration sequence length: 512
• Calibration batch size: 1
• KV-cache quantization: disabled
• FlashAttention Python package during PTQ: not required
• PTQ language attention backend: eager
• PTQ vision attention backend: sdpa
• PTQ uses a local staging copy of the dots.mocr custom model code so the upstream unconditional flash_attn import does not block model loading.

---

Build environment

The following versions were used to produce this checkpoint:

• PyTorch: 2.8.0+cu128
• Transformers: 4.57.6
• huggingface_hub: 0.36.2
• NVIDIA ModelOpt: 0.0.1.dev1+g87f7d1432

Source revisions:

• dots.mocr commit: 23f3e5612fb8066d4034d5ecfc8f33a9243533eb

• NVIDIA Model Optimizer commit: 87f7d1432f6dccffe67069c84b9a18877a35019d

---

vLLM deployment with Docker

A recent vLLM release can load ModelOpt NVFP4 and W4A16_NVFP4 checkpoints using:

modelopt_fp4

Start vLLM OpenAI server

If this Hugging Face repository is private:

export HF_TOKEN="hf_xxx"

Then run:

docker run --rm \
  --gpus all \
  --ipc=host \
  --shm-size=16g \
  -p 8000:8000 \
  -e HF_TOKEN \
  vllm/vllm-openai:latest \
  --model Konthee/dots-mocr-w4a16-nvfp4 \
  --quantization modelopt_fp4 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --chat-template-content-format string \
  --served-model-name model \
  --trust-remote-code

For a public repository, -e HF_TOKEN can be omitted.

Runtime note

For W4A16_NVFP4, vLLM uses the ModelOpt FP4 runtime path. Current vLLM documentation says auto backend selection resolves to Marlin for W4A16_NVFP4. With BF16 activations, recent vLLM versions can also use --linear-backend flashinfer_cutedsl; benchmark both on your RTX 5090 and keep the faster/stable option for your workload.

The PTQ staging checkpoint uses sdpa for the Transformers vision module so that flash-attn is not required during export. Recent vLLM releases have a native DotsOCRForCausalLM implementation, so this Transformers-only staging choice is not the vLLM attention backend.

It is generally better to let vLLM select the linear backend automatically first.

Do not force a backend unless you have benchmarked it on your specific GPU and vLLM version.

---

Check vLLM

curl http://localhost:8000/v1/models

Expected API endpoint:

http://localhost:8000/v1

---

Multimodal OpenAI API

dots.mocr should be used with the prompts provided by the upstream project for best document parsing behavior.

Images can be supplied as an HTTP URL or a base64 data URL.

Example request body:

{
  "model": "model",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,<BASE64_IMAGE>"
          }
        },
        {
          "type": "text",
          "text": "<DOTS_MOCR_PROMPT>"
        }
      ]
    }
  ],
  "temperature": 0,
  "max_tokens": 4096
}

For document parsing, use the prompt definitions provided by the upstream dots.mocr project rather than replacing them with a generic OCR prompt.

Useful upstream locations include:

dots_mocr/utils/prompts.py
demo/demo_vllm.py
dots_mocr/model/inference.py

---

Architecture / precision notes

This checkpoint intentionally uses the following structure:

Image
  |
  v
Vision tower
high precision
  |
  v
Multimodal projection / integration
high precision
  |
  v
Language model
W4A16 NVFP4
  |
  v
Output tokens

This allows the vision side of dots.mocr to remain at higher precision while reducing the memory / compute cost of the language model.

---

Accuracy considerations

Quantization can affect:

• recognized text
• reading order
• bounding boxes
• document layout
• table HTML
• mathematical formulas
• structured graphics
• SVG generation
• very small text
• old / degraded scans

For production use, compare this checkpoint against the original BF16 model on a representative validation set.

Recommended comparison:

BF16 original
    vs
W4A16 NVFP4
    vs
W4A4 NVFP4

Useful metrics include:

• character error rate
• word error rate
• exact match
• table structure accuracy
• layout / bounding-box accuracy
• end-to-end document parse accuracy

---

Benchmark results

Tested on 2026-09-23 against the original dots-studio/dots.mocr endpoint using the latest Runpod concurrency sweep (20260923T112115437972Z). This checkpoint is the W4A16 variant: language-model weights use NVFP4 and activations remain BF16/FP16.

Per-endpoint concurrency	Successful pages	Throughput (pages/s)	vs. FP16 throughput	p95 latency	CER vs. FP16	Agreement (1 − CER)
16	320/320 (100.00%)	0.902	+5.7%	24.138 s	0.5367%	99.4633%
32	319/320 (99.69%)	1.004	+2.2%	48.840 s	0.7256%	99.2744%
64	318/320 (99.38%)	0.920	+4.6%	101.977 s	7.9066%	92.0934%

Across the three disjoint 320-page sets, this model completed 957/960 requests (99.69%). The character-weighted CER was 3.0440% (96.9560% agreement) over successful FP16 reference outputs.

Test method and interpretation

• Each concurrency level used 320 unique page images sampled from 77 PDFs (3,226 pages) in assets/fixtures. The three concurrency levels used disjoint image sets to avoid vLLM prefix-cache reuse; within a level, all three model endpoints received the same encoded images.
• All three endpoints started together at the listed per-endpoint concurrency. Each was warmed with five synthetic pages. No automatic retries were used.
• Throughput is successful pages divided by endpoint wall time. p95 latency includes failed requests; timing includes network and inference, but excludes PDF rendering and image preprocessing.
• OCR text was concatenated in output order, normalized with Unicode NFC, and stripped of whitespace before CER calculation. CER is character-weighted against FP16 outputs for pages where FP16 succeeded. Missing or failed candidate output counts as a deletion.
• FP16 is a comparison reference, not human-verified ground truth. These agreement values do not establish absolute OCR accuracy or layout/bounding-box accuracy.
• At concurrency 32, all three endpoints had an InternalServerError on the same sample ID. At concurrency 64, this model had one InternalServerError and one MODEL_OUTPUT_TRUNCATED result; no retries were made.
• Concurrency-64 had higher CER on its own disjoint sample set. This result alone cannot establish that higher concurrency caused the difference.

For this run, W4A16 at concurrency 32 was the recommended balance: 99.2744% agreement and 1.004 pages/s (+2.2% vs. FP16 at the same concurrency). Concurrency 16 had the lowest CER and p95 latency for W4A16, with 0.902 pages/s. Consider validating on a representative, human-labeled OCR set before selecting a production configuration.

Hardware notes

Native NVFP4 acceleration is most relevant on NVIDIA hardware with native FP4 support, especially Blackwell-class GPUs.

Runtime behavior on other NVIDIA GPU generations depends on the kernels available in the installed vLLM version.

---

License

This repository is a quantized derivative of:

dots-studio/dots.mocr

The original model's license and usage terms continue to apply.

Review the upstream model repository before redistribution or production use.