Instructions to use qda-lab/QDA-Gemma-4-31B-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use qda-lab/QDA-Gemma-4-31B-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="qda-lab/QDA-Gemma-4-31B-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("qda-lab/QDA-Gemma-4-31B-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("qda-lab/QDA-Gemma-4-31B-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use qda-lab/QDA-Gemma-4-31B-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "qda-lab/QDA-Gemma-4-31B-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qda-lab/QDA-Gemma-4-31B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/qda-lab/QDA-Gemma-4-31B-W4A16

SGLang

How to use qda-lab/QDA-Gemma-4-31B-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "qda-lab/QDA-Gemma-4-31B-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qda-lab/QDA-Gemma-4-31B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "qda-lab/QDA-Gemma-4-31B-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qda-lab/QDA-Gemma-4-31B-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use qda-lab/QDA-Gemma-4-31B-W4A16 with Docker Model Runner:
```
docker model run hf.co/qda-lab/QDA-Gemma-4-31B-W4A16
```

QDA-Gemma-4-31B-W4A16

4-bit weight quantization of google/gemma-4-31B-it. 2.4× smaller than BF16 (25.9 GB vs. 62.5 GB), on par with BF16 accuracy.

Evaluation

Benchmark	BF16 62.5 GB	QDA W4A16 25.9 GB	Google QAT W4A16 23.3 GB	ParoQuant 19.3 GB
AIME 2025	84.0	84.4	82.8	81.8
HMMT Feb 2026	77.1	78.5	74.0	75.3
GPQA-Diamond	85.7	85.3	83.3	84.4
IFBench	80.4	80.2	79.1	80.4
MMLU-Pro	86.9	86.4	86.3	86.2
LiveCodeBench	83.9	85.1	83.1	83.4
Average	83.0	83.3	81.4	81.9

Thinking mode, pass@1 averaged over 1–32 seeds. Same sampling, token caps and seeds for every model; the ParoQuant column ran on its own serving stack (vLLM 0.19.1 with a config override it needs to load, and the chat template shipped in its repo, which differs from Google's current template by one newline in the system turn).

Speed on one H100 80GB, 512 generated tokens per request (tok/s = generated tokens / wall time, mean of 2 runs): 95 tok/s at batch 1 and 1,101 to 1,251 at batch 16 (the two runs); ParoQuant 66 and 891 (both on vLLM 0.19.1).

Usage

Runs on NVIDIA H100, A100, RTX Pro 6000 and B200 GPUs.

vLLM:

vllm serve qda-lab/QDA-Gemma-4-31B-W4A16

SGLang:

python -m sglang.launch_server --model-path qda-lab/QDA-Gemma-4-31B-W4A16

How it was made

Weights are 4-bit integers in groups of 32, quantized with GPTQ; the group scales were then fine-tuned by distillation. The vision tower stays in BF16. The recipe was chosen by QDA's automated search.

Reproduce

hf download qda-lab/qda-recipe-inputs --repo-type dataset --include "QDA-Gemma-4-31B-W4A16/*" --local-dir inputs
qda build inputs/QDA-Gemma-4-31B-W4A16/gemma-4-31b-w4a16.yaml --set "devices=[0,1]"

Needs the QDA library (the qda command). Recipe: gemma-4-31b-w4a16.yaml; inputs (calibration sample and distillation data): QDA-Gemma-4-31B-W4A16/ in qda-lab/qda-recipe-inputs. Takes about 1 h on 2× H100 (measured). The rebuild matches this checkpoint's KL to BF16 within 5 %; weights are not bit-identical.

License

Apache-2.0, the same as the base model (Gemma 4 license).

Citation

TBD.

Downloads last month: 71

Safetensors

Model size

31B params

Tensor type

BF16

I32

Model tree for qda-lab/QDA-Gemma-4-31B-W4A16

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(363)

this model