Instructions to use kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK")
model = AutoModelForMultimodalLM.from_pretrained("kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK

SGLang

How to use kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK with Docker Model Runner:
```
docker model run hf.co/kaivoss/diffusiongemma-26B-A4B-it-ALWAYS-THINK
```

Replace upstream model card with a short one describing this repo

by Akicou - opened Sep 19

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

+40

-264

Akicou

Sep 19

The README was a verbatim copy of Google's DiffusionGemma card, which does not say what this repo changes. Replaced with a short card covering:

weights are unmodified; the only intended difference is chat_template.jinja
what actually switches thinking on (the <|think|> system marker), and that pre-filling <|channel>thought in the generation prompt duplicates a token the model emits itself
the caveat that the marker is necessary but not sufficient: on 14 held-out prompts the base model still produced an empty thought block every time
the abandoned sriq-sft-v1.5 LoRA attempt, with the measured numbers and why it likely failed
two practical notes (do not merge the adapter; needs a transformers build with the DiffusionGemma classes)

See also the chat template fix in PR #2.

Replace upstream model card with a short one describing this repo3226f17f

kaivoss changed pull request status to merged Sep 19

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment