Instructions to use GSAI-ML/LLaDA-o with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use GSAI-ML/LLaDA-o with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="GSAI-ML/LLaDA-o")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("GSAI-ML/LLaDA-o", dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use GSAI-ML/LLaDA-o with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "GSAI-ML/LLaDA-o"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GSAI-ML/LLaDA-o",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/GSAI-ML/LLaDA-o

SGLang

How to use GSAI-ML/LLaDA-o with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "GSAI-ML/LLaDA-o" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GSAI-ML/LLaDA-o",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "GSAI-ML/LLaDA-o" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "GSAI-ML/LLaDA-o",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use GSAI-ML/LLaDA-o with Docker Model Runner:
```
docker model run hf.co/GSAI-ML/LLaDA-o
```

JSON Required for quantization

by VaLtEc-BoY - opened Mar 27

Discussion

VaLtEc-BoY

Mar 27

Hello !! would like to request the JSON file for QUANTIZATION of the LLaDA model for popularization in the HF community.

(malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0)

If possible I thank you!

https://huggingface.co/mradermacher/model_requests/discussions/2080

RichardErkhov

Mar 27

renaming llm_config.json -> config.json should fix it, and then it's upto llama cpp to support it or not

VaLtEc-BoY

Mar 27

Thank you very much if we can it would be very cool for the whole HF community!!if you can create your own gguf model in the future it would also be of great appreciation to the HF Open Source community.thank you very much for the instruction!

RichardErkhov

Mar 28

I sadly dont have enough time and motivation for my own training, I would be doing some heretics, but I have trouble with huggingface on my account and pew heretic code isnt very ready yet for my configuration, so maybe later...

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment