Instructions to use Qwen/Qwen3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
AMD Developer Cloud
Local Apps Settings

vLLM

How to use Qwen/Qwen3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3-8B

SGLang

How to use Qwen/Qwen3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3-8B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3-8B
```

Cyrillic homoglyphs appearing in English model outputs

#31

by LinkaG - opened Mar 13

Discussion

LinkaG

Mar 13

Model outputs Cyrillic homoglyphs instead of Latin characters in English answers

Description

When evaluating Qwen-based models, we observed that the model sometimes outputs Cyrillic characters that visually resemble Latin letters in responses to English questions.

Visually the text looks correct, but in Unicode these are different code points. As a result, automatic evaluation methods (substring match, token recall, exact match) mark correct answers as incorrect.

This behavior leads to systematic evaluation errors.

Examples

Model: Qwen3-8B

Example 1

Question

What is Tsutomu Seki's occupation?

Expected answer

astronomer

Model output

Tsutomу Sekі is an amateur аstronomer

Character substitutions

Cyrillic	Latin
у (U+0443)	y (U+0079)
і (U+0456)	i (U+0069)
а (U+0430)	a (U+0061)

Example 2

Question

In what city was Jorge Betancourt born?

Expected answer

Matanzas

Model output

Matanzас

Substitution

Cyrillic	Latin
а (U+0430)	a (U+0061)
с (U+0441)	c (U+0063)

Example 3

Question

What genre is Culture?

Expected answer

roots reggae

Model output

Roots reggaе

Substitution

Cyrillic	Latin
е (U+0435)	e (U+0065)

Example 4

Question

Who is the father of Seol Chong?

Expected answer

Wonhyo

Model output

Wonhyо

Substitution

Cyrillic	Latin
о (U+043E)	o (U+006F)

Unicode homoglyphs observed

Common substitutions we observed:

Looks like	Cyrillic	Latin
a	U+0430	U+0061
e	U+0435	U+0065
o	U+043E	U+006F
c	U+0441	U+0063
p	U+0440	U+0070
x	U+0445	U+0078
y	U+0443	U+0079
i	U+0456	U+0069
A	U+0410	U+0041
B	U+0412	U+0042
E	U+0415	U+0045
K	U+041A	U+004B
M	U+041C	U+004D
H	U+041D	U+0048
O	U+041E	U+004F
P	U+0420	U+0050
C	U+0421	U+0043
T	U+0422	U+0054
X	U+0425	U+0058

Why this matters

Even though the answer is semantically correct, evaluation pipelines that rely on exact string matching fail.

Example:

astronomer
аstronomer

The first character differs:

a  U+0061 (Latin)
а  U+0430 (Cyrillic)

This leads to incorrect benchmark results.

Questions

Is this a known issue in Qwen tokenization or decoding?
Could this originate from training data containing Unicode homoglyphs?
Is there a recommended normalization step for evaluation?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment