2026-11-10T03:43:03.248556
image/svg+xml
Matplotlib v3.11.2, https://matplotlib.org/
GS1-0.8B
GS1-2B
GS1-4B
0
20
40
60
80
Complete decision latency (ms)
54.1
53.9
72.9
54.6
54.3
73.8
12.5
18.6
35.2
256-token budget · 1 question
GS1-0.8B
GS1-2B
GS1-4B
0
25
50
75
100
125
Complete decision latency (ms)
56.2
57.7
105.2
56.9
58.1
118.5
23.6
42.3
86.3
1,024-token budget · 1 question
GS1-0.8B
GS1-2B
GS1-4B
0
200
400
600
Complete decision latency (ms)
422.9
422.7
566.2
436.5
434.1
589.2
70.0
121.2
240.4
256-token budget · 8 questions
GS1-0.8B
GS1-2B
GS1-4B
0
200
400
600
800
1000
Complete decision latency (ms)
436.8
443.5
673.4
450.3
459.8
939.5
178.4
347.9
754.3
1,024-token budget · 8 questions
GS1 inference latency · current release
RTX 4090 (48 GiB reported) · BF16 · 5 warmups + 30 calls per group
Bars: P50; whiskers: P95 · includes compilation + probabilities; excludes load, capture and HTTP
GS1 Default
GS1 eager
GS1 CUDA Graph