performance is great
#5
by erichartford - opened
just wanna let you know:
with 100 concurrent requests, your quant gets 10k tok/s on 8x H200
with Qwen's official FP8, I get 8k tok/s with the same test. (no other changes)
So your FP8-dynamic is faster (in vLLM anyway)