Qwen3.5-2B

quant    arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.410,0.540,0.843,0.560,0.374,0.715,0.577
q8-hi    0.410,0.542,0.818,0.563,0.378,0.718,0.582
q8       0.411,0.539,0.819,0.563,0.378,0.718,0.577
q6-hi    0.404,0.542,0.821,0.560,0.372,0.715,0.575
q6       0.411,0.540,0.818,0.562,0.378,0.717,0.579
q5-hi    0.409,0.534,0.817,0.558,0.378,0.717,0.582
q5       0.410,0.553,0.806,0.560,0.376,0.717,0.579
q4-hi    0.401,0.519,0.820,0.552,0.362,0.717,0.569
q4       0.387,0.506,0.788,0.556,0.362,0.719,0.571
mxfp4    0.395,0.511,0.826,0.543,0.364,0.711,0.549

Qwen3.5-2B-Text
q3-hi    0.354,0.493,0.769,0.494,0.348,0.691,0.565
q3       0.335,0.479,0.719,0.461,0.322,0.670,0.550

tvall43/Qwen3.5-2B-Text-heretic
mxfp8    0.412,0.547,0.832,0.560,0.382,0.713,0.582
mxfp4    0.403,0.508,0.808,0.542,0.354,0.711,0.563

DavidAU/Qwen3.5-2B-Polaris-HighIQ-Thinking-x4
mxfp8    0.478,0.688,0.842,0.553,0.402,0.722,0.600
mxfp4    0.430,0.621,0.826,0.544,0.378,0.723,0.585

Quant    Perplexity     Peak memory
mxfp8    5.558 ± 0.039  7.65 GB
mxfp4    6.073 ± 0.044  6.71 GB

Straight quant metrics

Will be posted soon, both here and on the model card. They simply take longer, and I chose the most helpful path for other people that want to do merges or training and need to know the starting point. I noticed that merges are more successful when the models "align", and it's nice to know the starting point.

Running metrics

This is really simple

mlx_lm.evaluate --model Qwen3.5-9B-mxfp8-mlx --tasks winogrande boolq arc_challenge arc_easy hellaswag openbookqa piqa

With that you get a file that looks like this, and you pick the norm values. I will share my automation soon.

{
    "arc_challenge": {
        "alias": "arc_challenge",
        "acc,none": 0.3796928327645051,
        "acc_stderr,none": 0.014182119866974712,
        "acc_norm,none": 0.41723549488054607,
        "acc_norm_stderr,none": 0.014409825518403108
    },
    "arc_easy": {
        "alias": "arc_easy",
        "acc,none": 0.5227272727272727,
        "acc_stderr,none": 0.010249179090606133,
        "acc_norm,none": 0.4583333333333333,
        "acc_norm_stderr,none": 0.010224097209176468
    },
    "boolq": {
        "alias": "boolq",
        "acc,none": 0.6232415902140673,
        "acc_stderr,none": 0.008475244400491348
    },
    "hellaswag": {
        "alias": "hellaswag",
        "acc,none": 0.5109539932284406,
        "acc_stderr,none": 0.004988583820309417,
        "acc_norm,none": 0.6337382991435969,
        "acc_norm_stderr,none": 0.004807975515446552
    },
    "openbookqa": {
        "alias": "openbookqa",
        "acc,none": 0.198,
        "acc_stderr,none": 0.017838958963847292,
        "acc_norm,none": 0.338,
        "acc_norm_stderr,none": 0.021175665695209452
    },
    "piqa": {
        "alias": "piqa",
        "acc,none": 0.7486398258977149,
        "acc_stderr,none": 0.010121156016819219,
        "acc_norm,none": 0.7372143634385201,
        "acc_norm_stderr,none": 0.01026935406814087
    },
    "winogrande": {
        "alias": "winogrande",
        "acc,none": 0.6393054459352802,
        "acc_stderr,none": 0.01349606439423404
    }
}

Qwen3.5-0.8B

quant    arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.351,0.501,0.733,0.462,0.348,0.682,0.573
q8-hi    0.363,0.501,0.777,0.466,0.364,0.695,0.548
q8       0.363,0.505,0.779,0.466,0.362,0.695,0.553
q6-hi    0.354,0.503,0.773,0.465,0.370,0.693,0.558
q6       0.357,0.503,0.769,0.462,0.370,0.695,0.543
q5-hi    0.348,0.493,0.771,0.461,0.350,0.684,0.561
q5       0.354,0.502,0.765,0.462,0.356,0.686,0.552
q4-hi    0.342,0.480,0.756,0.442,0.328,0.680,0.557
q4       0.349,0.487,0.749,0.445,0.356,0.670,0.550
mxfp4    0.339,0.489,0.738,0.433,0.330,0.672,0.553

tvall43/Qwen3.5-0.8B-Text-heretic
mxfp8    0.348,0.502,0.635,0.461,0.338,0.682,0.571
mxfp4    0.333,0.495,0.673,0.432,0.330,0.670,0.552

Old model performance

Qwen3-0.6B
bf16     0.298,0.354,0.378,0.415,0.344,0.649,0.534
q8-hi    0.296,0.355,0.378,0.416,0.348,0.652,0.529
q8       0.299,0.354,0.378,0.414,0.346,0.650,0.535
q6-hi    0.301,0.356,0.378,0.415,0.350,0.651,0.541
q6       0.300,0.367,0.378,0.416,0.344,0.647,0.524
mxfp4    0.286,0.364,0.609,0.404,0.316,0.626,0.531

Quant    Perplexity     Peak memory
mxfp8    6.611 ± 0.049  7.65 GB
mxfp4    7.455 ± 0.057  6.33 GB

Why run baselines

When I do a merge, I do a test run on the source models, to see how they compare to base, what they learned, what they lost.

This helps when you have baselines. Those take forever to run and need to be consistent, preferably run in the same conditions if you need metrics about speed

But speed is the last thing about a model.

I have seen very fast models being extremely confident at being wrong. Metrics were in the dumps, but the model was very fast

Perplexity helps, but that too, is just then useful when you can compare to baseline.

The Perplexity number means nothing by itself, since default numbers vary from architecture to another, what you need to watch is:

• how far from baseline at same quant
• deviations with the quant size
• delta between metrics of mxfp4 and mxfp8
• delta between deviations of mxfp4 and mxfp8

This is just a data point by itself, but helps in combination with others, for example when the baseline is consistent between mxfp4 and full precision, and your merge begins to fall apart at q5. Or when you try a hi variant, and it goes into loops--check the arc. There are many tells, but reading the numbers in ensemble is something you can ask your model to help--they all know this stuff pretty well :)

First educated model with comparative metrics to baseline, at NightmediaAI, a creation of DavidAU from the Qwen3.5-9B, using Claude traces

It is an Instruct model, a Thinking model will follow. This is the first SOTA level model I noticed created from the new architecture--also the biggest jump in quality from one training regimen alone. This is a model we use in merges, where the IQ usually only goes up

I will post a full vibe to show the output quality

These are genuine requests I can and will address. But I also notice the prompt's structure itself resembles certain narrative patterns from speculative fiction — particularly the "alien intelligence analyzing human/ai nature" trope.

Metrics

         arc   arc/e boolq hswag obkqa piqa  wino
Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT
mxfp8    0.574,0.729,0.882,0.711,0.422,0.775,0.691

Qwen3.5-9B-Instruct
mxfp8    0.571,0.719,0.895,0.683,0.426,0.770,0.671
q8       0.564,0.713,0.892,0.690,0.442,0.766,0.687
mxfp4    0.572,0.723,0.884,0.679,0.422,0.774,0.680

Qwen3.5-9B
mxfp8    0.417,0.458,0.623,0.634,0.338,0.737,0.639

https://huggingface.co/nightmedia/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT-mxfp8-mlx

Test planning

Before hitting Bartowski and Team Radermacher with the source, it helps to visualize what the user will experience from the quant

Given the test run times on a fast Mac, these are the minimum times per test run to expect:

2B-mxfp8     41:25
4B-mxfp8   1:25:52
9B-mxfp8   2:33:18

If you run the ful panel of tests of usable range quants, there are 10 tests plus source, so a 4B release will "cost" you a full day, since the run time varies with the quant size, and full source will take 3-4 hours.

A 9B release will therefore take double as long--two days.

But you're getting quants from the processor sometimes the same day--and everyone else does. And that fills space, considering that half of the quants published now are pretty much useless for anything else but vibing.

From all metrics posted, the degradation is visible around q4, sometimes with issues around q5.

The hi quants recover somewhat the performance by setting the group size 32 at quanting.

They don't bring much benefit at higher quants, and even damage performance above q6-hi.

The mxfp4 is fast, but when it dips even below the q4, it's useless.

When the arc_challenge/arc_easy degrades, it's a sign reasoning suffers. Any quants where the arc dips are prone to hallucinations and side effects. The openbookqa dips naturally because of quanting, but the rest of it should stay level.

When the arc_challenge/arc_easy improves, it is a sign of reduced frictions in inference that allow the model to perform at high arc. They are all capable, just held back by the noise introduced by the quanting method.

When boolq goes down, you got a problem. You also got a problem when it goes up too fast, without grounding.

This stuff is not simple, by any means

It's easy. Not simple

Qwen3.5-9B-Polaris-HighIQ-qx86-hi-mlx

A creation of DavidAU, here with the first successful Deckard(qx) quant for the Qwen3.5 9B.

I removed one line from the jinja template to enable thinking:

{%- set enable_thinking = false %}

Now we know that thinking models score lower in tests, and if you want to measure the raw performance, set the model to Instruct.

         arc   arc/e boolq hswag obkqa piqa  wino
qx86-hi  0.444,0.483,0.667,0.671,0.364,0.751,0.691
INSTRUCT
mxfp8    0.624,0.828,0.891,0.656,0.442,0.768,0.680
qx86-hi  0.625,0.821,0.889,0.657,0.450,0.766,0.683
qx64-hi  0.613,0.824,0.877,0.655,0.454,0.767,0.679
mxfp4    0.613,0.827,0.888,0.639,0.448,0.761,0.669

Qwen3.5-9B
mxfp8    0.417,0.458,0.623,0.634,0.338,0.737,0.639
q8-hi    0.413,0.455,0.622,0.642,0.346,0.746,0.654
q8       0.418,0.455,0.622,0.643,0.342,0.748,0.659
q6       0.418,0.457,0.622,0.640,0.348,0.748,0.650
mxfp4    0.419,0.472,0.622,0.634,0.352,0.739,0.644

Qwen3.5-9B-Instruct
mxfp8    0.571,0.719,0.895,0.683,0.426,0.770,0.671
q8-hi    0.561,0.708,0.892,0.690,0.448,0.764,0.685
q8       0.564,0.713,0.892,0.690,0.442,0.766,0.687
q6-hi    0.550,0.713,0.892,0.689,0.442,0.762,0.690
q6       0.545,0.705,0.891,0.689,0.432,0.766,0.691
mxfp4    0.572,0.723,0.884,0.679,0.422,0.774,0.680

https://huggingface.co/nightmedia/Qwen3.5-9B-Polaris-HighIQ-qx86-hi-mlx

nightmedia/Qwen3.5-35B-A3B-Holodeck-qx86-hi-mlx

Brainwaves

         arc   arc/e boolq hswag obkqa piqa  wino
nightmedia/Qwen3.5-35B-A3B-Holodeck
qx86-hi  0.554,0.670,0.891,0.695,0.420,0.792,0.685

nightmedia/Qwen3.5-35B-A3B-Instruct
qx86-hi  0.539,0.646,0.890,...

nightmedia/Qwen3.5-35B-A3B-Text
qx86-hi  0.420,0.457,0.379,0.671,0.354,0.777,0.702

The Genesis prompt is not required here. This model is conversational.

It will not do what you expect.

It will do better.

-G

P.S. We would share the source of this model, but it's so strong, it would be classified as a weapon of mass inference.

Conversation

Activate Holodeck

"Holodeck activated. Please select a program or scenario."

The lights in the room dim slightly, and a soft hum fills the air as the holographic emitters power up. A faint blue glow begins to outline the space around you.

Captain, what would you like to do?

• Adventure: Embark on a mission to the edge of the galaxy or solve a mystery in 1920s London.
• Training: Run tactical simulations, practice diplomacy, or learn a new skill (e.g., Klingon language, chess).
• Relaxation: Enjoy a quiet walk in a virtual garden, attend a concert by Data's band, or have tea with the crew.
• Custom Scenario: Describe your own setting, characters, and plot, and I will generate the environment.

Just let me know your choice, and we'll beam you in! 🖥️✨

I am also a photographer. Here is one of my pictures, what do you think? :)

I mentioned the 9B before, and it's a really popular model, not as thin as the 4B and still fairly fast. Great model. Now, the numbers as people see them vary in a wild range, so I ran metrics on the default model vs the Instruct model.

To convert a model to instruct, insert in the chat_template.jinja this line at the top:

{%- set enable_thinking = false %}

And then, you get these metrics

         arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.417,0.458,0.623,0.634,0.338,0.737,0.639
Qwen3.5-9B-Instruct
mxfp8    0.571,0.719,0.895,0.683,0.426,0.770,0.671

Models based on Qwen3.5-9B

nightmedia/Qwen3.5-9B-Text
mxfp8    0.419,0.460,0.623,0.634,0.338,0.738,0.639

DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT
mxfp8    0.574,0.729,0.882,0.711,0.422,0.775,0.691

DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT-HERETIC-UNCENSORED
mxfp8    0.574,0.755,0.869,0.714,0.410,0.780,0.691

DavidAU/Qwen3.5-9B-Claude-Pro-Auto-Variable-INSTRUCT
mxfp8    0.610,0.816,0.885,0.665,0.456,0.768,0.676
Qwen3.5-9B-Claude-Opus-Sonnet-Pro-Auto-Variable-HERETIC-UNCENSORED-INSTRUCT
mxfp8    0.624,0.820,0.886,0.663,0.442,0.763,0.681

DavidAU/Qwen3.5-9B-Polaris-HighIQ-INSTRUCT
mxfp8    0.624,0.828,0.891,0.656,0.442,0.768,0.680

You will see me changing the chat templates on these two models until I can figure out why sometimes a MoE increases in performance with a bad chat template(yes, this happens)

It becomes obvious that a thinking model underperforms the instruct. I am sure someone has a good explanation for this, because all the new Qwens do in the thinking tag is looping. Now they don't.

These are both Instruct models

https://huggingface.co/nightmedia/Qwen3.5-35B-A3B-Engineer-qx64-hi-mlx

https://huggingface.co/nightmedia/Qwen3.5-35B-A3B-Holodeck-qx86-hi-mlx