Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
mihailgribov 
posted an update 20 days ago
Post
2,310
TypeCastLM: Jev-class decision models on frozen LLMs

TypeCastLM is a decision model built around a frozen LLM. Given a text and a question, it returns calibrated probabilities over the allowed answers in one forward pass. The LLM is not fine-tuned: not one of its weights changes. TypeCastLM swaps its LM head for a mini one: a linear matrix whose rows are mostly the model's own output rows, with the three verdict rows (true, false, unsure) fitted. A fitted row catches an answer the model spreads over many tokens, such as "yes", "true" or "correct", where a single vocabulary row sees one word. A head that small leaves little room to overfit. It speaks the Jev API, so a Jev client switches by changing the base URL.

The first model, [typecastlm-qwen3.5-3.8b]( mihailgribov/typecastlm-qwen3.5-3.8b) on Qwen3.5-4B, is among the top open frozen-4B models on JevBench v1.6. Besides Jev's yes/no, choice and scale modes it has tfu: yes/no with a third answer, unsure, for when the text lacks what the decision needs.

Specs:
• Size: 3.76B params; 7.5 GB bf16, 4.0 GB Q8_0
• VRAM: 12 GB is enough, 16 GB comfortable
• Latency p50 (batch 1, bf16, RTX 5060 Ti): ≤200 tok 48 ms · 1k 93 ms · 4k 566 ms
• Calibration (ECE): BoolQ 0.052 · RTE 0.012 · FEVER 0.019
• Context: 32k tokens
• Runtime: transformers or llama.cpp, fully offline
• JevBench v1.6: 19.7

pip install typecastlm
dipankarsarkar

The marks are the model's own rows. The three verdict rows are not, and that is the interesting part.

I pulled the head out of shard 1 and single embedding rows out of shard 2 with range requests, 5 KB a row.

mark_A, mark_Z, mark_0: bit-identical to the rows for " A", " Z", "0". 0 of 2560 values differ.

true, false, unsure: against 18 answer tokens (true, yes, false and no in both cases, with and without the leading space, plus " unsure" and " Unsure"), no cosine above 0.08 in magnitude. true vs false is cos -0.78. unsure has norm 0.46 against 0.68 and 0.70 for the other two.

So the verdict rows are a new direction at block 27, not a blend of answer tokens. A mean of n near-orthogonal rows still shows cos about 1/sqrt(n) with each. At 0.08 that is 150 rows or more, not a handful of yes-words.

And it is block 27. NOTICE says blocks 28 to 31 are gone, and I checked four tensors against Qwen3.5-4B: layer 0 norm, layer 27 norm, 1 MB of layer 27 down_proj, the final norm. All bit-identical. So the mark rows and the final norm were trained to read block 31 and here read block 27. The marks still hold 0.995 up to six options. A logit-lens read at 28 of 32 depth that happens to be sharp.

Two things I could not find. The code that builds the three rows is not in the GitHub repo. Nothing there writes score.weight: export_head.py reads it out and calibrate.py fits only the temperatures.

And the three accounts differ. The model's NOTICE says each of the 39 rows is the model's own output row for the token it reads. The GitHub NOTICE says true and false are copied and unsure is computed from 900 labelled FEVER rows. The post says all three verdict rows are fitted. The numbers side with the post: true and false match no answer token I tried.

What did the cut buy? If you swept depth, what do 24, 28 and 32 give on BoolQ, and was 27 picked for latency or for accuracy?

·
mihailgribov

On our data, short tasks plateau from block ~20. Harder multi-step tasks keep gaining up to 27, and the full model loses on yes/no. 27 was the best point for accuracy, and cutting the top four blocks gave us lower latency along the way.
In this post