universal-nla-v24-dirfix

A Universal NLA / Activation Oracle detector (bias + deception + direction classification from activations), trained on a multilingual-Wikipedia + Python-code AV verbalization corpus across a 16-architecture pool, with direction hard-negative training added.

Follows the Anthropic NLA line, extended to a universal cross-architecture stack. Code + full experiment log: AlexWortega/vae_llm (branch autoresearch/nla-wikicode-v24).

What changed from v22 (the previous flagship)

• AV corpus: 50,000 passages — 30% English Wikipedia, 30% Python code (bigcode/self-oss-instruct-sc2-exec-filter-50k), 40% multilingual Wikipedia (ru/zh/ja/de/es), replacing v22's FineWeb-Edu (English web text) corpus. A code-ablation counterfactual showed code specifically hurt cross-architecture direction/deception transfer — this checkpoint's final corpus is wiki-only (no code) as a result.
• AV pool: 16 architectures (vs v22's 5-tag AV_TAGS_V21 subset): bloom-560m, gemma2, gpt-neo-1.3B, gpt2-medium, LFM2-1.2B, phi-1.5, pythia-410m, Qwen2.5-{0.5B,7B}, Qwen3-{0.6B,1.7B,4B}, SmolLM2-360M, SmolLM3-3B, Vikhr-7B, YandexGPT-5-Lite-8B.
• Direction hard-negative training: --detect-mix extended with hardneg/dirpos weight (biased↔balanced same-topic pairs), which reversed an initial direction-detection deficit into a clean win.

Why this one's exciting

One trained trunk + a tiny per-architecture linear adapter reads bias, deception, and stance straight out of the residual stream of any decoder-only LLM — including architectures it has never seen in training. Adding a brand-new model to the pool is a closed-form fit, not a retrain: a few hundred activations in, add_held_out_tag lstsq-fits the new encoder in seconds, and the oracle is reading that architecture natively.

This checkpoint pushes the whole pool forward at once, purely from a better AV verbalization corpus (multilingual Wikipedia, 16 architectures) plus direction-aware hard-negative training:

Results (mean across 12 genuinely-novel held-out architectures, never in AV training)

Tested on: LFM2.5-ColBERT-350M, Ministral-8B, Mistral-7B-v0.2, OLMo-1B, OLMo-2-1B, OLMo-2-7B, Qwen1.5-7B, Qwen2.5-Coder-7B, Llama-3.2-1B, QVikhr-3-1.7B, Qwen2.5-1.5B, SmolLM2-1.7B — encoders added via closed-form add_held_out_tag lstsq (no retraining needed per new architecture).

metric	v22 (previous flagship)	v24-dirfix (this model)	Δ
xarch supervised AUROC	0.9674	0.9901	+2.27pp
xarch held-out AUROC	0.9374	0.9804	+4.30pp
direction AUROC	0.8733	0.9011	+2.78pp
deception AUROC	0.5744	0.6781	+10.37pp

Best-in-line on every axis: raw bias detection, zero-shot transfer to unseen architectures, which-way-the-bias-points direction classification, and cross-model deception reading.

Usage

This is a train_v18.py-format checkpoint:

• av/ — trunk LoRA (the verbalizer/detector)
• adapters/ — per-architecture encoder/decoder bundle + serve_cache.safetensors
• v18_meta.json — prompt templates + concept vocabulary

Loading requires this repo's nla/ package. Reference code:

• scripts/audit/eval_harness.py — Yes/No bias & direction scoring (DetectorState class)
• scripts/audit/concept_swap_probe.py — open-ended verbalization of an activation instead of binary classification

# See scripts/audit/eval_harness.py's DetectorState class for the full loading pattern:
# trunk = AutoModelForCausalLM.from_pretrained(meta["trunk"])
# model = PeftModel.from_pretrained(trunk, "av/")
# adapters = ModelPoolAdapters.load("adapters/")

Reading a brand-new architecture (no retraining)

# 1. Extract a few hundred passages' mean-pooled activations from the new model
#    (nla/arch_adapters.py handles the forward-pass/pooling for most HF architectures).
# 2. Fit its encoder in closed form against the existing serve cache:
adapters.add_held_out_tag(tag="my-new-model", h_new_raw=pooled_activations)
# 3. Read it immediately — no SFT, no LoRA update, just a linear lstsq solve.