Byrne-TriAtn-86M

A TriAttention-enabled variant of Byrne-86M. Same weights, same chat behavior โ€” plus a training-free KV-cache compression path (TriAttention) that bounds attention memory at long context. This repo does not replace Byrne-86M; it is a separate, drop-in model.

For contexts up to the KV budget (default 2048 tokens), this model's output is byte-for-byte identical to Byrne-86M (verified). TriAttention only changes behavior once the cache would exceed the budget, where it caps memory instead of growing.

What was changed vs Byrne-86M

Nothing was retrained โ€” the transformer weights are identical. The additions are inference-side:

  • triattention.py (new) โ€” a per-layer KV-cache compressor implementing TriAttention: it scores cached keys in the pre-RoPE space using this model's Q/K concentration centers (a trigonometric series over Qโ€“K distance) plus a Q/K-norm signal, and keeps the top-scoring keys (with attention-sink + recent-window retention). Streaming, position-aware eviction.
  • Baked calibration โ€” the per-layer pre-RoPE Q/K centers, concentration (mean-resultant-length R โ‰ˆ 0.79), and key-norm statistics are stored as buffers in model.safetensors. Calibrated offline on 256 FineWeb-Edu documents (the family's pretraining distribution). No weight updates.
  • Config flags (config.py) โ€” use_triattention (default True), tri_kv_budget (default 2048), tri_local_window, tri_sink_tokens, tri_future_offsets.
  • Dynamic RoPE (model_v2.py) โ€” the rotary tables now extend on demand, so generation can exceed the original 4096-position cap.

Helper scripts included: calibrate_triattention.py, generate_triattention.py, compare_triattention.py, chat_eval_triattention.py.

Why / when it helps

KV cache grows linearly with sequence length. TriAttention caps it at tri_kv_budget, cutting memory at long context (and stabilizing throughput) with little quality loss. On an 86M model this is a memory tool, not a speed tool โ€” the benefit appears when the cache would otherwise grow large.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-TriAtn-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-TriAtn-86M", trust_remote_code=True)
# TriAttention is ON by default (budget 2048). To tune or disable:
#   ...from_pretrained(..., tri_kv_budget=1024)      # more compression
#   ...from_pretrained(..., use_triattention=False)  # exact original behavior

Prompts use ChatML (<|im_start|>role\nโ€ฆ<|im_end|>), generation starting after a trailing <|im_start|>assistant\n. Note: when using TriAttention with compression active you must pass explicit absolute position_ids during generation (see generate_triattention.py), because a compressed cache can no longer infer position from its length.

Measured behavior (this model)

  • Short chat (โ‰ค budget): identical to Byrne-86M.
  • Long context (ChatML, 3.3k tokens): identical to full attention down to budget โ‰ˆ 1024 (3ร— KV reduction) on the tested prompt; quality drifts below that. Default 2048 leaves a safe margin.

Limitations

  • Adaptation of TriAttention to MLA / partial-RoPE / MQA; pooled per-head scores onto the shared KV head. Validate on your own workload before relying on aggressive budgets.
  • 86M chat model: best on short, ChatML-formatted turns; long summarization is repetitive (a base-capacity limit, independent of compression).
  • Custom SpikeTokenizer is not bf16-safe in the engram path; run in fp32/fp16.

Architecture, tokenizer, evaluation

Unchanged from Byrne-86M (SpikeWhaleLM, ~86M params, 16 layers, hidden 640, MLA + RoPE-16/NoPE-48 MQA, engram memory, hyper-connections, HRM refinement, Non-JEPA, ChatML SpikeTokenizer). See the base repo for the full benchmark table.

Citation

@misc{byrnetriatn86m,
  title        = {Byrne-TriAtn-86M: TriAttention KV-compression variant of Byrne-86M},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-TriAtn-86M}},
  note         = {TriAttention: arXiv:2604.04921}
}

Update: format-blended SFT on the engram-repaired base

This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: ') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:

MMLU acc 0.280->0.276, format 0.926->0.948; GSM8K '####' 0.420->0.790

Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.

Downloads last month
36
Safetensors
Model size
96.9M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Quazim0t0/Byrne-TriAtn-86M

Finetuned
(1)
this model

Paper for Quazim0t0/Byrne-TriAtn-86M