Quazim0t0/Positronic-144M

Positronic v1

Model will be ungated for open download soon! These models are undertrained, and are NOT meant to be finished. These are Research Artifacts only.

144M causal LM with a non-standard channel mixer. Token mixing is still attention. Per-layer MLP is a QuazimotoBlock: coupled phase oscillators (Kuramoto) on concentric rings, a few differentiable Euler steps, readout [cos θ, sin θ].

Positronic is from Data's positronic brain in Star Trek - a mind on weird internal dynamics, not a standard FFN stack.

Same tokenizer, traits, and data as the sibling Quazimoto models (MyCel-LM). Only the mixer is swapped. Controlled experiment: can a transformer generalize if the FFN is a self-organizing dynamical system instead?

Research / hobby architecture, not a product. 144M, short run. Fluent ChatML, can reproduce <think>…</think> structure. Facts and arithmetic are unreliable. Interesting for what it is, not what it knows.

• Base checkpoint: pretraining, ~64k steps (base weights not shipped in this repo)
• This release (SFT): chkpt/quazimoto_sft.pt (chat/instruction SFT, 8k steps)
• Params: 144.3M
• Vocab: 16,512 (byte-level length-max tokenizer + appended special tokens)
• Context: block_size 2024 (RoPE max_position_embeddings 4096)

---

Architecture at a glance

Component	Setting
Layers	10
d_model	768
Attention heads	12 (GQA: 4 KV heads)
Head dim	64 = 32 NoPE + 32 RoPE (partial RoPE)
Q / O projections	MLA low-rank (rank 384)
QK-Norm	on
Channel mixer	QuazimotoBlock (508 oscillators / token)
Oscillator rings	7 concentric: (4, 8, 16, 32, 64, 128, 256)
Kuramoto Euler steps	4 (osc_dt 0.5)
Readout MLP expansion	3× on the [cos, sin] vector
Norm	RMSNorm (pre-norm), no biases
LM head	tied to token embedding, learned logit_scale
Logit discipline	z-loss (1e-4)

Every layer is x → x + Attention(norm(x)) then x → x + QuazimotoBlock(x). All the exotic machinery lives in (or around) the QuazimotoBlock.

1. QuazimotoBlock - the oscillator channel mixer

This is the defining component. Instead of an MLP, each token's hidden state drives a bank of N = 508 phase oscillators laid out on 7 concentric rings:

1. Project to phases and frequencies. Two linear maps produce initial phases θ [B,T,N] and bounded natural frequencies ω = tanh(...).
2. Structured Kuramoto coupling. The oscillators are coupled through a learnable R×R (7×7) block-gain matrix between rings - strong on-diagonal (within-ring) coupling, weaker neighbour-ring coupling at init. Because coupling is block-constant, the update uses per-ring mean fields (order parameters), giving an O(N·R) update instead of the O(N²) pairwise Kuramoto sum. A per-ring learnable frustration α shifts the coupling phase.
3. A rotating "center ball." A single global phase (center_phase, advancing at center_omega) pulls every oscillator toward it via a per-ring center_gain - a hierarchy-to-center coupling that is the "Quazimoto" twist on plain oscillatory neurons.
4. Integrate. osc_steps=4 differentiable Euler steps advance θ under ω + coupling + center-pull + external drive.
5. Read out. Concatenate [cos θ, sin θ] → [B,T,2N], pass through a 3×-expansion GELU MLP back to d_model.
6. Family gate + soft-clamp. Output is scaled by tanh(gate) and passed through soft_clamp (an erf saturating clamp, bound 10) for BPTT stability. Unlike the auxiliary traits below, this gate is initialised open (~0.9) because the oscillator bank is the main mixer.

Degenerate check: set ring coupling to dense and α=0 and the block reduces toward AKOrN-style oscillatory neurons. The ring structure + center coupling is what makes it Quazimoto.

2. Attention - MLA / partial-RoPE / GQA

Ported from a "v2" transformer. Multi-head Latent Attention (low-rank q/o projections, rank 384), partial RoPE (each 64-dim head splits into a 32-dim no-position part and a 32-dim RoPE part), QK-Norm (per-head RMSNorm before RoPE), and GQA (4 KV heads shared across 12 query heads). Optional DERF (erf-attention) and XSA (value-subspace removal) exist but are off in this checkpoint. A GQA-compact KV cache supports incremental decoding.

3. Interstitial rings (use_rings=True)

Between each pair of adjacent oscillator rings sits a slot that absorbs context once and injects a constant drive into its two neighbour rings during the Kuramoto integration:

• PhaseAttentionRing - causal self-attention in phase space ([cos,sin] of the two neighbour rings) producing an injection current.
• EngramRing - an n-gram hash memory: a frozen LSH-style compressor hashes 1-3-token windows into per-head embedding tables, gated by a DERF context gate, projected into an injection current.

4. Per-ring self-organizing controllers (use_ring_controllers=True)

One tiny RingController per ring (shared across layers). Each observes its ring's order-parameter magnitude/phase and mean frequency, and emits no-op-at-init modulations of that ring's coupling, frustration, center-pull, and injection. Its core is a fast-weight predictor updated online by a delta rule (one gradient step on its own prediction error - surprise minimization), not by backprop; only a small zero-init decoder is backprop-trained.

5. Per-ring memory specialists (use_ring_specialists=True)

A small MoE of 7 mini "memory specialists" per ring (top-2 routed). Each specialist owns two test-time-writable stores (store_in, store_out) that accumulate an addressable context memory during generation via a gated EMA write. Tokens route to specialists by similarity to a learnable identity key plus a read of the accumulated input memory; the routed store_out is decoded into a ring injection current.

Important for inference: these stores are mutated at inference time and persist across forward calls. See "Known issue" below.

6. Trunk refinements & auxiliary heads

Applied after the layer stack, before the final RMSNorm + LM head:

• HRM (use_hrm=True) - Hierarchical Reasoning Module: iterative gated refinement that starts from a random learned state z0 and reconciles it against the trunk over 3 steps, contributing only the reasoning delta. Gates start open.
• MoE SwiGLU (use_moe=True) - shared + top-2-of-4 routed SwiGLU experts with load-balance aux loss; down-projections zero-init (no-op at start).
• MTP (use_mtp=True) - 4 multi-token-prediction draft heads. A train-time auxiliary loss, and at inference they enable self-speculative decoding (--speculative): the heads draft the next 4 tokens, the main head verifies them in one parallel forward. Output is bit-identical to greedy decoding, only faster.
• JEPA (use_jepa=True) - representation-space k-ahead prediction with a stop-grad target. Train-time only (auxiliary loss); no effect at inference.

7. Fractal (Mandelbrot) phase seeding (use_fractal_phase_seed=True)

Each token id maps to a complex point c (spread over the Mandelbrot region by a 2D Halton sequence); the angles of its z ← z² + c orbit seed the oscillator phases, added through a zero-init gate. A frozen, parameter-free, token-specific dynamical prior congruent with what the Kuramoto block already does. The [16512, 508] table is regenerated deterministically on load (not stored in the checkpoint).

Family "safe-at-init" contract: every auxiliary block (rings, engram, specialists, HRM, MoE, MTP, JEPA, fractal seed) is gated so it is a no-op at initialization while its content weights still receive gradient - the model boots as a clean attention+oscillator transformer and the optimizer opens each trait only if it helps.

---

Verification (this checkpoint)

I confirmed the released SFT checkpoint actually uses the full custom architecture (not a fallback):

• Strict state_dict load succeeds - every checkpoint tensor maps onto the model with none missing or unexpected. (generate.py loads with strict=False, but strict passes too.)
• All traits are active in family_config: HRM, MoE, MTP, JEPA, Rings, Controllers, Specialists(7/ring), FractalSeed.
• Training opened the oscillator block: per-layer output gates tanh(gate) climb from ~0.31 (layer 0) to ~0.99 (deep layers) - the oscillators dominate the later layers.
• Ablations change real-prompt logits (max |Δlogit| on one assistant-turn prompt): zeroing the oscillator gates 31.3 (argmax flips), MoE 12.7, ring controllers 8.1, HRM 3.2, fractal seed 0.63, interstitial rings 0.12, ring specialists 0.06. So the big mixers are load-bearing; the fractal seed and interstitial rings are present but contribute weakly at this checkpoint.
• KV-cache path is correct: greedy decoding with use_cache=True is bit-identical to full recompute over 40 tokens.

---

What this is for / what it isn't

Research into oscillatory / Kuramoto channel mixing, custom-architecture inference, small ChatML/<think> demo.

Not for factual QA, arithmetic, or anything that needs to be correct. 144M, short run. Format is often right; content is often wrong or incoherent.

No safety / RLHF. Outputs can be nonsense. English-centric.

---

Usage

transformers note

Custom architecture (QuazimotoLM in model.py). Does not load via AutoModelForCausalLM. Use this repo's model.py + generate.py. Tokenizer is HF-compatible via SpikeTokenizer (spike_tokenizer.py).

Chat generation (recommended for the SFT model)

python generate.py --ckpt chkpt/quazimoto_sft.pt --chat \
    --prompt "Hello, who are you?" --max_new_tokens 200

Chat mode wraps the prompt in ChatML (<|im_start|><|user|> … <|im_end|> then an <|assistant|> header) and stops on <|im_end|>.

Reasoning with <think>

The SFT mix included short-reasoning data, so the model knows the <think>…</think> format but does not always open the tag on its own. To force it, prime the assistant turn ending in <think>.

Speculative decoding (faster, greedy-identical)

python generate.py --ckpt chkpt/quazimoto_sft.pt --chat --speculative --spec_stats \
    --prompt "Explain the plan." --max_new_tokens 200

Programmatic

import torch
from model import QuazimotoLM, QuazimotoConfig
from generate import load_tokenizer, generate, build_prompt, resolve_stop_ids

tok = load_tokenizer(".")
ck = torch.load("chkpt/quazimoto_sft.pt", map_location="cpu", weights_only=False)
model = QuazimotoLM(QuazimotoConfig(**ck["family_config"]))
model.load_state_dict(ck["model"]); model.eval()

model.reset_ring_memory()               # clear test-time specialist memory per request
ids, _ = build_prompt(tok, "Hello!", chat=True, system="")
x = torch.tensor([ids])
out = generate(model, model.cfg, x, 200, stop_ids=resolve_stop_ids(tok, True))
print(tok.decode(out[0, len(ids):].tolist(), skip_special_tokens=True))

Special tokens

ChatML (<|im_start|>, <|im_end|>, <|system|>, <|user|>, <|assistant|>), reasoning (<think>, </think>, <begin_solution>, <end_solution>), tool calls (<tool_call>, <tool_response>), code FIM (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>), and <|endoftext|>.

---

Statefulness note (ring-specialist test-time memory)

The ring specialists write to persistent stores that carry over across forward calls (store_out norm goes 0.0 → ~44.5 after one forward), so in principle memory from one prompt can leak into the next. At this checkpoint the practical effect is negligible: the specialist output gates barely opened during the short SFT (tanh(scale) in -0.10…+0.17), so cross-prompt contamination perturbs next-token logits by only ~0.007 (max) and did not flip a single greedy token in testing. It is the weakest of all the model's components.

It is still handled correctly for reproducibility and to future-proof checkpoints where those gates open wider: generate.py now calls model.reset_ring_memory() before every prompt (REPL turn / single run). Flags:

• --keep_ring_memory - opt out of the per-prompt reset (let memory accumulate across turns).
• --freeze_ring_memory - disable online writes entirely for fully stateless, reproducible generation (model.set_ring_memory_writing(False)).

A Gradio wrapper that reuses one loaded model gets clean per-request behavior through the same run_once path; if you call generate() directly, call model.reset_ring_memory() yourself at the start of each request.

---

Training data

• Pretraining blend: ~35% Ultra-FineWeb-L3, 25% FineWeb-Edu, 25% FineMath, 15% Quazim0t0/PretrainNew.
• SFT blend (this release): ~equal quarters of mlabonne/ultrachat_200k_sft, mlabonne/ultrafeedback-sft, openbmb/UltraData-SFT-2605 (Knowledge, no-think split), and volcanos/OpenThoughts2-1M-ShortThink (reasoning). Loss is masked to assistant turns (standard instruction SFT).

Files

model.py (architecture), family.py (trait modules), fractal.py (Mandelbrot seeding), generate.py (inference harness), spike_tokenizer.py + tokenizer.json (tokenizer), special_tokens.py (special-token registry), chkpt/quazimoto_sft.pt (this release).

Citation

If you use this model, please cite:

@misc{positronic144m,
  title        = {Positronic-144M: A ~144M-parameter Quazimoto oscillator LM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Positronic-144M}},
  note         = {Quazim0t0/Positronic-144M}
}

Update: format-blended SFT

Short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train ('#### N' reasoning), and MMLU-style ('Answer: ') examples, so chat fluency improves while the benchmark output-formats are preserved. Held-out (test-split) before->after:

held-out 1.96->1.18, MMLU acc 0.040->0.255, format 0.225->0.995 (A/B test showed opening the inert EngramRings adds nothing over plain SFT, so this ships with rings unchanged.)

Note: fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.

Update: longer-trained checkpoint + format-blended SFT

Same 144.3M-param architecture, trained further (to step 8250, ~1B tokens, with the context window extended 2048 -> 4096), then given the format-blended SFT (smoltalk + GSM8K '#### N' + MMLU 'Answer: '). Held-out (test-split) before->after: held-out loss 2.01 -> 1.12, MMLU acc 0.010 -> 0.255, answer-format rate 0.095 -> 1.000.

Best decoding (from a decode_lab.py sweep on this checkpoint): temperature ~0.7, top-k 40, repetition-penalty 1.15, no-repeat-ngram 3 (the "nrng" recipe; distinct-2 0.945 / rep-4 0.000). Do NOT use CFG on this model -- it over-amplifies and degenerates ("first first first ..."). Greedy + no-repeat-ngram is also stable.