Repopulate as fully-oscillatory KOLM-Alpha (KOLM-True data); hybrid moved to KOLM-Hybrid-1

README.md

@@ -4,113 +4,117 @@ tags:
   - kuramoto
   - oscillator
   - language-model
+  - synchronization-routing
   - tinystories
 library_name: pytorch
 ---
 
-# KOLM-Alpha — Kuramoto Oscillator Language Model
+# KOLM-Alpha — Fully-Oscillatory Kuramoto Language Model
 
 *Delon Swartz — AI Researcher | Engineer*
 
-**KOLM-Alpha is, to our knowledge, the first language model whose per-token
-processing layers are networks of coupled Kuramoto oscillators, trained
-natively from scratch.** Each layer uses standard causal attention for
-routing between tokens and a **Kuramoto block** for processing: *H*
-oscillators, each a unit vector on a sphere, relax over *K* settling steps
-under trained pairwise couplings, conditioned on the attention output. The
-oscillator block replaces the Transformer's feed-forward network; attention
-is retained.
+**KOLM-Alpha is, to our knowledge, the first language model with *no
+transformer computation anywhere* — both routing between tokens and
+processing within them are performed by networks of coupled Kuramoto
+oscillators.** It is the successor to [KOLM-Hybrid-1](https://huggingface.co/Dbrent/KOLM-Hybrid-1),
+which kept standard attention and made only the feed-forward slot
+oscillatory. KOLM-Alpha replaces attention itself with **synchronization
+routing**.
+
+> **Naming note:** the name *KOLM-Alpha* previously labelled the hybrid twin
+> study, now published as **KOLM-Hybrid-1**. Under the project naming
+> registry (2026-07), *KOLM-Alpha* denotes this fully-oscillatory line
+> (working name "KOLM-True").
+
+## The mechanism: KuramotoRouter A
+
+Each layer's routing is a competitive synchronization process, not static
+attention:
+
+- **Softmax competition** over the causal past — sharp selection, not
+  blurry averaging.
+- **RoPE on the routing scores** — position is *in* the routing, not just
+  the moved content.
+- **Dynamic query modulation** — routing weights are *recomputed at every
+  settling step*, modulated by the evolving oscillator state. This is
+  genuinely dynamic routing that static attention cannot express.
+
+The routed value drives each token's oscillators, which settle via Kuramoto
+dynamics under trained couplings, natural rotations, and (optionally)
+frustrated (phase-offset) coupling. Normalization is **SphereNorm**. Every
+block is an exact identity at initialization.
 
 ## The result
 
-Under a strictly controlled **twin protocol** — identical tokenizer, data,
-data order, context length, optimizer, schedule and seed, with *only* the
-feed-forward slot differing — KOLM-Alpha is compared against **TMT**
-(Transformer Model Twin), a standard attention + MLP model. Both were
-trained from scratch on 11.3M tokens of TinyStories on a single consumer
-laptop.
+Under the same controlled protocol as KOLM-Hybrid-1 (identical tokenizer,
+data, order, budget, seed; 11.26M TinyStories tokens; single laptop):
 
-![Validation loss](assets/val_loss_curve.png)
-
-TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens,
-and finishes ahead — **with 3.5% fewer parameters.**
-
-![Final scores](assets/final_scores.png)
-
-| Model | Parameters | Val loss | Perplexity |
+| Model | Params | Val loss | Perplexity |
 |---|---|---|---|
-| **KOLM-Alpha** | **16,916,224** | **2.6946** | **14.80** |
-| TMT (Transformer twin) | 17,538,816 | 2.7188 | 15.16 |
-
-Both models generate coherent TinyStories-grade prose. Prompt *"Once upon a
-time there was a little red shoe"*:
-
-> **KOLM-Alpha:** …The little boy would come home and play with his friends.
-> One day, the little girl was playing with the big castle in the sky…
+| **KOLM-Alpha (Router A)** | 17.06M | **2.3170** | **10.14** |
+| KOLM-Hybrid-1 (attention + osc-FFN) | 16.92M | 2.6946 | 14.80 |
+| Transformer twin (TMT) | 17.54M | 2.7188 | 15.16 |
+
+**KOLM-Alpha wins decisively — 0.38 nats below the hybrid, 0.40 below the
+transformer twin** — while doing *zero* transformer computation. It crossed
+the hybrid's final score at 5M tokens (41% through training).
+
+An honest negative precedes this win: **Router v1** (sigmoid-gated,
+position-blind, routing frozen across settle steps) *lost* at 3.0273. The
+three diagnosed failures motivated Router A's softmax + RoPE + dynamic
+modulation.
+
+## Scaling to 42M from scratch (the flagship)
+
+**KOLM-Alpha-42M** ("W42") is the same architecture at 16 layers, d=384,
+H=320, frustrated coupling, randomized settling depth *K ~ U[1,4]*, trained
+from scratch on 188M tokens (32k vocabulary, TinyStories + Simple English
+Wikipedia):
+
+- **Val loss 1.7141 / perplexity 5.55** — from scratch, matching
+  function-preserving-transfer-warmed comparables (K-sweep K1 1.7162 /
+  K2 1.6848 / K4 1.7044).
+- **Chat**, via staged SFT: broad chat mixture → single-turn discipline
+  (Alpaca). The staged order beats Alpaca-from-base by 0.35 nats
+  (alpaca-val 3.19 vs 3.55). Turn discipline is clean (answers once, stops
+  at `<|end|>`); simple facts land ("The capital of France is Paris").
+  Entity binding and counting remain beyond 42M — capacity limits, not
+  recipe bugs.
 
 ## Why it matters
 
-A Transformer computes with a fixed number of operations per token. A
-Kuramoto block computes by **iterative settling** — the oscillator field
-relaxes toward a configuration over *K* steps — which opens a second axis of
-scale that costs no additional memory, only settling time. KOLM-Alpha
-establishes the first point: at matched parameters, oscillatory processing
-is not merely competitive with a Transformer feed-forward network — it edges
-it out.
+The hybrid showed oscillators can *process* as well as an MLP. KOLM-Alpha
+shows they can *route* better than attention — the whole layer is now
+synchronization dynamics, with a settling-depth **compute dial** that adds a
+memory-free axis of scale. Randomized-depth training makes that dial
+extrapolate past its trained depth.
 
-## Files
+## Files (weights to follow)
 
 | File | Purpose |
 |---|---|
-| `native_kolm.py` | model + twin trainer (`--arch kolm` / `--arch transformer`) |
-| `kuramoto_torch.py` | the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference) |
-| `kolm.py`, `olm.py` | pure-NumPy reference core with hand-derived, gradient-checked backprop |
-| `sample_native.py` | generation from the trained weights |
-| `native_kolm.pt` | **KOLM-Alpha weights** (16.9M) |
-| `native_transformer.pt` | TMT weights (17.5M) |
-| `tiny8k.json` | tokenizer (required to run the model) |
-| `curve_kolm.csv`, `curve_transformer.csv` | validation-loss curves |
-
-## Run it
-
-```bash
-python3 -m venv --system-site-packages .venv
-.venv/bin/pip install torch tokenizers
-
-# generate from the released weights
-.venv/bin/python sample_native.py --prompt "Once upon a time"
-
-# verify the hand-derived gradients of the NumPy reference (no PyTorch needed)
-python3 kolm.py --gradcheck
-```
-
-## Reproduce the comparison
-
-```bash
-.venv/bin/pip install transformers accelerate
-.venv/bin/python native_kolm.py --prep                 # build tokenizer + data
-.venv/bin/python native_kolm.py --arch kolm        --steps 5500
-.venv/bin/python native_kolm.py --arch transformer --steps 5500
-```
+| `kolm_true.py` | full-oscillator model + trainer (Router A) |
+| `kuramoto_torch.py` | Kuramoto block (parity-tested to 2.2e-16 vs NumPy) |
+| `chat_true.py` | REPL / sampler with the `/think K` settling dial |
+| `native_true_A.pt` | **KOLM-Alpha 17M weights** (Router A) |
+| `native_w42.pt` | **KOLM-Alpha-42M** pretrain weights |
+| `native_w42chat_alp.pt` | 42M chat model (staged SFT) |
+| `tiny32k.json` | 32k tokenizer |
 
 ## Roadmap
 
-KOLM-Alpha is the first release. In progress:
-
-- **KOLM-Beta** — a larger model with frustrated coupling and a broader
-  training corpus, grown from Alpha via function-preserving depth growth.
-- **KOLM-Chat** — an instruction-tuned conversational KOLM, exposing the
-  settling depth as a user-facing "think-harder" control.
+- **KOLM-Beta-T** ([repo](https://huggingface.co/Dbrent/KOLM-Beta-T)) —
+  transplanting pretrained transformer weights into this architecture to
+  reach billion-parameter capability without pretraining.
+- Larger from-scratch KOLM-Alpha rungs exposing the settling dial as a
+  user-facing "think-harder" control.
 
-## Details
+## Status & limitations
 
-- **Architecture:** 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE.
-  Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling
-  steps, 32 order-parameter readouts).
-- **Training:** AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed.
-- **Hardware:** one Apple-silicon laptop (MPS), ~6 hours.
-- **Status:** research preview. The headline comparison is single-seed; a
-  second seed is queued.
+Research preview. Headline comparisons are single-seed; the 17M Router-A run
+used a settling/lr protocol that differs slightly from the twin runs (a
+matched rerun is queued before external citation). Corpora are easy and all
+budgets are far below Chinchilla-optimal; factual recall at 42M is limited.
 
 ## License
 
@@ -119,4 +123,5 @@ Apache-2.0.
 ## Acknowledgements
 
 Builds on the Kuramoto-oscillator formulation of **AKOrN** (Miyato et al.,
-ICLR 2025) and the **TinyStories** corpus (Eldan & Li, 2023).
+ICLR 2025), the frustrated-synchronization principle of **FSN**
+(arXiv:2606.18694), and the **TinyStories** corpus (Eldan & Li, 2023).