add README.md

README.md

@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+tags:
+  - kuramoto
+  - oscillator
+  - language-model
+  - tinystories
+library_name: pytorch
+---
+
+# KOLM-Alpha — a native Kuramoto-Oscillator Language Model
+
+**KOLM-Alpha is, to our knowledge, the first language model whose per-token
+processing layers are networks of coupled Kuramoto oscillators, trained
+natively from scratch.** Each layer uses standard causal attention for
+routing between tokens and a **Kuramoto block** for processing: *H*
+oscillators, each a unit vector on a sphere, relax over *K* settling steps
+under trained pairwise couplings, conditioned on the attention output. The
+oscillator block replaces the Transformer's feed-forward network; attention
+is retained.
+
+## The result
+
+Under a strictly controlled **twin protocol** — identical tokenizer, data,
+data order, context length, optimizer, schedule and seed, with *only* the
+feed-forward slot differing — KOLM-Alpha is compared against **TMT**
+(Transformer Model Twin), a standard attention + MLP model. Both were
+trained from scratch on 11.3M tokens of TinyStories on a single consumer
+laptop.
+
+![Validation loss](assets/val_loss_curve.png)
+
+TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens,
+and finishes ahead — **with 3.5% fewer parameters.**
+
+![Final scores](assets/final_scores.png)
+
+| Model | Parameters | Val loss | Perplexity |
+|---|---|---|---|
+| **KOLM-Alpha** | **16,916,224** | **2.6946** | **14.80** |
+| TMT (Transformer twin) | 17,538,816 | 2.7188 | 15.16 |
+
+Both models generate coherent TinyStories-grade prose. Prompt *"Once upon a
+time there was a little red shoe"*:
+
+> **KOLM-Alpha:** …The little boy would come home and play with his friends.
+> One day, the little girl was playing with the big castle in the sky…
+
+## Why it matters
+
+A Transformer computes with a fixed number of operations per token. A
+Kuramoto block computes by **iterative settling** — the oscillator field
+relaxes toward a configuration over *K* steps — which opens a second axis of
+scale that costs no additional memory, only settling time. KOLM-Alpha
+establishes the first point: at matched parameters, oscillatory processing
+is not merely competitive with a Transformer feed-forward network — it edges
+it out.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `native_kolm.py` | model + twin trainer (`--arch kolm` / `--arch transformer`) |
+| `kuramoto_torch.py` | the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference) |
+| `kolm.py`, `olm.py` | pure-NumPy reference core with hand-derived, gradient-checked backprop |
+| `sample_native.py` | generation from the trained weights |
+| `native_kolm.pt` | **KOLM-Alpha weights** (16.9M) |
+| `native_transformer.pt` | TMT weights (17.5M) |
+| `tiny8k.json` | tokenizer (required to run the model) |
+| `curve_kolm.csv`, `curve_transformer.csv` | validation-loss curves |
+
+## Run it
+
+```bash
+python3 -m venv --system-site-packages .venv
+.venv/bin/pip install torch tokenizers
+
+# generate from the released weights
+.venv/bin/python sample_native.py --prompt "Once upon a time"
+
+# verify the hand-derived gradients of the NumPy reference (no PyTorch needed)
+python3 kolm.py --gradcheck
+```
+
+## Reproduce the comparison
+
+```bash
+.venv/bin/pip install transformers accelerate
+.venv/bin/python native_kolm.py --prep                 # build tokenizer + data
+.venv/bin/python native_kolm.py --arch kolm        --steps 5500
+.venv/bin/python native_kolm.py --arch transformer --steps 5500
+```
+
+## Roadmap
+
+KOLM-Alpha is the first release. In progress:
+
+- **KOLM-Beta** — a larger model with frustrated coupling and a broader
+  training corpus, grown from Alpha via function-preserving depth growth.
+- **KOLM-Chat** — an instruction-tuned conversational KOLM, exposing the
+  settling depth as a user-facing "think-harder" control.
+
+## Details
+
+- **Architecture:** 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE.
+  Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling
+  steps, 32 order-parameter readouts).
+- **Training:** AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed.
+- **Hardware:** one Apple-silicon laptop (MPS), ~6 hours.
+- **Status:** research preview. The headline comparison is single-seed; a
+  second seed is queued.
+
+## License
+
+Apache-2.0.
+
+## Acknowledgements
+
+Builds on the Kuramoto-oscillator formulation of **AKOrN** (Miyato et al.,
+ICLR 2025) and the **TinyStories** corpus (Eldan & Li, 2023).