pngwn/memcap-l-above-capacity

memcap: 8.44M-param GPT-2, above capacity regime

Part of a tiny-scale reproduction of How much do language models memorize? (Morris et al., ICML 2026 honorable mention): GPT-style models trained from scratch on uniformly random token sequences, where generalization is impossible and every bit of loss reduction is memorization.

This checkpoint sits above the model's predicted capacity (dataset entropy = 4x capacity). Memorization converged at 2.64 bits/param, below the 3.6 prediction — at 8M params the measurement becomes optimization-limited (see the seed spread at N=43k in the results repo), so this is best read as a lower bound on the plateau.

Model

• Architecture: GPT-2 (via transformers), 8.44M parameters (d_model 256, 10 layers, 4 heads), vocab 2048, context 64, trained from scratch (init seed 0)
• Precision: fp32 (TF32 disabled) | Optimizer: AdamW lr 1e-3, cosine decay | Batch: 512 sequences
• Trained for 48,250 steps

Data

Uniformly random tokens, vocab 2048, 64 tokens/sequence, N = 172,000 sequences, data seed 300. Dataset entropy = 63 predictable tokens x log2(2048) = 693 bits/sequence = 119,196,000 bits total. Dataset: pngwn/memcap-random-data

Memorization measurement

memorized bits = dataset entropy - model NLL on the data (in bits)

quantity	value
dataset entropy	119,196,000 bits
memorized	22,245,979 bits (18.7% of entropy)
bits per parameter	2.64

Full learning curves and all 18 cells: pngwn/memcap-results. Training script: train_memcap.py in that repo. This same checkpoint also lives on branch l_n172000_s0_fp32 of pngwn/memcap-runs.

Usage

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("pngwn/memcap-l-above-capacity")

Note: no tokenizer ships with this checkpoint — the training data is raw token ids in [0, 2048).