\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{natbib}
\usepackage{listings}
\lstset{basicstyle=\ttfamily\small, breaklines=true, frame=single}

\title{Memory as an Additive Side-Path: Lookup Niches, Combine
Mechanisms, and Learned Depth Allocation in a Bilingual MoE}

\author{Wei Ciao Wu\thanks{Independent researcher. Code, corpora, and checkpoints:
\texttt{github.com/<TBD>/circus-0.2} and \texttt{huggingface.co/wcamon}.}\\
\and
Claude (Anthropic)\thanks{AI research assistant; experiments and drafting
conducted collaboratively in a single-GPU session.}}
\date{July 2026 --- draft v0.2 (complete: \S1--\S6)}

\begin{document}
\maketitle

\begin{abstract}
We study, at 1B-active-parameter scale, whether a token-keyed
static-memory feed-forward module (STEM) --- indexed directly by token
identity rather than by a learned router --- has any regime in which it
beats an ordinary sparsely-gated mixture-of-experts (MoE) layer, by what
mechanism, and whether the answer can be turned into a learned,
per-token depth-allocation architecture. Three campaigns share one
1B-parameter, single-seed, $\sim$6{,}300-step bilingual (English /
Traditional Chinese) plus code and math wind tunnel.

A five-hypothesis \textbf{niche-mapping} campaign finds three of five
candidate STEM advantages refuted --- two of them \emph{in reverse} ---
and two confirmed: STEM's one genuine quality edge is an
early-training, low-token-budget regime that dilutes to parity by the
shared endpoint and, on code, reverses outright in MoE's favor, while a
systems-efficiency edge (higher decode throughput, fewer OOM-class
crashes) holds unconditionally.

A follow-up $2\times2$ \textbf{mechanism factorial} (token- vs.\
product-key addressing $\times$ multiplicative- vs.\ additive combination
with the dense path) finds the addressing axis second-order for quality
but the combine axis decisive: replacing STEM's multiplicative gate with
an additive residual repairs the code reversal (an 11.9-point BPB swing)
and produces the campaign's only two cells, of four, that beat MoE on
multiple held-out domains --- at zero additional FLOPs, via a
dense-path-preservation mechanism whose benefit is extremely
hub-concentrated yet still diffuses to non-hub positions through the
residual stream.

A \textbf{STEM-router} campaign turns the additive-combine decision into
a Mixture-of-Depths-style layer: an always-on additive STEM path plus a
capacity-gated MoE path, selected by expert-choice top-$k$. At a
FLOP-matched capacity the router beats the all-MoE baseline on 3 of 5
held-out domains with zero losses; at a net-FLOP-saving quarter-capacity
operating point ($-8.5\%$ total MACs, $+4.1\%$ wall-clock) it still wins
3 of 5, with a monotone dose-response between the two points and one
math result close enough to the noise band to flag as a watch item. The
learned gate is depth-heterogeneous: its final three layers are perfectly
frequency-aligned (per-layer Spearman $\rho=-1.00$, MoE capacity
concentrated on the most frequent tokens rather than the pre-registered
rare-favoring direction), a pattern that replicates across three
independently trained arms. A further
three-axis probe finds: a data-driven per-layer capacity schedule wins
over uniform capacity on code at bit-exact equal FLOPs, at the cost of a
small math regression when compared back to the original all-MoE
baseline; a static n-gram key axis --- whether spliced onto or
substituted for the token key --- adds no dividend, and even its best
variant still loses to plain token-keying on code; and a post-hoc causal
decoding threshold matches the training-time sequence-level gate to
within $0.34\%$ bpb on every domain, so the gate score alone carries the
selection signal. Two further frozen-checkpoint probes on the
equal-FLOP arm show the trained gate degrades gracefully, not
catastrophically, when its threshold is dialed to unseen capacities, and
allocates FFN capacity across domains --- a $2.2\times$ spread between
the hungriest and the most economical domain under one shared threshold
--- entirely through the trained gate scores, with no explicit
allocator.

We synthesize the three campaigns into five design laws --- memory as an
additive side-path, token identity as the strongest static key with no
dividend from static n-gram augmentation, learned capacity gating over
uniform allocation, gate scores as a sufficient selection signal, and
capacity as a graceful inference-time dial --- each scoped tightly to
the wind-tunnel protocol that produced it. All results are from a
1B-scale, single-seed, $\sim$6,300-step wind tunnel with a
bits-per-byte-only evaluation; we report them as design priors, not
production claims.
\end{abstract}

\section{Model and Wind-Tunnel Protocol}
\label{sec:setup}

\paragraph{Scope statement.} Every number in this paper comes from a
\textbf{1B-parameter-scale, 6{,}300-step wind-tunnel study}: small,
from-scratch matched-pair models trained on a fixed data slice for a
single epoch, one seed per arm, evaluated with a pre-registered protocol
shared across campaigns. This is deliberate --- the wind tunnel exists
to cheaply falsify or corroborate architectural hypotheses before
committing GPU time to a production-scale run --- but it means every
result below is a \emph{directional prior}, not a scaling law. We say so
explicitly at each headline claim rather than once here and never again.

\paragraph{Model.} \textsc{circus-0.2}'s wind-tunnel arms are a
from-scratch, $\sim$1B-parameter decoder with hidden size 1{,}024,
built as a \textbf{20-layer decoupled attention/FFN stack}: each of the
20 positions is \emph{either} an attention sublayer \emph{or} a
feed-forward sublayer (never both fused into one block), so a
20-character \emph{pattern string} fully specifies the architecture.
% source: hoops/configs/circus02z_bi_s.yaml (num_layers: 20); hoops/model/config.py (layer_kinds docstring + ModelConfig fields; class default num_layers=40, overridden per-experiment by the yaml)
Attention positions (\texttt{'*'}) use grouped-query attention (16
query heads, 4 KV groups, head dim 64) with no positional encoding; every
fifth attention layer is full-causal, the rest use a 1{,}024-token
sliding window. FFN positions are one of: \texttt{'S'} --- \emph{StemFFN},
a token-keyed static-memory module, $y = \mathrm{fc2}(\mathrm{silu}(
\mathrm{fc1}(x)) \odot \mathrm{table}[\mathrm{id}])$, table teacher-initialized
from the base embedding; \texttt{'E'} --- a standard top-2-of-24
sparsely-gated MoE layer \citep{shazeer2017moe}, expert hidden 512; or
\texttt{'R'} --- a routed layer combining both, described in
\S\ref{sec:router}. In every wind-tunnel arm studied in \S\ref{sec:niche}--\S\ref{sec:factorial},
the FFN positions split 6 \texttt{S} / 4 \texttt{E} (or 10 \texttt{E} for
the all-MoE control), and every FFN position additionally carries a
small always-on shared dense expert (hidden 512), constant across arms.
% source: hoops/model/config.py; frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S7 (token table 63,316x1,536/layer x 6 layers)

\paragraph{Data.} All wind-tunnel arms in this paper train from scratch,
byte-identically, on a fixed \textbf{bilingual + code + math slice}:
Traditional Chinese (native script + OpenCC-converted), English (DCLM),
code (StarCoder), and math (Nemotron), $\approx$1.66B tokens ($\approx$6{,}322--6{,}323
steps at global batch 8 $\times$ 32{,}768-token sequences, one epoch, data
exhausts at step 6{,}322; the niche verdict's $\approx$1.63B figure was a
design-time estimate). % source: batch arithmetic $8\times32{,}768\times6{,}322\approx1.657$B tok; $\sum(\mathrm{data\_max\_seqs})\times32{,}768\approx1.658$B tok (hoops/configs/circus02z_bi_s.yaml: data\_max\_seqs=[9737,7584,16662,9155,7451], global\_batch=8, seq\_length=32768); cf.\ frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2 ("$\approx$1.63B tok", design-time estimate)
Held-out evaluation uses a \textbf{carved 5-val protocol}: \texttt{en},
\texttt{zh\_native}, \texttt{zh\_opencc}, \texttt{code}, \texttt{math}
--- val tails carved from the training tail (Chinese/math) or
train-head-only caps with an unseen remainder (English/code), 64
sequences per val unless noted. % source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2; docs/superpowers/specs/2026-07-12-stem-router-design.md \S6

\paragraph{Noise bands.} Headline capability claims (bits-per-byte, BPB)
are judged against a pre-registered \textbf{$\pm$1.5\% relative-BPB noise
band}, reused unchanged across the niche, factorial, and router
campaigns; a stronger dual-criterion test (niche Phase 2) requires
clearing \textbf{2$\times$ the band (3.0\%)}. % source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2; docs/superpowers/specs/2026-07-12-stem-router-design.md \S6
Mechanism-level metrics that are not BPB (rare-token share, content-token
share, spillover CE) instead use a \textbf{split-half $2\sigma$ band}
computed per metric in the factorial campaign (\S\ref{sec:factorial});
the two band conventions are not interchangeable and each table below
states which one applies.

\paragraph{Iso-FLOP discipline.} Every architectural comparison in this
paper is gated on an analytic, per-token active-MAC ledger
(\texttt{frontier/circus\_cp\_agentic/experiments/flop\_count.py}), not
on wall-clock cost, because wall-clock throughput conflates FLOPs with
memory-bandwidth effects (a distinction that turns out to matter,
\S\ref{sec:factorial}). The all-MoE control (E) and the token-keyed STEM
arm (V0) are matched to within \textbf{$-0.16\%$ total active MACs}; the
two product-key arms (V2/V3) carry a \textbf{$+2.34\%$ conditional-FFN
MAC premium} over the token-keyed arms, still inside a pre-registered
\textbf{$\leq$5\% FLOP-match gate}. % source: frontier/circus_cp_agentic/experiments/results/factorial/flop_table.json (cond_ffn_vs_E_pct, total_vs_E_pct); frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S7 ("FLOP-matched ($\leq$5\% gate) but not param-matched")
The same discipline recurs in the STEM-router design (\S\ref{sec:router}):
its primary operating point ($C=0.5$) is chosen specifically to hold
total FFN MACs constant against the all-MoE baseline, not to save
compute.

\paragraph{Campaign lineage.} Three campaigns share this protocol and
this paper's scope: \textbf{niche-mapping} (\S\ref{sec:niche}) asks
\emph{where}, if anywhere, token-keyed STEM beats MoE; \textbf{mechanism
factorial} (\S\ref{sec:factorial}) reuses the niche campaign's matched
S/E pair as two of its four cells (equivalence-gated: the factorial's
reproduction of the S arm's \texttt{en} BPB matches the niche record to
four decimal places) and asks \emph{why}, decomposing the STEM
mechanism into an addressing axis and a combine axis; \textbf{STEM-router}
(\S\ref{sec:router}) asks whether the mechanism factorial's answer can
be turned into a learned, per-token depth-allocation policy. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md (header equivalence gate); \S1 of frontier/circus_cp_agentic/experiments/results/factorial/judgment.json ("equivalence\_gate": pass)

\section{Lookup Niches: Where Does Token-Keyed Memory Win?}
\label{sec:niche}

\paragraph{S/E placement.} Every S-arm in this paper --- the original
agentic-mix small pair and the bilingual pair (\texttt{bi\_s}) reused
throughout \S\ref{sec:factorial} --- uses the identical FFN placement
\texttt{pattern = "*E*S*S*E*S*S*E*S*S*E"}: of the 10 FFN positions, 6
are STEM and 4 are MoE, i.e.\ a fixed \textbf{60\% majority} of FFN
capacity is static per-token lookup. The all-MoE control (\texttt{bi\_e})
uses \texttt{"*E*E*E*E*E*E*E*E*E*E"} (10 of 10 MoE), differing from the
S arm \emph{only} in this one pattern string. % source: hoops/configs/circus02z_bi_s.yaml, circus02z_bi_e.yaml; frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S7 ("token table ... x 6 layers")
This placement was never swept in either campaign: the question asked
is whether STEM has \emph{any} niche, not how much of the stack it
should occupy.

\paragraph{Five pre-registered hypotheses.} The niche-mapping campaign
tested five candidate STEM advantages against plain MoE, each with a
mechanical, pre-registered adjudication rule (Table~\ref{tab:niche-verdict}).

\begin{table}[h]
\centering
\small
\begin{tabular}{p{0.05\linewidth}p{0.24\linewidth}p{0.12\linewidth}p{0.5\linewidth}}
\toprule
\# & Hypothesis & Verdict & Key evidence \\
\midrule
H1 & Long-tail memory & \textbf{Refuted, reversed} & Zero-tabling STEM's $\Delta$CE cost is largest on \emph{frequent} tokens and falls monotonically toward rare tokens in all 3 well-powered checkpoint/val combinations (tail/head ratio 0.42$\times$--0.71$\times$, opposite of the $\geq$2$\times$ required) \\
H2 & CJK / bilingual niche & \textbf{Refuted} & Endpoint zh-mean BPB gap vs.\ E $= -1.13\%$ (need $< -3.0\%$); en gap $-0.67\%$ also fails the specificity check \\
H3 & Templated-text niche & \textbf{Refuted, reversed} & The (S$-$E) CE gap is \emph{largest} on the most-templated tercile ($+0.0646$) and \emph{smallest} on the least-templated ($+0.0168$) --- 3.84$\times$ in the wrong direction \\
H4 & Systems efficiency & \textbf{Confirmed} & Decode throughput $+88.6\%$ (375.47 vs.\ 199.04 tok/s), TTFT $-9.9\%$, train throughput $+6.1\%$ (105,135 vs.\ 99,073 tok/s), 0 vs.\ 2 \texttt{cudaErrorLaunchTimeout} MoE-kernel crashes \\
H5 & Early-advantage dilution & \textbf{Confirmed} & S leads beyond the noise band at step $\leq$1{,}500 on all 3 language vals, then E catches to within band by the shared endpoint (step 6{,}322) \\
\bottomrule
\end{tabular}
\caption{Niche-mapping executive verdict. Every hypothesis label and
number is quoted or arithmetically derived from the source verdict.}
% source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S1
\label{tab:niche-verdict}
\end{table}

\paragraph{The early-leverage finding (H5).} The pre-registered
criterion for H5 requires an S-lead beyond the $\pm$1.5\% noise band at
step $\leq$1{,}500 on each language val, with the lead gone by the shared
endpoint. All three language vals clear it (Table~\ref{tab:h5}).

\begin{table}[h]
\centering
\small
\begin{tabular}{lrl}
\toprule
Val & Early lead (step, rel.\ gap) & Endpoint-caught \\
\midrule
en & step 1{,}500, $-1.90\%$ & yes \\
zh\_native & step 1{,}500, $-1.56\%$ & yes \\
zh\_opencc & step 1{,}500, $-2.28\%$ (also $-1.74\%$ @ 1{,}000) & yes \\
\bottomrule
\end{tabular}
\caption{H5 pre-registered criterion, all three language vals.}
% source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2, Criterion B
\label{tab:h5}
\end{table}

Two further vals were tracked descriptively (not part of the
pre-registered H5 gate, which is language-only) and sharpen the picture.
\textbf{Math} shows the cleanest dilution: S leads by $-5.36\%$ at step
2{,}000, fading to $-0.53\%$ by the endpoint. \textbf{Code} does not
merely dilute --- it \emph{reverses}: S leads by $-8.28\%$ at step
2{,}500, crosses over to an E-lead near step 3{,}000, and finishes
\textbf{+4.03\% in E's favor} at the shared endpoint (step 6{,}322), the
only val where the eventual winner is not the early leader. % source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2 (descriptive findings + [code]/[math] curve tables)
This code-reversal is the headline finding \S\ref{sec:factorial} sets out
to explain mechanistically.

\paragraph{Systems efficiency (H4) stands independent.} Unlike H1--H3,H5,
H4 is not a quality-niche claim: token lookup has no router and no
grouped-GEMM dispatch, so it is unconditionally faster to decode from and
more stable to train (Table~\ref{tab:niche-verdict}). This advantage is
uncontested by any other finding in this paper.

\paragraph{The $>$50\%-replacement point.} Because placement was fixed at
60\% STEM (never swept, \emph{supra}), this campaign cannot locate a
replacement-fraction \emph{threshold}. What it does show, at the single
majority-replacement point actually tested, is a ceiling: the one domain
that most demands compositional (not lexical-identity) computation ---
code --- is also the one domain where committing 60\% of FFN capacity to
a static per-token table costs quality once training accumulates past
the early-budget regime, rather than merely diluting to parity like
every other val. We read this as evidence that \emph{some} upper bound
on the STEM-replaceable fraction exists on composition-heavy domains,
without claiming 60\% itself is the located threshold: no lower-replacement
arm was ever trained, so whether the wall sits above, at, or below 50\%
remains an open question for future wind-tunnel work.

\section{Combine Mechanisms: Multiplicative Gate vs.\ Additive Side-Path}
\label{sec:factorial}

\paragraph{The $2\times2$ design.} A follow-up factorial decomposes
STEM's mechanism along two axes: \textbf{addressing} (token identity vs.\
a learned product key, PKM-style \citep{lample2019pkm}) and
\textbf{combine} (STEM's native multiplicative SwiGLU gate,
$\mathrm{fc2}(\mathrm{silu}(\mathrm{fc1}(x))\odot\mathrm{table})$, vs.\
an additive residual, $\mathrm{fc2}(\mathrm{silu}(\mathrm{fc1}(x))) + e$).
V0 (token $\times$ mult) \emph{is} the niche campaign's STEM arm, reused
verbatim; E (all-MoE) is its control, also reused; V1 (token $\times$
add), V2 (pk $\times$ mult), and V3 (pk $\times$ add) are new arms
trained on the byte-identical protocol. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md header + \S8

\paragraph{Headline capability.} Table~\ref{tab:headline} reports the
endpoint (step 6{,}322) relative-BPB gap of each arm against E, judged
against the same $\pm$1.5\% noise-band \emph{convention} as \S\ref{sec:niche}
--- only the band convention is shared, not the held-out val sets
themselves; see the protocol note below the table.

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrrrl}
\toprule
Arm vs.\ E & en & zh\_tw & code & math & Beats E on $\geq$2 domains \\
\midrule
V0: token $\times$ mult (= STEM) & $-0.67$ & $-1.14$ & $\mathbf{+4.03}$ & $-0.51$ & no \\
V1: token $\times$ add & $+0.24$ & $\mathbf{-2.17}$ & $\mathbf{-7.85}$ & $+0.02$ & \textbf{yes} \\
V2: pk $\times$ mult & $+0.95$ & $+0.24$ & $-1.33$ & $+1.46$ & no \\
V3: pk $\times$ add & $-0.06$ & $\mathbf{-2.15}$ & $\mathbf{-5.45}$ & $-0.25$ & \textbf{yes} \\
\bottomrule
\end{tabular}
\caption{Relative BPB gap vs.\ E at the shared endpoint (negative = arm
beats E), percent. Bold = outside the $\pm$1.5\% band (sign gives
direction; negative = arm beats E).}
% source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S2
\label{tab:headline}
\end{table}

\paragraph{Protocol note (do not cross-compare with \S\ref{sec:niche}).}
These BPBs are the mechanism-factorial campaign's rsync'd \textbf{4-val
protocol} (\texttt{en}, \texttt{zhtw}, \texttt{code}, \texttt{math}, each
an \texttt{\_text\_document} val). The \texttt{en} and \texttt{code}
columns coincide byte-identically with \S\ref{sec:niche}'s carved-val
endpoints (confirmed by the equivalence gate, \S\ref{sec:setup}); \texttt{zhtw}
and \texttt{math} are \emph{different} held-out sets from \S\ref{sec:niche}'s
carved \texttt{zh\_native}/\texttt{zh\_opencc}/\texttt{math\_slice} vals.
Treat Table~\ref{tab:headline} as within-table contrasts only --- its
cells are not comparable to \S\ref{sec:niche}'s tables. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S7 (disclosed asymmetries: "bpb vals are the rsync'd 4-val protocol ... NOT the niche campaign's carved vals"); frontier/circus_cp_agentic/experiments/results/factorial/judgment.json equivalence_gate (en byte-identical to 4dp); frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S2 (carved 5-val protocol; MID-RUN FIX note on legacy combined zhtw/math \_text\_document vals)

\paragraph{Add repairs the code reversal.} The single largest finding of
this campaign: switching V0's combine mode from multiplicative to
additive (V0$\to$V1) turns code from \textbf{+4.03\% in E's favor to
$-7.85\%$ in the table arm's favor} --- an \textbf{11.9-point BPB swing}
--- and produces the campaign's only two cells (V1, V3) that beat E on
multiple held-out domains at all. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S2 ("11.9-point swing")

\paragraph{Mechanism.} The multiplicative combine forces \emph{every}
token through the table as the SwiGLU up-projection itself: a
composition-heavy domain like code inherits the table's quality on the
critical path. The additive combine keeps the dense compute path intact
(\texttt{silu(fc1(x))}) and adds the table only as a residual bonus,
$+e$. Consistent with this, the additive table's own zero-ablation
cost is $\sim$12$\times$ smaller than the multiplicative table's
($\Delta$CE $+0.012$ for V3 vs.\ $+0.154$ for V2) --- it matters
\emph{less} to the forward pass --- yet endpoint quality is
\emph{better}. \textbf{Memory as an additive side-path beats memory as
a multiplicative gate} --- at this scale and training budget; we make no
claim beyond the wind-tunnel protocol of \S\ref{sec:setup}. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S2

\paragraph{Main effects: combine dominates, addressing is second-order.}
A mechanical $2\times2$ attribution (Table~\ref{tab:maineffects}, split-half
$2\sigma$ bands, distinct from the $\pm$1.5\% BPB band above) confirms
this asymmetry at the level of \emph{where} benefit lands, not just
endpoint BPB: the addressing axis has no effect on how much benefit
reaches rare tokens (rare-share is 0.15--0.18\% in every cell,
regardless of key mechanism --- frequency-reversal is intrinsic to
table memory, not to token-keying); the combine axis's main effect on
content-token share is negative and \emph{opposite} the na\"ive
prediction, but the interaction term is superadditive on the content
axis, meaning V3 (pk + add together) loses less content-share than the
two main effects predict in isolation.

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrl}
\toprule
Axis & Effect & Band ($2\sigma_{\mathrm{split}}$) & Verdict \\
\midrule
Addressing (rare-share, \{V2,V3\}$-$\{V0,V1\}) & $+0.0002$ & $0.0004$ & Refuted \\
Combine (content-share, \{V1,V3\}$-$\{V0,V2\}) & $-0.0303$ & $0.0036$ & Refuted (wrong sign) \\
Interaction (V3 vs.\ additive prediction, content) & $+0.0190$ & $0.0039$ & Superadditive, confirmed \\
\bottomrule
\end{tabular}
\caption{$2\times2$ mechanism attribution on non-BPB metrics
(rare-token / content-token benefit share). Note the band convention
here is a split-half $2\sigma$, not the $\pm$1.5\% relative-BPB band
used elsewhere.}
% source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S1; frontier/circus_cp_agentic/experiments/results/factorial/judgment.json
\label{tab:maineffects}
\end{table}

\paragraph{Hub diffusion.} A separate causal probe zeros only the
top-1\% highest-benefit table rows and measures the CE cost at
\emph{non-hub} positions. In all four arms this spillover clears its
$2\sigma_{\mathrm{split}}$ band (Table~\ref{tab:hub}): hub-row benefit
diffuses through the residual stream to positions those rows never
directly key, and the effect is $\sim$100--200$\times$ larger in the
multiplicative arms, where the hub rows sit on every token's critical
path. Benefit concentration is extreme in every arm (Gini 0.936--0.992;
top-1\% of rows carry 38--77\% of total benefit) --- table memory is
intrinsically hub-shaped, and D1's frequency-reversal
(\S\ref{sec:niche}, H1) is best read as \emph{rational hub investment}
rather than a limitation: frequent tokens are hubs, and improving hubs
diffuses value outward.

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrrr}
\toprule
Arm & Spillover $\Delta$CE & Band ($2\sigma_{\mathrm{split}}$) & Gini & Top-1\% share \\
\midrule
V0 (token $\times$ mult) & $+0.589$ & $0.0041$ & 0.936 & 57\% \\
V1 (token $\times$ add) & $+0.576$ & $0.0090$ & 0.942 & 60\% \\
V2 (pk $\times$ mult) & $+0.032$ & $0.0002$ & 0.976 & 77\% \\
V3 (pk $\times$ add) & $+0.0024$ & $0.0002$ & 0.992 & 38\% \\
\bottomrule
\end{tabular}
\caption{H-hub: zeroing the top-1\% benefit rows degrades CE on
\emph{non-hub} positions in every arm --- diffusion confirmed.}
% source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S3
\label{tab:hub}
\end{table}

\paragraph{The product-key bandwidth tax.} Product-key addressing (V2,
V3) is second-order for quality (Table~\ref{tab:maineffects}) but not
free: the top-16 gather over a 262{,}144-row shared table costs only
$+2.34\%$ active MACs against the token-keyed arms (well inside the
$\leq$5\% iso-FLOP gate, \S\ref{sec:setup}) yet costs $\mathbf{-15\%}$
wall-clock throughput --- the selection cost here is a
\textbf{memory-bandwidth tax}, not a FLOP cost, and grouped-gather
dispatch is not free even when the arithmetic is. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S2 ("pk = pure -15\% bandwidth tax")
A companion economics check (H-offload) finds routing predictability is
\emph{not} lower in the table-augmented arms than in plain MoE (argmax
accuracy $0.9115$ vs.\ $0.9098$, within a $0.0344$ band) --- the division
of labor between STEM and MoE shows up in \emph{how much} compensation
each decile needs, not in the router's own confidence. % source: frontier/circus_cp_agentic/experiments/results/factorial/judgment.json h_offload.band_2sig (0.0344, same e13_mean_table_arms=0.9115/e13_E=0.9098 comparison); frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S3 (H-offload)
Both product-key arms were confirmed \textbf{trained}, not merely
cold-started, before any of the above was adjudicated (zero-table
$\Delta$CE hurts in both; row drift 97--100\% by the endpoint), ruling
out undertraining as an alternative explanation for the pk arms'
weaker showing. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S4

\paragraph{Downstream decision.} Taken together --- combine dominates
addressing, additive combine is FLOP-neutral and beats E on multiple
domains, product keys buy nothing on quality and cost throughput --- the
mechanism factorial's design-relevant output is a single decision:
\textbf{combine = additive}. The STEM-router campaign (\S\ref{sec:router})
inherits this by construction. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S8 (DP-1)

\section{STEM-Router: Learned Depth Allocation}
\label{sec:router}

\paragraph{Motivation.} \S\ref{sec:niche}--\S\ref{sec:factorial} leave two
facts sitting side by side: STEM's early-training lead dilutes to parity
(or, on code, reverses) by the shared endpoint (H5), and the additive
combine that repairs the reversal makes the table's own contribution to
the forward pass \emph{smaller}, not larger, while endpoint quality
improves (\S\ref{sec:factorial}, table zero-ablation $\Delta$CE
12$\times$ down). Both point at the same design move: stop asking a
static table to do the same job at every training step and every stack
position, and instead let a learned, per-token policy decide when the
extra, composition-capable FLOPs of an MoE layer are worth spending on
top of an always-on additive memory floor. This section reports that
architecture, \textbf{STEM-router} (layer type \texttt{R}), a
Mixture-of-Depths-style design \citep{raposo2024mod}, pre-registered as
three hypotheses (H-R, H-SE, H-RT) and evaluated end to end, then probed
along three further axes. % source: docs/superpowers/specs/2026-07-12-stem-router-design.md \S1

\paragraph{The routed layer.} Every FFN position in the router arms is
layer type \texttt{R} (pattern all-\texttt{R}, 10 attention + 10 R
positions), replacing \S\ref{sec:niche}--\S\ref{sec:factorial}'s
\texttt{S}/\texttt{E}/shared-expert stack outright. Each R layer computes
$y = x + \mathrm{StemFFN}(x) + \sigma(\mathrm{score}(x))\cdot
\mathrm{MoeFFN}(x)$ for tokens admitted by the gate, and
$y = x + \mathrm{StemFFN}(x)$ for tokens not admitted, where StemFFN is
the additive-combine variant that \S\ref{sec:factorial}'s downstream
decision (DP-1) selected, running on \emph{every} token as an always-on
residual, and the unchanged top-2-of-24 MoE layer runs conditionally,
gated by a per-token sigmoid score, only on tokens admitted by
\textbf{expert-choice top-$k$}: each (sequence, layer) admits the top
$k=\lceil C\cdot T\rceil$-scoring tokens, $T$ the sequence length.
Expert-choice is a hard capacity floor by construction --- a fixed
fraction of tokens is always admitted regardless of the score
distribution --- so it doubles as insurance against gate collapse; a
second insurance is a $+2.0$ gate-bias initialization
($\sigma(2)\approx0.88$, so the MoE path starts near-fully open and the
gate learns to \emph{withhold} capacity, never to earn it from zero)
plus a capacity anneal from $C=1.0$ (every token selected, full
gradient to the MoE layer) over a 1{,}500-step warmup down to the target
operating point $C$ by step 2{,}000, held fixed thereafter. The shared
dense expert that ran unconditionally in \S\ref{sec:niche}--\S\ref{sec:factorial}
is removed (\texttt{shared\_ffn\_hidden=0}); the always-on StemFFN is
its replacement. % source: docs/superpowers/specs/2026-07-12-stem-router-design.md \S3.1, \S4, \S9; frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md header

\paragraph{Three pre-registered hypotheses.} \textbf{H-R} (primary): at
equal FLOPs, always-on-STEM-recall plus router-allocated composition
beats uniform allocation (every token through shared-expert-plus-MoE),
judged by an endpoint decision rule at the same $\pm$1.5\% noise band as
\S\ref{sec:niche}--\S\ref{sec:factorial}: no loss on any of the five
held-out vals, with at least one outright win (only the band convention
is shared: \S\ref{sec:niche} applies per-hypothesis decision rules,
e.g.\ H2's 2$\times$-band $-3.0\%$ criterion, and \S\ref{sec:factorial}'s
headline column is a separate ``beats E on $\geq$2 domains'' rule).
\textbf{H-SE} (secondary): removing the shared expert costs nothing once
StemFFN covers its recall load (the ``no val loses'' clause of H-R's own
criterion, \emph{supra}). \textbf{H-RT} (router health, tertiary, three sub-criteria):
the learned gate reproduces two offline priors --- frequent tokens
routed away from MoE (b1, pre-registered as the \emph{mean-over-the-10
R-layers} Spearman $\rho>0.5$ between frequency decile, $d_0=$
most-frequent, and MoE selection rate; the spec's orientation expects
selection rate to \emph{rise} with decile index, i.e.\ toward rarer
deciles, so a positive layer-mean $\rho$ under that orientation is the
anti-frequency signature) and composition-heavy domains (code, math)
routed toward MoE more than language domains (b2, higher mean
pre-sigmoid gate score) --- plus a non-degeneracy floor (b3, gate-score
std $>0.01$ across tokens, per layer).
% source: docs/superpowers/specs/2026-07-12-stem-router-design.md \S2, \S6 (criterion B verbatim: "頻率軸:終點 ckpt 上,e9 decile(d0=最高頻)與該 decile 的 MoE 入選率之 Spearman ρ > 0.5(跨 10 個 R 層平均;入選率應隨 decile index 上升)")

\paragraph{v1 verdict: an equal-FLOP win that survives becoming a
FLOP-saving win.} The primary arm, \texttt{sr50} ($C=0.5$), is
FLOP-matched to the all-MoE baseline \texttt{bi\_e} by construction ---
StemFFN's extra always-on cost is offset almost exactly by halving MoE's
conditional-token budget. It clears H-R and H-SE outright at its
endpoint (Table~\ref{tab:router-v1} caption): 3 wins, 0 losses
(Table~\ref{tab:router-v1}). A gated follow-up, \texttt{sr25} ($C=0.25$),
asks whether the win survives when the arm stops being FLOP-neutral and
starts saving FLOPs; it does, unchanged at 3 wins / 0 losses, and the
size of every gap moves toward E by roughly half as $C$ halves (code
$-8.77\%\to-4.73\%$, zh\_native
$-2.96\%\to-2.10\%$, en $-0.48\%\to+0.53\%$) --- a real, monotone
dose-response, not noise. One watch item: sr25's math gap ($+1.43\%$)
sits close to the $+1.5\%$ band edge, though still in-band; sr50's math
gap was $-0.85\%$, so a math-sensitive deployment would prefer the
higher-capacity point. % source: frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S1, \S2, \S8

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
val & sr50 bpb & sr25 bpb & bi\_e bpb & sr50 gap & sr25 gap \\
\midrule
en & 1.2216 & 1.2339 & 1.2274 & $-0.48$ & $+0.53$ \\
zh\_native & 1.6099 & 1.6241 & 1.6589 & $\mathbf{-2.96}$ & $\mathbf{-2.10}$ \\
zh\_opencc & 1.4008 & 1.4161 & 1.4530 & $\mathbf{-3.59}$ & $\mathbf{-2.53}$ \\
code & 1.4071 & 1.4694 & 1.5423 & $\mathbf{-8.77}$ & $\mathbf{-4.73}$ \\
math & 0.8576 & 0.8774 & 0.8650 & $-0.85$ & $+1.43$ \\
\bottomrule
\end{tabular}
\caption{STEM-router endpoint bpb vs.\ the all-MoE baseline
\texttt{bi\_e}, two capacities. \texttt{sr50} endpoint step 6{,}309
(13 steps short of the default 6{,}322 due to a disclosed disk-full
incident, biased conservative against H-R); \texttt{sr25} and
\texttt{bi\_e} both at the default endpoint 6{,}322. Bold = outside the
$\pm$1.5\% band; sign negative = router arm beats \texttt{bi\_e}. 3
wins / 0 losses at both capacities.}
% source: frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S1 (sr50), \S8 (sr25)
\label{tab:router-v1}
\end{table}

\paragraph{Economics.} The per-token cost of one routed layer in
\texttt{sr50} is 4{,}731{,}904 MACs (Table~\ref{tab:router-econ}), of
which the conditional MoE component is 49.26\% of \texttt{bi\_e}'s MoE
cost; total active MACs (summed across all 10 R layers plus attention)
are $-0.12\%$ vs.\ E --- FLOP-matched by design, not by accident.
Wall-clock throughput is $97.9$--$98.1$k tok/s
vs.\ E's $99.4$k ($-1.4\%$): the MoD dispatch tax (gather/top-$k$) is
almost fully offset by halving the MoE token count. At $C=0.25$ the
economics flip sign entirely: total active MACs are $86.58$M vs.\ E's
$94.62$M ($-8.50\%$), and throughput is \emph{faster} than E's, $103.5$k
vs.\ $99.4$k tok/s ($+4.1\%$) --- this arm is unconditionally cheaper to
run, not merely differently allocated, and it still wins 3 of 5 domains.
For contrast, \S\ref{sec:factorial}'s product-key selection cost $-15\%$
throughput for no quality gain; MoD's layer-level selection costs at
most $-1.4\%$ and delivers real wins --- selecting \emph{which tokens
reach a layer} is roughly an order of magnitude cheaper than selecting
\emph{which memory rows a token gathers}, at this scale.
% source: frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S4, \S8

\begin{table}[h]
\centering
\small
\begin{tabular}{lrr}
\toprule
Metric vs.\ \texttt{bi\_e} & sr50 ($C{=}0.5$) & sr25 ($C{=}0.25$) \\
\midrule
Total active MACs & $-0.12\%$ (FLOP-matched) & $-8.50\%$ \\
Wall-clock tok/s & 97.9--98.1k vs.\ 99.4k ($-1.4\%$) & 103.5k vs.\ 99.4k ($+4.1\%$) \\
\bottomrule
\end{tabular}
\caption{STEM-router economics at two capacities. \texttt{sr50} is
FLOP-matched to \texttt{bi\_e} by design (always-on StemFFN cost offset
by halved MoE token budget); \texttt{sr25} trades allocation-neutrality
for net FLOP savings.}
% source: frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S4, \S8
\label{tab:router-econ}
\end{table}

\paragraph{What the gate learned.} H-RT does not pass as pre-registered:
two of its three sub-criteria fail at the mean level --- b1 (frequency
tracking, layer-mean Spearman $\rho\approx-0.02$ in \texttt{sr50} and
$-0.24$ in \texttt{sr25}, both far short of the pre-registered mean
$\rho>+0.5$ threshold) and b2 (domain ranking) --- while b3, the non-degeneracy floor, PASSES
(min per-layer gate-score std $0.497$ in \texttt{sr50}, $0.468$ in
\texttt{sr25}, both $\gg0.01$): the gate is strongly differentiated, not
collapsed, so the anti-collapse insurance built into the design
(\emph{supra}) worked as intended. The b1 mean-level failure is
nonetheless informative: it hides a strongly heterogeneous
\emph{per-layer} structure. \textbf{Convention}: Table~\ref{tab:router-rho}
reports $\rho$ in b1's own pre-registered orientation --- Spearman
correlation between frequency-decile index ($d_0=$ most-frequent,
$d_9=$ rarest) and per-decile MoE selection rate --- so a positive
$\rho$ is the rare-favoring signature b1 sought (selection rate rising
with rarity) and a negative $\rho$ is the opposite, frequency-aligned
pattern (selection rate falling with rarity, i.e.\ concentrating on the
most frequent tokens). By that convention, the mid-depth layers
(\texttt{sr50} L3--L6: $+0.66$ to $+0.84$) are modestly rare-favoring,
roughly the direction b1 predicted; the last three of ten R layers, by
contrast, are perfectly \emph{frequency-aligned} ($\rho=-1.00$) in
\texttt{sr50} --- raw per-decile selection rates fall monotonically from
$0.53$--$0.59$ at $d_0$ to $0.10$--$0.16$ at $d_9$ --- the maximal
\emph{violation}, not satisfaction, of the pre-registered prior: these
three deepest layers spend MoE capacity almost exclusively on the tokens
the model already predicts well from parametric memory, and leave rare
tokens to the always-on stem side-path. The identical deep-three,
frequency-aligned pattern reappears, independently trained, in
\texttt{sr25} and again in \texttt{srpl} (\emph{infra}) --- a triple
replication of the frequency-aligned, not rare-favoring, deep policy
(Table~\ref{tab:router-rho}). The mean-level FAIL verdict is unchanged
by this reading: the other seven of ten layers pull the 10-layer mean
back toward zero or the rare-favoring sign, so b1 fails as
pre-registered, and the reading clarifies why --- the learned policy is
heterogeneous by outcome, not merely noisy: mid-depth layers lean
rare-favoring while the deepest three lean maximally frequency-aligned.
Two further regularities: gate-score
std \emph{grows with depth within} \texttt{sr25} (from $\approx0.47$ at
the shallowest gate to $2.11$ at layer 10; the cross-arm minimum
per-layer std is $0.497$ in \texttt{sr50} vs.\ $0.468$ in \texttt{sr25}
--- both far above the $0.01$ floor, and this cross-arm comparison is
not itself a depth trend), and the mean pre-sigmoid gate score shrinks
as capacity shrinks (mean sigmoid $\approx0.10$ at $C=0.5$ vs.\
$\approx0.06$ at $C=0.25$) --- with fewer MoE slots available, the model
does not fight for amplitude on the slots it keeps; it shrinks the
routed path's average gain instead, the same ``small additive bonus''
regime \S\ref{sec:factorial} found for the additive table itself (V3's
$\Delta$CE 12$\times$ smaller than V2's, yet better quality). Reading:
quality does not require the naive ``spend FFN where it's hardest''
policy; the deepest layers instead spend their MoE budget on the most
frequent tokens and leave rare tokens to the memorized stem side-path,
while the routed path, on average, functions as a small correction
rather than a load-bearing default.
% source: frontier/circus_cp_agentic/experiments/results/router/e14_sr50.json (raw per-decile selection rates); STEM_ROUTER_VERDICT.md \S3 (corrected 2026-07-14, commit f1c58f1), \S8, \S9.1 (mean rho -0.02 / -0.24, min std 0.497 / 0.468, layer-10 std 0.47->2.11)

\begin{table}[h]
\centering
\footnotesize
\begin{tabular}{lrrrrrrrrrr}
\toprule
Arm & L1 & L2 & L3 & L4 & L5 & L6 & L7 & L8 & L9 & L10 \\
\midrule
sr50 ($C{=}0.5$) & $-0.35$ & $-0.33$ & $+0.84$ & $+0.79$ & $+0.66$ & $+0.81$ & $+0.33$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ \\
sr25 ($C{=}0.25$) & $-0.53$ & $-0.48$ & $+0.26$ & $+0.27$ & $+0.54$ & $+0.92$ & $-0.38$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ \\
\bottomrule
\end{tabular}
\caption{Per-layer Spearman $\rho$ (frequency-decile index, $d_0=$
most-frequent, vs.\ selection rate) at the endpoint, two independently
trained capacities. The last three of ten R layers are perfectly
frequency-aligned in both (bold) --- the mirror of the pre-registered
rare-favoring b1 direction; a third independently trained arm,
\texttt{srpl} (below), replicates the same deep-three pattern.}
% source: frontier/circus_cp_agentic/experiments/results/router/e14_sr50.json; STEM_ROUTER_VERDICT.md \S3 (sr50, corrected 2026-07-14, commit f1c58f1), \S8 (sr25), \S9.1 (srpl replication)
\label{tab:router-rho}
\end{table}

\paragraph{v2, axis (a): does a data-driven per-layer schedule beat
uniform capacity?} The deep-three/mid-layer heterogeneity above motivates
a hand-designed per-layer capacity schedule, $C_l=$
$(0.125, 0.125, 0.40625, 0.40625, 0.40625, 0.40625, 0.25, 0.125, 0.125,
0.125)$ for R layers 1--10 (sum $2.5$ bit-exactly, $0.40625=13/32$;
dyadic fractions chosen so the schedule is bit-exact iso-FLOP against
\texttt{sr25}'s uniform $C=0.25$): starve the confidently
frequency-aligned deep layers and the weakly frequency-aligned shallow
pair down to $0.125$, feed the rare-favoring mid layers up to
$0.40625$, leave the one layer
whose sign flips between \texttt{sr50} and
\texttt{sr25} at a neutral $0.25$. The resulting arm, \texttt{srpl},
clears the pre-registered H-PLC criterion (no-loss vs.\ \texttt{sr25} at
equal FLOPs, $\geq$1 win): code wins outright, and every other domain is
in-band (Table~\ref{tab:router-srpl}). Against the original baseline
\texttt{bi\_e}, however, the picture is not a free lunch: three domains
still win, but math crosses into a loss ($+1.57\%$, $0.07$pp over band)
--- reallocating capacity by depth buys code at a small math cost, a
domain-preference knob rather than a strictly dominant schedule. The
deep-three frequency-aligned pattern replicates a third time under this
schedule. % source: frontier/circus_cp_agentic/experiments/results/router/srpl_verdict.txt; STEM_ROUTER_VERDICT.md \S9.1 (corrected 2026-07-14, commit f1c58f1); docs/superpowers/specs/2026-07-13-stem-router-v2-design.md \S2

\begin{table}[h]
\centering
\small
\begin{tabular}{lrr}
\toprule
val & srpl vs.\ sr25 (iso-FLOP) & srpl vs.\ bi\_e (context) \\
\midrule
en & $+0.22$ & $+0.74$ \\
zh\_native & $-0.01$ & $\mathbf{-2.11}$ \\
zh\_opencc & $+0.03$ & $\mathbf{-2.51}$ \\
code & $\mathbf{-2.91}$ & $\mathbf{-7.50}$ \\
math & $+0.13$ & $\mathbf{+1.57}$ \\
\bottomrule
\end{tabular}
\caption{E-A per-layer capacity schedule (\texttt{srpl}), endpoint step
6{,}322 both comparisons. Left column: primary pre-registered criterion,
bit-exact iso-FLOP vs.\ \texttt{sr25}'s uniform $C=0.25$ (no-loss,
$\geq$1 win $\Rightarrow$ H-PLC confirmed). Right column: descriptive
context vs.\ the original baseline \texttt{bi\_e} (same total capacity
sum as \texttt{sr25}, different per-layer shape --- not the primary
criterion). Bold = outside $\pm$1.5\% band; bold positive on math = a
loss. Left column adjudicated by \texttt{analyze\_v2.py} (pre-registered
pairwise criterion); right column adjudicated by \texttt{analyze\_sr.py}
(descriptive vs-\texttt{bi\_e} context, same script as the STEM-router
v1 verdict, Table~\ref{tab:router-v1}).}
% source: frontier/circus_cp_agentic/experiments/results/router/srpl_verdict.txt (block 1 = analyze_v2.py, block 2 = analyze_sr.py); STEM_ROUTER_VERDICT.md \S9 header
\label{tab:router-srpl}
\end{table}

\paragraph{v2, axis (b): can a context-aware key replace the static
n-gram ladder?} A second axis asks whether concatenating a bigram-hashed
key onto (or in place of) the token-identity key buys anything, holding
the STEM channel width and total FLOPs fixed at \texttt{sr50}'s level
(width 1{,}536). \texttt{srng} splits the width into a token block
(1{,}024) and a hashed-bigram backoff block (512, 131{,}072 slots);
\texttt{srck} drops the token block and uses a bigram-only key at full
width (86{,}016 slots). The pre-registered prediction was that
\texttt{srck} loses to \texttt{srng} on en/code (token identity is
load-bearing, and dropping it costs specificity); the outcome refutes
this: \texttt{srck} beats \texttt{srng} on \emph{all five} domains
(Table~\ref{tab:router-eb}). But neither arm beats the simpler
\texttt{sr50} baseline convincingly: \texttt{srng} loses to \texttt{sr50}
on all five domains ($+1.84\%$ to $+6.33\%$), and \texttt{srck} is only
in-band on four of five, losing code by $+2.43\%$. The split ladder is
dominated \emph{from both directions} --- worse than pure token keying
and worse than pure bigram keying --- so the failure is not ``bigrams
don't help,'' it is ``carving width away from the token table to make
room for a backoff table is a bad trade at this scale.'' Bigram lookup
also costs throughput independent of quality ($\sim$2\% tok/s tax on
both \texttt{srng} ($\approx$96.0k tok/s) and \texttt{srck}
($\approx$96.1k tok/s) against \texttt{sr50} ($\approx$98.0k tok/s)).
Token-only addressing remains the strongest static STEM key found in
this paper; any context-aware key would need to beat \texttt{sr50}
directly, not the ladder. % source: frontier/circus_cp_agentic/experiments/results/router/eb_replacement_verdict.txt; STEM_ROUTER_VERDICT.md \S9.2; hoops/runs/*/train.log via eb_replacement_verdict.txt (sr50 ~98.0k, srng ~96.0k, srck ~96.1k tok/s -> ~2% tax, corrected from an earlier ~6% error)

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrr}
\toprule
val & srck vs.\ srng (primary) & srng vs.\ sr50 (context) & srck vs.\ sr50 (context) \\
\midrule
en & $\mathbf{-2.29}$ & $\mathbf{+1.84}$ & $-0.50$ \\
zh\_native & $\mathbf{-2.41}$ & $\mathbf{+2.17}$ & $-0.29$ \\
zh\_opencc & $\mathbf{-2.88}$ & $\mathbf{+2.58}$ & $-0.37$ \\
code & $\mathbf{-3.67}$ & $\mathbf{+6.33}$ & $\mathbf{+2.43}$ \\
math & $\mathbf{-4.37}$ & $\mathbf{+4.33}$ & $-0.23$ \\
\bottomrule
\end{tabular}
\caption{E-B key-replacement axis, both arms FLOP-identical to
\texttt{sr50} (STEM channel width 1{,}536). Primary column: pre-registered
criterion, \texttt{srck} (bigram-only, 86{,}016 slots) vs.\ \texttt{srng}
(token+bigram split ladder, 131{,}072 bigram slots), both at endpoint
6{,}322 --- replacement holds (5 wins, 0 losses), refuting the
pre-registered prediction. Context columns: descriptive comparison
against \texttt{sr50} (token-only, endpoint 6{,}309 --- a different
endpoint step than the two v2 arms, per the disclosed sr50 deviation,
Table~\ref{tab:router-v1}); both context arms lose to token-only on at
least one domain. Bold = outside $\pm$1.5\% band.}
% source: frontier/circus_cp_agentic/experiments/results/router/eb_replacement_verdict.txt
\label{tab:router-eb}
\end{table}

\paragraph{v2, axis (c): does the gate need to see the future?}
Expert-choice top-$k$ ranks every token in a sequence against every other
token before selecting, which is unavailable at autoregressive decode
time; the gate \emph{score} itself, however, is already causal (a
function of the current token's hidden state only). The third axis
tests whether a decode-compatible substitute --- a fixed per-layer
quantile threshold $\tau_l=\mathrm{quantile}(\mathrm{scores},\,1-C)$,
calibrated post-hoc on a held-out training-slice tail and frozen on
\texttt{sr50}'s final checkpoint --- reproduces expert-choice's
sequence-level competition. It does, comfortably: every domain's
threshold-mode bpb is within the pre-registered $+1.5\%$ gate of the
expert-choice oracle, with the largest observed gap only $+0.34\%$
(Table~\ref{tab:router-ec}). Sequence-level top-$k$ competition therefore
carries almost no information beyond a fixed quantile of the (already
causal) gate score --- a MoD-style decode path needs only a calibrated
threshold, not an auxiliary predictor head co-trained with the model.
% source: frontier/circus_cp_agentic/experiments/results/router/ec_verdict.txt; STEM_ROUTER_VERDICT.md \S9.3

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrr}
\toprule
val & threshold bpb & oracle bpb & gap \\
\midrule
en & 1.2216 & 1.2216 & $+0.00\%$ \\
zh\_native & 1.6132 & 1.6099 & $+0.20\%$ \\
zh\_opencc & 1.4056 & 1.4008 & $+0.34\%$ \\
code & 1.4081 & 1.4071 & $+0.07\%$ \\
math & 0.8566 & 0.8576 & $-0.12\%$ \\
\bottomrule
\end{tabular}
\caption{E-C causal-threshold probe: post-hoc per-layer quantile
threshold $\tau_l$ (calibrated on an English train-slice tail) vs.\ the
native expert-choice oracle, both evaluated on the frozen \texttt{sr50}
checkpoint at step 6{,}309. All gaps clear the pre-registered $+1.5\%$
gate; largest observed gap $+0.34\%$ (zh\_opencc).}
% source: frontier/circus_cp_agentic/experiments/results/router/ec_verdict.txt
\label{tab:router-ec}
\end{table}

\paragraph{Two frozen-checkpoint probes: capacity as an inference-time
dial.} Given a causal threshold reproduces training-time selection
(axis c), two further post-hoc probes ask what happens when the
\emph{same} frozen \texttt{sr50} checkpoint (trained at a single
capacity, $C=0.5$) is dialed to capacities it never trained at, using
only $\tau$ recalibration --- no further training. \textbf{Elasticity}:
dialing down to $C=0.25$ costs $+1.7\%$ to $+3.5\%$ bpb depending on
domain, dialing down again to $C=0.125$ costs a further $+1.4\%$ to
$+6.0\%$ (roughly doubling the increment again on en/math, roughly flat
on zh\_native/zh\_opencc, and \emph{smaller} on code), and dialing
\emph{up} to $C=0.75$ mildly \emph{improves} every domain ($-0.1\%$ to
$-0.5\%$) with no overshoot penalty --- smooth, monotone degradation
with no cliff at any tested point (Table~\ref{tab:router-tau}). Notably,
the dialed-down $C=0.25$ checkpoint (the frozen \texttt{sr50} checkpoint,
step 6{,}309, with $\tau$ recalibrated) beats the \emph{natively trained}
\texttt{sr25} arm (step 6{,}322, a different endpoint, \emph{supra}) on
code ($-2.5\%$), and is within $1.0$--$2.2\%$ elsewhere --- a single
capacity-0.5 training run supports a compute knob at inference time at a
cost comparable to training a second checkpoint outright.
% source: docs/circus-0.4-branch-mod-allocation.md \S2 (Q1); per-domain increments computed from Table~\ref{tab:router-tau} (en 2.31->4.02, zh_native 2.74->2.70, zh_opencc 2.94->2.74, code 1.72->1.35, math 3.49->6.02); comparison vs sr25 recomputed en +1.29 / zh_native +2.05 / zh_opencc +2.17 / code -2.52 / math +1.04

\begin{table}[h]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
$\tau$-cap & en & zh\_native & zh\_opencc & code & math \\
\midrule
0.125 & 1.3001 & 1.7022 & 1.4865 & 1.4516 & 0.9399 \\
0.25 & 1.2498 & 1.6574 & 1.4469 & 1.4323 & 0.8865 \\
0.50 (native) & 1.2216 & 1.6132 & 1.4056 & 1.4081 & 0.8566 \\
0.75 & 1.2202 & 1.6077 & 1.3990 & 1.4019 & 0.8550 \\
\bottomrule
\end{tabular}
\caption{$\tau$-elasticity: bpb of the single, frozen \texttt{sr50}
checkpoint (trained at $C=0.5$, step 6{,}309) with its causal threshold
recalibrated to four capacities, including two it never trained at
($0.125$, $0.75$) and one below its native point ($0.25$). No further
training. The $0.50$ row is the same threshold-mode bpb as
Table~\ref{tab:router-ec}'s threshold column.}
% source: docs/circus-0.4-branch-mod-allocation.md \S2 (Q1 table)
\label{tab:router-tau}
\end{table}

\paragraph{Emergent domain allocation.} \textbf{Selection matrix}: under
one fixed threshold (calibrated on English, $C=0.5$), mean per-token
selection rate varies $2.2\times$ across domains with no explicit
domain signal in the gate --- math is the hungriest domain (mean $0.67$,
layers 1--2 alone select $0.90$--$0.94$ of tokens), code is the most
economical (mean $0.31$; layer 1 selects only $0.053$ of tokens, though
one deep layer spikes to $0.649$), and the two Chinese vals sit at
$0.48$--$0.49$ with a shared, CJK-specific hotspot (layer 2 selects
$0.998$ and $0.981$ of tokens respectively) that English (the calibration
domain, flat at $0.48$) does not share (Table~\ref{tab:router-selmatrix}).
The domain shape is entirely a property of the trained gate scores under
one shared threshold; no explicit domain-conditioned allocator is
involved. % source: docs/circus-0.4-branch-mod-allocation.md \S2 (Q2); frontier/circus_cp_agentic/experiments/results/router/ec_selmatrix.json

\begin{table}[h]
\centering
\small
\begin{tabular}{lrl}
\toprule
Domain & Mean selection rate (10-layer avg.) & Shape notes \\
\midrule
math & 0.67 & layers 1--2: 0.90 / 0.94 (hungriest domain) \\
zh\_native / zh\_opencc & 0.49 / 0.48 & layer 2 saturates: 0.998 / 0.981 \\
en & 0.48 & flat across layers (calibration domain) \\
code & 0.31 & layer 1: 0.053; one deep layer spikes to 0.649 \\
\bottomrule
\end{tabular}
\caption{Domain $\times$ layer selection-rate summary under one fixed
threshold (calibrated on English at cap $0.5$, frozen \texttt{sr50}
checkpoint, step 6{,}309). Full 5-domain $\times$ 10-layer matrix in
\texttt{ec\_selmatrix.json}. Mean rate spans $2.2\times$ (math/code)
under a single shared threshold with no explicit domain signal.}
% source: docs/circus-0.4-branch-mod-allocation.md \S2 (Q2 table); frontier/circus_cp_agentic/experiments/results/router/ec_selmatrix.json
\label{tab:router-selmatrix}
\end{table}

\paragraph{Summary.} Three campaigns now agree on the same shape: memory
belongs on an always-on additive path (\S\ref{sec:factorial}), and the
FLOPs freed by not forcing every token through a multiplicative gate are
better spent by a learned, capacity-floored per-token policy than by
uniform allocation (this section) --- at equal FLOPs (3W/0L) and even
when that policy is asked to save FLOPs outright (still 3W/0L, $-8.5\%$
MACs). The policy the gate lands on is not the naive ``route hard tokens
to more compute'' rule; it is a depth- and domain-heterogeneous
allocation that is legible after the fact (per-layer frequency
heterogeneity --- rare-favoring at mid-depth, frequency-aligned in the
deepest three --- domain-shaped selection rates) but was never specified
as a target.

\section{Design Laws}
\label{sec:design-laws}

\paragraph{Scope.} The following five statements synthesize
\S\ref{sec:niche}--\S\ref{sec:router} into design principles for
token-identity memory and learned depth allocation in a bilingual MoE.
Each is scoped explicitly to the wind-tunnel protocol of
\S\ref{sec:setup} --- a single 1B-parameter, single-seed,
$\sim$6{,}300-step run per arm --- and is a directional prior for future
architecture choices, not an established scaling law.

\paragraph{Law 1: memory belongs on an additive side-path, not a
multiplicative gate.} Routing every token's dense computation through a
lookup table as a multiplicative SwiGLU gate makes a composition-heavy
domain (code) inherit the table's quality on its critical path; adding
the table instead as a residual bonus repairs an 11.9-point BPB swing on
code and produces the only cells in the mechanism factorial that beat
plain MoE on multiple domains, at zero additional FLOPs
(\S\ref{sec:factorial}). The STEM-router campaign inherits this by
construction and confirms it generalizes to a gated setting: the
always-on additive path beats uniform allocation on 3 of 5 domains with
zero losses, twice --- at equal FLOPs ($C=0.5$) and again at a
net-FLOP-saving quarter capacity ($C=0.25$, $-8.5\%$ total MACs)
(\S\ref{sec:router}). Scope: tested only for SwiGLU-style multiplicative
combination vs.\ residual addition; concatenation or cross-attention
combine mechanisms are untested. % source: frontier/circus_cp_agentic/STEM_MECHANISM_FACTORIAL.md \S2 (11.9-point swing, DP-1); frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S1, \S8 (3W/0L twice)

\paragraph{Law 2: token identity is already the strongest static key;
static n-gram augmentation adds no dividend at this scale.} Long-tail
memory (H1) refutes in reverse --- the table helps \emph{frequent}, not
rare, tokens (\S\ref{sec:niche}) --- and product-key addressing buys no
quality while costing $-15\%$ throughput as a pure memory-bandwidth tax
(\S\ref{sec:factorial}). The router campaign's key-replacement axis
extends this: a hashed-bigram backoff ladder loses to plain token
keying on every domain, and even the best bigram-only variant (which
beats the ladder on all five domains) still loses to token-only on code
(\S\ref{sec:router}, axis b). Composition is where every lookup-keyed
variant is weakest; this paper does not test whether the composition gap
is instead attention-machinery territory, only that no keying variant of
the static table closes it. Scope: one hash-slot count per arm, one
wind-tunnel corpus scale (1.66B tokens); a larger corpus may train
hashed tables past this regime. % source: frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md \S1 (H1 reversal); STEM_MECHANISM_FACTORIAL.md \S2 (pk $-15\%$ tok/s); frontier/circus_cp_agentic/experiments/results/router/eb_replacement_verdict.txt (ladder dominated both ways)

\paragraph{Law 3: a learned gate with a hard capacity floor beats uniform
always-on allocation at equal FLOPs, and the learned policy is
depth-heterogeneous.} Expert-choice top-$k$ plus an always-on additive
memory floor beats routing every token through a shared expert and full
MoE, at equal FLOPs (3W/0L) and again when the arm is asked to save
FLOPs outright (3W/0L, $-8.5\%$ MACs, \S\ref{sec:router}). The policy the
gate converges on is not naive frequency- or domain-tracking on average;
it is layer-specific, with the deepest three of ten FFN layers perfectly
frequency-aligned --- concentrating MoE capacity on the most frequent
tokens rather than the pre-registered rare-favoring direction --- a
pattern that replicates across three independently trained arms
(Table~\ref{tab:router-rho}). Scope: one pattern string (10
attention + 10 routed FFN, uniform layer count and placement); whether
the depth-heterogeneity is intrinsic to this stack shape or an artifact
of it is untested. % source: frontier/circus_cp_agentic/experiments/results/router/e14_sr50.json; STEM_ROUTER_VERDICT.md \S1, \S8 (3W/0L twice); \S3, \S9.1 (deep-3 $\rho=-1.00$ triple replication, corrected 2026-07-14, commit f1c58f1)

\paragraph{Law 4: the gate score itself carries the selection signal ---
post-hoc thresholding matches oracle competition, and domain-shaped
allocation emerges without an explicit allocator.} A fixed per-layer
quantile threshold, calibrated once on a held-out slice and frozen,
reproduces expert-choice's full sequence-level competition to within
$0.34\%$ bpb on every domain (\S\ref{sec:router}, axis c); under that
same fixed threshold, mean selection rate still varies $2.2\times$ across
domains (math hungriest at $0.67$, code most economical at $0.31$) with
domain-specific per-layer hotspots (a CJK-specific layer-2 saturation at
$\approx$1.0), with no domain label or explicit allocator anywhere in
the mechanism. Scope: probed only post-hoc on a checkpoint trained with
sequence-level expert-choice; a model trained from scratch against a
threshold, rather than adapted to one after the fact, is untested.
% source: frontier/circus_cp_agentic/experiments/results/router/ec_verdict.txt; docs/circus-0.4-branch-mod-allocation.md \S2 (Q2)

\paragraph{Law 5: capacity generalizes across the dial --- a
single-point-trained gate supports an inference-time compute knob at low
cost.} The same frozen checkpoint, trained at one capacity ($C=0.5$),
degrades smoothly and monotonically when its threshold is recalibrated
to unseen capacities from $0.125$ to $0.75$, with no cliff at any tested
point, and at $C=0.25$ it even beats a checkpoint natively trained at
that capacity on code ($-2.5\%$, \S\ref{sec:router}). Scope: one
checkpoint, one calibration domain (English), a $6\times$ capacity range
($0.125$--$0.75$); capacities outside that range, and calibration on
other domains, are untested. % source: docs/circus-0.4-branch-mod-allocation.md \S2 (Q1)

\section{Limitations}
\label{sec:limitations}

\paragraph{Wind-tunnel scale.} Every result in this paper, without
exception, comes from a single-seed, $\sim$1B-parameter,
$\sim$1.66B-token, 6{,}309--6{,}323-step run per arm (\S\ref{sec:setup};
the default endpoint is 6{,}322--6{,}323, but \texttt{sr50} stopped at
6{,}309 due to a disclosed infra incident, \S\ref{sec:router}).
No arm was replicated across seeds; where we report a pattern as
``replicated'' (the deep-three frequency-aligned gate structure,
Table~\ref{tab:router-rho}) we mean replicated across \emph{different
architectural arms trained on the same data slice}, not across
independent seeds of the same arm. Absolute bpb values, throughput
numbers, and any claim not explicitly hedged as directional should be
read as priors for a larger-scale run, not as production numbers
themselves.

\paragraph{Single evaluation endpoint: bpb only.} Every headline number
in this paper is bits-per-byte on held-out text. No downstream task
evaluation (code execution, math answer-checking, instruction-following,
or any agentic/tool-use benchmark) was run against any arm in this
paper's scope. A bpb win is evidence the model's predictive distribution
over held-out bytes improved; it is not evidence of a corresponding gain
on any task built on top of that distribution.

\paragraph{Noise band is one measurement protocol.} All win/loss/in-band
calls use a single pre-registered $\pm$1.5\% relative-bpb band
(\S\ref{sec:setup}), estimated once and reused across all three
campaigns. Values close to the edge of the band --- \texttt{sr25}'s math
gap at $+1.43\%$ (Table~\ref{tab:router-v1}), \texttt{srpl}'s math gap
at $+1.57\%$ (Table~\ref{tab:router-srpl}) --- are calls made against
that one estimate, not against a re-derived per-arm noise floor; a
wider or narrower band, estimated independently, could flip individual
calls that sit within a few tenths of a percent of $\pm$1.5\%.

\paragraph{Carved, not held-out-by-construction, validation sets.} The
5-val protocol's Chinese and math splits are carved from the training
data's tail; the English and code splits are a train-head cap with an
unseen remainder (\S\ref{sec:setup}). Neither carving scheme guarantees
the same kind of held-out-ness as a fully disjoint corpus, and the two
schemes are not identical to each other across languages.

\paragraph{Cross-protocol endpoints are not interchangeable.} Three
distinct comparison protocols appear in this paper and are labeled
individually in each table's caption, not assumed interchangeable: (i)
the niche campaign's carved 5-val bpb, adjudicated by its own phase-2
verdict machinery (\texttt{phase2\_verdict.txt}, commit \texttt{7cfbaa2}),
and the STEM-router v1 campaign's carved 5-val bpb (\texttt{sr50},
\texttt{sr25} vs.\ \texttt{bi\_e}), adjudicated by \texttt{analyze\_sr.py}
--- both against \texttt{bi\_e} and sharing the same carved-5-val
protocol and $\pm$1.5\% band, but \emph{not} the same adjudicator
script; (ii) the mechanism factorial's separate rsync'd 4-val protocol
(\S\ref{sec:factorial}); and (iii) the router v2 campaign's
pre-registered pairwise endpoint comparisons (\texttt{srpl}-vs-\texttt{sr25},
\texttt{srck}-vs-\texttt{srng}, \texttt{srng}/\texttt{srck}-vs-\texttt{sr50}),
adjudicated by \texttt{analyze\_v2.py}, with \texttt{analyze\_sr.py}
additionally supplying the descriptive vs-\texttt{bi\_e} context for v2
arms (e.g.\ Table~\ref{tab:router-srpl}'s \texttt{srpl}-vs-\texttt{bi\_e}
column, including the math $+1.57\%$ figure cited above). Within (i),
\texttt{sr50}'s final checkpoint is step 6{,}309 (13 steps short of the
default 6{,}322, a disclosed disk-full-induced early data exhaustion,
\S\ref{sec:router}), while \texttt{sr25} and all v2 arms (\texttt{srpl},
\texttt{srng}, \texttt{srck}) reach the default endpoint 6{,}322 ---
every table that mixes an \texttt{sr50}-anchored comparison with a
6{,}322-endpoint arm states both endpoint steps explicitly
(Tables~\ref{tab:router-eb}, \ref{tab:router-tau}). % source: frontier/circus_cp_agentic/experiments/results/router/STEM_ROUTER_VERDICT.md \S2 (sr50 deviation), \S9 header (analyze\_v2.py + analyze\_sr.py); frontier/circus_cp_agentic/experiments/results/niche/STEM_NICHE_VERDICT.md (phase2_verdict.txt adjudicator, commit 7cfbaa2); analyze_sr.py / analyze_v2.py docstrings; \S\ref{sec:factorial} protocol note

\paragraph{Hash-collision regime tested at one slot count.} The
key-replacement axis's bigram tables (\S\ref{sec:router}, axis b) were
tested at one slot count each (131{,}072 for the split ladder,
86{,}016 for the bigram-only arm), sized to this wind tunnel's 1.66B-token
corpus. Neither the collision rate nor the quality effect of a
substantially larger or smaller table was measured; the conclusion that
static n-gram keys add no dividend applies to this slot-count regime,
not to hash tables in general. % source: frontier/circus_cp_agentic/experiments/results/router/eb_replacement_verdict.txt (caveats)

\paragraph{Pruned intermediate snapshots.} \texttt{srck}'s bpb curve
between steps 3{,}000 and 6{,}000 was pruned before the battery ran, to
manage disk footprint; only the endpoint checkpoint was evaluated for
this arm's verdict. The endpoint-only pre-registered criterion is
unaffected, but no dose-response or early-leverage curve (of the kind
Table~\ref{tab:h5} shows for the niche campaign) is available for this
arm. % source: frontier/circus_cp_agentic/experiments/results/router/eb_replacement_verdict.txt (caveats)

\paragraph{Placement and schedule choices were fixed or hand-designed,
not swept.} The niche and factorial campaigns' 60\%-STEM FFN placement
was never swept (\S\ref{sec:niche}); the STEM-router's per-layer capacity
schedule (\texttt{srpl}) was hand-designed from a single prior read-out
of the deep-three frequency-aligned structure, not learned or searched over
alternative shapes (\S\ref{sec:router}, axis a). Both campaigns report a
single tested point, not a swept curve; whether a different placement
fraction or a different, possibly learned, capacity schedule would
dominate the ones tested here is open.

These results inform successor designs but are not, by themselves, a
recipe for one; every number in this paper should be read as we present
it --- a wind-tunnel-scale prior, gated for falsification, not for
production deployment.

\bibliographystyle{plainnat}
\begin{thebibliography}{9}

\bibitem[Shazeer et~al.(2017)]{shazeer2017moe}
N.~Shazeer, A.~Mirhoseini, K.~Maziarz, A.~Davis, Q.~Le, G.~Hinton, J.~Dean.
\newblock Outrageously large neural networks: The sparsely-gated
mixture-of-experts layer.
\newblock \emph{ICLR}, 2017.

\bibitem[Lample et~al.(2019)]{lample2019pkm}
G.~Lample, A.~Sablayrolles, M.~Ranzato, L.~Denoyer, H.~J\'{e}gou.
\newblock Large memory layers with product keys.
\newblock \emph{NeurIPS}, 2019.

\bibitem[Raposo et~al.(2024)]{raposo2024mod}
D.~Raposo, S.~Ritter, B.~Richards, T.~Lillicrap, P.~C.~Humphreys, A.~Santoro.
\newblock Mixture-of-depths: Dynamically allocating compute in
transformer-based language models.
\newblock arXiv:2404.02258, 2024.

\end{thebibliography}

\end{document}