| \documentclass[11pt]{article} |
| \usepackage[margin=1in]{geometry} |
| \usepackage{amsmath,amssymb} |
| \usepackage{booktabs} |
| \usepackage{graphicx} |
| \usepackage{xcolor} |
| \usepackage{hyperref} |
| \usepackage{natbib} |
| \usepackage{listings} |
| \lstset{basicstyle=\ttfamily\small, breaklines=true, frame=single} |
|
|
| \title{Memory as an Additive Side-Path: Lookup Niches, Combine |
| Mechanisms, and Learned Depth Allocation in a Bilingual MoE} |
|
|
| \author{Wei Ciao Wu\thanks{Independent researcher. Code, corpora, and checkpoints: |
| \texttt{github.com/<TBD>/circus-0.2} and \texttt{huggingface.co/wcamon}.}\\ |
| \and |
| Claude (Anthropic)\thanks{AI research assistant; experiments and drafting |
| conducted collaboratively in a single-GPU session.}} |
| \date{July 2026 --- draft v0.2 (complete: \S1--\S6)} |
|
|
| \begin{document} |
| \maketitle |
|
|
| \begin{abstract} |
| We study, at 1B-active-parameter scale, whether a token-keyed |
| static-memory feed-forward module (STEM) --- indexed directly by token |
| identity rather than by a learned router --- has any regime in which it |
| beats an ordinary sparsely-gated mixture-of-experts (MoE) layer, by what |
| mechanism, and whether the answer can be turned into a learned, |
| per-token depth-allocation architecture. Three campaigns share one |
| 1B-parameter, single-seed, $\sim$6{,}300-step bilingual (English / |
| Traditional Chinese) plus code and math wind tunnel. |
|
|
| A five-hypothesis \textbf{niche-mapping} campaign finds three of five |
| candidate STEM advantages refuted --- two of them \emph{in reverse} --- |
| and two confirmed: STEM's one genuine quality edge is an |
| early-training, low-token-budget regime that dilutes to parity by the |
| shared endpoint and, on code, reverses outright in MoE's favor, while a |
| systems-efficiency edge (higher decode throughput, fewer OOM-class |
| crashes) holds unconditionally. |
|
|
| A follow-up $2\times2$ \textbf{mechanism factorial} (token- vs.\ |
| product-key addressing $\times$ multiplicative- vs.\ additive combination |
| with the dense path) finds the addressing axis second-order for quality |
| but the combine axis decisive: replacing STEM's multiplicative gate with |
| an additive residual repairs the code reversal (an 11.9-point BPB swing) |
| and produces the campaign's only two cells, of four, that beat MoE on |
| multiple held-out domains --- at zero additional FLOPs, via a |
| dense-path-preservation mechanism whose benefit is extremely |
| hub-concentrated yet still diffuses to non-hub positions through the |
| residual stream. |
|
|
| A \textbf{STEM-router} campaign turns the additive-combine decision into |
| a Mixture-of-Depths-style layer: an always-on additive STEM path plus a |
| capacity-gated MoE path, selected by expert-choice top-$k$. At a |
| FLOP-matched capacity the router beats the all-MoE baseline on 3 of 5 |
| held-out domains with zero losses; at a net-FLOP-saving quarter-capacity |
| operating point ($-8.5\%$ total MACs, $+4.1\%$ wall-clock) it still wins |
| 3 of 5, with a monotone dose-response between the two points and one |
| math result close enough to the noise band to flag as a watch item. The |
| learned gate is depth-heterogeneous: its final three layers are perfectly |
| frequency-aligned (per-layer Spearman $\rho=-1.00$, MoE capacity |
| concentrated on the most frequent tokens rather than the pre-registered |
| rare-favoring direction), a pattern that replicates across three |
| independently trained arms. A further |
| three-axis probe finds: a data-driven per-layer capacity schedule wins |
| over uniform capacity on code at bit-exact equal FLOPs, at the cost of a |
| small math regression when compared back to the original all-MoE |
| baseline; a static n-gram key axis --- whether spliced onto or |
| substituted for the token key --- adds no dividend, and even its best |
| variant still loses to plain token-keying on code; and a post-hoc causal |
| decoding threshold matches the training-time sequence-level gate to |
| within $0.34\%$ bpb on every domain, so the gate score alone carries the |
| selection signal. Two further frozen-checkpoint probes on the |
| equal-FLOP arm show the trained gate degrades gracefully, not |
| catastrophically, when its threshold is dialed to unseen capacities, and |
| allocates FFN capacity across domains --- a $2.2\times$ spread between |
| the hungriest and the most economical domain under one shared threshold |
| --- entirely through the trained gate scores, with no explicit |
| allocator. |
|
|
| We synthesize the three campaigns into five design laws --- memory as an |
| additive side-path, token identity as the strongest static key with no |
| dividend from static n-gram augmentation, learned capacity gating over |
| uniform allocation, gate scores as a sufficient selection signal, and |
| capacity as a graceful inference-time dial --- each scoped tightly to |
| the wind-tunnel protocol that produced it. All results are from a |
| 1B-scale, single-seed, $\sim$6,300-step wind tunnel with a |
| bits-per-byte-only evaluation; we report them as design priors, not |
| production claims. |
| \end{abstract} |
|
|
| \section{Model and Wind-Tunnel Protocol} |
| \label{sec:setup} |
|
|
| \paragraph{Scope statement.} Every number in this paper comes from a |
| \textbf{1B-parameter-scale, 6{,}300-step wind-tunnel study}: small, |
| from-scratch matched-pair models trained on a fixed data slice for a |
| single epoch, one seed per arm, evaluated with a pre-registered protocol |
| shared across campaigns. This is deliberate --- the wind tunnel exists |
| to cheaply falsify or corroborate architectural hypotheses before |
| committing GPU time to a production-scale run --- but it means every |
| result below is a \emph{directional prior}, not a scaling law. We say so |
| explicitly at each headline claim rather than once here and never again. |
|
|
| \paragraph{Model.} \textsc{circus-0.2}'s wind-tunnel arms are a |
| from-scratch, $\sim$1B-parameter decoder with hidden size 1{,}024, |
| built as a \textbf{20-layer decoupled attention/FFN stack}: each of the |
| 20 positions is \emph{either} an attention sublayer \emph{or} a |
| feed-forward sublayer (never both fused into one block), so a |
| 20-character \emph{pattern string} fully specifies the architecture. |
| |
| Attention positions (\texttt{'*'}) use grouped-query attention (16 |
| query heads, 4 KV groups, head dim 64) with no positional encoding; every |
| fifth attention layer is full-causal, the rest use a 1{,}024-token |
| sliding window. FFN positions are one of: \texttt{'S'} --- \emph{StemFFN}, |
| a token-keyed static-memory module, $y = \mathrm{fc2}(\mathrm{silu}( |
| \mathrm{fc1}(x)) \odot \mathrm{table}[\mathrm{id}])$, table teacher-initialized |
| from the base embedding; \texttt{'E'} --- a standard top-2-of-24 |
| sparsely-gated MoE layer \citep{shazeer2017moe}, expert hidden 512; or |
| \texttt{'R'} --- a routed layer combining both, described in |
| \S\ref{sec:router}. In every wind-tunnel arm studied in \S\ref{sec:niche}--\S\ref{sec:factorial}, |
| the FFN positions split 6 \texttt{S} / 4 \texttt{E} (or 10 \texttt{E} for |
| the all-MoE control), and every FFN position additionally carries a |
| small always-on shared dense expert (hidden 512), constant across arms. |
| |
|
|
| \paragraph{Data.} All wind-tunnel arms in this paper train from scratch, |
| byte-identically, on a fixed \textbf{bilingual + code + math slice}: |
| Traditional Chinese (native script + OpenCC-converted), English (DCLM), |
| code (StarCoder), and math (Nemotron), $\approx$1.66B tokens ($\approx$6{,}322--6{,}323 |
| steps at global batch 8 $\times$ 32{,}768-token sequences, one epoch, data |
| exhausts at step 6{,}322; the niche verdict's $\approx$1.63B figure was a |
| design-time estimate). |
| Held-out evaluation uses a \textbf{carved 5-val protocol}: \texttt{en}, |
| \texttt{zh\_native}, \texttt{zh\_opencc}, \texttt{code}, \texttt{math} |
| --- val tails carved from the training tail (Chinese/math) or |
| train-head-only caps with an unseen remainder (English/code), 64 |
| sequences per val unless noted. |
|
|
| \paragraph{Noise bands.} Headline capability claims (bits-per-byte, BPB) |
| are judged against a pre-registered \textbf{$\pm$1.5\% relative-BPB noise |
| band}, reused unchanged across the niche, factorial, and router |
| campaigns; a stronger dual-criterion test (niche Phase 2) requires |
| clearing \textbf{2$\times$ the band (3.0\%)}. |
| Mechanism-level metrics that are not BPB (rare-token share, content-token |
| share, spillover CE) instead use a \textbf{split-half $2\sigma$ band} |
| computed per metric in the factorial campaign (\S\ref{sec:factorial}); |
| the two band conventions are not interchangeable and each table below |
| states which one applies. |
|
|
| \paragraph{Iso-FLOP discipline.} Every architectural comparison in this |
| paper is gated on an analytic, per-token active-MAC ledger |
| (\texttt{frontier/circus\_cp\_agentic/experiments/flop\_count.py}), not |
| on wall-clock cost, because wall-clock throughput conflates FLOPs with |
| memory-bandwidth effects (a distinction that turns out to matter, |
| \S\ref{sec:factorial}). The all-MoE control (E) and the token-keyed STEM |
| arm (V0) are matched to within \textbf{$-0.16\%$ total active MACs}; the |
| two product-key arms (V2/V3) carry a \textbf{$+2.34\%$ conditional-FFN |
| MAC premium} over the token-keyed arms, still inside a pre-registered |
| \textbf{$\leq$5\% FLOP-match gate}. |
| The same discipline recurs in the STEM-router design (\S\ref{sec:router}): |
| its primary operating point ($C=0.5$) is chosen specifically to hold |
| total FFN MACs constant against the all-MoE baseline, not to save |
| compute. |
|
|
| \paragraph{Campaign lineage.} Three campaigns share this protocol and |
| this paper's scope: \textbf{niche-mapping} (\S\ref{sec:niche}) asks |
| \emph{where}, if anywhere, token-keyed STEM beats MoE; \textbf{mechanism |
| factorial} (\S\ref{sec:factorial}) reuses the niche campaign's matched |
| S/E pair as two of its four cells (equivalence-gated: the factorial's |
| reproduction of the S arm's \texttt{en} BPB matches the niche record to |
| four decimal places) and asks \emph{why}, decomposing the STEM |
| mechanism into an addressing axis and a combine axis; \textbf{STEM-router} |
| (\S\ref{sec:router}) asks whether the mechanism factorial's answer can |
| be turned into a learned, per-token depth-allocation policy. |
|
|
| \section{Lookup Niches: Where Does Token-Keyed Memory Win?} |
| \label{sec:niche} |
|
|
| \paragraph{S/E placement.} Every S-arm in this paper --- the original |
| agentic-mix small pair and the bilingual pair (\texttt{bi\_s}) reused |
| throughout \S\ref{sec:factorial} --- uses the identical FFN placement |
| \texttt{pattern = "*E*S*S*E*S*S*E*S*S*E"}: of the 10 FFN positions, 6 |
| are STEM and 4 are MoE, i.e.\ a fixed \textbf{60\% majority} of FFN |
| capacity is static per-token lookup. The all-MoE control (\texttt{bi\_e}) |
| uses \texttt{"*E*E*E*E*E*E*E*E*E*E"} (10 of 10 MoE), differing from the |
| S arm \emph{only} in this one pattern string. |
| This placement was never swept in either campaign: the question asked |
| is whether STEM has \emph{any} niche, not how much of the stack it |
| should occupy. |
|
|
| \paragraph{Five pre-registered hypotheses.} The niche-mapping campaign |
| tested five candidate STEM advantages against plain MoE, each with a |
| mechanical, pre-registered adjudication rule (Table~\ref{tab:niche-verdict}). |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{p{0.05\linewidth}p{0.24\linewidth}p{0.12\linewidth}p{0.5\linewidth}} |
| \toprule |
| \# & Hypothesis & Verdict & Key evidence \\ |
| \midrule |
| H1 & Long-tail memory & \textbf{Refuted, reversed} & Zero-tabling STEM's $\Delta$CE cost is largest on \emph{frequent} tokens and falls monotonically toward rare tokens in all 3 well-powered checkpoint/val combinations (tail/head ratio 0.42$\times$--0.71$\times$, opposite of the $\geq$2$\times$ required) \\ |
| H2 & CJK / bilingual niche & \textbf{Refuted} & Endpoint zh-mean BPB gap vs.\ E $= -1.13\%$ (need $< -3.0\%$); en gap $-0.67\%$ also fails the specificity check \\ |
| H3 & Templated-text niche & \textbf{Refuted, reversed} & The (S$-$E) CE gap is \emph{largest} on the most-templated tercile ($+0.0646$) and \emph{smallest} on the least-templated ($+0.0168$) --- 3.84$\times$ in the wrong direction \\ |
| H4 & Systems efficiency & \textbf{Confirmed} & Decode throughput $+88.6\%$ (375.47 vs.\ 199.04 tok/s), TTFT $-9.9\%$, train throughput $+6.1\%$ (105,135 vs.\ 99,073 tok/s), 0 vs.\ 2 \texttt{cudaErrorLaunchTimeout} MoE-kernel crashes \\ |
| H5 & Early-advantage dilution & \textbf{Confirmed} & S leads beyond the noise band at step $\leq$1{,}500 on all 3 language vals, then E catches to within band by the shared endpoint (step 6{,}322) \\ |
| \bottomrule |
| \end{tabular} |
| \caption{Niche-mapping executive verdict. Every hypothesis label and |
| number is quoted or arithmetically derived from the source verdict.} |
| |
| \label{tab:niche-verdict} |
| \end{table} |
|
|
| \paragraph{The early-leverage finding (H5).} The pre-registered |
| criterion for H5 requires an S-lead beyond the $\pm$1.5\% noise band at |
| step $\leq$1{,}500 on each language val, with the lead gone by the shared |
| endpoint. All three language vals clear it (Table~\ref{tab:h5}). |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrl} |
| \toprule |
| Val & Early lead (step, rel.\ gap) & Endpoint-caught \\ |
| \midrule |
| en & step 1{,}500, $-1.90\%$ & yes \\ |
| zh\_native & step 1{,}500, $-1.56\%$ & yes \\ |
| zh\_opencc & step 1{,}500, $-2.28\%$ (also $-1.74\%$ @ 1{,}000) & yes \\ |
| \bottomrule |
| \end{tabular} |
| \caption{H5 pre-registered criterion, all three language vals.} |
| |
| \label{tab:h5} |
| \end{table} |
|
|
| Two further vals were tracked descriptively (not part of the |
| pre-registered H5 gate, which is language-only) and sharpen the picture. |
| \textbf{Math} shows the cleanest dilution: S leads by $-5.36\%$ at step |
| 2{,}000, fading to $-0.53\%$ by the endpoint. \textbf{Code} does not |
| merely dilute --- it \emph{reverses}: S leads by $-8.28\%$ at step |
| 2{,}500, crosses over to an E-lead near step 3{,}000, and finishes |
| \textbf{+4.03\% in E's favor} at the shared endpoint (step 6{,}322), the |
| only val where the eventual winner is not the early leader. |
| This code-reversal is the headline finding \S\ref{sec:factorial} sets out |
| to explain mechanistically. |
|
|
| \paragraph{Systems efficiency (H4) stands independent.} Unlike H1--H3,H5, |
| H4 is not a quality-niche claim: token lookup has no router and no |
| grouped-GEMM dispatch, so it is unconditionally faster to decode from and |
| more stable to train (Table~\ref{tab:niche-verdict}). This advantage is |
| uncontested by any other finding in this paper. |
|
|
| \paragraph{The $>$50\%-replacement point.} Because placement was fixed at |
| 60\% STEM (never swept, \emph{supra}), this campaign cannot locate a |
| replacement-fraction \emph{threshold}. What it does show, at the single |
| majority-replacement point actually tested, is a ceiling: the one domain |
| that most demands compositional (not lexical-identity) computation --- |
| code --- is also the one domain where committing 60\% of FFN capacity to |
| a static per-token table costs quality once training accumulates past |
| the early-budget regime, rather than merely diluting to parity like |
| every other val. We read this as evidence that \emph{some} upper bound |
| on the STEM-replaceable fraction exists on composition-heavy domains, |
| without claiming 60\% itself is the located threshold: no lower-replacement |
| arm was ever trained, so whether the wall sits above, at, or below 50\% |
| remains an open question for future wind-tunnel work. |
|
|
| \section{Combine Mechanisms: Multiplicative Gate vs.\ Additive Side-Path} |
| \label{sec:factorial} |
|
|
| \paragraph{The $2\times2$ design.} A follow-up factorial decomposes |
| STEM's mechanism along two axes: \textbf{addressing} (token identity vs.\ |
| a learned product key, PKM-style \citep{lample2019pkm}) and |
| \textbf{combine} (STEM's native multiplicative SwiGLU gate, |
| $\mathrm{fc2}(\mathrm{silu}(\mathrm{fc1}(x))\odot\mathrm{table})$, vs.\ |
| an additive residual, $\mathrm{fc2}(\mathrm{silu}(\mathrm{fc1}(x))) + e$). |
| V0 (token $\times$ mult) \emph{is} the niche campaign's STEM arm, reused |
| verbatim; E (all-MoE) is its control, also reused; V1 (token $\times$ |
| add), V2 (pk $\times$ mult), and V3 (pk $\times$ add) are new arms |
| trained on the byte-identical protocol. |
|
|
| \paragraph{Headline capability.} Table~\ref{tab:headline} reports the |
| endpoint (step 6{,}322) relative-BPB gap of each arm against E, judged |
| against the same $\pm$1.5\% noise-band \emph{convention} as \S\ref{sec:niche} |
| --- only the band convention is shared, not the held-out val sets |
| themselves; see the protocol note below the table. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrrrl} |
| \toprule |
| Arm vs.\ E & en & zh\_tw & code & math & Beats E on $\geq$2 domains \\ |
| \midrule |
| V0: token $\times$ mult (= STEM) & $-0.67$ & $-1.14$ & $\mathbf{+4.03}$ & $-0.51$ & no \\ |
| V1: token $\times$ add & $+0.24$ & $\mathbf{-2.17}$ & $\mathbf{-7.85}$ & $+0.02$ & \textbf{yes} \\ |
| V2: pk $\times$ mult & $+0.95$ & $+0.24$ & $-1.33$ & $+1.46$ & no \\ |
| V3: pk $\times$ add & $-0.06$ & $\mathbf{-2.15}$ & $\mathbf{-5.45}$ & $-0.25$ & \textbf{yes} \\ |
| \bottomrule |
| \end{tabular} |
| \caption{Relative BPB gap vs.\ E at the shared endpoint (negative = arm |
| beats E), percent. Bold = outside the $\pm$1.5\% band (sign gives |
| direction; negative = arm beats E).} |
| |
| \label{tab:headline} |
| \end{table} |
|
|
| \paragraph{Protocol note (do not cross-compare with \S\ref{sec:niche}).} |
| These BPBs are the mechanism-factorial campaign's rsync'd \textbf{4-val |
| protocol} (\texttt{en}, \texttt{zhtw}, \texttt{code}, \texttt{math}, each |
| an \texttt{\_text\_document} val). The \texttt{en} and \texttt{code} |
| columns coincide byte-identically with \S\ref{sec:niche}'s carved-val |
| endpoints (confirmed by the equivalence gate, \S\ref{sec:setup}); \texttt{zhtw} |
| and \texttt{math} are \emph{different} held-out sets from \S\ref{sec:niche}'s |
| carved \texttt{zh\_native}/\texttt{zh\_opencc}/\texttt{math\_slice} vals. |
| Treat Table~\ref{tab:headline} as within-table contrasts only --- its |
| cells are not comparable to \S\ref{sec:niche}'s tables. |
|
|
| \paragraph{Add repairs the code reversal.} The single largest finding of |
| this campaign: switching V0's combine mode from multiplicative to |
| additive (V0$\to$V1) turns code from \textbf{+4.03\% in E's favor to |
| $-7.85\%$ in the table arm's favor} --- an \textbf{11.9-point BPB swing} |
| --- and produces the campaign's only two cells (V1, V3) that beat E on |
| multiple held-out domains at all. |
|
|
| \paragraph{Mechanism.} The multiplicative combine forces \emph{every} |
| token through the table as the SwiGLU up-projection itself: a |
| composition-heavy domain like code inherits the table's quality on the |
| critical path. The additive combine keeps the dense compute path intact |
| (\texttt{silu(fc1(x))}) and adds the table only as a residual bonus, |
| $+e$. Consistent with this, the additive table's own zero-ablation |
| cost is $\sim$12$\times$ smaller than the multiplicative table's |
| ($\Delta$CE $+0.012$ for V3 vs.\ $+0.154$ for V2) --- it matters |
| \emph{less} to the forward pass --- yet endpoint quality is |
| \emph{better}. \textbf{Memory as an additive side-path beats memory as |
| a multiplicative gate} --- at this scale and training budget; we make no |
| claim beyond the wind-tunnel protocol of \S\ref{sec:setup}. |
|
|
| \paragraph{Main effects: combine dominates, addressing is second-order.} |
| A mechanical $2\times2$ attribution (Table~\ref{tab:maineffects}, split-half |
| $2\sigma$ bands, distinct from the $\pm$1.5\% BPB band above) confirms |
| this asymmetry at the level of \emph{where} benefit lands, not just |
| endpoint BPB: the addressing axis has no effect on how much benefit |
| reaches rare tokens (rare-share is 0.15--0.18\% in every cell, |
| regardless of key mechanism --- frequency-reversal is intrinsic to |
| table memory, not to token-keying); the combine axis's main effect on |
| content-token share is negative and \emph{opposite} the na\"ive |
| prediction, but the interaction term is superadditive on the content |
| axis, meaning V3 (pk + add together) loses less content-share than the |
| two main effects predict in isolation. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrl} |
| \toprule |
| Axis & Effect & Band ($2\sigma_{\mathrm{split}}$) & Verdict \\ |
| \midrule |
| Addressing (rare-share, \{V2,V3\}$-$\{V0,V1\}) & $+0.0002$ & $0.0004$ & Refuted \\ |
| Combine (content-share, \{V1,V3\}$-$\{V0,V2\}) & $-0.0303$ & $0.0036$ & Refuted (wrong sign) \\ |
| Interaction (V3 vs.\ additive prediction, content) & $+0.0190$ & $0.0039$ & Superadditive, confirmed \\ |
| \bottomrule |
| \end{tabular} |
| \caption{$2\times2$ mechanism attribution on non-BPB metrics |
| (rare-token / content-token benefit share). Note the band convention |
| here is a split-half $2\sigma$, not the $\pm$1.5\% relative-BPB band |
| used elsewhere.} |
| |
| \label{tab:maineffects} |
| \end{table} |
|
|
| \paragraph{Hub diffusion.} A separate causal probe zeros only the |
| top-1\% highest-benefit table rows and measures the CE cost at |
| \emph{non-hub} positions. In all four arms this spillover clears its |
| $2\sigma_{\mathrm{split}}$ band (Table~\ref{tab:hub}): hub-row benefit |
| diffuses through the residual stream to positions those rows never |
| directly key, and the effect is $\sim$100--200$\times$ larger in the |
| multiplicative arms, where the hub rows sit on every token's critical |
| path. Benefit concentration is extreme in every arm (Gini 0.936--0.992; |
| top-1\% of rows carry 38--77\% of total benefit) --- table memory is |
| intrinsically hub-shaped, and D1's frequency-reversal |
| (\S\ref{sec:niche}, H1) is best read as \emph{rational hub investment} |
| rather than a limitation: frequent tokens are hubs, and improving hubs |
| diffuses value outward. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrrr} |
| \toprule |
| Arm & Spillover $\Delta$CE & Band ($2\sigma_{\mathrm{split}}$) & Gini & Top-1\% share \\ |
| \midrule |
| V0 (token $\times$ mult) & $+0.589$ & $0.0041$ & 0.936 & 57\% \\ |
| V1 (token $\times$ add) & $+0.576$ & $0.0090$ & 0.942 & 60\% \\ |
| V2 (pk $\times$ mult) & $+0.032$ & $0.0002$ & 0.976 & 77\% \\ |
| V3 (pk $\times$ add) & $+0.0024$ & $0.0002$ & 0.992 & 38\% \\ |
| \bottomrule |
| \end{tabular} |
| \caption{H-hub: zeroing the top-1\% benefit rows degrades CE on |
| \emph{non-hub} positions in every arm --- diffusion confirmed.} |
| |
| \label{tab:hub} |
| \end{table} |
|
|
| \paragraph{The product-key bandwidth tax.} Product-key addressing (V2, |
| V3) is second-order for quality (Table~\ref{tab:maineffects}) but not |
| free: the top-16 gather over a 262{,}144-row shared table costs only |
| $+2.34\%$ active MACs against the token-keyed arms (well inside the |
| $\leq$5\% iso-FLOP gate, \S\ref{sec:setup}) yet costs $\mathbf{-15\%}$ |
| wall-clock throughput --- the selection cost here is a |
| \textbf{memory-bandwidth tax}, not a FLOP cost, and grouped-gather |
| dispatch is not free even when the arithmetic is. |
| A companion economics check (H-offload) finds routing predictability is |
| \emph{not} lower in the table-augmented arms than in plain MoE (argmax |
| accuracy $0.9115$ vs.\ $0.9098$, within a $0.0344$ band) --- the division |
| of labor between STEM and MoE shows up in \emph{how much} compensation |
| each decile needs, not in the router's own confidence. |
| Both product-key arms were confirmed \textbf{trained}, not merely |
| cold-started, before any of the above was adjudicated (zero-table |
| $\Delta$CE hurts in both; row drift 97--100\% by the endpoint), ruling |
| out undertraining as an alternative explanation for the pk arms' |
| weaker showing. |
|
|
| \paragraph{Downstream decision.} Taken together --- combine dominates |
| addressing, additive combine is FLOP-neutral and beats E on multiple |
| domains, product keys buy nothing on quality and cost throughput --- the |
| mechanism factorial's design-relevant output is a single decision: |
| \textbf{combine = additive}. The STEM-router campaign (\S\ref{sec:router}) |
| inherits this by construction. |
|
|
| \section{STEM-Router: Learned Depth Allocation} |
| \label{sec:router} |
|
|
| \paragraph{Motivation.} \S\ref{sec:niche}--\S\ref{sec:factorial} leave two |
| facts sitting side by side: STEM's early-training lead dilutes to parity |
| (or, on code, reverses) by the shared endpoint (H5), and the additive |
| combine that repairs the reversal makes the table's own contribution to |
| the forward pass \emph{smaller}, not larger, while endpoint quality |
| improves (\S\ref{sec:factorial}, table zero-ablation $\Delta$CE |
| 12$\times$ down). Both point at the same design move: stop asking a |
| static table to do the same job at every training step and every stack |
| position, and instead let a learned, per-token policy decide when the |
| extra, composition-capable FLOPs of an MoE layer are worth spending on |
| top of an always-on additive memory floor. This section reports that |
| architecture, \textbf{STEM-router} (layer type \texttt{R}), a |
| Mixture-of-Depths-style design \citep{raposo2024mod}, pre-registered as |
| three hypotheses (H-R, H-SE, H-RT) and evaluated end to end, then probed |
| along three further axes. |
|
|
| \paragraph{The routed layer.} Every FFN position in the router arms is |
| layer type \texttt{R} (pattern all-\texttt{R}, 10 attention + 10 R |
| positions), replacing \S\ref{sec:niche}--\S\ref{sec:factorial}'s |
| \texttt{S}/\texttt{E}/shared-expert stack outright. Each R layer computes |
| $y = x + \mathrm{StemFFN}(x) + \sigma(\mathrm{score}(x))\cdot |
| \mathrm{MoeFFN}(x)$ for tokens admitted by the gate, and |
| $y = x + \mathrm{StemFFN}(x)$ for tokens not admitted, where StemFFN is |
| the additive-combine variant that \S\ref{sec:factorial}'s downstream |
| decision (DP-1) selected, running on \emph{every} token as an always-on |
| residual, and the unchanged top-2-of-24 MoE layer runs conditionally, |
| gated by a per-token sigmoid score, only on tokens admitted by |
| \textbf{expert-choice top-$k$}: each (sequence, layer) admits the top |
| $k=\lceil C\cdot T\rceil$-scoring tokens, $T$ the sequence length. |
| Expert-choice is a hard capacity floor by construction --- a fixed |
| fraction of tokens is always admitted regardless of the score |
| distribution --- so it doubles as insurance against gate collapse; a |
| second insurance is a $+2.0$ gate-bias initialization |
| ($\sigma(2)\approx0.88$, so the MoE path starts near-fully open and the |
| gate learns to \emph{withhold} capacity, never to earn it from zero) |
| plus a capacity anneal from $C=1.0$ (every token selected, full |
| gradient to the MoE layer) over a 1{,}500-step warmup down to the target |
| operating point $C$ by step 2{,}000, held fixed thereafter. The shared |
| dense expert that ran unconditionally in \S\ref{sec:niche}--\S\ref{sec:factorial} |
| is removed (\texttt{shared\_ffn\_hidden=0}); the always-on StemFFN is |
| its replacement. |
|
|
| \paragraph{Three pre-registered hypotheses.} \textbf{H-R} (primary): at |
| equal FLOPs, always-on-STEM-recall plus router-allocated composition |
| beats uniform allocation (every token through shared-expert-plus-MoE), |
| judged by an endpoint decision rule at the same $\pm$1.5\% noise band as |
| \S\ref{sec:niche}--\S\ref{sec:factorial}: no loss on any of the five |
| held-out vals, with at least one outright win (only the band convention |
| is shared: \S\ref{sec:niche} applies per-hypothesis decision rules, |
| e.g.\ H2's 2$\times$-band $-3.0\%$ criterion, and \S\ref{sec:factorial}'s |
| headline column is a separate ``beats E on $\geq$2 domains'' rule). |
| \textbf{H-SE} (secondary): removing the shared expert costs nothing once |
| StemFFN covers its recall load (the ``no val loses'' clause of H-R's own |
| criterion, \emph{supra}). \textbf{H-RT} (router health, tertiary, three sub-criteria): |
| the learned gate reproduces two offline priors --- frequent tokens |
| routed away from MoE (b1, pre-registered as the \emph{mean-over-the-10 |
| R-layers} Spearman $\rho>0.5$ between frequency decile, $d_0=$ |
| most-frequent, and MoE selection rate; the spec's orientation expects |
| selection rate to \emph{rise} with decile index, i.e.\ toward rarer |
| deciles, so a positive layer-mean $\rho$ under that orientation is the |
| anti-frequency signature) and composition-heavy domains (code, math) |
| routed toward MoE more than language domains (b2, higher mean |
| pre-sigmoid gate score) --- plus a non-degeneracy floor (b3, gate-score |
| std $>0.01$ across tokens, per layer). |
| |
|
|
| \paragraph{v1 verdict: an equal-FLOP win that survives becoming a |
| FLOP-saving win.} The primary arm, \texttt{sr50} ($C=0.5$), is |
| FLOP-matched to the all-MoE baseline \texttt{bi\_e} by construction --- |
| StemFFN's extra always-on cost is offset almost exactly by halving MoE's |
| conditional-token budget. It clears H-R and H-SE outright at its |
| endpoint (Table~\ref{tab:router-v1} caption): 3 wins, 0 losses |
| (Table~\ref{tab:router-v1}). A gated follow-up, \texttt{sr25} ($C=0.25$), |
| asks whether the win survives when the arm stops being FLOP-neutral and |
| starts saving FLOPs; it does, unchanged at 3 wins / 0 losses, and the |
| size of every gap moves toward E by roughly half as $C$ halves (code |
| $-8.77\%\to-4.73\%$, zh\_native |
| $-2.96\%\to-2.10\%$, en $-0.48\%\to+0.53\%$) --- a real, monotone |
| dose-response, not noise. One watch item: sr25's math gap ($+1.43\%$) |
| sits close to the $+1.5\%$ band edge, though still in-band; sr50's math |
| gap was $-0.85\%$, so a math-sensitive deployment would prefer the |
| higher-capacity point. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrrrr} |
| \toprule |
| val & sr50 bpb & sr25 bpb & bi\_e bpb & sr50 gap & sr25 gap \\ |
| \midrule |
| en & 1.2216 & 1.2339 & 1.2274 & $-0.48$ & $+0.53$ \\ |
| zh\_native & 1.6099 & 1.6241 & 1.6589 & $\mathbf{-2.96}$ & $\mathbf{-2.10}$ \\ |
| zh\_opencc & 1.4008 & 1.4161 & 1.4530 & $\mathbf{-3.59}$ & $\mathbf{-2.53}$ \\ |
| code & 1.4071 & 1.4694 & 1.5423 & $\mathbf{-8.77}$ & $\mathbf{-4.73}$ \\ |
| math & 0.8576 & 0.8774 & 0.8650 & $-0.85$ & $+1.43$ \\ |
| \bottomrule |
| \end{tabular} |
| \caption{STEM-router endpoint bpb vs.\ the all-MoE baseline |
| \texttt{bi\_e}, two capacities. \texttt{sr50} endpoint step 6{,}309 |
| (13 steps short of the default 6{,}322 due to a disclosed disk-full |
| incident, biased conservative against H-R); \texttt{sr25} and |
| \texttt{bi\_e} both at the default endpoint 6{,}322. Bold = outside the |
| $\pm$1.5\% band; sign negative = router arm beats \texttt{bi\_e}. 3 |
| wins / 0 losses at both capacities.} |
| |
| \label{tab:router-v1} |
| \end{table} |
|
|
| \paragraph{Economics.} The per-token cost of one routed layer in |
| \texttt{sr50} is 4{,}731{,}904 MACs (Table~\ref{tab:router-econ}), of |
| which the conditional MoE component is 49.26\% of \texttt{bi\_e}'s MoE |
| cost; total active MACs (summed across all 10 R layers plus attention) |
| are $-0.12\%$ vs.\ E --- FLOP-matched by design, not by accident. |
| Wall-clock throughput is $97.9$--$98.1$k tok/s |
| vs.\ E's $99.4$k ($-1.4\%$): the MoD dispatch tax (gather/top-$k$) is |
| almost fully offset by halving the MoE token count. At $C=0.25$ the |
| economics flip sign entirely: total active MACs are $86.58$M vs.\ E's |
| $94.62$M ($-8.50\%$), and throughput is \emph{faster} than E's, $103.5$k |
| vs.\ $99.4$k tok/s ($+4.1\%$) --- this arm is unconditionally cheaper to |
| run, not merely differently allocated, and it still wins 3 of 5 domains. |
| For contrast, \S\ref{sec:factorial}'s product-key selection cost $-15\%$ |
| throughput for no quality gain; MoD's layer-level selection costs at |
| most $-1.4\%$ and delivers real wins --- selecting \emph{which tokens |
| reach a layer} is roughly an order of magnitude cheaper than selecting |
| \emph{which memory rows a token gathers}, at this scale. |
| |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrr} |
| \toprule |
| Metric vs.\ \texttt{bi\_e} & sr50 ($C{=}0.5$) & sr25 ($C{=}0.25$) \\ |
| \midrule |
| Total active MACs & $-0.12\%$ (FLOP-matched) & $-8.50\%$ \\ |
| Wall-clock tok/s & 97.9--98.1k vs.\ 99.4k ($-1.4\%$) & 103.5k vs.\ 99.4k ($+4.1\%$) \\ |
| \bottomrule |
| \end{tabular} |
| \caption{STEM-router economics at two capacities. \texttt{sr50} is |
| FLOP-matched to \texttt{bi\_e} by design (always-on StemFFN cost offset |
| by halved MoE token budget); \texttt{sr25} trades allocation-neutrality |
| for net FLOP savings.} |
| |
| \label{tab:router-econ} |
| \end{table} |
|
|
| \paragraph{What the gate learned.} H-RT does not pass as pre-registered: |
| two of its three sub-criteria fail at the mean level --- b1 (frequency |
| tracking, layer-mean Spearman $\rho\approx-0.02$ in \texttt{sr50} and |
| $-0.24$ in \texttt{sr25}, both far short of the pre-registered mean |
| $\rho>+0.5$ threshold) and b2 (domain ranking) --- while b3, the non-degeneracy floor, PASSES |
| (min per-layer gate-score std $0.497$ in \texttt{sr50}, $0.468$ in |
| \texttt{sr25}, both $\gg0.01$): the gate is strongly differentiated, not |
| collapsed, so the anti-collapse insurance built into the design |
| (\emph{supra}) worked as intended. The b1 mean-level failure is |
| nonetheless informative: it hides a strongly heterogeneous |
| \emph{per-layer} structure. \textbf{Convention}: Table~\ref{tab:router-rho} |
| reports $\rho$ in b1's own pre-registered orientation --- Spearman |
| correlation between frequency-decile index ($d_0=$ most-frequent, |
| $d_9=$ rarest) and per-decile MoE selection rate --- so a positive |
| $\rho$ is the rare-favoring signature b1 sought (selection rate rising |
| with rarity) and a negative $\rho$ is the opposite, frequency-aligned |
| pattern (selection rate falling with rarity, i.e.\ concentrating on the |
| most frequent tokens). By that convention, the mid-depth layers |
| (\texttt{sr50} L3--L6: $+0.66$ to $+0.84$) are modestly rare-favoring, |
| roughly the direction b1 predicted; the last three of ten R layers, by |
| contrast, are perfectly \emph{frequency-aligned} ($\rho=-1.00$) in |
| \texttt{sr50} --- raw per-decile selection rates fall monotonically from |
| $0.53$--$0.59$ at $d_0$ to $0.10$--$0.16$ at $d_9$ --- the maximal |
| \emph{violation}, not satisfaction, of the pre-registered prior: these |
| three deepest layers spend MoE capacity almost exclusively on the tokens |
| the model already predicts well from parametric memory, and leave rare |
| tokens to the always-on stem side-path. The identical deep-three, |
| frequency-aligned pattern reappears, independently trained, in |
| \texttt{sr25} and again in \texttt{srpl} (\emph{infra}) --- a triple |
| replication of the frequency-aligned, not rare-favoring, deep policy |
| (Table~\ref{tab:router-rho}). The mean-level FAIL verdict is unchanged |
| by this reading: the other seven of ten layers pull the 10-layer mean |
| back toward zero or the rare-favoring sign, so b1 fails as |
| pre-registered, and the reading clarifies why --- the learned policy is |
| heterogeneous by outcome, not merely noisy: mid-depth layers lean |
| rare-favoring while the deepest three lean maximally frequency-aligned. |
| Two further regularities: gate-score |
| std \emph{grows with depth within} \texttt{sr25} (from $\approx0.47$ at |
| the shallowest gate to $2.11$ at layer 10; the cross-arm minimum |
| per-layer std is $0.497$ in \texttt{sr50} vs.\ $0.468$ in \texttt{sr25} |
| --- both far above the $0.01$ floor, and this cross-arm comparison is |
| not itself a depth trend), and the mean pre-sigmoid gate score shrinks |
| as capacity shrinks (mean sigmoid $\approx0.10$ at $C=0.5$ vs.\ |
| $\approx0.06$ at $C=0.25$) --- with fewer MoE slots available, the model |
| does not fight for amplitude on the slots it keeps; it shrinks the |
| routed path's average gain instead, the same ``small additive bonus'' |
| regime \S\ref{sec:factorial} found for the additive table itself (V3's |
| $\Delta$CE 12$\times$ smaller than V2's, yet better quality). Reading: |
| quality does not require the naive ``spend FFN where it's hardest'' |
| policy; the deepest layers instead spend their MoE budget on the most |
| frequent tokens and leave rare tokens to the memorized stem side-path, |
| while the routed path, on average, functions as a small correction |
| rather than a load-bearing default. |
| |
|
|
| \begin{table}[h] |
| \centering |
| \footnotesize |
| \begin{tabular}{lrrrrrrrrrr} |
| \toprule |
| Arm & L1 & L2 & L3 & L4 & L5 & L6 & L7 & L8 & L9 & L10 \\ |
| \midrule |
| sr50 ($C{=}0.5$) & $-0.35$ & $-0.33$ & $+0.84$ & $+0.79$ & $+0.66$ & $+0.81$ & $+0.33$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ \\ |
| sr25 ($C{=}0.25$) & $-0.53$ & $-0.48$ & $+0.26$ & $+0.27$ & $+0.54$ & $+0.92$ & $-0.38$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ & $\mathbf{-1.00}$ \\ |
| \bottomrule |
| \end{tabular} |
| \caption{Per-layer Spearman $\rho$ (frequency-decile index, $d_0=$ |
| most-frequent, vs.\ selection rate) at the endpoint, two independently |
| trained capacities. The last three of ten R layers are perfectly |
| frequency-aligned in both (bold) --- the mirror of the pre-registered |
| rare-favoring b1 direction; a third independently trained arm, |
| \texttt{srpl} (below), replicates the same deep-three pattern.} |
| |
| \label{tab:router-rho} |
| \end{table} |
|
|
| \paragraph{v2, axis (a): does a data-driven per-layer schedule beat |
| uniform capacity?} The deep-three/mid-layer heterogeneity above motivates |
| a hand-designed per-layer capacity schedule, $C_l=$ |
| $(0.125, 0.125, 0.40625, 0.40625, 0.40625, 0.40625, 0.25, 0.125, 0.125, |
| 0.125)$ for R layers 1--10 (sum $2.5$ bit-exactly, $0.40625=13/32$; |
| dyadic fractions chosen so the schedule is bit-exact iso-FLOP against |
| \texttt{sr25}'s uniform $C=0.25$): starve the confidently |
| frequency-aligned deep layers and the weakly frequency-aligned shallow |
| pair down to $0.125$, feed the rare-favoring mid layers up to |
| $0.40625$, leave the one layer |
| whose sign flips between \texttt{sr50} and |
| \texttt{sr25} at a neutral $0.25$. The resulting arm, \texttt{srpl}, |
| clears the pre-registered H-PLC criterion (no-loss vs.\ \texttt{sr25} at |
| equal FLOPs, $\geq$1 win): code wins outright, and every other domain is |
| in-band (Table~\ref{tab:router-srpl}). Against the original baseline |
| \texttt{bi\_e}, however, the picture is not a free lunch: three domains |
| still win, but math crosses into a loss ($+1.57\%$, $0.07$pp over band) |
| --- reallocating capacity by depth buys code at a small math cost, a |
| domain-preference knob rather than a strictly dominant schedule. The |
| deep-three frequency-aligned pattern replicates a third time under this |
| schedule. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrr} |
| \toprule |
| val & srpl vs.\ sr25 (iso-FLOP) & srpl vs.\ bi\_e (context) \\ |
| \midrule |
| en & $+0.22$ & $+0.74$ \\ |
| zh\_native & $-0.01$ & $\mathbf{-2.11}$ \\ |
| zh\_opencc & $+0.03$ & $\mathbf{-2.51}$ \\ |
| code & $\mathbf{-2.91}$ & $\mathbf{-7.50}$ \\ |
| math & $+0.13$ & $\mathbf{+1.57}$ \\ |
| \bottomrule |
| \end{tabular} |
| \caption{E-A per-layer capacity schedule (\texttt{srpl}), endpoint step |
| 6{,}322 both comparisons. Left column: primary pre-registered criterion, |
| bit-exact iso-FLOP vs.\ \texttt{sr25}'s uniform $C=0.25$ (no-loss, |
| $\geq$1 win $\Rightarrow$ H-PLC confirmed). Right column: descriptive |
| context vs.\ the original baseline \texttt{bi\_e} (same total capacity |
| sum as \texttt{sr25}, different per-layer shape --- not the primary |
| criterion). Bold = outside $\pm$1.5\% band; bold positive on math = a |
| loss. Left column adjudicated by \texttt{analyze\_v2.py} (pre-registered |
| pairwise criterion); right column adjudicated by \texttt{analyze\_sr.py} |
| (descriptive vs-\texttt{bi\_e} context, same script as the STEM-router |
| v1 verdict, Table~\ref{tab:router-v1}).} |
| |
| \label{tab:router-srpl} |
| \end{table} |
|
|
| \paragraph{v2, axis (b): can a context-aware key replace the static |
| n-gram ladder?} A second axis asks whether concatenating a bigram-hashed |
| key onto (or in place of) the token-identity key buys anything, holding |
| the STEM channel width and total FLOPs fixed at \texttt{sr50}'s level |
| (width 1{,}536). \texttt{srng} splits the width into a token block |
| (1{,}024) and a hashed-bigram backoff block (512, 131{,}072 slots); |
| \texttt{srck} drops the token block and uses a bigram-only key at full |
| width (86{,}016 slots). The pre-registered prediction was that |
| \texttt{srck} loses to \texttt{srng} on en/code (token identity is |
| load-bearing, and dropping it costs specificity); the outcome refutes |
| this: \texttt{srck} beats \texttt{srng} on \emph{all five} domains |
| (Table~\ref{tab:router-eb}). But neither arm beats the simpler |
| \texttt{sr50} baseline convincingly: \texttt{srng} loses to \texttt{sr50} |
| on all five domains ($+1.84\%$ to $+6.33\%$), and \texttt{srck} is only |
| in-band on four of five, losing code by $+2.43\%$. The split ladder is |
| dominated \emph{from both directions} --- worse than pure token keying |
| and worse than pure bigram keying --- so the failure is not ``bigrams |
| don't help,'' it is ``carving width away from the token table to make |
| room for a backoff table is a bad trade at this scale.'' Bigram lookup |
| also costs throughput independent of quality ($\sim$2\% tok/s tax on |
| both \texttt{srng} ($\approx$96.0k tok/s) and \texttt{srck} |
| ($\approx$96.1k tok/s) against \texttt{sr50} ($\approx$98.0k tok/s)). |
| Token-only addressing remains the strongest static STEM key found in |
| this paper; any context-aware key would need to beat \texttt{sr50} |
| directly, not the ladder. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrr} |
| \toprule |
| val & srck vs.\ srng (primary) & srng vs.\ sr50 (context) & srck vs.\ sr50 (context) \\ |
| \midrule |
| en & $\mathbf{-2.29}$ & $\mathbf{+1.84}$ & $-0.50$ \\ |
| zh\_native & $\mathbf{-2.41}$ & $\mathbf{+2.17}$ & $-0.29$ \\ |
| zh\_opencc & $\mathbf{-2.88}$ & $\mathbf{+2.58}$ & $-0.37$ \\ |
| code & $\mathbf{-3.67}$ & $\mathbf{+6.33}$ & $\mathbf{+2.43}$ \\ |
| math & $\mathbf{-4.37}$ & $\mathbf{+4.33}$ & $-0.23$ \\ |
| \bottomrule |
| \end{tabular} |
| \caption{E-B key-replacement axis, both arms FLOP-identical to |
| \texttt{sr50} (STEM channel width 1{,}536). Primary column: pre-registered |
| criterion, \texttt{srck} (bigram-only, 86{,}016 slots) vs.\ \texttt{srng} |
| (token+bigram split ladder, 131{,}072 bigram slots), both at endpoint |
| 6{,}322 --- replacement holds (5 wins, 0 losses), refuting the |
| pre-registered prediction. Context columns: descriptive comparison |
| against \texttt{sr50} (token-only, endpoint 6{,}309 --- a different |
| endpoint step than the two v2 arms, per the disclosed sr50 deviation, |
| Table~\ref{tab:router-v1}); both context arms lose to token-only on at |
| least one domain. Bold = outside $\pm$1.5\% band.} |
| |
| \label{tab:router-eb} |
| \end{table} |
|
|
| \paragraph{v2, axis (c): does the gate need to see the future?} |
| Expert-choice top-$k$ ranks every token in a sequence against every other |
| token before selecting, which is unavailable at autoregressive decode |
| time; the gate \emph{score} itself, however, is already causal (a |
| function of the current token's hidden state only). The third axis |
| tests whether a decode-compatible substitute --- a fixed per-layer |
| quantile threshold $\tau_l=\mathrm{quantile}(\mathrm{scores},\,1-C)$, |
| calibrated post-hoc on a held-out training-slice tail and frozen on |
| \texttt{sr50}'s final checkpoint --- reproduces expert-choice's |
| sequence-level competition. It does, comfortably: every domain's |
| threshold-mode bpb is within the pre-registered $+1.5\%$ gate of the |
| expert-choice oracle, with the largest observed gap only $+0.34\%$ |
| (Table~\ref{tab:router-ec}). Sequence-level top-$k$ competition therefore |
| carries almost no information beyond a fixed quantile of the (already |
| causal) gate score --- a MoD-style decode path needs only a calibrated |
| threshold, not an auxiliary predictor head co-trained with the model. |
| |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrr} |
| \toprule |
| val & threshold bpb & oracle bpb & gap \\ |
| \midrule |
| en & 1.2216 & 1.2216 & $+0.00\%$ \\ |
| zh\_native & 1.6132 & 1.6099 & $+0.20\%$ \\ |
| zh\_opencc & 1.4056 & 1.4008 & $+0.34\%$ \\ |
| code & 1.4081 & 1.4071 & $+0.07\%$ \\ |
| math & 0.8566 & 0.8576 & $-0.12\%$ \\ |
| \bottomrule |
| \end{tabular} |
| \caption{E-C causal-threshold probe: post-hoc per-layer quantile |
| threshold $\tau_l$ (calibrated on an English train-slice tail) vs.\ the |
| native expert-choice oracle, both evaluated on the frozen \texttt{sr50} |
| checkpoint at step 6{,}309. All gaps clear the pre-registered $+1.5\%$ |
| gate; largest observed gap $+0.34\%$ (zh\_opencc).} |
| |
| \label{tab:router-ec} |
| \end{table} |
|
|
| \paragraph{Two frozen-checkpoint probes: capacity as an inference-time |
| dial.} Given a causal threshold reproduces training-time selection |
| (axis c), two further post-hoc probes ask what happens when the |
| \emph{same} frozen \texttt{sr50} checkpoint (trained at a single |
| capacity, $C=0.5$) is dialed to capacities it never trained at, using |
| only $\tau$ recalibration --- no further training. \textbf{Elasticity}: |
| dialing down to $C=0.25$ costs $+1.7\%$ to $+3.5\%$ bpb depending on |
| domain, dialing down again to $C=0.125$ costs a further $+1.4\%$ to |
| $+6.0\%$ (roughly doubling the increment again on en/math, roughly flat |
| on zh\_native/zh\_opencc, and \emph{smaller} on code), and dialing |
| \emph{up} to $C=0.75$ mildly \emph{improves} every domain ($-0.1\%$ to |
| $-0.5\%$) with no overshoot penalty --- smooth, monotone degradation |
| with no cliff at any tested point (Table~\ref{tab:router-tau}). Notably, |
| the dialed-down $C=0.25$ checkpoint (the frozen \texttt{sr50} checkpoint, |
| step 6{,}309, with $\tau$ recalibrated) beats the \emph{natively trained} |
| \texttt{sr25} arm (step 6{,}322, a different endpoint, \emph{supra}) on |
| code ($-2.5\%$), and is within $1.0$--$2.2\%$ elsewhere --- a single |
| capacity-0.5 training run supports a compute knob at inference time at a |
| cost comparable to training a second checkpoint outright. |
| |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrrrrr} |
| \toprule |
| $\tau$-cap & en & zh\_native & zh\_opencc & code & math \\ |
| \midrule |
| 0.125 & 1.3001 & 1.7022 & 1.4865 & 1.4516 & 0.9399 \\ |
| 0.25 & 1.2498 & 1.6574 & 1.4469 & 1.4323 & 0.8865 \\ |
| 0.50 (native) & 1.2216 & 1.6132 & 1.4056 & 1.4081 & 0.8566 \\ |
| 0.75 & 1.2202 & 1.6077 & 1.3990 & 1.4019 & 0.8550 \\ |
| \bottomrule |
| \end{tabular} |
| \caption{$\tau$-elasticity: bpb of the single, frozen \texttt{sr50} |
| checkpoint (trained at $C=0.5$, step 6{,}309) with its causal threshold |
| recalibrated to four capacities, including two it never trained at |
| ($0.125$, $0.75$) and one below its native point ($0.25$). No further |
| training. The $0.50$ row is the same threshold-mode bpb as |
| Table~\ref{tab:router-ec}'s threshold column.} |
| |
| \label{tab:router-tau} |
| \end{table} |
|
|
| \paragraph{Emergent domain allocation.} \textbf{Selection matrix}: under |
| one fixed threshold (calibrated on English, $C=0.5$), mean per-token |
| selection rate varies $2.2\times$ across domains with no explicit |
| domain signal in the gate --- math is the hungriest domain (mean $0.67$, |
| layers 1--2 alone select $0.90$--$0.94$ of tokens), code is the most |
| economical (mean $0.31$; layer 1 selects only $0.053$ of tokens, though |
| one deep layer spikes to $0.649$), and the two Chinese vals sit at |
| $0.48$--$0.49$ with a shared, CJK-specific hotspot (layer 2 selects |
| $0.998$ and $0.981$ of tokens respectively) that English (the calibration |
| domain, flat at $0.48$) does not share (Table~\ref{tab:router-selmatrix}). |
| The domain shape is entirely a property of the trained gate scores under |
| one shared threshold; no explicit domain-conditioned allocator is |
| involved. |
|
|
| \begin{table}[h] |
| \centering |
| \small |
| \begin{tabular}{lrl} |
| \toprule |
| Domain & Mean selection rate (10-layer avg.) & Shape notes \\ |
| \midrule |
| math & 0.67 & layers 1--2: 0.90 / 0.94 (hungriest domain) \\ |
| zh\_native / zh\_opencc & 0.49 / 0.48 & layer 2 saturates: 0.998 / 0.981 \\ |
| en & 0.48 & flat across layers (calibration domain) \\ |
| code & 0.31 & layer 1: 0.053; one deep layer spikes to 0.649 \\ |
| \bottomrule |
| \end{tabular} |
| \caption{Domain $\times$ layer selection-rate summary under one fixed |
| threshold (calibrated on English at cap $0.5$, frozen \texttt{sr50} |
| checkpoint, step 6{,}309). Full 5-domain $\times$ 10-layer matrix in |
| \texttt{ec\_selmatrix.json}. Mean rate spans $2.2\times$ (math/code) |
| under a single shared threshold with no explicit domain signal.} |
| |
| \label{tab:router-selmatrix} |
| \end{table} |
|
|
| \paragraph{Summary.} Three campaigns now agree on the same shape: memory |
| belongs on an always-on additive path (\S\ref{sec:factorial}), and the |
| FLOPs freed by not forcing every token through a multiplicative gate are |
| better spent by a learned, capacity-floored per-token policy than by |
| uniform allocation (this section) --- at equal FLOPs (3W/0L) and even |
| when that policy is asked to save FLOPs outright (still 3W/0L, $-8.5\%$ |
| MACs). The policy the gate lands on is not the naive ``route hard tokens |
| to more compute'' rule; it is a depth- and domain-heterogeneous |
| allocation that is legible after the fact (per-layer frequency |
| heterogeneity --- rare-favoring at mid-depth, frequency-aligned in the |
| deepest three --- domain-shaped selection rates) but was never specified |
| as a target. |
|
|
| \section{Design Laws} |
| \label{sec:design-laws} |
|
|
| \paragraph{Scope.} The following five statements synthesize |
| \S\ref{sec:niche}--\S\ref{sec:router} into design principles for |
| token-identity memory and learned depth allocation in a bilingual MoE. |
| Each is scoped explicitly to the wind-tunnel protocol of |
| \S\ref{sec:setup} --- a single 1B-parameter, single-seed, |
| $\sim$6{,}300-step run per arm --- and is a directional prior for future |
| architecture choices, not an established scaling law. |
|
|
| \paragraph{Law 1: memory belongs on an additive side-path, not a |
| multiplicative gate.} Routing every token's dense computation through a |
| lookup table as a multiplicative SwiGLU gate makes a composition-heavy |
| domain (code) inherit the table's quality on its critical path; adding |
| the table instead as a residual bonus repairs an 11.9-point BPB swing on |
| code and produces the only cells in the mechanism factorial that beat |
| plain MoE on multiple domains, at zero additional FLOPs |
| (\S\ref{sec:factorial}). The STEM-router campaign inherits this by |
| construction and confirms it generalizes to a gated setting: the |
| always-on additive path beats uniform allocation on 3 of 5 domains with |
| zero losses, twice --- at equal FLOPs ($C=0.5$) and again at a |
| net-FLOP-saving quarter capacity ($C=0.25$, $-8.5\%$ total MACs) |
| (\S\ref{sec:router}). Scope: tested only for SwiGLU-style multiplicative |
| combination vs.\ residual addition; concatenation or cross-attention |
| combine mechanisms are untested. |
|
|
| \paragraph{Law 2: token identity is already the strongest static key; |
| static n-gram augmentation adds no dividend at this scale.} Long-tail |
| memory (H1) refutes in reverse --- the table helps \emph{frequent}, not |
| rare, tokens (\S\ref{sec:niche}) --- and product-key addressing buys no |
| quality while costing $-15\%$ throughput as a pure memory-bandwidth tax |
| (\S\ref{sec:factorial}). The router campaign's key-replacement axis |
| extends this: a hashed-bigram backoff ladder loses to plain token |
| keying on every domain, and even the best bigram-only variant (which |
| beats the ladder on all five domains) still loses to token-only on code |
| (\S\ref{sec:router}, axis b). Composition is where every lookup-keyed |
| variant is weakest; this paper does not test whether the composition gap |
| is instead attention-machinery territory, only that no keying variant of |
| the static table closes it. Scope: one hash-slot count per arm, one |
| wind-tunnel corpus scale (1.66B tokens); a larger corpus may train |
| hashed tables past this regime. |
|
|
| \paragraph{Law 3: a learned gate with a hard capacity floor beats uniform |
| always-on allocation at equal FLOPs, and the learned policy is |
| depth-heterogeneous.} Expert-choice top-$k$ plus an always-on additive |
| memory floor beats routing every token through a shared expert and full |
| MoE, at equal FLOPs (3W/0L) and again when the arm is asked to save |
| FLOPs outright (3W/0L, $-8.5\%$ MACs, \S\ref{sec:router}). The policy the |
| gate converges on is not naive frequency- or domain-tracking on average; |
| it is layer-specific, with the deepest three of ten FFN layers perfectly |
| frequency-aligned --- concentrating MoE capacity on the most frequent |
| tokens rather than the pre-registered rare-favoring direction --- a |
| pattern that replicates across three independently trained arms |
| (Table~\ref{tab:router-rho}). Scope: one pattern string (10 |
| attention + 10 routed FFN, uniform layer count and placement); whether |
| the depth-heterogeneity is intrinsic to this stack shape or an artifact |
| of it is untested. |
|
|
| \paragraph{Law 4: the gate score itself carries the selection signal --- |
| post-hoc thresholding matches oracle competition, and domain-shaped |
| allocation emerges without an explicit allocator.} A fixed per-layer |
| quantile threshold, calibrated once on a held-out slice and frozen, |
| reproduces expert-choice's full sequence-level competition to within |
| $0.34\%$ bpb on every domain (\S\ref{sec:router}, axis c); under that |
| same fixed threshold, mean selection rate still varies $2.2\times$ across |
| domains (math hungriest at $0.67$, code most economical at $0.31$) with |
| domain-specific per-layer hotspots (a CJK-specific layer-2 saturation at |
| $\approx$1.0), with no domain label or explicit allocator anywhere in |
| the mechanism. Scope: probed only post-hoc on a checkpoint trained with |
| sequence-level expert-choice; a model trained from scratch against a |
| threshold, rather than adapted to one after the fact, is untested. |
| |
|
|
| \paragraph{Law 5: capacity generalizes across the dial --- a |
| single-point-trained gate supports an inference-time compute knob at low |
| cost.} The same frozen checkpoint, trained at one capacity ($C=0.5$), |
| degrades smoothly and monotonically when its threshold is recalibrated |
| to unseen capacities from $0.125$ to $0.75$, with no cliff at any tested |
| point, and at $C=0.25$ it even beats a checkpoint natively trained at |
| that capacity on code ($-2.5\%$, \S\ref{sec:router}). Scope: one |
| checkpoint, one calibration domain (English), a $6\times$ capacity range |
| ($0.125$--$0.75$); capacities outside that range, and calibration on |
| other domains, are untested. |
|
|
| \section{Limitations} |
| \label{sec:limitations} |
|
|
| \paragraph{Wind-tunnel scale.} Every result in this paper, without |
| exception, comes from a single-seed, $\sim$1B-parameter, |
| $\sim$1.66B-token, 6{,}309--6{,}323-step run per arm (\S\ref{sec:setup}; |
| the default endpoint is 6{,}322--6{,}323, but \texttt{sr50} stopped at |
| 6{,}309 due to a disclosed infra incident, \S\ref{sec:router}). |
| No arm was replicated across seeds; where we report a pattern as |
| ``replicated'' (the deep-three frequency-aligned gate structure, |
| Table~\ref{tab:router-rho}) we mean replicated across \emph{different |
| architectural arms trained on the same data slice}, not across |
| independent seeds of the same arm. Absolute bpb values, throughput |
| numbers, and any claim not explicitly hedged as directional should be |
| read as priors for a larger-scale run, not as production numbers |
| themselves. |
|
|
| \paragraph{Single evaluation endpoint: bpb only.} Every headline number |
| in this paper is bits-per-byte on held-out text. No downstream task |
| evaluation (code execution, math answer-checking, instruction-following, |
| or any agentic/tool-use benchmark) was run against any arm in this |
| paper's scope. A bpb win is evidence the model's predictive distribution |
| over held-out bytes improved; it is not evidence of a corresponding gain |
| on any task built on top of that distribution. |
|
|
| \paragraph{Noise band is one measurement protocol.} All win/loss/in-band |
| calls use a single pre-registered $\pm$1.5\% relative-bpb band |
| (\S\ref{sec:setup}), estimated once and reused across all three |
| campaigns. Values close to the edge of the band --- \texttt{sr25}'s math |
| gap at $+1.43\%$ (Table~\ref{tab:router-v1}), \texttt{srpl}'s math gap |
| at $+1.57\%$ (Table~\ref{tab:router-srpl}) --- are calls made against |
| that one estimate, not against a re-derived per-arm noise floor; a |
| wider or narrower band, estimated independently, could flip individual |
| calls that sit within a few tenths of a percent of $\pm$1.5\%. |
|
|
| \paragraph{Carved, not held-out-by-construction, validation sets.} The |
| 5-val protocol's Chinese and math splits are carved from the training |
| data's tail; the English and code splits are a train-head cap with an |
| unseen remainder (\S\ref{sec:setup}). Neither carving scheme guarantees |
| the same kind of held-out-ness as a fully disjoint corpus, and the two |
| schemes are not identical to each other across languages. |
|
|
| \paragraph{Cross-protocol endpoints are not interchangeable.} Three |
| distinct comparison protocols appear in this paper and are labeled |
| individually in each table's caption, not assumed interchangeable: (i) |
| the niche campaign's carved 5-val bpb, adjudicated by its own phase-2 |
| verdict machinery (\texttt{phase2\_verdict.txt}, commit \texttt{7cfbaa2}), |
| and the STEM-router v1 campaign's carved 5-val bpb (\texttt{sr50}, |
| \texttt{sr25} vs.\ \texttt{bi\_e}), adjudicated by \texttt{analyze\_sr.py} |
| --- both against \texttt{bi\_e} and sharing the same carved-5-val |
| protocol and $\pm$1.5\% band, but \emph{not} the same adjudicator |
| script; (ii) the mechanism factorial's separate rsync'd 4-val protocol |
| (\S\ref{sec:factorial}); and (iii) the router v2 campaign's |
| pre-registered pairwise endpoint comparisons (\texttt{srpl}-vs-\texttt{sr25}, |
| \texttt{srck}-vs-\texttt{srng}, \texttt{srng}/\texttt{srck}-vs-\texttt{sr50}), |
| adjudicated by \texttt{analyze\_v2.py}, with \texttt{analyze\_sr.py} |
| additionally supplying the descriptive vs-\texttt{bi\_e} context for v2 |
| arms (e.g.\ Table~\ref{tab:router-srpl}'s \texttt{srpl}-vs-\texttt{bi\_e} |
| column, including the math $+1.57\%$ figure cited above). Within (i), |
| \texttt{sr50}'s final checkpoint is step 6{,}309 (13 steps short of the |
| default 6{,}322, a disclosed disk-full-induced early data exhaustion, |
| \S\ref{sec:router}), while \texttt{sr25} and all v2 arms (\texttt{srpl}, |
| \texttt{srng}, \texttt{srck}) reach the default endpoint 6{,}322 --- |
| every table that mixes an \texttt{sr50}-anchored comparison with a |
| 6{,}322-endpoint arm states both endpoint steps explicitly |
| (Tables~\ref{tab:router-eb}, \ref{tab:router-tau}). |
|
|
| \paragraph{Hash-collision regime tested at one slot count.} The |
| key-replacement axis's bigram tables (\S\ref{sec:router}, axis b) were |
| tested at one slot count each (131{,}072 for the split ladder, |
| 86{,}016 for the bigram-only arm), sized to this wind tunnel's 1.66B-token |
| corpus. Neither the collision rate nor the quality effect of a |
| substantially larger or smaller table was measured; the conclusion that |
| static n-gram keys add no dividend applies to this slot-count regime, |
| not to hash tables in general. |
|
|
| \paragraph{Pruned intermediate snapshots.} \texttt{srck}'s bpb curve |
| between steps 3{,}000 and 6{,}000 was pruned before the battery ran, to |
| manage disk footprint; only the endpoint checkpoint was evaluated for |
| this arm's verdict. The endpoint-only pre-registered criterion is |
| unaffected, but no dose-response or early-leverage curve (of the kind |
| Table~\ref{tab:h5} shows for the niche campaign) is available for this |
| arm. |
|
|
| \paragraph{Placement and schedule choices were fixed or hand-designed, |
| not swept.} The niche and factorial campaigns' 60\%-STEM FFN placement |
| was never swept (\S\ref{sec:niche}); the STEM-router's per-layer capacity |
| schedule (\texttt{srpl}) was hand-designed from a single prior read-out |
| of the deep-three frequency-aligned structure, not learned or searched over |
| alternative shapes (\S\ref{sec:router}, axis a). Both campaigns report a |
| single tested point, not a swept curve; whether a different placement |
| fraction or a different, possibly learned, capacity schedule would |
| dominate the ones tested here is open. |
|
|
| These results inform successor designs but are not, by themselves, a |
| recipe for one; every number in this paper should be read as we present |
| it --- a wind-tunnel-scale prior, gated for falsification, not for |
| production deployment. |
|
|
| \bibliographystyle{plainnat} |
| \begin{thebibliography}{9} |
|
|
| \bibitem[Shazeer et~al.(2017)]{shazeer2017moe} |
| N.~Shazeer, A.~Mirhoseini, K.~Maziarz, A.~Davis, Q.~Le, G.~Hinton, J.~Dean. |
| \newblock Outrageously large neural networks: The sparsely-gated |
| mixture-of-experts layer. |
| \newblock \emph{ICLR}, 2017. |
|
|
| \bibitem[Lample et~al.(2019)]{lample2019pkm} |
| G.~Lample, A.~Sablayrolles, M.~Ranzato, L.~Denoyer, H.~J\'{e}gou. |
| \newblock Large memory layers with product keys. |
| \newblock \emph{NeurIPS}, 2019. |
|
|
| \bibitem[Raposo et~al.(2024)]{raposo2024mod} |
| D.~Raposo, S.~Ritter, B.~Richards, T.~Lillicrap, P.~C.~Humphreys, A.~Santoro. |
| \newblock Mixture-of-depths: Dynamically allocating compute in |
| transformer-based language models. |
| \newblock arXiv:2404.02258, 2024. |
|
|
| \end{thebibliography} |
|
|
| \end{document} |
|
|
