taekyung-k/beneath-cot-export

Released artifacts

Two things live here, both keyed by model:

• annotations/ — the LLM-annotated reasoning traces (the output of pipeline/02_annotate_spans.py + pipeline/03_build_span_dataset.py). Human-readable JSON.
• probes/ — the fitted PCA→LDA operation probes (the output of pipeline/04_fit_lda_probes.py). These are what you need to score new text.

release/ is git-ignored; it is distributed separately (see the repository README).

---

Reasoning operations

All probes and annotations use the same eight operations, in this index order (the integer labels in y_tr / y_te and in label_names):

idx	label
0	[EXTRACTION]
1	[SYMBOLIZATION.DIRECT-MAPPING]
2	[STRUCTURAL-ANALYSIS.DECOMPOSITION]
3	[RETRIEVAL.RECALL]
4	[INFERENCE-FLOW.CHAINING.DEDUCTION]
5	[EXECUTION.OPERATION.ALGEBRAIC-MANIPULATION]
6	[EXECUTION.OPERATION.ARITHMETIC-COMPUTATION]
7	[FINAL-ANSWER]

The annotation taxonomy is larger than this (see code/prompt.py); these eight are the operations that survived the frequency threshold and are analysed in the paper. The annotation files retain all tags, so you can re-derive a different label subset by editing LABEL_CONFIG in pipeline/04_fit_lda_probes.py.

---

annotations/<Model>/

file	what it is
annotated.json	every trace with word-level operation spans
spans.json	same, plus token-level span positions
spans_train.json / spans_test.json	the split the probes were fit on
spans_test_MATH-500.json, spans_test_gpqa-diamond.json	held-out-domain test sets (no training counterpart)
span_stats.json, span_stats.csv	per-operation span counts
erroneous/	Qwen3-8B only — the incorrect-trace analysis (§ Erroneous execution)

Counts

Model	traces	train / test	domains	held-out domains
Qwen3-8B	900	640 / 161	dapo17k 450, TheoremQA 450	MATH-500, GPQA-Diamond
Qwen2.5-7B	704	540 / 145	dapo17k 352, TheoremQA 352	MATH-500, GPQA-Diamond
Gemma4-31B	942	750 / 191	dapo17k 500, TheoremQA 442	MATH-500, GPQA-Diamond
Llama3-8B	314	179 / 45	dapo17k 117, TheoremQA 197	—

Llama3-8B yielded far fewer usable correct traces, so it ran with the default train split (179) rather than the larger split used for the other three. Its per-class training counts are also unbalanced (35–100 spans/class vs. a flat 100 for the other models), which is why it is reported separately.

What the indices mean

Each item in annotated.json / spans.json carries four identifiers:

• query_idx — index of the question in the source HF dataset, in dataset order, capped by --max-samples. Not unique across domains: dapo17k query_idx=5 and TheoremQA query_idx=5 are different questions, so the identity of a question is the pair (domain, query_idx).
• try_idx — which sampled generation for that question. 00_extract_inference.py samples up to 10 attempts per question and keeps at most 5 correct and 1 incorrect, so one question can contribute several traces.
• _original_index — row id in the round-robin interleaving of the per-domain annotation files, assigned by pipeline/03_build_span_dataset.py step 1. This is the stable key downstream scripts use to refer back to a trace.
• result_type — correct or incorrect, i.e. whether the trace's final answer was graded correct.

The corresponding raw files on disk are named {query_idx:06d}_{try_idx}.json (text, tokens, per-token entropy) and {query_idx:06d}_{try_idx}.pt (hidden states), under INFERENCE/<model>/<domain>/<result_type>/.

What the span offsets mean

• reasoning_type: a list of {"start": i, "end": j, "tag": "..."}. start and end are inclusive word indices into word_list — the span text is " ".join(word_list[start : end + 1]).
• semantic_token_spans: {tag: [{"token_pos_index": [s, e], "token": [...], "text_span": "..."}]}. Here s/e are inclusive token indices into the model's own tokenization of the trace, aligned with the .pt hidden-state rows. token holds the token ids and text_span the decoded string.

Correct vs. incorrect traces

Everything under annotations/<Model>/*.json is correct traces — all probes in the paper are trained and evaluated on correct reasoning only.

Incorrect traces are used in exactly one place, the erroneous-execution analysis (Qwen3-8B), and live in annotations/Qwen3-8B/erroneous/:

file	what it is
filtered_incorrect.json	short (≤700-token), non-Chinese incorrect traces
filtered_correct.json	correct traces paired to those by (domain, query_idx)
annotated_incorrect.json	operation labels and per-chunk factual-error judgements
review.csv	one row per factual-error chunk, for human review

The probes applied to these are the ones in probes/Qwen3-8B/ — trained on correct traces only, applied without retraining.

---

probes/<Model>/

res/res_L<layer>_<token_sel>.joblib, one file per (layer, token-pooling) combination.

• rep_type — res only (residual stream, the paper's main result). res_i is the output of decoder layer i, i.e. the post-block residual stream.
• layer — 00 … n_layers-1.
• token_sel — how a span was pooled into one vector: mean, first, last, middle. mean is the one used throughout the paper, and the one to use for per-token scoring.

Model	layers	probes (res)	paper's layer band
Qwen3-8B	36	144	10–30
Qwen2.5-7B	28	112	8–23
Gemma4-31B	60	240	13–40
Llama3-8B	32	128	10–28

Each .joblib is a dict:

{
  "pca":         sklearn PCA(n_components=128),      # fit on L2-normalised hidden states
  "lda":         sklearn LinearDiscriminantAnalysis, # fit on the PCA scores
  "directions":  {label: unit vector, shape (n_classes-1,)},
  "label_names": [label, ...],                       # index order of y_tr / y_te
}

Scoring one hidden state h against operation c:

h_n = h / ||h||
z   = lda.transform(pca.transform(h_n))
a_c = z[:k] . directions[c][:k]          # k = 8 in the paper

code/tools/export_token_scores.py implements exactly this, folded into a single matrix multiply, and is the intended entry point if you only want to use the probes.

---

Reproducing vs. reusing

• To reuse the probes on your own text: code/tools/export_token_scores.py.
• To reproduce the pipeline from scratch: code/scripts/run_main.sh. It writes to data/ and outputs/, never into release/, so a replication run cannot overwrite anything here.