reaperdoesntknow commited on
Commit
32e5c8c
·
verified ·
1 Parent(s): 2f3aa73

OPSEC: minimize author name (remove surname)

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -14,7 +14,7 @@ tags:
14
  ---
15
  # From Three Teachers to Dual Cognition: Topology-Aware Multi-Teacher Distillation and Role-Conditioned Self-Critique at 1.7B Scale
16
 
17
- **Roy S. Colca**
18
  Convergent Intelligence LLC: Research Division
19
 
20
  **Abstract.** We present a four-stage pipeline for producing small language models (1.7B parameters) that exhibit self-critiquing dual-cognition reasoning from a 30B-parameter teacher. The pipeline chains: (1) proof-weighted knowledge distillation from three teacher variants (Instruct, Thinking, Coder) of Qwen3-30B-A3B, each producing distinct capability profiles in the student; (2) Topological Knowledge Distillation (TKD), which uses the Discrepancy Calculus (DISC) framework to decompose the teacher's output distribution into smooth, jump, and singular-continuous components via bounded variation theory, allocating training capacity to structural boundaries that standard KD smears across; (3) multi-teacher ghost imprinting, where sequential distillation from different teachers creates residual discrepancy fields in weight space that produce emergent capabilities absent from any individual teacher; and (4) DualMind, a role-conditioned generation scheme that collapses multi-architecture dialectical reasoning into a single model via `<explore>`, `<examine>`, and `<response>` tokens. Trained on H100 at BF16 precision, the resulting models demonstrate dual-cognition reasoning — free derivation followed by adversarial self-critique followed by clean synthesis — at a parameter count where such behavior is not typically observed. We release 43 models (12,000+ downloads), training code, and this methodology under Apache 2.0.
@@ -226,8 +226,8 @@ All models, training code, and this methodology are released under Apache 2.0.
226
  ## References
227
 
228
  - Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network.
229
- - Colca, R. S. (2026). Structure Over Scale. DOI: 10.57967/hf/8165.
230
- - Colca, R. S. (2025-2026). Discrepancy Calculus (DISC): A Measure-Theoretic Framework for Singularities. Convergent Intelligence LLC.
231
  - Ambrosio, L., Fusco, N., & Pallara, D. (2000). Functions of Bounded Variation and Free Discontinuity Problems. Oxford.
232
 
233
  ---
 
14
  ---
15
  # From Three Teachers to Dual Cognition: Topology-Aware Multi-Teacher Distillation and Role-Conditioned Self-Critique at 1.7B Scale
16
 
17
+ **Roy C**
18
  Convergent Intelligence LLC: Research Division
19
 
20
  **Abstract.** We present a four-stage pipeline for producing small language models (1.7B parameters) that exhibit self-critiquing dual-cognition reasoning from a 30B-parameter teacher. The pipeline chains: (1) proof-weighted knowledge distillation from three teacher variants (Instruct, Thinking, Coder) of Qwen3-30B-A3B, each producing distinct capability profiles in the student; (2) Topological Knowledge Distillation (TKD), which uses the Discrepancy Calculus (DISC) framework to decompose the teacher's output distribution into smooth, jump, and singular-continuous components via bounded variation theory, allocating training capacity to structural boundaries that standard KD smears across; (3) multi-teacher ghost imprinting, where sequential distillation from different teachers creates residual discrepancy fields in weight space that produce emergent capabilities absent from any individual teacher; and (4) DualMind, a role-conditioned generation scheme that collapses multi-architecture dialectical reasoning into a single model via `<explore>`, `<examine>`, and `<response>` tokens. Trained on H100 at BF16 precision, the resulting models demonstrate dual-cognition reasoning — free derivation followed by adversarial self-critique followed by clean synthesis — at a parameter count where such behavior is not typically observed. We release 43 models (12,000+ downloads), training code, and this methodology under Apache 2.0.
 
226
  ## References
227
 
228
  - Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network.
229
+ - CIx, R. S. (2026). Structure Over Scale. DOI: 10.57967/hf/8165.
230
+ - CIx, R. S. (2025-2026). Discrepancy Calculus (DISC): A Measure-Theoretic Framework for Singularities. Convergent Intelligence LLC.
231
  - Ambrosio, L., Fusco, N., & Pallara, D. (2000). Functions of Bounded Variation and Free Discontinuity Problems. Oxford.
232
 
233
  ---