--- license: apache-2.0 language: - en tags: - from-scratch - pretrained - trc - tpu - maxtext - jax - grouped-query-attention datasets: - mlfoundations/dclm-baseline-1.0 - HuggingFaceFW/fineweb-edu model-index: - name: kisoku-3b-base results: - task: type: text-generation dataset: name: HellaSwag type: hellaswag metrics: - name: acc_norm (0-shot, 500 samples) type: accuracy value: 0.30 - task: type: text-generation dataset: name: ARC-Challenge type: arc_challenge metrics: - name: acc_norm (0-shot, 500 samples) type: accuracy value: 0.262 - task: type: text-generation dataset: name: WinoGrande type: winogrande metrics: - name: acc (0-shot, 500 samples) type: accuracy value: 0.508 - task: type: text-generation dataset: name: TruthfulQA MC2 type: truthful_qa metrics: - name: acc (0-shot, 500 samples) type: accuracy value: 0.4925 --- > **Superseded.** This is Kisoku v1, an earlier 3B model. The current release is [Kisoku 1.6B](https://huggingface.co/0arch-io/kisoku-1.6b). Training log, benchmark charts and run instructions: [0arch.io/kisoku](https://0arch.io/kisoku). # Kisoku 3B Base A 3B parameter language model trained **entirely from scratch** on Google Cloud TPUs using [MaxText](https://github.com/AI-Hypercomputer/maxtext) (JAX), supported by [Google's TPU Research Cloud (TRC)](https://sites.research.google/trc/). ## Overview Kisoku 3B is an independent research project by a solo researcher, trained from scratch with no pretrained weight initialization. This model represents a learning journey in open-source LLM pretraining on TPU infrastructure. **This is the base (pretrained) model.** For the instruction-tuned version, see [kisoku-3b-sft](https://huggingface.co/0arch-io/kisoku-3b-sft). ## Architecture | Parameter | Value | |-----------|-------| | Architecture | LlamaForCausalLM | | Parameters | ~3B | | Layers | 28 | | Hidden size | 3072 | | FFN size | 8192 | | Attention heads | 24 | | KV heads | 6 (Grouped-Query Attention) | | Head dim | 128 | | Vocab size | 128,256 | | Context length | 4,096 | | Position encoding | RoPE (theta=500,000) | | Activation | SiLU | | Tied embeddings | Yes | ## Training Details | Detail | Value | |--------|-------| | Framework | MaxText (JAX) on Google Cloud TPU | | Hardware | TPU v4-32 (32 chips), on-demand | | Training steps | 460,000 | | Training data | DCLM-Baseline 1.0, FineWeb-Edu | | Precision | bfloat16 | | Compute provider | Google TPU Research Cloud (TRC) | ## Benchmarks Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.11, 0-shot, 500 samples per task: | Benchmark | Metric | Score | |-----------|--------|-------| | HellaSwag | acc_norm | 0.300 | | ARC-Challenge | acc_norm | 0.262 | | WinoGrande | acc | 0.508 | | TruthfulQA MC2 | acc | 0.493 | **Note:** Benchmarks indicate the model is significantly undertrained. Competitive 3B models require 2-11T tokens of training data. This model was trained on substantially fewer tokens. A v2 with dramatically more training data is planned. ## Usage ## Limitations - Undertrained: needs significantly more tokens to reach competitive performance - English-focused - No safety alignment or content filtering applied - Base model only (not instruction-tuned) ## Acknowledgments Research supported with Cloud TPUs from Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/). ## License Apache 2.0