kisoku-3b-base / README.md
0arch-io's picture
Mark as superseded, link the Kisoku 1.6B release and 0arch.io/kisoku
69d4c3d verified
|
Raw History Blame Contribute Delete
3.59 kB
metadata
license: apache-2.0
language:
  - en
tags:
  - from-scratch
  - pretrained
  - trc
  - tpu
  - maxtext
  - jax
  - grouped-query-attention
datasets:
  - mlfoundations/dclm-baseline-1.0
  - HuggingFaceFW/fineweb-edu
model-index:
  - name: kisoku-3b-base
    results:
      - task:
          type: text-generation
        dataset:
          name: HellaSwag
          type: hellaswag
        metrics:
          - name: acc_norm (0-shot, 500 samples)
            type: accuracy
            value: 0.3
      - task:
          type: text-generation
        dataset:
          name: ARC-Challenge
          type: arc_challenge
        metrics:
          - name: acc_norm (0-shot, 500 samples)
            type: accuracy
            value: 0.262
      - task:
          type: text-generation
        dataset:
          name: WinoGrande
          type: winogrande
        metrics:
          - name: acc (0-shot, 500 samples)
            type: accuracy
            value: 0.508
      - task:
          type: text-generation
        dataset:
          name: TruthfulQA MC2
          type: truthful_qa
        metrics:
          - name: acc (0-shot, 500 samples)
            type: accuracy
            value: 0.4925

Superseded. This is Kisoku v1, an earlier 3B model. The current release is Kisoku 1.6B. Training log, benchmark charts and run instructions: 0arch.io/kisoku.

Kisoku 3B Base

A 3B parameter language model trained entirely from scratch on Google Cloud TPUs using MaxText (JAX), supported by Google's TPU Research Cloud (TRC).

Overview

Kisoku 3B is an independent research project by a solo researcher, trained from scratch with no pretrained weight initialization. This model represents a learning journey in open-source LLM pretraining on TPU infrastructure.

This is the base (pretrained) model. For the instruction-tuned version, see kisoku-3b-sft.

Architecture

Parameter Value
Architecture LlamaForCausalLM
Parameters ~3B
Layers 28
Hidden size 3072
FFN size 8192
Attention heads 24
KV heads 6 (Grouped-Query Attention)
Head dim 128
Vocab size 128,256
Context length 4,096
Position encoding RoPE (theta=500,000)
Activation SiLU
Tied embeddings Yes

Training Details

Detail Value
Framework MaxText (JAX) on Google Cloud TPU
Hardware TPU v4-32 (32 chips), on-demand
Training steps 460,000
Training data DCLM-Baseline 1.0, FineWeb-Edu
Precision bfloat16
Compute provider Google TPU Research Cloud (TRC)

Benchmarks

Evaluated with lm-evaluation-harness v0.4.11, 0-shot, 500 samples per task:

Benchmark Metric Score
HellaSwag acc_norm 0.300
ARC-Challenge acc_norm 0.262
WinoGrande acc 0.508
TruthfulQA MC2 acc 0.493

Note: Benchmarks indicate the model is significantly undertrained. Competitive 3B models require 2-11T tokens of training data. This model was trained on substantially fewer tokens. A v2 with dramatically more training data is planned.

Usage

Limitations

  • Undertrained: needs significantly more tokens to reach competitive performance
  • English-focused
  • No safety alignment or content filtering applied
  • Base model only (not instruction-tuned)

Acknowledgments

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC).

License

Apache 2.0