Upload BabyLM 2026 Accuracy-Morph Strict-Small checkpoint

README.md

@@ -0,0 +1,73 @@
+---
+library_name: transformers
+tags:
+- babylm
+- babylm-2026
+- strict-small
+- masked-language-modeling
+- bert
+- accuracy-morph
+pipeline_tag: fill-mask
+---
+
+# BabyLM 2026 Strict-Small Accuracy-Morph
+
+This is the final Accuracy-Morph seed-2 checkpoint from the BabyLM 2026 Strict-Small masking experiments.
+
+The model is a compact BERT-style masked language model trained from scratch under a controlled masking-only setup. The experiment keeps the dataset, tokenizer, architecture, optimizer, schedule, training length, and total 15% MLM mask rate fixed, changing only how masked positions are selected.
+
+## Intended Use
+
+This checkpoint is intended for BabyLM 2026 shared-task evaluation and for reproducing the accompanying controlled masking study. It is not intended as a general-purpose language model.
+
+## Training Setup
+
+- Track: BabyLM 2026 Strict-Small
+- Training data: `BabyLM-community/BabyLM-2026-Strict-Small`
+- Tokenizer: `bert-base-uncased`
+- Architecture: compact BERT masked language model
+- Hidden size: 256
+- Layers: 4
+- Attention heads: 4
+- Intermediate size: 1024
+- Parameters: approximately 11.2M
+- Maximum sequence length: 128
+- Batch size: 64
+- Training steps: 10,000
+- Learning rate: 5e-4
+- Weight decay: 0.01
+- Warmup fraction: 0.06
+- Total MLM mask rate: 15%
+- Seed: 2
+
+## Masking Policy
+
+Accuracy-Morph uses smoothed top-1 prediction correctness as the online difficulty signal. During warmup, masking is random. After warmup, the fixed 15% MLM mask budget is allocated as:
+
+- 80% random masking
+- 15% token-level correctness-guided masking
+- 5% character-trigram correctness-guided masking
+
+Sketches count masked-token exposures and wrong top-1 predictions. Candidate difficulty is estimated from smoothed error rates rather than raw cross-entropy loss.
+
+## Local Validation Result
+
+The final local validation MLM loss for this seed-2 checkpoint was:
+
+```text
+3.0105
+```
+
+In the matched seed-2 comparison, the random MLM baseline validation loss was 3.0296.
+
+## Files
+
+- `model.safetensors`: model weights
+- `config.json`: model configuration
+- `tokenizer.json`, `tokenizer_config.json`: tokenizer files
+- `training_config.yaml`: training and masking configuration used for the run
+- `metadata.json`: run metadata
+
+## Caveats
+
+This is a research checkpoint from a small-data controlled experiment. The associated paper frames the result cautiously: Accuracy-Morph is the most promising tested masking signal, but it is not a settled robust improvement without broader multi-seed confirmation.

config.json

@@ -0,0 +1,28 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.7.0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 30522
+}

metadata.json

@@ -0,0 +1,4 @@
+{
+  "seed": 2,
+  "git_hash": "unknown"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:717f3e8dbaf8036e4a3b386bfaefc2cecf629e41eb87a4fa986e1dfece1d6bc4
+size 44813832

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_config.yaml

@@ -0,0 +1,43 @@
+name: accuracy_morph_seed2
+output_root: outputs
+data:
+  dataset_name: BabyLM-community/BabyLM-2026-Strict-Small
+  text_column: text
+  validation_size: 2000
+  num_proc: 4
+tokenizer:
+  name: bert-base-uncased
+model:
+  hidden_size: 256
+  num_hidden_layers: 4
+  num_attention_heads: 4
+  intermediate_size: 1024
+  max_position_embeddings: 512
+training:
+  max_length: 128
+  batch_size: 64
+  max_steps: 10000
+  learning_rate: 0.0005
+  weight_decay: 0.01
+  warmup_fraction: 0.06
+  max_grad_norm: 1.0
+  eval_every: 500
+  eval_batches: 40
+masking:
+  type: accuracy_morph
+  mask_rate: 0.15
+  random_share: 0.8
+  hard_token_share: 0.15
+  hard_morph_share: 0.05
+  adaptive_warmup_fraction: 0.3
+  update_every_steps: 200
+  smoothing_alpha: 1.0
+  min_seen: 3.0
+  score_cap: 0.9
+  char_ngram_n: 3
+  add_boundaries: true
+  token_cms_width: 65536
+  token_cms_depth: 4
+  morph_cms_width: 65536
+  morph_cms_depth: 4
+  conservative_update: true