Turkmen MMS ASR

Turkmen speech โ†’ Turkmen Latin text. A standalone Wav2Vec2/MMS CTC model with one embedded tuk-script_latin adapter and one flat vocabulary. It does not require downloading a separate language adapter or switching languages.

This is an experimental, non-commercial release by Pirnazar, based on Meta's facebook/mms-1b-all. It is a Transformers model, not a Whisper/CTranslate2 checkpoint.

Results

The exported checkpoint was selected using validation before the held-out test. Both the original Turkmen MMS adapter and this export were evaluated on the same audio and references.

Evaluation Original Turkmen MMS This release
Held-out TurkmenSpeech test: strict WER 37.05% 24.58%
Held-out TurkmenSpeech test: strict CER 5.95% 4.11%
  • Held-out test: 3,879 clips, 8.42 hours, 61,733 reference words, 17 source recordings. None of those source recordings or full normalized transcript strings occur in the training or validation splits.
  • Improvement was observed on all 17 test sources. Paired bootstrap over source recordings (2,000 resamples): 95% interval for the strict WER reduction is 11.72โ€“13.09 percentage points.
  • Speaker identities are unknown. Source-disjoint evaluation does not establish speaker-disjoint evaluation or performance on every dialect.
  • Strict metrics preserve case, punctuation and Turkmen diacritics after NFC normalization. Normalized metrics additionally casefold and map punctuation to spaces; diacritics are retained. CER includes spaces.

These figures are from this release's custom split, not an official benchmark leaderboard or a comparison against all Turkmen ASR systems. Conversational, social-media, noisy and musical audio remain challenging.

Local usage

Tested with Python 3.11/3.12, PyTorch 2.9.1+cu126, Transformers 5.17.0, and safetensors 0.8.0, on CPU and NVIDIA CUDA.

pip install "torch==2.9.1" "transformers==5.17.0" "safetensors==0.8.0" soundfile numpy
python transcribe.py speech.wav --device cpu
# With a suitable CUDA PyTorch build:
python transcribe.py speech.wav --device cuda

The demo expects 16 kHz WAV audio. Stereo input is averaged to mono. Convert other sample rates locally, for example:

ffmpeg -i recording.ogg -ar 16000 -ac 1 -c:a pcm_s16le speech.wav

For a model folder already downloaded to disk:

python transcribe.py speech.wav --model ./mms-1b-turkmen --local-files-only --device cpu

The first Hub load downloads weights. Recognition runs locally; no cloud STT or LLM correction is used. The demo processes long audio in overlapping windows.

Transformers example

from contextlib import nullcontext
import soundfile as sf
import torch
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

repo = "Pirnazar/mms-1b-turkmen"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
processor = Wav2Vec2Processor.from_pretrained(repo)
model = Wav2Vec2ForCTC.from_pretrained(
    repo, dtype=dtype, attn_implementation="sdpa"
).to(device).eval()

audio, sr = sf.read("speech.wav", dtype="float32")
if audio.ndim == 2:
    audio = audio.mean(axis=1)
if sr != 16000:
    raise ValueError("Convert the audio to 16 kHz first")

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
inputs = {name: value.to(device) for name, value in inputs.items()}
amp = torch.autocast("cuda", dtype=dtype) if device == "cuda" else nullcontext()
with torch.inference_mode(), amp:
    logits = model(**inputs).logits
text = processor.batch_decode(logits.argmax(-1))[0].strip()
print(text)

Use short utterances (up to approximately 30 seconds) with the direct example. Use transcribe.py for longer recordings. Do not call load_adapter() or set_target_lang() on this standalone release. No trust_remote_code is needed.

Training and provenance

  • Base: facebook/mms-1b-all, revision 3d33597edbdaaba14a8e858e2c8caa76e3cec0cd.
  • The existing trained tuk-script_latin adapter was preserved and further trained; it was not reinitialized. The backbone remained frozen.
  • Approximately 965M total parameters, 2.28M trainable adapter/head parameters.
  • Training data: mamed0v/TurkmenSpeech.
  • Train: 104,713 clips / 195.47 h / 370 sources. Validation: 6,146 clips / 11.36 h / 24 sources. Test: 3,879 clips / 8.42 h / 17 sources.
  • Source grouping preceded duration filtering. Full clips above the duration bound were not shortened. 2,808 clips with repeated normalized transcript strings spanning splits were quarantined, including every cross-split copy.
  • One epoch, 13,090 optimizer updates, microbatch 1, gradient accumulation 8, AdamW, learning rate 1e-4, weight decay 0.01, 5% warmup, linear decay with a 5%-of-peak floor, gradient clipping 1.0, seed 42, gradient checkpointing.
  • The run continued from a laptop in BF16 to a second GPU in FP16, preserving adapter weights, AdamW state and data cursor. GradScaler was initialized for FP16. This was not a bitwise-identical cross-hardware continuation.
  • Selected checkpoint: 11,000, using strict WER on a fixed 64-clip validation sample spanning all 24 validation sources. Validation WER: 24.84%.
  • No private recordings were used for training. No LLM-generated corrections or synthetic additional training examples were added in this run.
  • The vocabulary retains the original token IDs and appends characters for case, digits, punctuation and names. The output has 102 tokens. Character availability alone does not establish reliable brand-name recognition.

Public aggregate results and sanitized provenance are in evaluation_results.json and training_provenance.json. No private audio, transcripts, user identifiers, machine paths, optimizer state or credentials are included in this repository.

Limitations

  • This is a single-language Turkmen Latin ASR model. Translation, diarization, speaker identification and reliable multilingual code switching are not provided.
  • Dialects, informal endings, names and product brands can still be misrecognized.
  • The training corpus and evaluation references may contain transcription errors.
  • Runtime depends on hardware and audio length. The evaluation was on a working laptop and involved a process restart; its timings are not a controlled speed comparison. No universal latency or memory guarantee is made.
  • This snapshot cannot be loaded by faster-whisper; use Transformers/PyTorch.

License and attribution

CC BY-NC 4.0 โ€” non-commercial use only; attribution is required. See LICENSE and NOTICE.

Credit Meta / the MMS authors for the base weights, mamed0v for TurkmenSpeech, and Pirnazar for this adapter fine-tune and standalone export. This release changes the Turkmen adapter/output head and vocabulary and packages one language as a self-contained FP16 snapshot. The upstream model and dataset retain their respective notices and conditions.

Downloads last month
45
Safetensors
Model size
1.0B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Pirnazar/mms-1b-turkmen

Finetuned
(443)
this model

Evaluation results

  • WER (strict, percent) on TurkmenSpeech (custom source-disjoint held-out test)
    test set self-reported
    24.580
  • CER (strict, percent) on TurkmenSpeech (custom source-disjoint held-out test)
    test set self-reported
    4.106