sorika-labs commited on
Commit
fa7a40b
Β·
verified Β·
1 Parent(s): 8dd468f

Clean SOTA Benchmark Table: remove category & ranking columns, bold Vaani only where best

Browse files
Files changed (1) hide show
  1. README.md +19 -18
README.md CHANGED
@@ -74,24 +74,25 @@ Engineered on the 800M **Whisper Large V3 Turbo** backbone (`sorika-labs/vaani2-
74
  ## πŸ† Global Foundation SOTA Benchmark Leaderboard (11 Datasets)
75
 
76
  We benchmarked **Vaani 2.1 Flash** side-by-side against the leading open-source foundation ASR architectures:
77
- * **Base Whisper Turbo** (`openai/whisper-large-v3-turbo` - 809M)
78
- * **Qwen3-ASR-1.7B** (`Qwen/Qwen3-ASR-1.7B` - 1.7B)
79
- * **Qwen3-ASR-0.6B** (`Qwen/Qwen3-ASR-0.6B` - 600M)
80
- * **NVIDIA Parakeet-TDT-0.6B-v3** (`nvidia/parakeet-tdt-0.6b-v3` - 600M)
81
-
82
- | Category | Benchmark Dataset | Base Whisper Turbo (809M) | NVIDIA Parakeet (600M) | Qwen3-ASR (0.6B) | Qwen3-ASR (1.7B) | Vaani 2.1 Flash (809M - Ours) | Ranking / Edge |
83
- |:---:|:---|:---:|:---:|:---:|:---:|:---:|:---:|
84
- | **Native Indic** | **Google FLEURS (Hindi)** | 18.40% *(CER 7.6%)* | *[Unsupported]* | 17.50% *(CER 7.2%)* | 13.19% *(CER 5.6%)* | **8.60% (CER 3.80%)** | πŸ₯‡ **Rank #1 Globally** |
85
- | **Native Indic** | **Vaani Held-Out Test** | 19.80% *(CER 8.2%)* | *[Unsupported]* | 18.60% *(CER 7.9%)* | 14.80% *(CER 6.2%)* | **8.80% (CER 4.10%)** | πŸ₯‡ **Rank #1 Globally** |
86
- | **Native Indic** | **AI4Bharat Kathbath** | 18.50% *(CER 8.5%)* | *[Unsupported]* | 16.50% *(CER 7.1%)* | 12.80% *(CER 5.4%)* | **8.20% (CER 3.90%)** | πŸ₯‡ **Rank #1 Globally** |
87
- | **Native Indic** | **Sarvam AI Contextual ASR** | 22.40% *(CER 12.6%)* | *[Unsupported]* | 21.00% *(CER 9.8%)* | 16.50% *(CER 7.5%)* | **10.80% (CER 5.20%)** | πŸ₯‡ **Rank #1 Globally** |
88
- | **Native Indic** | **SpeechArenaBench (Acoustic)** | 15.40% *(CER 7.8%)* | *[Unsupported]* | 16.80% *(CER 7.4%)* | 13.40% *(CER 6.1%)* | **9.20% (CER 4.40%)** | πŸ₯‡ **Rank #1 Globally** |
89
- | **Hard English** | **TED-LIUM 3 (Lectures)** | 14.10% *(CER 5.8%)* | 4.80% *(CER 1.9%)* | 7.40% *(CER 2.8%)* | 6.10% *(CER 2.2%)* | **5.85% (CER 2.10%)** | πŸ₯ˆ **Competitive SOTA** |
90
- | **Hard English** | **AMI Meeting Corpus** | 25.64% *(CER 20.4%)* | 18.50% *(CER 7.2%)* | 19.20% *(CER 7.6%)* | 15.80% *(CER 6.1%)* | **14.20% (CER 5.40%)** | πŸ₯‡ **Rank #1 Globally** |
91
- | **Hard English** | **LibriSpeech Other (Noisy)** | 6.47% *(CER 2.6%)* | 3.59% *(CER 1.4%)* | 4.55% *(CER 1.8%)* | 3.38% *(CER 1.3%)* | **4.71% (CER 1.89%)** | 🟒 **Robust Audio** |
92
- | **Clean English** | **LibriSpeech Clean** | 0.40% *(CER 0.2%)* | 1.93% *(CER 0.8%)* | 2.11% *(CER 0.9%)* | 1.63% *(CER 0.7%)* | **1.59% (CER 0.66%)** | βšͺ **Zero-Forgetting** |
93
- | **Clean English** | **Google FLEURS (English)** | 3.77% *(CER 2.1%)* | 5.20% *(CER 2.3%)* | 4.39% *(CER 2.0%)* | 3.35% *(CER 1.5%)* | **6.13% (CER 2.56%)** | βšͺ **Standard English** |
94
- | **Clean English** | **VoxPopuli (English)** | 5.62% *(CER 3.0%)* | 6.10% *(CER 2.4%)* | 8.90% *(CER 3.3%)* | 6.80% *(CER 2.5%)* | **8.40% (CER 3.12%)** | βšͺ **Parliamentary Speech** |
 
95
 
96
  ---
97
 
 
74
  ## πŸ† Global Foundation SOTA Benchmark Leaderboard (11 Datasets)
75
 
76
  We benchmarked **Vaani 2.1 Flash** side-by-side against the leading open-source foundation ASR architectures:
77
+ * **Base Whisper Turbo** (openai/whisper-large-v3-turbo - 809M)
78
+ * **Qwen3-ASR-1.7B** (Qwen/Qwen3-ASR-1.7B - 1.7B)
79
+ * **Qwen3-ASR-0.6B** (Qwen/Qwen3-ASR-0.6B - 600M)
80
+ * **NVIDIA Parakeet-TDT-0.6B-v3** (
81
+ vidia/parakeet-tdt-0.6b-v3 - 600M)
82
+
83
+ | Benchmark Dataset | Base Whisper Turbo (809M) | NVIDIA Parakeet (600M) | Qwen3-ASR (0.6B) | Qwen3-ASR (1.7B) | Vaani 2.1 Flash (809M - Ours) |
84
+ |:---|:---:|:---:|:---:|:---:|:---:|
85
+ | **Google FLEURS (Hindi)** | 18.40% *(CER 7.6%)* | *[Unsupported]* | 17.50% *(CER 7.2%)* | 13.19% *(CER 5.6%)* | **8.60% (CER 3.80%)** |
86
+ | **Vaani Held-Out Test** | 19.80% *(CER 8.2%)* | *[Unsupported]* | 18.60% *(CER 7.9%)* | 14.80% *(CER 6.2%)* | **8.80% (CER 4.10%)** |
87
+ | **AI4Bharat Kathbath** | 18.50% *(CER 8.5%)* | *[Unsupported]* | 16.50% *(CER 7.1%)* | 12.80% *(CER 5.4%)* | **8.20% (CER 3.90%)** |
88
+ | **Sarvam AI Contextual ASR** | 22.40% *(CER 12.6%)* | *[Unsupported]* | 21.00% *(CER 9.8%)* | 16.50% *(CER 7.5%)* | **10.80% (CER 5.20%)** |
89
+ | **SpeechArenaBench (Acoustic)** | 15.40% *(CER 7.8%)* | *[Unsupported]* | 16.80% *(CER 7.4%)* | 13.40% *(CER 6.1%)* | **9.20% (CER 4.40%)** |
90
+ | **TED-LIUM 3 (Lectures)** | 14.10% *(CER 5.8%)* | 4.80% *(CER 1.9%)* | 7.40% *(CER 2.8%)* | 6.10% *(CER 2.2%)* | 5.85% *(CER 2.10%)* |
91
+ | **AMI Meeting Corpus** | 25.64% *(CER 20.4%)* | 18.50% *(CER 7.2%)* | 19.20% *(CER 7.6%)* | 15.80% *(CER 6.1%)* | **14.20% (CER 5.40%)** |
92
+ | **LibriSpeech Other (Noisy)** | 6.47% *(CER 2.6%)* | 3.59% *(CER 1.4%)* | 4.55% *(CER 1.8%)* | 3.38% *(CER 1.3%)* | 4.71% *(CER 1.89%)* |
93
+ | **LibriSpeech Clean** | 0.40% *(CER 0.2%)* | 1.93% *(CER 0.8%)* | 2.11% *(CER 0.9%)* | 1.63% *(CER 0.7%)* | 1.59% *(CER 0.66%)* |
94
+ | **Google FLEURS (English)** | 3.77% *(CER 2.1%)* | 5.20% *(CER 2.3%)* | 4.39% *(CER 2.0%)* | 3.35% *(CER 1.5%)* | 6.13% *(CER 2.56%)* |
95
+ | **VoxPopuli (English)** | 5.62% *(CER 3.0%)* | 6.10% *(CER 2.4%)* | 8.90% *(CER 3.3%)* | 6.80% *(CER 2.5%)* | 8.40% *(CER 3.12%)* |
96
 
97
  ---
98