Clean SOTA Benchmark Table: remove category & ranking columns, bold Vaani only where best
Browse files
README.md
CHANGED
|
@@ -74,24 +74,25 @@ Engineered on the 800M **Whisper Large V3 Turbo** backbone (`sorika-labs/vaani2-
|
|
| 74 |
## π Global Foundation SOTA Benchmark Leaderboard (11 Datasets)
|
| 75 |
|
| 76 |
We benchmarked **Vaani 2.1 Flash** side-by-side against the leading open-source foundation ASR architectures:
|
| 77 |
-
* **Base Whisper Turbo** (
|
| 78 |
-
* **Qwen3-ASR-1.7B** (
|
| 79 |
-
* **Qwen3-ASR-0.6B** (
|
| 80 |
-
* **NVIDIA Parakeet-TDT-0.6B-v3** (
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
|
| 84 |
-
|
|
| 85 |
-
| **
|
| 86 |
-
| **
|
| 87 |
-
| **
|
| 88 |
-
| **
|
| 89 |
-
| **
|
| 90 |
-
| **
|
| 91 |
-
| **
|
| 92 |
-
| **
|
| 93 |
-
| **
|
| 94 |
-
| **
|
|
|
|
| 95 |
|
| 96 |
---
|
| 97 |
|
|
|
|
| 74 |
## π Global Foundation SOTA Benchmark Leaderboard (11 Datasets)
|
| 75 |
|
| 76 |
We benchmarked **Vaani 2.1 Flash** side-by-side against the leading open-source foundation ASR architectures:
|
| 77 |
+
* **Base Whisper Turbo** (openai/whisper-large-v3-turbo - 809M)
|
| 78 |
+
* **Qwen3-ASR-1.7B** (Qwen/Qwen3-ASR-1.7B - 1.7B)
|
| 79 |
+
* **Qwen3-ASR-0.6B** (Qwen/Qwen3-ASR-0.6B - 600M)
|
| 80 |
+
* **NVIDIA Parakeet-TDT-0.6B-v3** (
|
| 81 |
+
vidia/parakeet-tdt-0.6b-v3 - 600M)
|
| 82 |
+
|
| 83 |
+
| Benchmark Dataset | Base Whisper Turbo (809M) | NVIDIA Parakeet (600M) | Qwen3-ASR (0.6B) | Qwen3-ASR (1.7B) | Vaani 2.1 Flash (809M - Ours) |
|
| 84 |
+
|:---|:---:|:---:|:---:|:---:|:---:|
|
| 85 |
+
| **Google FLEURS (Hindi)** | 18.40% *(CER 7.6%)* | *[Unsupported]* | 17.50% *(CER 7.2%)* | 13.19% *(CER 5.6%)* | **8.60% (CER 3.80%)** |
|
| 86 |
+
| **Vaani Held-Out Test** | 19.80% *(CER 8.2%)* | *[Unsupported]* | 18.60% *(CER 7.9%)* | 14.80% *(CER 6.2%)* | **8.80% (CER 4.10%)** |
|
| 87 |
+
| **AI4Bharat Kathbath** | 18.50% *(CER 8.5%)* | *[Unsupported]* | 16.50% *(CER 7.1%)* | 12.80% *(CER 5.4%)* | **8.20% (CER 3.90%)** |
|
| 88 |
+
| **Sarvam AI Contextual ASR** | 22.40% *(CER 12.6%)* | *[Unsupported]* | 21.00% *(CER 9.8%)* | 16.50% *(CER 7.5%)* | **10.80% (CER 5.20%)** |
|
| 89 |
+
| **SpeechArenaBench (Acoustic)** | 15.40% *(CER 7.8%)* | *[Unsupported]* | 16.80% *(CER 7.4%)* | 13.40% *(CER 6.1%)* | **9.20% (CER 4.40%)** |
|
| 90 |
+
| **TED-LIUM 3 (Lectures)** | 14.10% *(CER 5.8%)* | 4.80% *(CER 1.9%)* | 7.40% *(CER 2.8%)* | 6.10% *(CER 2.2%)* | 5.85% *(CER 2.10%)* |
|
| 91 |
+
| **AMI Meeting Corpus** | 25.64% *(CER 20.4%)* | 18.50% *(CER 7.2%)* | 19.20% *(CER 7.6%)* | 15.80% *(CER 6.1%)* | **14.20% (CER 5.40%)** |
|
| 92 |
+
| **LibriSpeech Other (Noisy)** | 6.47% *(CER 2.6%)* | 3.59% *(CER 1.4%)* | 4.55% *(CER 1.8%)* | 3.38% *(CER 1.3%)* | 4.71% *(CER 1.89%)* |
|
| 93 |
+
| **LibriSpeech Clean** | 0.40% *(CER 0.2%)* | 1.93% *(CER 0.8%)* | 2.11% *(CER 0.9%)* | 1.63% *(CER 0.7%)* | 1.59% *(CER 0.66%)* |
|
| 94 |
+
| **Google FLEURS (English)** | 3.77% *(CER 2.1%)* | 5.20% *(CER 2.3%)* | 4.39% *(CER 2.0%)* | 3.35% *(CER 1.5%)* | 6.13% *(CER 2.56%)* |
|
| 95 |
+
| **VoxPopuli (English)** | 5.62% *(CER 3.0%)* | 6.10% *(CER 2.4%)* | 8.90% *(CER 3.3%)* | 6.80% *(CER 2.5%)* | 8.40% *(CER 3.12%)* |
|
| 96 |
|
| 97 |
---
|
| 98 |
|