Decision Index 0.3

README.md

@@ -105,19 +105,9 @@ Hugging Face models and Spaces are first-class artifacts here, not only GitHub r
 
 ## Decision Index (`index.html`)
 
-The **Index** tab (switch at the top of every page) is the leaderboard: every open reproduction that finished the frozen suite, scored with one number, the Decision Index, plus per-category radars against Jev and a full page per model (`index.html?model=<engine>`; `?model=jev` is Jev's own report).
+The **Index** tab (switch at the top of every page) is the leaderboard: every open reproduction and Jev, ranked by the Full score: 20% public benchmarks, 50% private tests of the same skills and 30% private tasks from new domains, all on one scale. Most of the score comes from tests nobody can train on, so a public-only advantage counts for little. The switch on the main chart offers Full score, Public only and Your own (set your own part and area weights). Below the chart sit the per-area radars, every benchmark with its public and private scores, and a full page per model (`index.html?model=<engine>`; `?model=jev` is Jev's own report). A Vision board ranks the image-capable entrants the same way.
 
-All numbers come from one static bundle, `data/index.json`, written by `evaluation/reproductions/build_leaderboard.py` in the `typesafe-diffusion-lab` checkout from `capability-indices.json`, `entrant-metadata.json`, each run's `benchmark-summary.json` and the Jev release report. Re-run that script and commit the JSON to refresh the page; the HTML never needs to change for a data update.
-
-The edition is read from the data too: `suite.edition` and `suite.label` in `data/index.json` (and `edition.label` / `suite.note` in `data/methodology.json`) drive the page titles, the header, the footers and the suite description; when those fields are absent the pages fall back to Decision Index 0.1 on release-v1. `build_leaderboard.py --suite release-v2` and `build_methodology.py --suite release-v2` write `data/index-v2.json` and `data/methodology-v2.json` (Decision Index 0.2: the same suite minus a stratified subset cut of ToolRet and BRIGHT); copying them over `index.json` and `methodology.json` switches the whole site to that edition. The site now serves 0.2 this way; the 0.1 bundle is archived as `data/index-v0.1.json` and `data/methodology-v0.1.json`, which is also where a `--suite release-v1` build now writes. The static `<title>` and `og:title` strings in the HTML are the no-JavaScript fallback and should be bumped by hand when an edition ships.
-
-the answered-rate explanations come from `evaluation/reproductions/answer_gaps.py`, which mines every run's refusal messages into one line per cause and writes `answer-gaps.json` for the build to pick up.
-
-Editorial choices baked into the build: the six interactive environments in the frozen panel are left out of every model's index (Jev included) because no reproduction has run them. Decision Index 0.1 is a 19-benchmark panel across five areas (ChessBench folded into Knowledge & Reasoning, Language split into Language Understanding and Retrieval & Classification). Decision Index 0.2 averages 40 benchmarks in the same five equal-weight areas (the edition panel `PANEL_V02` in `evaluation/reproductions/edition.py`): every static benchmark except MMLU, ARC-Easy, ARC-Challenge and SimpleBench, which stay on the board outside the index, plus the seven benchmarks added in 0.2; ForecastBench enters against its baseline as clip((0.25 − Brier) / 0.25) × coverage. An entrant goes on the 0.2 board only with results on all 40; iSarcasmEval's headline is track A (English sarcasm F1) with the other tracks listed under each result; 0.1 shows the balanced raw index; from 0.2 the headline is the chance-corrected index (`suite.headline` = `balanced_skill`: each benchmark mapped to (score − chance) / (1 − chance) before averaging, so 0 is random guessing and 100 perfect), with raw accuracy kept on model pages and the breadth variant computed but hidden.
-
-## Regenerating the social image
-
-`og-index.html` is the 1200×630 stage that produces `og.png`, the Space thumbnail (title lockup plus the Decision Index bar chart, read live from `data/index.json`; render at 2× with headless Chrome after regenerating the data). `og-news.html` is the older stage for the News tab and produces `og-news.png`.
+The public suite is 37 benchmarks in five areas, each chance-corrected so 0 is random guessing and 100 is perfect; an unanswered request counts as wrong. The private tests and the new-domain tasks are never published. The **Methodology** page has the full rules, every benchmark's subset and baseline, and the changelog.
 
 ## Contributing
 

data/own.json

@@ -0,0 +1 @@
+{"formula":"v0.3-final","parts":{"public":0.24,"same_skills":0.6,"new_domains":0.36},"areas":[{"id":"knowledge","label":"Knowledge & Reasoning","w":0.258494},{"id":"language","label":"Language Understanding","w":0.258494},{"id":"retrieval","label":"Retrieval & Classification","w":0.200229},{"id":"tools","label":"Tools & Automation","w":0.182783},{"id":"arts","label":"Arts & Human Taste","w":0.1}],"eq":{"mB":23.4973,"sB":16.539081,"mS":24.268626,"sS":16.350451,"mO":23.682619,"sO":12.286109},"models":[{"e":"decision-27b-ckpt-nothink","a":{"knowledge":[45.753833,1,38.77743,0.9],"language":[59.736809,1,52.457021,0.893617],"retrieval":[58.422424,1,60.996061,1],"tools":[83.593333,1,82.44,1],"arts":[48.166667,1,44.606667,0.833333]},"o":52.870125},{"e":"decider-chat-gemma4-31b","a":{"knowledge":[45.456,1,40.790417,0.9],"language":[59.492766,1,49.972553,0.893617],"retrieval":[63.081818,1,63.74303,1],"tools":[75.573704,1,70.741111,1],"arts":[42.176667,1,37.393333,0.833333]},"o":50.3485},{"e":"rune-26b-a4b-v3","a":{"knowledge":[45.954,1,38.28203,0.9],"language":[61.932128,1,52.724255,0.893617],"retrieval":[63.520909,1,62.640606,1],"tools":[71.221852,1,69.272593,1],"arts":[46.503333,1,43.683333,0.833333]},"o":49.381625},{"e":"decision-2.0-vega-27b","a":{"knowledge":[45.957667,1,34.551857,0.9],"language":[59.037234,1,48.552553,0.893617],"retrieval":[60.880303,1,61.410606,1],"tools":[73.535185,1,77.634444,1],"arts":[42.031667,1,38.418333,0.833333]},"o":47.907625},{"e":"kev-27b","a":{"knowledge":[45.6655,1,35.015063,0.9],"language":[62.598936,1,51.679787,0.893617],"retrieval":[55.749394,1,55.99,1],"tools":[72.645926,1,77.698519,1],"arts":[42.665,1,37.981667,0.833333]},"o":55.488125},{"e":"xor-26b-a4b","a":{"knowledge":[41.227333,1,38.28982,0.9],"language":[60.06383,1,52.588511,0.893617],"retrieval":[64.294242,1,60.131818,1],"tools":[70.695185,1,71.032222,1],"arts":[43.23,1,40.665,0.833333]},"o":47.370375},{"e":"jebadiah-27b","a":{"knowledge":[40.783167,1,35.336463,0.9],"language":[59.655532,1,51.957447,0.893617],"retrieval":[53.888788,1,56.935455,1],"tools":[78.127037,1,76.70037,1],"arts":[40.733333,1,37.08,0.833333]},"o":48.363625},{"e":"blink-v0.3-26b-a4b-nvfp4","a":{"knowledge":[45.124333,1,37.18073,0.9],"language":[63.563404,1,53.26766,0.893617],"retrieval":[63.023636,1,63.429394,1],"tools":[70.043333,1,69.613704,1],"arts":[46.975,1,43.956667,0.833333]},"o":50.084375},{"e":"simple-jev-qwen3.8-27b","a":{"knowledge":[36.916667,1,31.11834,0.9],"language":[61.223617,1,50.519574,0.893617],"retrieval":[63.25303,1,63.822727,1],"tools":[76.209259,1,75.920741,1],"arts":[40.815,1,38.206667,0.833333]},"o":38.55775},{"e":"jev-27b","a":{"knowledge":[41.406667,1,35.671767,0.9],"language":[56.444468,1,46.121277,0.893617],"retrieval":[53.741515,1,58.860303,1],"tools":[73.445185,1,74.08963,1],"arts":[41.605,1,39.518333,0.833333]},"o":50.83525},{"e":"eikos-27b-fp8","a":{"knowledge":[39.978833,1,34.365067,0.9],"language":[52.557447,1,49.641915,0.893617],"retrieval":[55.865455,1,59.684242,1],"tools":[74.434444,1,78.642593,1],"arts":[41.416667,1,39.675,0.833333]},"o":48.360625},{"e":"jade","a":{"knowledge":[41.244667,1,33.79223,0.9],"language":[56.456809,1,50.88234,0.893617],"retrieval":[54.398788,1,56.408182,1],"tools":[73.177407,1,73.802222,1],"arts":[38.968333,1,38.125,0.833333]},"o":50.656125},{"e":"jev-gemma4-26b-a4b","a":{"knowledge":[43.736333,1,35.38982,0.9],"language":[61.23,1,50.288511,0.893617],"retrieval":[67.884242,1,63.612424,1],"tools":[69.725926,1,66.581852,1],"arts":[45.558333,1,40.893333,0.833333]},"o":49.01675},{"e":"xor-1.2","a":{"knowledge":[35.471667,1,32.759497,0.9],"language":[56.862553,1,48.528511,0.893617],"retrieval":[54.457273,1,58.236061,1],"tools":[71.190741,1,71.828148,1],"arts":[44.961667,1,40.471667,0.833333]},"o":41.16675},{"e":"reflex-27b-v2","a":{"knowledge":[34.843833,1,30.225513,0.9],"language":[53.587447,1,47.058936,0.893617],"retrieval":[57.806364,1,58.557273,1],"tools":[74.12037,1,76.482963,1],"arts":[42.92,1,38.406667,0.833333]},"o":42.21875},{"e":"decider-chat-qwen3.6-27b","a":{"knowledge":[38.304167,1,34.74049,0.9],"language":[55.938936,1,47.36617,0.893617],"retrieval":[52.231515,1,54.186061,1],"tools":[71.423333,1,70.071111,1],"arts":[36.955,1,36.175,0.833333]},"o":43.57275},{"e":"clef","a":{"knowledge":[52.804333,1,33.227877,0.9],"language":[61.268936,1,43.149787,0.893617],"retrieval":[62.839697,1,60.132121,1],"tools":[80.714815,1,75.689259,1],"arts":[48.885,1,37.703333,0.833333]},"o":41.88375},{"e":"joshua-diffusion-full","a":{"knowledge":[32.463833,1,29.99149,0.9],"language":[54.309362,1,46.786383,0.893617],"retrieval":[58.373939,1,54.592727,1],"tools":[70.164815,1,67.168148,1],"arts":[31.981667,1,34.59,0.833333]},"o":37.232},{"e":"winnow-12b-q8","a":{"knowledge":[34.754667,1,28.724727,0.9],"language":[56.985957,1,45.602979,0.893617],"retrieval":[54.047273,1,55.020909,1],"tools":[71.022222,1,63.23037,1],"arts":[31.376667,1,29.813333,0.833333]},"o":44.15625},{"e":"jevfire-uncapped","a":{"knowledge":[30.253833,1,22.75185,0.9],"language":[53.895319,1,40.383404,0.893617],"retrieval":[56.216061,1,54.123939,1],"tools":[72.332222,1,72.411111,1],"arts":[35.68,1,31.828333,0.833333]},"o":36.984875},{"e":"decision-2.0-lux-9b","a":{"knowledge":[35.457333,1,27.847143,0.9],"language":[46.848511,1,41.640426,0.893617],"retrieval":[55.652424,1,54.722121,1],"tools":[60.36963,1,65.298889,1],"arts":[33.078333,1,31.836667,0.833333]},"o":44.45275},{"e":"nimble-v3","a":{"knowledge":[40.96,1,30.18736,0.9],"language":[60.680426,1,49.157447,0.893617],"retrieval":[55.477273,1,53.296061,1],"tools":[84.244444,1,73.957037,1],"arts":[44.05,1,39.715,0.833333]},"o":49.0475},{"e":"jevk5-9b-v0.3.3","a":{"knowledge":[31.644167,1,29.5824,0.9],"language":[54.675532,1,48.355957,0.893617],"retrieval":[49.984848,1,52.492424,1],"tools":[64.144444,1,61.176667,1],"arts":[35.698333,1,29.76,0.833333]},"o":43.33},{"e":"jpt-9b","a":{"knowledge":[30.921167,1,26.791723,0.9],"language":[57.678936,1,42.007234,0.893617],"retrieval":[44.585455,1,42.861818,1],"tools":[67.024074,1,61.387407,1],"arts":[28.781667,1,28.226667,0.833333]},"o":39.9115},{"e":"jev-9b","a":{"knowledge":[30.717333,1,27.485467,0.9],"language":[44.151915,1,36.902766,0.893617],"retrieval":[46.450606,1,48.927879,1],"tools":[62.058148,1,57.188889,1],"arts":[32.825,1,29.346667,0.833333]},"o":44.40275},{"e":"sieve-9b","a":{"knowledge":[28.9625,1,23.340527,0.9],"language":[45.554468,1,40.954894,0.893617],"retrieval":[46.065152,1,46.348485,1],"tools":[61.297778,1,62.467037,1],"arts":[21.865,1,25.345,0.833333]},"o":43.53675},{"e":"kev-9b-v2","a":{"knowledge":[30.797167,1,26.35679,0.9],"language":[46.678298,1,40.078723,0.893617],"retrieval":[46.794848,1,47.95,1],"tools":[56.858519,1,59.024444,1],"arts":[17.506667,1,18.411667,0.833333]},"o":41.029625},{"e":"jev-omni","a":{"knowledge":[26.091333,1,21.800603,0.9],"language":[51.335319,1,43.463191,0.893617],"retrieval":[35.825152,1,39.932121,1],"tools":[60.770741,1,63.711852,1],"arts":[27.425,1,27.153333,0.833333]},"o":39.241},{"e":"decision-2.0-nox-4b","a":{"knowledge":[31.370167,1,26.178033,0.9],"language":[46.381064,1,38.195106,0.893617],"retrieval":[52.437879,1,52.73303,1],"tools":[60.118519,1,58.799259,1],"arts":[26.248333,1,27.136667,0.833333]},"o":42.185},{"e":"decider-35b-nvfp4","a":{"knowledge":[35.856167,1,29.433963,0.9],"language":[52.746809,1,43.729362,0.893617],"retrieval":[54.664242,1,52.577576,1],"tools":[56.537407,1,48.737407,1],"arts":[35.18,1,36.165,0.833333]},"o":47.567875},{"e":"djev","a":{"knowledge":[25.934833,1,20.819623,0.9],"language":[50.033617,1,43.834255,0.893617],"retrieval":[41.770303,1,45.364545,1],"tools":[51.681111,1,53.354815,1],"arts":[34.406667,1,31.676667,0.833333]},"o":38.588},{"e":"jpt-4b","a":{"knowledge":[26.513833,1,25.101327,0.9],"language":[53.73234,1,37.383191,0.893617],"retrieval":[45.006061,1,40.997273,1],"tools":[57.196296,1,56.548148,1],"arts":[26.111667,1,26.036667,0.833333]},"o":39.171875},{"e":"jet-v6.2","a":{"knowledge":[25.422667,1,24.201243,0.9],"language":[44.205106,1,36.521489,0.893617],"retrieval":[48.172727,1,48.185758,1],"tools":[62.938148,1,54.971852,1],"arts":[30.226667,1,26.728333,0.833333]},"o":34.464625},{"e":"nimble-v2","a":{"knowledge":[30.4765,1,24.614213,0.9],"language":[44.30617,1,40.128936,0.893617],"retrieval":[34.927879,1,36.678182,1],"tools":[59.212222,1,53.404815,1],"arts":[27.56,1,29.028333,0.833333]},"o":39.681125},{"e":"hopper-g-1.2","a":{"knowledge":[24.5905,1,24.471,0.9],"language":[45.66617,1,36.112553,0.893617],"retrieval":[45.417576,1,47.056061,1],"tools":[60.064444,1,62.529259,1],"arts":[29.166667,1,28.888333,0.833333]},"o":39.60925},{"e":"razorback-one-read-3e296f08","a":{"knowledge":[25.694,1,21.338577,0.9],"language":[45.503617,1,41.786383,0.893617],"retrieval":[37.01697,1,38.242727,1],"tools":[47.186667,1,54.378148,1],"arts":[33.068333,1,30.416667,0.833333]},"o":38.2045},{"e":"kev-4b-r10","a":{"knowledge":[26.299833,1,21.644387,0.9],"language":[41.782766,1,32.368298,0.893617],"retrieval":[42.481212,1,42.223939,1],"tools":[55.01037,1,50.574815,1],"arts":[17.9,1,19.81,0.833333]},"o":41.24725},{"e":"lev","a":{"knowledge":[22.783,1,19.46748,0.9],"language":[39.131064,1,26.983404,0.893617],"retrieval":[47.920303,1,42.879697,1],"tools":[57.524074,1,58.358148,1],"arts":[30.815,1,33.346667,0.833333]},"o":31.982875},{"e":"solomon-v11-bf16-encoding","a":{"knowledge":[27.805,1,20.013473,0.9],"language":[56.663404,1,48.815319,0.893617],"retrieval":[25.972727,1,27.62697,1],"tools":[46.968148,1,46.07,1],"arts":[24.466667,1,26.141667,0.833333]},"o":43.567375},{"e":"jevk5-v0.3","a":{"knowledge":[25.9425,1,23.86624,0.9],"language":[46.56617,1,41.834894,0.893617],"retrieval":[44.468485,1,46.14697,1],"tools":[56.937037,1,53.400741,1],"arts":[30.825,1,28.43,0.833333]},"o":37.7305},{"e":"neohorse-jev-4b","a":{"knowledge":[23.606,1,21.624643,0.9],"language":[39.621702,1,31.158298,0.893617],"retrieval":[43.908788,1,45.187576,1],"tools":[58.879259,1,56.124074,1],"arts":[11.025,1,13.665,0.833333]},"o":37.009},{"e":"ryotide-qwen-v0.3.1","a":{"knowledge":[20.954667,1,17.149413,0.9],"language":[35.988511,1,30.24383,0.893617],"retrieval":[47.520606,1,46.359394,1],"tools":[55.084815,1,52.174074,1],"arts":[29.811667,1,30.303333,0.833333]},"o":34.747125},{"e":"intern-decision-4b","a":{"knowledge":[22.962667,1,21.34866,0.9],"language":[42.544468,1,37.694468,0.893617],"retrieval":[40.341515,1,40.111515,1],"tools":[55.656296,1,56.208519,1],"arts":[28.181667,1,27.971667,0.833333]},"o":34.354625},{"e":"winnow-e4b-q8","a":{"knowledge":[22.763833,1,18.77626,0.9],"language":[46.770851,1,34.082979,0.893617],"retrieval":[43.79697,1,41.348788,1],"tools":[62.509259,1,53.952222,1],"arts":[26.933333,1,24.133333,0.833333]},"o":37.1195},{"e":"clef-flash","a":{"knowledge":[52.975833,1,29.854243,0.9],"language":[47.218511,1,36.7,0.893617],"retrieval":[52.155758,1,41.42,1],"tools":[81.867037,1,66.235556,1],"arts":[48.406667,1,33.22,0.833333]},"o":42.504875},{"e":"decider-4b","a":{"knowledge":[27.697167,1,24.380443,0.9],"language":[43.985532,1,30.841702,0.893617],"retrieval":[44.674242,1,42.785152,1],"tools":[58.624444,1,48.222222,1],"arts":[30.041667,1,29.308333,0.833333]},"o":40.4045},{"e":"vllm-pr57250","a":{"knowledge":[24.9885,1,21.585173,0.9],"language":[46.479574,1,43.076383,0.893617],"retrieval":[24.169091,1,23.794545,1],"tools":[33.985185,1,37.2,1],"arts":[31.695,1,28.228333,0.833333]},"o":39.328},{"e":"intelif-qwen3-4b","a":{"knowledge":[19.471833,1,18.969643,0.9],"language":[31.211915,1,33.252553,0.893617],"retrieval":[39.547879,1,37.272121,1],"tools":[50.962963,1,55.355556,1],"arts":[21.34,1,23.235,0.833333]},"o":31.2855},{"e":"jobe","a":{"knowledge":[18.453667,1,15.358783,0.9],"language":[34.852128,1,32.738511,0.893617],"retrieval":[36.690303,1,38.113939,1],"tools":[48.805185,1,48.654074,1],"arts":[28.868333,1,29.761667,0.833333]},"o":30.082625},{"e":"gevva-e4b","a":{"knowledge":[17.912,1,15.786803,0.9],"language":[27.611489,1,25.941277,0.893617],"retrieval":[39.969091,1,33.517879,1],"tools":[56.936296,1,48.931481,1],"arts":[22.616667,1,24.18,0.833333]},"o":34.36775},{"e":"pngwn-space-uncapped-full","a":{"knowledge":[22.7915,1,22.175077,0.9],"language":[34.680426,1,33.514681,0.893617],"retrieval":[29.113939,1,24.742424,1],"tools":[45.73037,1,37.29,1],"arts":[14.451667,1,17.618333,0.833333]},"o":34.28475},{"e":"tev1-4b","a":{"knowledge":[24.345333,1,21.180777,0.9],"language":[39.18383,1,37.525319,0.893617],"retrieval":[15.48,1,19.29697,1],"tools":[41.828519,1,44.730741,1],"arts":[23.476667,1,23.383333,0.833333]},"o":37.645625},{"e":"openvons","a":{"knowledge":[19.491167,1,18.601737,0.9],"language":[30.730426,1,28.96234,0.893617],"retrieval":[25.368788,1,30.71697,1],"tools":[44.64963,1,47.351111,1],"arts":[24.4,1,27.781667,0.833333]},"o":32.9685},{"e":"gevva-e2b","a":{"knowledge":[11.9435,1,13.306903,0.9],"language":[24.980426,1,19.688298,0.893617],"retrieval":[38.604848,1,39.107576,1],"tools":[52.53037,1,45.495926,1],"arts":[16.523333,1,11.583333,0.833333]},"o":28.385625},{"e":"metask-jev-4b","a":{"knowledge":[21.552833,1,19.96154,0.9],"language":[40.05234,1,33.580851,0.893617],"retrieval":[12.912727,1,14.973333,1],"tools":[37.592222,1,38.967037,1],"arts":[21.941667,1,19.123333,0.833333]},"o":35.691625},{"e":"semif","a":{"knowledge":[17.893333,1,14.52659,0.9],"language":[34.019574,1,32.639149,0.893617],"retrieval":[15.719394,1,18.646364,1],"tools":[39.095185,1,40.235926,1],"arts":[27.92,1,28.013333,0.833333]},"o":30.6815},{"e":"jev-style-2b-v3","a":{"knowledge":[17.161167,1,16.8377,0.9],"language":[29.922128,1,27.39383,0.893617],"retrieval":[51.605152,1,49.926667,1],"tools":[52.768148,1,47.672593,1],"arts":[19.015,1,20,0.833333]},"o":25.19225},{"e":"mini-jev","a":{"knowledge":[15.829,1,17.01331,0.9],"language":[29.785532,1,26.649787,0.893617],"retrieval":[8.659394,1,10.905758,1],"tools":[28.415926,1,30.304074,1],"arts":[25.29,1,27.175,0.833333]},"o":30.04125},{"e":"sieve-2b","a":{"knowledge":[10.43,1,12.001327,0.9],"language":[20.784894,1,20.086383,0.893617],"retrieval":[28.46303,1,28.727273,1],"tools":[35.277778,1,32.412222,1],"arts":[19.456667,1,20.291667,0.833333]},"o":17.763},{"e":"decider-2b-fp8-http","a":{"knowledge":[17.1805,1,14.650057,0.9],"language":[32.205106,1,25.103617,0.893617],"retrieval":[37.284848,1,37.31,1],"tools":[42.451481,1,36.294074,1],"arts":[17.568333,1,20.991667,0.833333]},"o":21.0355},{"e":"decision-2.0-sol-2b","a":{"knowledge":[16.219167,1,12.480247,0.9],"language":[24.631915,1,22.13383,0.893617],"retrieval":[45.58,1,41.015455,1],"tools":[48.798889,1,47.117407,1],"arts":[14.411667,1,14.318333,0.833333]},"o":23.04},{"e":"intern-decision-2b","a":{"knowledge":[12.01,1,9.38917,0.9],"language":[21.260851,1,26.346809,0.893617],"retrieval":[15.654848,1,18.547273,1],"tools":[33.334444,1,37.503704,1],"arts":[14.355,1,18.468333,0.833333]},"o":24.732625},{"e":"jpt-0.8b","a":{"knowledge":[8.157,1,6.838413,0.9],"language":[25.167872,1,17.935532,0.893617],"retrieval":[16.402727,1,15.46,1],"tools":[36.234074,1,34.671481,1],"arts":[8.443333,1,7.311667,0.833333]},"o":21.99475},{"e":"this-that-1.2","a":{"knowledge":[16.583,1,13.634277,0.9],"language":[31.538936,1,21.562979,0.893617],"retrieval":[32.825152,1,38.488788,1],"tools":[44.965185,1,38.953333,1],"arts":[14.193333,1,16.583333,0.833333]},"o":19.703875},{"e":"bosun-v3.1-1.7b","a":{"knowledge":[4.748167,1,3.605523,0.9],"language":[13.522553,1,13.905106,0.893617],"retrieval":[42.820303,1,38.534242,1],"tools":[34.778519,1,32.578519,1],"arts":[8.961667,1,7.876667,0.833333]},"o":10.954875},{"e":"dinah-0","a":{"knowledge":[13.069667,1,8.3785,0.9],"language":[22.374468,1,12.403617,0.893617],"retrieval":[47.752424,1,28.631818,1],"tools":[37.846296,1,30.606296,1],"arts":[4.213333,1,7.53,0.833333]},"o":5.47525},{"e":"kodiak-v0.2-1b-accuracy","a":{"knowledge":[13.512167,1,9.262613,0.9],"language":[15.308085,1,10.283404,0.893617],"retrieval":[36.17697,1,30.61303,1],"tools":[15.133333,1,13.666667,1],"arts":[15.891667,1,15.11,0.833333]},"o":6.868375},{"e":"onejev-0.8b","a":{"knowledge":[9.291833,1,8.86929,0.9],"language":[14.983191,1,14.802553,0.893617],"retrieval":[31.251818,1,31.378485,1],"tools":[47.132963,1,37.972593,1],"arts":[10.723333,1,9.191667,0.833333]},"o":19.92175},{"e":"decision-2.0-kai-0.6b","a":{"knowledge":[6.436,1,6.287973,0.9],"language":[11.562766,1,12.478511,0.893617],"retrieval":[31.96697,1,27.498485,1],"tools":[26.726296,1,25.163704,1],"arts":[7.441667,1,7.631667,0.833333]},"o":11.810875},{"e":"decision-2.0-eos-0.8b","a":{"knowledge":[7.0335,1,7.672303,0.9],"language":[15.288298,1,18.298085,0.893617],"retrieval":[38.414848,1,33.725455,1],"tools":[34.024815,1,29.912963,1],"arts":[8.883333,1,8.798333,0.833333]},"o":13.928},{"e":"tev1-0.8b","a":{"knowledge":[6.778167,1,4.88825,0.9],"language":[15.264043,1,13.634468,0.893617],"retrieval":[4.87697,1,4.412727,1],"tools":[31.965185,1,31.256667,1],"arts":[5.868333,1,5.551667,0.833333]},"o":14.73125},{"e":"intern-decision-0.8b","a":{"knowledge":[6.769167,1,6.757023,0.9],"language":[10.871702,1,8.371915,0.893617],"retrieval":[6.440303,1,9.43303,1],"tools":[28.382222,1,29.926667,1],"arts":[7.721667,1,7.583333,0.833333]},"o":14.664625},{"e":"kev-0.8b-r15","a":{"knowledge":[6.967833,1,7.508887,0.9],"language":[14.231064,1,11.865532,0.893617],"retrieval":[22.127273,1,17.580303,1],"tools":[35.522963,1,31.344074,1],"arts":[3.801667,1,4.728333,0.833333]},"o":13.2005},{"e":"von-1.3","a":{"knowledge":[9.277,1,6.419733,0.9],"language":[11.634043,1,13.944468,0.893617],"retrieval":[30.131212,1,21.662727,1],"tools":[11.742222,1,8.52037,1],"arts":[2.705,1,7.481667,0.833333]},"o":4.87225},{"e":"jevk5-lite","a":{"knowledge":[5.360333,1,5.141947,0.9],"language":[9.036383,1,13.121064,0.893617],"retrieval":[28.659394,1,21.158182,1],"tools":[3.095556,1,0.673333,1],"arts":[2.551667,1,5.613333,0.833333]},"o":5.01675},{"e":"gliner-decide","a":{"knowledge":[2.294,1,2.67586,0.9],"language":[8.26383,1,14.603404,0.893617],"retrieval":[25.320606,1,19.121515,1],"tools":[14.777778,1,13.180741,1],"arts":[9.788333,1,5.055,0.833333]},"o":9.640375},{"e":"bosun-v3.1-0.6b","a":{"knowledge":[2.632667,1,1.2791,0.9],"language":[4.31617,1,5.397234,0.893617],"retrieval":[34.152424,1,28.869091,1],"tools":[28.445556,1,27.204444,1],"arts":[6.68,1,6.288333,0.833333]},"o":4.217625},{"e":"jev-style-0.8b-v3","a":{"knowledge":[5.358167,1,3.588227,0.9],"language":[11.891489,1,11.510851,0.893617],"retrieval":[26.213333,1,22.453636,1],"tools":[35.358148,1,24.551111,1],"arts":[8.301667,1,7.403333,0.833333]},"o":10.242875},{"e":"lavoir","a":{"knowledge":[2.774,1,3.215393,0.9],"language":[11.480213,1,9.088936,0.893617],"retrieval":[11.995758,1,17.913636,1],"tools":[13.021481,1,10.724815,1],"arts":[3.873333,1,4.81,0.833333]},"o":1.605625},{"e":"jeff-uncapped","a":{"knowledge":[2.666667,1,1.736483,0.9],"language":[11.636809,1,7.785319,0.893617],"retrieval":[21.310606,1,18.020303,1],"tools":[1.188148,1,0.775556,1],"arts":[0.923333,1,1.653333,0.833333]},"o":13.924875},{"e":"lfm2600","a":{"knowledge":[6.5825,1,6.818717,0.9],"language":[5.905106,1,5.736383,0.893617],"retrieval":[10.534242,1,5.915152,1],"tools":[4.427037,1,9.046296,1],"arts":[6.296667,1,10.286667,0.833333]},"o":9.76575},{"e":"sifr-0.8b-v3.1","a":{"knowledge":[13.223,1,8.08648,0.9],"language":[23.095745,1,13.60617,0.893617],"retrieval":[45.509394,1,26.709394,1],"tools":[35.94963,1,31.967407,1],"arts":[5.77,1,8.015,0.833333]},"o":15.175625},{"e":"laya-1e28ac20","a":{"knowledge":[2.898,1,2.16079,0.9],"language":[10.051702,1,13.788511,0.893617],"retrieval":[7.149394,1,10.157273,1],"tools":[5.82,1,8.888519,1],"arts":[3.466667,1,6.606667,0.833333]},"o":4.053625},{"e":"clm-v0.1-8b","a":{"knowledge":[2.3855,1,1.013043,0.9],"language":[5.681277,1,5.974681,0.893617],"retrieval":[11.259394,1,5.590909,1],"tools":[13.98,1,13.171111,1],"arts":[5.156667,1,2.111667,0.833333]},"o":11.837625},{"e":"harsha","a":{"knowledge":[1.3995,1,2.330343,0.9],"language":[5.197872,1,7.762553,0.893617],"retrieval":[0.941818,1,2.029394,1],"tools":[6.997778,1,8.688148,1],"arts":[8.501667,1,11.713333,0.833333]},"o":7.412},{"e":"akash-gemma","a":{"knowledge":[1.273167,1,2.953,0.9],"language":[1.212766,1,2.414468,0.893617],"retrieval":[17.223636,1,9.730303,1],"tools":[3.308148,1,3.544444,1],"arts":[4.898333,1,6.9,0.833333]},"o":1.7675},{"e":"julia-1","a":{"knowledge":[1.429667,1,4.548397,0.9],"language":[1.391702,1,2.218936,0.893617],"retrieval":[9.911212,1,4.491515,1],"tools":[7.205556,1,7.033704,1],"arts":[2.085,1,2.846667,0.833333]},"o":6.075},{"e":"verdict-8af2496e","a":{"knowledge":[0.322,1,2.818,0.9],"language":[1.639362,1,0.795745,0.893617],"retrieval":[0,1,0.532727,1],"tools":[6.576296,1,6.542593,1],"arts":[1.513333,1,0,0.833333]},"o":1.206625},{"e":"lfm350-sdpa","a":{"knowledge":[0.806833,1,0.595373,0.9],"language":[0.593617,1,2.694255,0.893617],"retrieval":[0.024242,1,0.107576,1],"tools":[3.84,1,4.464444,1],"arts":[3.57,1,4.436667,0.833333]},"o":3.80725},{"e":"pplx-decider-v1.1-27b","a":{"knowledge":[51.959333,1,39.42911,0.9],"language":[67.48,1,54.084255,0.893617],"retrieval":[61.257273,1,66.385152,1],"tools":[78.88037,1,81.449259,1],"arts":[46.965,1,44.668333,0.833333]},"o":55.570125},{"e":"jpt-35b-a3b","a":{"knowledge":[36.4655,1,32.809807,0.9],"language":[61.207447,1,51.498723,0.893617],"retrieval":[55.572727,1,55.502424,1],"tools":[69.212222,1,66.664444,1],"arts":[36.551667,1,34.97,0.833333]},"o":43.301},{"e":"zev-latest","a":{"knowledge":[0.6715,1,0.432387,0.9],"language":[0.178298,1,2.525106,0.893617],"retrieval":[0.92,1,0.391212,1],"tools":[3.006667,1,3.544444,1],"arts":[5.99,1,1.78,0.833333]},"o":3.878},{"e":"exaone-4.0-1.2b-jev-v0.3","a":{"knowledge":[14.995333,1,14.44853,0.9],"language":[26.580851,1,20.535957,0.893617],"retrieval":[40.947576,1,31.847576,1],"tools":[56.595185,1,44.812222,1],"arts":[13.86,1,14.218333,0.833333]},"o":16.348625},{"e":"rsi-jev-v5.0-vl-3b","a":{"knowledge":[24.069167,1,20.33039,0.9],"language":[45.029574,1,38.307234,0.893617],"retrieval":[42.978182,1,48.832424,1],"tools":[52.808148,1,49.981852,1],"arts":[19.861667,1,19.055,0.833333]},"o":35.4045},{"e":"ajev-lora5","a":{"knowledge":[35.881,1,32.435017,0.9],"language":[56.752766,1,48.924255,0.893617],"retrieval":[56.074848,1,55.426061,1],"tools":[68.851111,1,65.315556,1],"arts":[39,1,36.841667,0.833333]},"o":41.5055},{"e":"ezjev-4b-s2","a":{"knowledge":[31.0035,1,26.327733,0.9],"language":[60.490426,1,42.499574,0.893617],"retrieval":[56.318485,1,52.414545,1],"tools":[69.92037,1,62.88963,1],"arts":[31.155,1,31.988333,0.833333]},"o":40.7345},{"e":"bekko-v0-17m","a":{"knowledge":[2.528,1,0.724943,0.9],"language":[2.704255,1,1.495957,0.893617],"retrieval":[21.264848,1,15.355152,1],"tools":[5.908519,1,7.75963,1],"arts":[7.476667,1,4.103333,0.833333]},"o":3.73575},{"e":"bekko-v0-68m","a":{"knowledge":[3.988167,1,1.730647,0.9],"language":[6.365532,1,6.65383,0.893617],"retrieval":[32.188788,1,29.259394,1],"tools":[14.102963,1,12.244074,1],"arts":[13.143333,1,14.751667,0.833333]},"o":6.1315},{"e":"bekko-v0-400m","a":{"knowledge":[11.642667,1,7.931023,0.9],"language":[23.984894,1,18.265319,0.893617],"retrieval":[42.180909,1,38.349697,1],"tools":[18.601481,1,19.061481,1],"arts":[13.903333,1,21.108333,0.833333]},"o":7.1595},{"e":"kas-4b","a":{"knowledge":[32.216333,1,24.25542,0.9],"language":[35.073404,1,34.191702,0.893617],"retrieval":[40.397576,1,41.340606,1],"tools":[48.862593,1,54.215556,1],"arts":[39.398333,1,26.871667,0.833333]},"o":30.962875},{"e":"lev1-3.7b","a":{"knowledge":[23.066,1,18.81837,0.9],"language":[44.333617,1,37.889149,0.893617],"retrieval":[47.109394,1,44.549394,1],"tools":[68.841481,1,64.01037,1],"arts":[32.411667,1,32.73,0.833333]},"o":30.13525},{"e":"decisio-0.4.0-qwen3.6-35b-a3b","a":{"knowledge":[33.417167,1,28.69028,0.9],"language":[50.081489,1,45.316596,0.893617],"retrieval":[53.557273,1,54.412424,1],"tools":[68.431481,1,67.211852,1],"arts":[36.133333,1,33.995,0.833333]},"o":40.142375},{"e":"decisio-0.4.0-gemma-4-12b","a":{"knowledge":[30.7565,1,27.591583,0.9],"language":[52.381915,1,39.775532,0.893617],"retrieval":[57.999697,1,56.876667,1],"tools":[71.61963,1,70.938519,1],"arts":[35.298333,1,32.755,0.833333]},"o":39.7865},{"e":"hopper-12b-trained","a":{"knowledge":[31.696,1,25.701097,0.9],"language":[53.784468,1,42.527021,0.893617],"retrieval":[53.171212,1,53.337273,1],"tools":[66.258519,1,66.774074,1],"arts":[34.165,1,31.601667,0.833333]},"o":41.851375},{"e":"dm-jepa-1.1","a":{"knowledge":[17.273667,1,1.282447,0.9],"language":[21.693617,1,3.245106,0.893617],"retrieval":[33.99697,1,5.271515,1],"tools":[7.893333,1,6.925185,1],"arts":[8.636667,1,3.875,0.833333]},"o":2.841125},{"e":"jevstral-8b","a":{"knowledge":[24.669333,1,20.9897,0.9],"language":[33.635957,1,33.844255,0.893617],"retrieval":[39.244545,1,37.222121,1],"tools":[52.927778,1,57.989259,1],"arts":[14.088333,1,15.408333,0.833333]},"o":34.189125},{"e":"jiwo-0.8b","a":{"knowledge":[10.212167,1,11.374667,0.9],"language":[34.977021,1,30.202766,0.893617],"retrieval":[44.480909,1,43.037879,1],"tools":[38.01037,1,31.247407,1],"arts":[11.843333,1,12.471667,0.833333]},"o":18.07},{"e":"jiwo-4b","a":{"knowledge":[28.2335,1,26.87736,0.9],"language":[49.635319,1,42.985532,0.893617],"retrieval":[51.571818,1,51.694545,1],"tools":[67.612222,1,62.786667,1],"arts":[29.448333,1,29.42,0.833333]},"o":33.321625},{"e":"jev","a":{"knowledge":[53.865667,1,39.272577,0.9],"language":[59.225106,1,52.046809,0.893617],"retrieval":[55.424848,1,64.103333,1],"tools":[75.087778,1,73.37037,1],"arts":[39.078333,1,41.22,0.833333]},"o":54.982875},{"e":"torchcast-decision-27b","a":{"knowledge":[46.656325,1,35.566497,0.9],"language":[72.039362,1,51.514894,0.893617],"retrieval":[67.542424,1,63.542727,1],"tools":[84.581481,1,79.001111,1],"arts":[54.358333,1,43.876667,0.833333]},"o":50.826625},{"e":"deck31b","a":{"knowledge":[46.66525,1,40.76595,0.9],"language":[60.018085,1,51.655745,0.893617],"retrieval":[62.759394,1,65.048788,1],"tools":[75.57963,1,75.741481,1],"arts":[42.285,1,37.393333,0.833333]},"o":51.484},{"e":"quyet-1.0-large","a":{"knowledge":[49.696975,1,42.766543,0.9],"language":[66.961702,1,52.252128,0.893617],"retrieval":[65.836061,1,63.52697,1],"tools":[72.552593,1,74.600741,1],"arts":[41.866667,1,37.981667,0.833333]},"o":49.119}]}

data/v03.json

@@ -0,0 +1 @@
+{"generated_utc":"2026-10-06T16:32:56Z","edition":"0.3","label":"Decision Index 0.3","weights":{"public":0.2,"same_skills":0.5,"new_domains":0.3},"public_index_note":"GSM8K rebuilt, ForecastBench retired","vision":{"entrants":21},"models":[{"engine":"decision-27b-ckpt-nothink","pub":59.06,"same_skills":59.49,"new_domains":52.87,"v":60.21,"lab":false,"jev":false,"rank_v":2,"rank_pub":5,"name":"Fastino GLiDE no-thinking (28B)","eq":{"S":59.13,"O":62.79},"rank_tie":2,"tied":true,"parts":{"public":11.81,"same_skills":29.56,"new_domains":18.84}},{"engine":"jev","pub":57.96,"same_skills":58.04,"new_domains":54.98,"v":60.11,"lab":false,"jev":true,"rank_v":3,"rank_pub":10,"name":"Jev","eq":{"S":57.66,"O":65.63},"rank_tie":2,"tied":true,"parts":{"public":11.59,"same_skills":28.83,"new_domains":19.69}},{"engine":"decider-chat-gemma4-31b","pub":57.79,"same_skills":56.88,"new_domains":50.35,"v":57.62,"lab":false,"jev":false,"rank_v":9,"rank_pub":11,"name":"Decider chat · Gemma-4-31B","eq":{"S":56.48,"O":59.39},"rank_tie":8,"tied":true,"parts":{"public":11.56,"same_skills":28.24,"new_domains":17.82}},{"engine":"rune-26b-a4b-v3","pub":58.28,"same_skills":57.09,"new_domains":49.38,"v":57.43,"lab":false,"jev":false,"rank_v":10,"rank_pub":6,"name":"Surogate Rune 26B-A4B v3","eq":{"S":56.7,"O":58.09},"rank_tie":8,"tied":true,"parts":{"public":11.66,"same_skills":28.35,"new_domains":17.43}},{"engine":"decision-2.0-vega-27b","pub":56.97,"same_skills":55.71,"new_domains":47.91,"v":55.88,"lab":false,"jev":false,"rank_v":13,"rank_pub":13,"name":"vLLM-SR Decision 2.0 Vega 27B","eq":{"S":55.3,"O":56.11},"rank_tie":11,"tied":true,"parts":{"public":11.39,"same_skills":27.65,"new_domains":16.83}},{"engine":"kev-27b","pub":56.69,"same_skills":55.51,"new_domains":55.49,"v":58.78,"lab":false,"jev":false,"rank_v":6,"rank_pub":14,"name":"Kev 27B","eq":{"S":55.1,"O":66.31},"rank_tie":5,"tied":true,"parts":{"public":11.34,"same_skills":27.55,"new_domains":19.89}},{"engine":"xor-26b-a4b","pub":56.3,"same_skills":56.54,"new_domains":47.37,"v":55.95,"lab":false,"jev":false,"rank_v":12,"rank_pub":15,"name":"Xor 26B-A4B","eq":{"S":56.14,"O":55.38},"rank_tie":11,"tied":true,"parts":{"public":11.26,"same_skills":28.07,"new_domains":16.62}},{"engine":"jebadiah-27b","pub":55.11,"same_skills":55.58,"new_domains":48.36,"v":55.63,"lab":false,"jev":false,"rank_v":15,"rank_pub":18,"name":"frontier-infra Jebadiah 27B","eq":{"S":55.17,"O":56.72},"rank_tie":11,"tied":true,"parts":{"public":11.02,"same_skills":27.59,"new_domains":17.02}},{"engine":"blink-v0.3-26b-a4b-nvfp4","pub":58.21,"same_skills":57.21,"new_domains":50.08,"v":57.76,"lab":false,"jev":false,"rank_v":8,"rank_pub":7,"name":"Blink v0.3 26B-A4B (NVFP4)","eq":{"S":56.81,"O":59.04},"rank_tie":8,"tied":true,"parts":{"public":11.64,"same_skills":28.41,"new_domains":17.71}},{"engine":"simple-jev-qwen3.8-27b","pub":56.05,"same_skills":55.46,"new_domains":38.56,"v":51.79,"lab":false,"jev":false,"rank_v":20,"rank_pub":17,"name":"simple-jev · Qwen3.8-27B (featherless)","eq":{"S":55.05,"O":43.52},"rank_tie":20,"tied":true,"parts":{"public":11.21,"same_skills":27.53,"new_domains":13.06}},{"engine":"jev-27b","pub":53.64,"same_skills":54.22,"new_domains":50.84,"v":55.64,"lab":false,"jev":false,"rank_v":14,"rank_pub":19,"name":"JEV-27B","eq":{"S":53.79,"O":60.05},"rank_tie":11,"tied":true,"parts":{"public":10.73,"same_skills":26.9,"new_domains":18.01}},{"engine":"eikos-27b-fp8","pub":52.85,"same_skills":55.92,"new_domains":48.36,"v":55.34,"lab":false,"jev":false,"rank_v":17,"rank_pub":21,"name":"Eikos-27B-FP8 Qwen3.8-27B LoRA","eq":{"S":55.52,"O":56.72},"rank_tie":11,"tied":true,"parts":{"public":10.57,"same_skills":27.76,"new_domains":17.02}},{"engine":"jade","pub":53.42,"same_skills":54.29,"new_domains":50.66,"v":55.56,"lab":false,"jev":false,"rank_v":16,"rank_pub":20,"name":"JADE","eq":{"S":53.86,"O":59.81},"rank_tie":11,"tied":true,"parts":{"public":10.68,"same_skills":26.93,"new_domains":17.94}},{"engine":"jev-gemma4-26b-a4b","pub":58.03,"same_skills":54.99,"new_domains":49.02,"v":56.17,"lab":false,"jev":false,"rank_v":11,"rank_pub":9,"name":"GEV-26B-Decide","eq":{"S":54.58,"O":57.6},"rank_tie":11,"tied":true,"parts":{"public":11.61,"same_skills":27.29,"new_domains":17.28}},{"engine":"xor-1.2","pub":52.28,"same_skills":53.6,"new_domains":41.17,"v":51.15,"lab":false,"jev":false,"rank_v":24,"rank_pub":23,"name":"Xor 1.2","eq":{"S":53.17,"O":47.03},"rank_tie":20,"tied":true,"parts":{"public":10.46,"same_skills":26.58,"new_domains":14.11}},{"engine":"reflex-27b-v2","pub":52.27,"same_skills":53.25,"new_domains":42.22,"v":51.4,"lab":false,"jev":false,"rank_v":22,"rank_pub":24,"name":"reflex Qwen3.8-27B-FP8 (wide choice)","eq":{"S":52.81,"O":48.45},"rank_tie":20,"tied":true,"parts":{"public":10.45,"same_skills":26.41,"new_domains":14.53}},{"engine":"decider-chat-qwen3.6-27b","pub":51.57,"same_skills":52.15,"new_domains":43.57,"v":51.25,"lab":false,"jev":false,"rank_v":23,"rank_pub":26,"name":"Decider chat · Qwen3.6-27B","eq":{"S":51.7,"O":50.27},"rank_tie":20,"tied":true,"parts":{"public":10.31,"same_skills":25.85,"new_domains":15.08}},{"engine":"clef","pub":61.71,"same_skills":53.11,"new_domains":41.88,"v":53.08,"lab":false,"jev":false,"rank_v":19,"rank_pub":3,"name":"Cloudflare clef","eq":{"S":52.67,"O":48},"rank_tie":19,"tied":false,"parts":{"public":12.34,"same_skills":26.33,"new_domains":14.4}},{"engine":"joshua-diffusion-full","pub":50.14,"same_skills":50.02,"new_domains":37.23,"v":47.32,"lab":false,"jev":false,"rank_v":33,"rank_pub":29,"name":"JoshuaSP diffgemma","eq":{"S":49.54,"O":41.74},"rank_tie":31,"tied":true,"parts":{"public":10.03,"same_skills":24.77,"new_domains":12.52}},{"engine":"winnow-12b-q8","pub":50.66,"same_skills":48.14,"new_domains":44.16,"v":49.27,"lab":false,"jev":false,"rank_v":26,"rank_pub":28,"name":"Winnow-12B","eq":{"S":47.64,"O":51.06},"rank_tie":25,"tied":true,"parts":{"public":10.13,"same_skills":23.82,"new_domains":15.32}},{"engine":"jevfire-uncapped","pub":49.8,"same_skills":46.86,"new_domains":36.98,"v":45.55,"lab":false,"jev":false,"rank_v":36,"rank_pub":30,"name":"Jevfire Qwen3.8-27B","eq":{"S":46.35,"O":41.4},"rank_tie":36,"tied":true,"parts":{"public":9.96,"same_skills":23.17,"new_domains":12.42}},{"engine":"decision-2.0-lux-9b","pub":46.76,"same_skills":47.35,"new_domains":44.45,"v":48.21,"lab":false,"jev":false,"rank_v":29,"rank_pub":37,"name":"vLLM-SR Decision 2.0 Lux 9B","eq":{"S":46.85,"O":51.46},"rank_tie":27,"tied":true,"parts":{"public":9.35,"same_skills":23.42,"new_domains":15.44}},{"engine":"nimble-v3","pub":57.19,"same_skills":52.34,"new_domains":49.05,"v":54.67,"lab":false,"jev":false,"rank_v":18,"rank_pub":12,"name":"Bespoke Nimble 9B v3","eq":{"S":51.89,"O":57.64},"rank_tie":18,"tied":false,"parts":{"public":11.44,"same_skills":25.94,"new_domains":17.29}},{"engine":"jevk5-9b-v0.3.3","pub":47.62,"same_skills":48.19,"new_domains":43.33,"v":48.35,"lab":false,"jev":false,"rank_v":28,"rank_pub":35,"name":"JevK5 9B v0.3.3","eq":{"S":47.69,"O":49.95},"rank_tie":27,"tied":true,"parts":{"public":9.52,"same_skills":23.85,"new_domains":14.98}},{"engine":"jpt-9b","pub":46.96,"same_skills":43.45,"new_domains":39.91,"v":44.45,"lab":false,"jev":false,"rank_v":40,"rank_pub":36,"name":"JPT-9B Qwen3.5-9B LoRA","eq":{"S":42.9,"O":45.34},"rank_tie":39,"tied":true,"parts":{"public":9.39,"same_skills":21.45,"new_domains":13.6}},{"engine":"jev-9b","pub":43.28,"same_skills":42.83,"new_domains":44.4,"v":45.21,"lab":false,"jev":false,"rank_v":37,"rank_pub":40,"name":"JEV-9B","eq":{"S":42.27,"O":51.39},"rank_tie":36,"tied":true,"parts":{"public":8.66,"same_skills":21.13,"new_domains":15.42}},{"engine":"sieve-9b","pub":41.88,"same_skills":42.85,"new_domains":43.54,"v":44.59,"lab":false,"jev":false,"rank_v":39,"rank_pub":44,"name":"Sieve 9B","eq":{"S":42.3,"O":50.22},"rank_tie":39,"tied":true,"parts":{"public":8.38,"same_skills":21.15,"new_domains":15.07}},{"engine":"kev-9b-v2","pub":41.54,"same_skills":42.37,"new_domains":41.03,"v":43.27,"lab":false,"jev":false,"rank_v":41,"rank_pub":45,"name":"Kev 9B v2","eq":{"S":41.81,"O":46.85},"rank_tie":41,"tied":true,"parts":{"public":8.31,"same_skills":20.9,"new_domains":14.05}},{"engine":"jev-omni","pub":41.04,"same_skills":42.18,"new_domains":39.24,"v":42.35,"lab":false,"jev":false,"rank_v":44,"rank_pub":49,"name":"Jev-Omni","eq":{"S":41.62,"O":44.44},"rank_tie":44,"tied":true,"parts":{"public":8.21,"same_skills":20.81,"new_domains":13.33}},{"engine":"decision-2.0-nox-4b","pub":44.21,"same_skills":43.72,"new_domains":42.19,"v":44.95,"lab":false,"jev":false,"rank_v":38,"rank_pub":39,"name":"vLLM-SR Decision 2.0 Nox 4B","eq":{"S":43.17,"O":48.4},"rank_tie":36,"tied":true,"parts":{"public":8.84,"same_skills":21.59,"new_domains":14.52}},{"engine":"decider-35b-nvfp4","pub":47.7,"same_skills":45.12,"new_domains":47.57,"v":48.53,"lab":false,"jev":false,"rank_v":27,"rank_pub":34,"name":"Decider 35B-A3B","eq":{"S":44.59,"O":55.65},"rank_tie":27,"tied":true,"parts":{"public":9.54,"same_skills":22.3,"new_domains":16.7}},{"engine":"djev","pub":40.89,"same_skills":41.63,"new_domains":38.59,"v":41.78,"lab":false,"jev":false,"rank_v":45,"rank_pub":50,"name":"djev diffgemma","eq":{"S":41.06,"O":43.56},"rank_tie":44,"tied":true,"parts":{"public":8.18,"same_skills":20.53,"new_domains":13.07}},{"engine":"jpt-4b","pub":42.82,"same_skills":40.11,"new_domains":39.17,"v":41.63,"lab":false,"jev":false,"rank_v":46,"rank_pub":41,"name":"JPT-4B Qwen3.5-4B LoRA","eq":{"S":39.52,"O":44.35},"rank_tie":44,"tied":true,"parts":{"public":8.56,"same_skills":19.76,"new_domains":13.3}},{"engine":"jet-v6.2","pub":42.17,"same_skills":40.93,"new_domains":34.46,"v":40.01,"lab":false,"jev":false,"rank_v":52,"rank_pub":43,"name":"Jet v6.2 Qwen3.5-4B LoRA","eq":{"S":40.35,"O":38.01},"rank_tie":51,"tied":true,"parts":{"public":8.43,"same_skills":20.18,"new_domains":11.4}},{"engine":"nimble-v2","pub":39.9,"same_skills":39.51,"new_domains":39.68,"v":40.95,"lab":false,"jev":false,"rank_v":48,"rank_pub":52,"name":"Bespoke Nimble 9B v2","eq":{"S":38.91,"O":45.03},"rank_tie":48,"tied":true,"parts":{"public":7.98,"same_skills":19.46,"new_domains":13.51}},{"engine":"hopper-g-1.2","pub":41.15,"same_skills":42.37,"new_domains":39.61,"v":42.61,"lab":false,"jev":false,"rank_v":43,"rank_pub":47,"name":"Hopper (G) 1.2 Qwen3.5-4B LoRA","eq":{"S":41.8,"O":44.94},"rank_tie":41,"tied":true,"parts":{"public":8.23,"same_skills":20.9,"new_domains":13.48}},{"engine":"razorback-one-read-3e296f08","pub":37.75,"same_skills":39.74,"new_domains":38.2,"v":40.04,"lab":false,"jev":false,"rank_v":51,"rank_pub":59,"name":"razorback16 diffgemma","eq":{"S":39.15,"O":43.05},"rank_tie":51,"tied":true,"parts":{"public":7.55,"same_skills":19.57,"new_domains":12.91}},{"engine":"kev-4b-r10","pub":37.95,"same_skills":36.17,"new_domains":41.25,"v":39.5,"lab":false,"jev":false,"rank_v":53,"rank_pub":58,"name":"Kev 4B r10","eq":{"S":35.54,"O":47.14},"rank_tie":51,"tied":true,"parts":{"public":7.59,"same_skills":17.77,"new_domains":14.14}},{"engine":"lev","pub":39.2,"same_skills":37.2,"new_domains":31.98,"v":36.53,"lab":false,"jev":false,"rank_v":61,"rank_pub":53,"name":"Interfaze lev Qwen3.5-4B LoRA","eq":{"S":36.58,"O":34.67},"rank_tie":59,"tied":true,"parts":{"public":7.84,"same_skills":18.29,"new_domains":10.4}},{"engine":"solomon-v11-bf16-encoding","pub":38.07,"same_skills":36.95,"new_domains":43.57,"v":40.85,"lab":false,"jev":false,"rank_v":49,"rank_pub":56,"name":"Solomon v1.1","eq":{"S":36.32,"O":50.27},"rank_tie":48,"tied":true,"parts":{"public":7.61,"same_skills":18.16,"new_domains":15.08}},{"engine":"jevk5-v0.3","pub":41.14,"same_skills":41.75,"new_domains":37.73,"v":41.54,"lab":false,"jev":false,"rank_v":47,"rank_pub":48,"name":"JevK5 v0.3","eq":{"S":41.18,"O":42.41},"rank_tie":44,"tied":true,"parts":{"public":8.23,"same_skills":20.59,"new_domains":12.72}},{"engine":"neohorse-jev-4b","pub":37,"same_skills":36.9,"new_domains":37.01,"v":37.97,"lab":false,"jev":false,"rank_v":58,"rank_pub":61,"name":"TokenRhythm NeoHorse-Jev-4B","eq":{"S":36.27,"O":41.44},"rank_tie":54,"tied":true,"parts":{"public":7.4,"same_skills":18.14,"new_domains":12.43}},{"engine":"ryotide-qwen-v0.3.1","pub":37.28,"same_skills":36.67,"new_domains":34.75,"v":36.99,"lab":false,"jev":false,"rank_v":59,"rank_pub":60,"name":"Ryotide Qwen v0.3.1","eq":{"S":36.04,"O":38.39},"rank_tie":59,"tied":true,"parts":{"public":7.46,"same_skills":18.02,"new_domains":11.52}},{"engine":"intern-decision-4b","pub":38,"same_skills":39.1,"new_domains":34.35,"v":38.21,"lab":false,"jev":false,"rank_v":57,"rank_pub":57,"name":"InternLM Intern-Decision Qwen3.5-4B","eq":{"S":38.5,"O":37.86},"rank_tie":54,"tied":true,"parts":{"public":7.6,"same_skills":19.25,"new_domains":11.36}},{"engine":"winnow-e4b-q8","pub":40.86,"same_skills":36.79,"new_domains":37.12,"v":38.73,"lab":false,"jev":false,"rank_v":54,"rank_pub":51,"name":"Winnow-E4B","eq":{"S":36.17,"O":41.59},"rank_tie":54,"tied":true,"parts":{"public":8.17,"same_skills":18.08,"new_domains":12.48}},{"engine":"clef-flash","pub":56.15,"same_skills":44.01,"new_domains":42.5,"v":47.61,"lab":false,"jev":false,"rank_v":31,"rank_pub":16,"name":"Cloudflare clef-flash","eq":{"S":43.46,"O":48.84},"rank_tie":31,"tied":true,"parts":{"public":11.23,"same_skills":21.73,"new_domains":14.65}},{"engine":"decider-4b","pub":41.19,"same_skills":37.19,"new_domains":40.4,"v":40.33,"lab":false,"jev":false,"rank_v":50,"rank_pub":46,"name":"Decider 4B","eq":{"S":36.57,"O":46.01},"rank_tie":48,"tied":true,"parts":{"public":8.24,"same_skills":18.28,"new_domains":13.8}},{"engine":"vllm-pr57250","pub":32.69,"same_skills":33.44,"new_domains":39.33,"v":36.3,"lab":false,"jev":false,"rank_v":62,"rank_pub":65,"name":"mmastrac diffgemma","eq":{"S":32.78,"O":44.56},"rank_tie":59,"tied":true,"parts":{"public":6.54,"same_skills":16.39,"new_domains":13.37}},{"engine":"intelif-qwen3-4b","pub":32.47,"same_skills":35.92,"new_domains":31.29,"v":34.25,"lab":false,"jev":false,"rank_v":64,"rank_pub":66,"name":"Intelif Qwen3-4B","eq":{"S":35.28,"O":33.73},"rank_tie":64,"tied":true,"parts":{"public":6.49,"same_skills":17.64,"new_domains":10.12}},{"engine":"jobe","pub":32.93,"same_skills":34.34,"new_domains":30.08,"v":33.06,"lab":false,"jev":false,"rank_v":66,"rank_pub":64,"name":"Jobe Qwen3.5-4B","eq":{"S":33.68,"O":32.11},"rank_tie":66,"tied":true,"parts":{"public":6.59,"same_skills":16.84,"new_domains":9.63}},{"engine":"gevva-e4b","pub":32.44,"same_skills":31.03,"new_domains":34.37,"v":33.02,"lab":false,"jev":false,"rank_v":67,"rank_pub":67,"name":"Gevva E4B","eq":{"S":30.34,"O":37.88},"rank_tie":66,"tied":true,"parts":{"public":6.49,"same_skills":15.17,"new_domains":11.36}},{"engine":"pngwn-space-uncapped-full","pub":30.49,"same_skills":30.03,"new_domains":34.28,"v":32.09,"lab":false,"jev":false,"rank_v":69,"rank_pub":69,"name":"open-jev (pngwn)","eq":{"S":29.33,"O":37.77},"rank_tie":69,"tied":true,"parts":{"public":6.1,"same_skills":14.66,"new_domains":11.33}},{"engine":"tev1-4b","pub":29.51,"same_skills":31.78,"new_domains":37.65,"v":34.14,"lab":false,"jev":false,"rank_v":65,"rank_pub":72,"name":"Together Tev1 Qwen3.5-4B","eq":{"S":31.09,"O":42.29},"rank_tie":64,"tied":true,"parts":{"public":5.9,"same_skills":15.55,"new_domains":12.69}},{"engine":"openvons","pub":28.66,"same_skills":32.13,"new_domains":32.97,"v":32.26,"lab":false,"jev":false,"rank_v":68,"rank_pub":74,"name":"openvons Qwen3-4B","eq":{"S":31.45,"O":36},"rank_tie":66,"tied":true,"parts":{"public":5.73,"same_skills":15.72,"new_domains":10.8}},{"engine":"gevva-e2b","pub":28.53,"same_skills":27.78,"new_domains":28.39,"v":28.18,"lab":false,"jev":false,"rank_v":73,"rank_pub":76,"name":"Gevva E2B","eq":{"S":27.05,"O":29.83},"rank_tie":72,"tied":true,"parts":{"public":5.71,"same_skills":13.52,"new_domains":8.95}},{"engine":"metask-jev-4b","pub":27.58,"same_skills":27.82,"new_domains":35.69,"v":30.96,"lab":false,"jev":false,"rank_v":71,"rank_pub":77,"name":"Metask-Jev-4B","eq":{"S":27.09,"O":39.66},"rank_tie":71,"tied":false,"parts":{"public":5.52,"same_skills":13.55,"new_domains":11.9}},{"engine":"semif","pub":26.5,"same_skills":28.04,"new_domains":30.68,"v":28.84,"lab":false,"jev":false,"rank_v":72,"rank_pub":78,"name":"SemIf Qwen3.5-4B Base","eq":{"S":27.32,"O":32.92},"rank_tie":72,"tied":true,"parts":{"public":5.3,"same_skills":13.66,"new_domains":9.88}},{"engine":"jev-style-2b-v3","pub":34.05,"same_skills":34.56,"new_domains":25.19,"v":31.42,"lab":false,"jev":false,"rank_v":70,"rank_pub":62,"name":"Jev-Style 2B v3","eq":{"S":33.91,"O":25.53},"rank_tie":69,"tied":true,"parts":{"public":6.81,"same_skills":16.96,"new_domains":7.66}},{"engine":"mini-jev","pub":21.25,"same_skills":23.36,"new_domains":30.04,"v":25.16,"lab":false,"jev":false,"rank_v":76,"rank_pub":84,"name":"mini-jev Qwen3-4B","eq":{"S":22.58,"O":32.06},"rank_tie":75,"tied":true,"parts":{"public":4.25,"same_skills":11.29,"new_domains":9.62}},{"engine":"sieve-2b","pub":22.16,"same_skills":23.66,"new_domains":17.76,"v":20.53,"lab":false,"jev":false,"rank_v":81,"rank_pub":83,"name":"Sieve 2B","eq":{"S":22.88,"O":15.53},"rank_tie":81,"tied":true,"parts":{"public":4.43,"same_skills":11.44,"new_domains":4.66}},{"engine":"decider-2b-fp8-http","pub":29.75,"same_skills":28.47,"new_domains":21.04,"v":25.8,"lab":false,"jev":false,"rank_v":75,"rank_pub":71,"name":"Decider 2B","eq":{"S":27.75,"O":19.93},"rank_tie":75,"tied":true,"parts":{"public":5.95,"same_skills":13.88,"new_domains":5.98}},{"engine":"decision-2.0-sol-2b","pub":30.05,"same_skills":29.25,"new_domains":23.04,"v":27.07,"lab":false,"jev":false,"rank_v":74,"rank_pub":70,"name":"vLLM-SR Decision 2.0 Sol 2B","eq":{"S":28.54,"O":22.63},"rank_tie":74,"tied":false,"parts":{"public":6.01,"same_skills":14.27,"new_domains":6.79}},{"engine":"intern-decision-2b","pub":19.26,"same_skills":23.28,"new_domains":24.73,"v":22.58,"lab":false,"jev":false,"rank_v":80,"rank_pub":88,"name":"InternLM Intern-Decision Qwen3.5-2B","eq":{"S":22.5,"O":24.91},"rank_tie":79,"tied":true,"parts":{"public":3.85,"same_skills":11.25,"new_domains":7.47}},{"engine":"jpt-0.8b","pub":19.37,"same_skills":17.82,"new_domains":21.99,"v":18.73,"lab":false,"jev":false,"rank_v":83,"rank_pub":87,"name":"JPT-0.8B Qwen3.5-0.8B LoRA","eq":{"S":16.97,"O":21.23},"rank_tie":83,"tied":false,"parts":{"public":3.87,"same_skills":8.48,"new_domains":6.37}},{"engine":"this-that-1.2","pub":28.65,"same_skills":27.51,"new_domains":19.7,"v":24.56,"lab":false,"jev":false,"rank_v":77,"rank_pub":75,"name":"FLock this-that 1.2","eq":{"S":26.78,"O":18.14},"rank_tie":77,"tied":true,"parts":{"public":5.73,"same_skills":13.39,"new_domains":5.44}},{"engine":"bosun-v3.1-1.7b","pub":20.55,"same_skills":20.41,"new_domains":10.95,"v":15.82,"lab":false,"jev":false,"rank_v":86,"rank_pub":86,"name":"Bosun v3.1 1.7B LoRA","eq":{"S":19.6,"O":6.36},"rank_tie":86,"tied":true,"parts":{"public":4.11,"same_skills":9.8,"new_domains":1.91}},{"engine":"dinah-0","pub":26.06,"same_skills":18.77,"new_domains":5.48,"v":13.87,"lab":false,"jev":false,"rank_v":88,"rank_pub":79,"name":"Dinah-0","eq":{"S":17.93,"O":0},"rank_tie":88,"tied":true,"parts":{"public":5.21,"same_skills":8.97,"new_domains":0}},{"engine":"kodiak-v0.2-1b-accuracy","pub":19.05,"same_skills":16.33,"new_domains":6.87,"v":11.8,"lab":false,"jev":false,"rank_v":93,"rank_pub":90,"name":"Kodiak v0.2 1B (accuracy)","eq":{"S":15.47,"O":0.86},"rank_tie":91,"tied":true,"parts":{"public":3.81,"same_skills":7.74,"new_domains":0.26}},{"engine":"onejev-0.8b","pub":22.22,"same_skills":21.79,"new_domains":19.92,"v":20.47,"lab":false,"jev":false,"rank_v":82,"rank_pub":82,"name":"OneJev 0.8B","eq":{"S":20.99,"O":18.43},"rank_tie":81,"tied":true,"parts":{"public":4.44,"same_skills":10.49,"new_domains":5.53}},{"engine":"decision-2.0-kai-0.6b","pub":16.68,"same_skills":16.9,"new_domains":11.81,"v":13.61,"lab":false,"jev":false,"rank_v":90,"rank_pub":93,"name":"vLLM-SR Decision 2.0 Kai 0.6B","eq":{"S":16.05,"O":7.52},"rank_tie":88,"tied":true,"parts":{"public":3.34,"same_skills":8.02,"new_domains":2.25}},{"engine":"decision-2.0-eos-0.8b","pub":20.57,"same_skills":21.31,"new_domains":13.93,"v":17.47,"lab":false,"jev":false,"rank_v":85,"rank_pub":85,"name":"vLLM-SR Decision 2.0 Eos 0.8B","eq":{"S":20.5,"O":10.37},"rank_tie":84,"tied":true,"parts":{"public":4.11,"same_skills":10.25,"new_domains":3.11}},{"engine":"tev1-0.8b","pub":13.1,"same_skills":12.84,"new_domains":14.73,"v":12.02,"lab":false,"jev":false,"rank_v":91,"rank_pub":96,"name":"Together Tev1 Qwen3.5-0.8B","eq":{"S":11.94,"O":11.45},"rank_tie":91,"tied":true,"parts":{"public":2.62,"same_skills":5.97,"new_domains":3.43}},{"engine":"intern-decision-0.8b","pub":11.81,"same_skills":12.93,"new_domains":14.66,"v":11.78,"lab":false,"jev":false,"rank_v":94,"rank_pub":98,"name":"InternLM Intern-Decision Qwen3.5-0.8B","eq":{"S":12.03,"O":11.36},"rank_tie":91,"tied":true,"parts":{"public":2.36,"same_skills":6.02,"new_domains":3.41}},{"engine":"kev-0.8b-r15","pub":16.78,"same_skills":15.84,"new_domains":13.2,"v":13.66,"lab":false,"jev":false,"rank_v":89,"rank_pub":92,"name":"Kev 0.8B r15","eq":{"S":14.97,"O":9.39},"rank_tie":88,"tied":true,"parts":{"public":3.36,"same_skills":7.49,"new_domains":2.82}},{"engine":"von-1.3","pub":13.86,"same_skills":12.8,"new_domains":4.87,"v":8.17,"lab":false,"jev":false,"rank_v":98,"rank_pub":95,"name":"Von 1.3","eq":{"S":11.9,"O":0},"rank_tie":98,"tied":true,"parts":{"public":2.77,"same_skills":5.95,"new_domains":0}},{"engine":"jevk5-lite","pub":10.28,"same_skills":10.37,"new_domains":5.02,"v":6.29,"lab":false,"jev":false,"rank_v":100,"rank_pub":100,"name":"JevK5 Lite","eq":{"S":9.44,"O":0},"rank_tie":100,"tied":true,"parts":{"public":2.06,"same_skills":4.72,"new_domains":0}},{"engine":"gliner-decide","pub":11.48,"same_skills":12.05,"new_domains":9.64,"v":9.24,"lab":false,"jev":false,"rank_v":95,"rank_pub":99,"name":"GLiNER2.5-Decide","eq":{"S":11.14,"O":4.59},"rank_tie":95,"tied":true,"parts":{"public":2.3,"same_skills":5.57,"new_domains":1.38}},{"engine":"bosun-v3.1-0.6b","pub":14.5,"same_skills":14.09,"new_domains":4.22,"v":8.69,"lab":false,"jev":false,"rank_v":96,"rank_pub":94,"name":"Bosun v3.1 0.6B LoRA","eq":{"S":13.21,"O":0},"rank_tie":95,"tied":true,"parts":{"public":2.9,"same_skills":6.6,"new_domains":0}},{"engine":"jev-style-0.8b-v3","pub":17,"same_skills":14.65,"new_domains":10.24,"v":11.91,"lab":false,"jev":false,"rank_v":92,"rank_pub":91,"name":"Jev-Style 0.8B v3","eq":{"S":13.77,"O":5.41},"rank_tie":91,"tied":true,"parts":{"public":3.4,"same_skills":6.89,"new_domains":1.62}},{"engine":"lavoir","pub":8.85,"same_skills":9.9,"new_domains":1.61,"v":4.39,"lab":false,"jev":false,"rank_v":104,"rank_pub":101,"name":"MoganAI Lavoir ModernBERT-large","eq":{"S":8.96,"O":0},"rank_tie":103,"tied":true,"parts":{"public":1.77,"same_skills":4.48,"new_domains":0}},{"engine":"jeff-uncapped","pub":8.27,"same_skills":6.86,"new_domains":13.92,"v":7.71,"lab":false,"jev":false,"rank_v":99,"rank_pub":102,"name":"jeff GLiFormer-large","eq":{"S":5.88,"O":10.36},"rank_tie":98,"tied":true,"parts":{"public":1.65,"same_skills":2.94,"new_domains":3.11}},{"engine":"lfm2600","pub":6.78,"same_skills":7.65,"new_domains":9.77,"v":6.13,"lab":false,"jev":false,"rank_v":102,"rank_pub":105,"name":"LFM2.5-2.6B-RLCD","eq":{"S":6.68,"O":4.76},"rank_tie":100,"tied":true,"parts":{"public":1.36,"same_skills":3.34,"new_domains":1.43}},{"engine":"sifr-0.8b-v3.1","pub":25.65,"same_skills":18.93,"new_domains":15.18,"v":17.79,"lab":false,"jev":false,"rank_v":84,"rank_pub":80,"name":"Sifr 0.8B v3.1","eq":{"S":18.09,"O":12.05},"rank_tie":84,"tied":true,"parts":{"public":5.13,"same_skills":9.05,"new_domains":3.61}},{"engine":"laya-1e28ac20","pub":6.19,"same_skills":9.08,"new_domains":4.05,"v":4.43,"lab":false,"jev":false,"rank_v":103,"rank_pub":106,"name":"Laya","eq":{"S":8.13,"O":0},"rank_tie":103,"tied":true,"parts":{"public":1.24,"same_skills":4.07,"new_domains":0}},{"engine":"clm-v0.1-8b","pub":7.41,"same_skills":5.96,"new_domains":11.84,"v":6.24,"lab":false,"jev":false,"rank_v":101,"rank_pub":104,"name":"CLM-v0.1-8B Qwen3-8B heads","eq":{"S":4.98,"O":7.55},"rank_tie":100,"tied":true,"parts":{"public":1.48,"same_skills":2.49,"new_domains":2.27}},{"engine":"harsha","pub":4.02,"same_skills":6.21,"new_domains":7.41,"v":3.9,"lab":false,"jev":false,"rank_v":106,"rank_pub":109,"name":"Qwen-2.5-1B-RLCD","eq":{"S":5.23,"O":1.59},"rank_tie":103,"tied":true,"parts":{"public":0.8,"same_skills":2.61,"new_domains":0.48}},{"engine":"akash-gemma","pub":5.19,"same_skills":5.03,"new_domains":1.77,"v":1.25,"lab":false,"jev":false,"rank_v":109,"rank_pub":107,"name":"system-one-gemma","eq":{"S":4.03,"O":0},"rank_tie":109,"tied":false,"parts":{"public":1.04,"same_skills":2.02,"new_domains":0}},{"engine":"julia-1","pub":4.24,"same_skills":4.54,"new_domains":6.08,"v":2.56,"lab":false,"jev":false,"rank_v":108,"rank_pub":108,"name":"Supersonic Labs Julia 1","eq":{"S":3.54,"O":0},"rank_tie":107,"tied":true,"parts":{"public":0.85,"same_skills":1.77,"new_domains":0}},{"engine":"verdict-8af2496e","pub":1.86,"same_skills":2.41,"new_domains":1.21,"v":0,"lab":false,"jev":false,"rank_v":110,"rank_pub":110,"name":"Verdict","eq":{"S":1.38,"O":0},"rank_tie":110,"tied":true,"parts":{"public":0.37,"same_skills":0.69,"new_domains":0}},{"engine":"lfm350-sdpa","pub":1.43,"same_skills":2.29,"new_domains":3.81,"v":0,"lab":false,"jev":false,"rank_v":111,"rank_pub":112,"name":"LFM2.5-350M-RLCD","eq":{"S":1.27,"O":0},"rank_tie":110,"tied":true,"parts":{"public":0.29,"same_skills":0.63,"new_domains":0}},{"engine":"pplx-decider-v1.1-27b","pub":62.25,"same_skills":61.1,"new_domains":55.57,"v":62.75,"lab":false,"jev":false,"rank_v":1,"rank_pub":2,"name":"Perplexity Decider v1.1 (27B)","eq":{"S":60.75,"O":66.42},"rank_tie":1,"tied":false,"parts":{"public":12.45,"same_skills":30.38,"new_domains":19.93}},{"engine":"jpt-35b-a3b","pub":52.68,"same_skills":52.25,"new_domains":43.3,"v":51.41,"lab":false,"jev":false,"rank_v":21,"rank_pub":22,"name":"JPT-35B-A3B","eq":{"S":51.8,"O":49.91},"rank_tie":20,"tied":true,"parts":{"public":10.54,"same_skills":25.9,"new_domains":14.97}},{"engine":"zev-latest","pub":1.55,"same_skills":1.79,"new_domains":3.88,"v":0,"lab":false,"jev":false,"rank_v":112,"rank_pub":111,"name":"Zev","eq":{"S":0.76,"O":0},"rank_tie":110,"tied":true,"parts":{"public":0.31,"same_skills":0.38,"new_domains":0}},{"engine":"exaone-4.0-1.2b-jev-v0.3","pub":30.68,"same_skills":26.92,"new_domains":16.35,"v":23.31,"lab":false,"jev":false,"rank_v":79,"rank_pub":68,"name":"EXAONE-4.0-1.2B-JEV v0.3","eq":{"S":26.18,"O":13.62},"rank_tie":79,"tied":true,"parts":{"public":6.14,"same_skills":13.09,"new_domains":4.09}},{"engine":"rsi-jev-v5.0-vl-3b","pub":38.11,"same_skills":38.69,"new_domains":35.4,"v":38.44,"lab":false,"jev":false,"rank_v":56,"rank_pub":55,"name":"RSI-Jev v5.0 VL 3B","eq":{"S":38.08,"O":39.28},"rank_tie":54,"tied":true,"parts":{"public":7.62,"same_skills":19.04,"new_domains":11.78}},{"engine":"ajev-lora5","pub":51.66,"same_skills":51.35,"new_domains":41.51,"v":50.02,"lab":false,"jev":false,"rank_v":25,"rank_pub":25,"name":"AJev lora5 (Gemma 4 12B)","eq":{"S":50.89,"O":47.49},"rank_tie":25,"tied":true,"parts":{"public":10.33,"same_skills":25.44,"new_domains":14.25}},{"engine":"ezjev-4b-s2","pub":50.82,"same_skills":46.22,"new_domains":40.73,"v":46.95,"lab":false,"jev":false,"rank_v":35,"rank_pub":27,"name":"ezjev 4B s2","eq":{"S":45.7,"O":46.45},"rank_tie":31,"tied":true,"parts":{"public":10.16,"same_skills":22.85,"new_domains":13.94}},{"engine":"bekko-v0-17m","pub":7.44,"same_skills":5.89,"new_domains":3.74,"v":2.93,"lab":false,"jev":false,"rank_v":107,"rank_pub":103,"name":"Bekko System One v0 17M","eq":{"S":4.91,"O":0},"rank_tie":107,"tied":true,"parts":{"public":1.49,"same_skills":2.45,"new_domains":0}},{"engine":"bekko-v0-68m","pub":13.01,"same_skills":12.62,"new_domains":6.13,"v":8.42,"lab":false,"jev":false,"rank_v":97,"rank_pub":97,"name":"Bekko System One v0 68M","eq":{"S":11.72,"O":0},"rank_tie":95,"tied":true,"parts":{"public":2.6,"same_skills":5.86,"new_domains":0}},{"engine":"bekko-v0-400m","pub":22.45,"same_skills":21.55,"new_domains":7.16,"v":15.24,"lab":false,"jev":false,"rank_v":87,"rank_pub":81,"name":"Bekko System One v0 400M","eq":{"S":20.75,"O":1.25},"rank_tie":86,"tied":true,"parts":{"public":4.49,"same_skills":10.38,"new_domains":0.38}},{"engine":"torchcast-decision-27b","pub":65.1,"same_skills":58.13,"new_domains":50.83,"v":59.91,"lab":false,"jev":false,"rank_v":4,"rank_pub":1,"name":"Torchcast Decision 27B","eq":{"S":57.75,"O":60.04},"rank_tie":2,"tied":true,"parts":{"public":13.02,"same_skills":28.88,"new_domains":18.01}},{"engine":"kas-4b","pub":38.35,"same_skills":38.69,"new_domains":30.96,"v":36.7,"lab":false,"jev":false,"rank_v":60,"rank_pub":54,"name":"Kas 4B","eq":{"S":38.09,"O":33.3},"rank_tie":59,"tied":true,"parts":{"public":7.67,"same_skills":19.04,"new_domains":9.99}},{"engine":"lev1-3.7b","pub":42.68,"same_skills":41.45,"new_domains":30.14,"v":38.63,"lab":false,"jev":false,"rank_v":55,"rank_pub":42,"name":"Lev1 3.7B","eq":{"S":40.88,"O":32.18},"rank_tie":54,"tied":true,"parts":{"public":8.54,"same_skills":20.44,"new_domains":9.66}},{"engine":"decisio-0.4.0-qwen3.6-35b-a3b","pub":48.43,"same_skills":49.15,"new_domains":40.14,"v":47.72,"lab":false,"jev":false,"rank_v":30,"rank_pub":32,"name":"decisio 0.4.0 (Qwen3.6-35B-A3B)","eq":{"S":48.67,"O":45.65},"rank_tie":27,"tied":true,"parts":{"public":9.69,"same_skills":24.33,"new_domains":13.7}},{"engine":"decisio-0.4.0-gemma-4-12b","pub":49.72,"same_skills":48.44,"new_domains":39.79,"v":47.47,"lab":false,"jev":false,"rank_v":32,"rank_pub":31,"name":"decisio 0.4.0 (Gemma 4 12B)","eq":{"S":47.94,"O":45.18},"rank_tie":31,"tied":true,"parts":{"public":9.94,"same_skills":23.97,"new_domains":13.55}},{"engine":"hopper-12b-trained","pub":48.27,"same_skills":46.97,"new_domains":41.85,"v":47.27,"lab":false,"jev":false,"rank_v":34,"rank_pub":33,"name":"Hopper 12B (trained)","eq":{"S":46.46,"O":47.96},"rank_tie":31,"tied":true,"parts":{"public":9.65,"same_skills":23.23,"new_domains":14.39}},{"engine":"dm-jepa-1.1","pub":19.19,"same_skills":4.17,"new_domains":2.84,"v":4.05,"lab":false,"jev":false,"rank_v":105,"rank_pub":89,"name":"DM-JEPA 1.1","eq":{"S":3.17,"O":0},"rank_tie":103,"tied":true,"parts":{"public":3.84,"same_skills":1.58,"new_domains":0}},{"engine":"jevstral-8b","pub":34.01,"same_skills":36.31,"new_domains":34.19,"v":35.93,"lab":false,"jev":false,"rank_v":63,"rank_pub":63,"name":"Jevstral 8B","eq":{"S":35.68,"O":37.64},"rank_tie":63,"tied":false,"parts":{"public":6.8,"same_skills":17.84,"new_domains":11.29}},{"engine":"jiwo-0.8b","pub":28.72,"same_skills":28.31,"new_domains":18.07,"v":24.32,"lab":false,"jev":false,"rank_v":78,"rank_pub":73,"name":"jiwo 0.8B","eq":{"S":27.58,"O":15.94},"rank_tie":77,"tied":true,"parts":{"public":5.74,"same_skills":13.79,"new_domains":4.78}},{"engine":"jiwo-4b","pub":45.76,"same_skills":46.05,"new_domains":33.32,"v":42.86,"lab":false,"jev":false,"rank_v":42,"rank_pub":38,"name":"jiwo 4B","eq":{"S":45.53,"O":36.47},"rank_tie":41,"tied":true,"parts":{"public":9.15,"same_skills":22.77,"new_domains":10.94}},{"engine":"deck31b","pub":58.19,"same_skills":58.6,"new_domains":51.48,"v":59.03,"lab":false,"jev":false,"rank_v":5,"rank_pub":8,"name":"deck31b (Gemma 4 31B)","eq":{"S":58.23,"O":60.92},"rank_tie":5,"tied":true,"parts":{"public":11.64,"same_skills":29.11,"new_domains":18.28}},{"engine":"quyet-1.0-large","pub":60.79,"same_skills":58.83,"new_domains":49.12,"v":58.71,"lab":false,"jev":false,"rank_v":7,"rank_pub":4,"name":"Quyet-1.0-Large","eq":{"S":58.46,"O":57.74},"rank_tie":5,"tied":true,"parts":{"public":12.16,"same_skills":29.23,"new_domains":17.32}}],"formula":{"id":"v0.3-final","decided_utc":"2026-10-06","equated":true,"tie_band":0.9,"equating":{"bases":21,"public":{"mean":23.5,"sd":16.54},"same_skills":{"mean":24.27,"sd":16.35},"new_domains":{"mean":23.68,"sd":12.29}}}}

data/vision.json

@@ -0,0 +1 @@
+{"generated_utc":"2026-10-06T16:32:57Z","weights":{"public":0.5,"private":0.5},"bench_weights":{"CV-Bench":1.0,"BLINK":1.0,"RealWorldQA":1.0,"CharXiv":1.0,"InfographicVQA":1.0,"Mind2Web":1.0,"Winoground":1.0,"KIE (CORD+FUNSD)":1.0,"Moderation (Hateful Memes)":1.0,"R-Bench-M":0.5,"MMMU-Pro vision":0.25},"private_sets":["CV-Bench","BLINK","RealWorldQA","CharXiv","InfographicVQA","Mind2Web","Winoground","KIE (CORD+FUNSD)","Moderation (Hateful Memes)"],"entrants":[{"engine":"solomon-v11-bf16-encoding","name":"Solomon v1.1 (27B)","url":"https://huggingface.co/DoccyHealth/Solomon","kind":"head / adapter","technique":"autoregressive","org":{"id":"DoccyHealth","name":"Doccy Health","logo":"data/orgs/DoccyHealth.png"},"pub":69.8,"priv":65.52,"full":67.66,"base":null,"caveat":"Answers up to 8 options: MMMU-Pro rows with more options count as failures.","bench":{"CV-Bench":{"pub":74.91,"private":42.71},"BLINK":{"pub":59.08,"private":59.33},"RealWorldQA":{"pub":75.31,"private":54.0},"CharXiv":{"pub":74.94,"private":63.24},"InfographicVQA":{"pub":95.63,"private":87.33},"Mind2Web":{"pub":79.72,"private":58.67},"Winoground":{"pub":83.75,"private":91.0},"KIE (CORD+FUNSD)":{"pub":91.78,"private":85.38},"Moderation (Hateful Memes)":{"pub":36.36,"private":48.0},"R-Bench-M":{"pub":18.24,"private":null},"MMMU-Pro vision":{"pub":-2.56,"private":null}},"params":27781427952,"active":null,"params_estimated":false,"cal":{"acc":0.7731,"conf":0.8488,"ece":0.0758,"over95":0.0287,"n":12255,"benchmarks":11},"ms_row":561.6,"gpu_h":3.006,"rank_full":1,"rank_pub":2},{"engine":"xor-1.2","name":"Xor 1.2 (35B-A3B)","url":"https://huggingface.co/juspay/xor","kind":"LoRA","technique":"autoregressive","org":{"id":"juspay","name":"Juspay Technologies","logo":"data/orgs/juspay.png"},"pub":70.32,"priv":58.27,"full":64.3,"base":{"name":"Qwen/Qwen3.6-35B-A3B","pub":69.49,"priv":56.53},"caveat":"Environment rebuilt from the reference image (no lane recipe existed); same model code.","bench":{"CV-Bench":{"pub":76.28,"private":44.05},"BLINK":{"pub":57.15,"private":47.33},"RealWorldQA":{"pub":66.43,"private":44.0},"CharXiv":{"pub":71.18,"private":54.75},"InfographicVQA":{"pub":95.63,"private":75.33},"Mind2Web":{"pub":75.61,"private":55.33},"Winoground":{"pub":84.25,"private":93.0},"KIE (CORD+FUNSD)":{"pub":92.72,"private":88.63},"Moderation (Hateful Memes)":{"pub":38.5,"private":22.0},"R-Bench-M":{"pub":35.39,"private":null},"MMMU-Pro vision":{"pub":40.73,"private":null}},"params":35950000000,"active":"A3B","params_estimated":false,"cal":{"acc":0.7638,"conf":0.7659,"ece":0.0084,"over95":0.0078,"n":13693,"benchmarks":11},"ms_row":242.2,"gpu_h":1.297,"rank_full":2,"rank_pub":1},{"engine":"rune-26b-a4b-v3","name":"Surogate Rune 26B-A4B v3","url":"https://huggingface.co/surogate/rune-26b-a4b-GGUF","kind":"full fine-tune","technique":"autoregressive","org":{"id":"surogate","name":"Surogate","logo":"data/orgs/surogate.png"},"pub":65.27,"priv":51.99,"full":58.63,"base":{"name":"google/gemma-4-26B-A4B-it","pub":58.25,"priv":47.34},"caveat":null,"bench":{"CV-Bench":{"pub":70.21,"private":42.04},"BLINK":{"pub":53.12,"private":4.67},"RealWorldQA":{"pub":54.22,"private":28.0},"CharXiv":{"pub":73.43,"private":52.87},"InfographicVQA":{"pub":89.62,"private":94.0},"Mind2Web":{"pub":63.82,"private":52.0},"Winoground":{"pub":78.5,"private":81.0},"KIE (CORD+FUNSD)":{"pub":90.98,"private":79.29},"Moderation (Hateful Memes)":{"pub":38.17,"private":34.0},"R-Bench-M":{"pub":33.22,"private":null},"MMMU-Pro vision":{"pub":30.72,"private":null}},"params":25805936206,"active":"A4B","params_estimated":false,"cal":{"acc":0.7265,"conf":0.8042,"ece":0.0797,"over95":0.0298,"n":13693,"benchmarks":11},"ms_row":143.4,"gpu_h":0.768,"rank_full":7,"rank_pub":5},{"engine":"jpt-9b","name":"JPT-9B","url":"https://huggingface.co/kirp/jpt-9b","kind":"LoRA","technique":"autoregressive","org":null,"pub":65.5,"priv":58.24,"full":61.87,"base":{"name":"Qwen/Qwen3.5-9B-Base","pub":63.47,"priv":54.09},"caveat":null,"bench":{"CV-Bench":{"pub":76.04,"private":45.06},"BLINK":{"pub":52.96,"private":38.0},"RealWorldQA":{"pub":65.6,"private":42.0},"CharXiv":{"pub":69.17,"private":45.33},"InfographicVQA":{"pub":93.44,"private":94.67},"Mind2Web":{"pub":69.3,"private":52.67},"Winoground":{"pub":79.25,"private":90.0},"KIE (CORD+FUNSD)":{"pub":91.92,"private":87.41},"Moderation (Hateful Memes)":{"pub":23.17,"private":29.0},"R-Bench-M":{"pub":18.42,"private":null},"MMMU-Pro vision":{"pub":34.3,"private":null}},"params":9653104368,"active":null,"params_estimated":false,"cal":{"acc":0.7273,"conf":0.8022,"ece":0.0749,"over95":0.0227,"n":13695,"benchmarks":11},"ms_row":300.8,"gpu_h":1.61,"rank_full":4,"rank_pub":4},{"engine":"jpt-4b","name":"JPT-4B","url":"https://huggingface.co/kirp/jpt-4b","kind":"LoRA","technique":"autoregressive","org":null,"pub":64.83,"priv":54.47,"full":59.65,"base":{"name":"Qwen/Qwen3.5-4B-Base","pub":59.56,"priv":46.8},"caveat":null,"bench":{"CV-Bench":{"pub":75.88,"private":43.38},"BLINK":{"pub":53.77,"private":43.33},"RealWorldQA":{"pub":63.1,"private":36.0},"CharXiv":{"pub":69.67,"private":45.33},"InfographicVQA":{"pub":91.26,"private":84.67},"Mind2Web":{"pub":66.84,"private":41.33},"Winoground":{"pub":78.0,"private":88.0},"KIE (CORD+FUNSD)":{"pub":90.32,"private":84.16},"Moderation (Hateful Memes)":{"pub":26.3,"private":24.0},"R-Bench-M":{"pub":19.51,"private":null},"MMMU-Pro vision":{"pub":29.0,"private":null}},"params":4659865088,"active":null,"params_estimated":false,"cal":{"acc":0.7183,"conf":0.7768,"ece":0.0584,"over95":0.0183,"n":13695,"benchmarks":11},"ms_row":177.9,"gpu_h":0.952,"rank_full":6,"rank_pub":6},{"engine":"winnow-12b-q8","name":"Winnow-12B","url":"https://huggingface.co/EldanRing/Winnow-12B","kind":"LoRA","technique":"autoregressive","org":null,"pub":64.38,"priv":55.29,"full":59.84,"base":{"name":"google/gemma-4-12B","pub":18.73,"priv":11.9},"caveat":"Its base answers at chance through the converted projector, so the base comparison is not meaningful.","bench":{"CV-Bench":{"pub":67.78,"private":30.99},"BLINK":{"pub":50.87,"private":22.0},"RealWorldQA":{"pub":53.39,"private":28.0},"CharXiv":{"pub":67.42,"private":60.41},"InfographicVQA":{"pub":87.43,"private":88.67},"Mind2Web":{"pub":76.29,"private":60.0},"Winoground":{"pub":82.0,"private":80.0},"KIE (CORD+FUNSD)":{"pub":90.85,"private":82.54},"Moderation (Hateful Memes)":{"pub":31.57,"private":45.0},"R-Bench-M":{"pub":24.92,"private":null},"MMMU-Pro vision":{"pub":30.65,"private":null}},"params":11959730224,"active":null,"params_estimated":false,"cal":{"acc":0.7196,"conf":0.7928,"ece":0.0788,"over95":0.029,"n":13695,"benchmarks":11},"ms_row":567.5,"gpu_h":3.038,"rank_full":5,"rank_pub":7},{"engine":"neohorse-jev-4b","name":"NeoHorse-Jev-4B","url":"https://huggingface.co/TokenRhythm/NeoHorse-Jev-4B","kind":"head / adapter","technique":"autoregressive","org":{"id":"TokenRhythm","name":"TokenRhythm","logo":"data/orgs/TokenRhythm.png"},"pub":57.21,"priv":44.25,"full":50.73,"base":null,"caveat":"Takes a single image only: multi-image rows (BLINK, Winoground) count as failures.","bench":{"CV-Bench":{"pub":74.1,"private":35.34},"BLINK":{"pub":-2.13,"private":-33.33},"RealWorldQA":{"pub":58.66,"private":22.0},"CharXiv":{"pub":64.16,"private":37.78},"InfographicVQA":{"pub":89.62,"private":76.0},"Mind2Web":{"pub":73.0,"private":54.0},"Winoground":{"pub":76.5,"private":84.0},"KIE (CORD+FUNSD)":{"pub":89.11,"private":82.13},"Moderation (Hateful Memes)":{"pub":19.87,"private":7.0},"R-Bench-M":{"pub":13.55,"private":null},"MMMU-Pro vision":{"pub":23.82,"private":null}},"params":4659865088,"active":null,"params_estimated":false,"cal":{"acc":0.7197,"conf":0.7286,"ece":0.0245,"over95":0.0129,"n":12321,"benchmarks":11},"ms_row":113.6,"gpu_h":0.608,"rank_full":10,"rank_pub":10},{"engine":"jev-omni","name":"Jev-Omni (12B)","url":"https://huggingface.co/akhilaaa3/Jev-Omni","kind":"LoRA + head","technique":"autoregressive","org":null,"pub":51.9,"priv":44.73,"full":48.32,"base":null,"caveat":"Takes a single image only: multi-image rows (BLINK, Winoground) count as failures.","bench":{"CV-Bench":{"pub":69.81,"private":34.0},"BLINK":{"pub":-5.52,"private":-33.33},"RealWorldQA":{"pub":47.56,"private":20.67},"CharXiv":{"pub":62.91,"private":38.73},"InfographicVQA":{"pub":70.49,"private":68.67},"Mind2Web":{"pub":46.15,"private":35.33},"Winoground":{"pub":75.5,"private":80.0},"KIE (CORD+FUNSD)":{"pub":90.45,"private":71.17},"Moderation (Hateful Memes)":{"pub":35.53,"private":54.0},"R-Bench-M":{"pub":5.79,"private":null},"MMMU-Pro vision":{"pub":19.05,"private":null}},"params":11959730224,"active":null,"params_estimated":false,"cal":{"acc":0.6722,"conf":0.8001,"ece":0.1279,"over95":0.0334,"n":12321,"benchmarks":11},"ms_row":102.9,"gpu_h":0.551,"rank_full":12,"rank_pub":12},{"engine":"winnow-e4b-q8","name":"Winnow-E4B","url":"https://huggingface.co/EldanRing/Winnow-E4B","kind":"LoRA","technique":"autoregressive","org":null,"pub":48.46,"priv":38.37,"full":43.41,"base":{"name":"google/gemma-4-E4B","pub":45.56,"priv":39.79},"caveat":null,"bench":{"CV-Bench":{"pub":49.08,"private":17.25},"BLINK":{"pub":34.92,"private":10.0},"RealWorldQA":{"pub":35.35,"private":19.33},"CharXiv":{"pub":43.36,"private":28.36},"InfographicVQA":{"pub":83.61,"private":61.33},"Mind2Web":{"pub":53.13,"private":46.0},"Winoground":{"pub":65.5,"private":68.0},"KIE (CORD+FUNSD)":{"pub":80.43,"private":76.04},"Moderation (Hateful Memes)":{"pub":15.91,"private":19.0},"R-Bench-M":{"pub":15.36,"private":null},"MMMU-Pro vision":{"pub":14.01,"private":null}},"params":7996156490,"active":null,"params_estimated":false,"cal":{"acc":0.6144,"conf":0.6313,"ece":0.0478,"over95":0.0187,"n":13695,"benchmarks":11},"ms_row":152.6,"gpu_h":0.817,"rank_full":13,"rank_pub":13},{"engine":"onejev-0.8b","name":"OneJev 0.8B","url":"https://huggingface.co/OmniJev/OneJev-0.8B","kind":"full fine-tune","technique":"autoregressive","org":{"id":"OmniJev","name":"OmniJev","logo":"data/orgs/OmniJev.png"},"pub":46.84,"priv":36.39,"full":41.61,"base":{"name":"Qwen/Qwen3.5-0.8B","pub":35.08,"priv":29.74},"caveat":null,"bench":{"CV-Bench":{"pub":55.8,"private":26.63},"BLINK":{"pub":39.43,"private":15.33},"RealWorldQA":{"pub":40.07,"private":24.67},"CharXiv":{"pub":52.13,"private":26.47},"InfographicVQA":{"pub":78.69,"private":54.67},"Mind2Web":{"pub":45.05,"private":29.33},"Winoground":{"pub":53.5,"private":69.0},"KIE (CORD+FUNSD)":{"pub":75.76,"private":72.39},"Moderation (Hateful Memes)":{"pub":12.61,"private":9.0},"R-Bench-M":{"pub":3.26,"private":null},"MMMU-Pro vision":{"pub":8.11,"private":null}},"params":870000000,"active":null,"params_estimated":false,"cal":{"acc":0.5954,"conf":0.611,"ece":0.0166,"over95":0.0042,"n":13695,"benchmarks":11},"ms_row":90.3,"gpu_h":0.483,"rank_full":15,"rank_pub":14},{"engine":"jpt-0.8b","name":"JPT-0.8B","url":"https://huggingface.co/kirp/jpt-0.8b","kind":"LoRA","technique":"autoregressive","org":null,"pub":44.72,"priv":40.98,"full":42.85,"base":{"name":"Qwen/Qwen3.5-0.8B-Base","pub":34.51,"priv":30.92},"caveat":null,"bench":{"CV-Bench":{"pub":51.35,"private":25.63},"BLINK":{"pub":33.31,"private":38.67},"RealWorldQA":{"pub":41.46,"private":26.67},"CharXiv":{"pub":52.13,"private":36.84},"InfographicVQA":{"pub":76.5,"private":54.67},"Mind2Web":{"pub":37.79,"private":33.33},"Winoground":{"pub":47.75,"private":62.0},"KIE (CORD+FUNSD)":{"pub":73.95,"private":74.01},"Moderation (Hateful Memes)":{"pub":12.78,"private":17.0},"R-Bench-M":{"pub":13.73,"private":null},"MMMU-Pro vision":{"pub":8.64,"private":null}},"params":873438784,"active":null,"params_estimated":false,"cal":{"acc":0.59,"conf":0.6536,"ece":0.0636,"over95":0.0067,"n":13695,"benchmarks":11},"ms_row":115.7,"gpu_h":0.619,"rank_full":14,"rank_pub":15},{"engine":"djev","name":"djev (26B-A4B)","url":"https://github.com/Davipar/djev-dev","kind":"inference technique","technique":"diffusion","org":null,"pub":44.51,"priv":29.21,"full":36.86,"base":null,"caveat":"Ran with its published canvas 128 and 32K context, since the long-context engine takes no images.","bench":{"CV-Bench":{"pub":50.38,"private":19.26},"BLINK":{"pub":27.18,"private":6.0},"RealWorldQA":{"pub":38.4,"private":18.67},"CharXiv":{"pub":35.34,"private":10.45},"InfographicVQA":{"pub":62.3,"private":60.67},"Mind2Web":{"pub":42.31,"private":32.67},"Winoground":{"pub":35.25,"private":38.0},"KIE (CORD+FUNSD)":{"pub":84.71,"private":56.14},"Moderation (Hateful Memes)":{"pub":38.5,"private":21.0},"R-Bench-M":{"pub":25.46,"private":null},"MMMU-Pro vision":{"pub":27.34,"private":null}},"params":25823778864,"active":"A4B","params_estimated":false,"cal":{"acc":0.6088,"conf":0.7669,"ece":0.1581,"over95":0.0548,"n":13695,"benchmarks":11},"ms_row":77.0,"gpu_h":0.412,"rank_full":16,"rank_pub":16},{"engine":"razorback-one-read-3e296f08","name":"razorback16 diffgemma (26B-A4B)","url":"https://github.com/vllm-project/vllm/pull/57250","kind":"inference technique","technique":"diffusion","org":null,"pub":43.91,"priv":28.8,"full":36.36,"base":null,"caveat":null,"bench":{"CV-Bench":{"pub":51.03,"private":20.6},"BLINK":{"pub":36.85,"private":-2.67},"RealWorldQA":{"pub":35.63,"private":21.33},"CharXiv":{"pub":25.81,"private":8.56},"InfographicVQA":{"pub":62.84,"private":63.33},"Mind2Web":{"pub":38.47,"private":32.67},"Winoground":{"pub":35.0,"private":47.0},"KIE (CORD+FUNSD)":{"pub":84.57,"private":53.71},"Moderation (Hateful Memes)":{"pub":41.14,"private":12.0},"R-Bench-M":{"pub":21.31,"private":null},"MMMU-Pro vision":{"pub":24.49,"private":null}},"params":25823778864,"active":"A4B","params_estimated":false,"cal":{"acc":0.6006,"conf":0.7814,"ece":0.1808,"over95":0.0598,"n":13695,"benchmarks":11},"ms_row":39.0,"gpu_h":0.209,"rank_full":18,"rank_pub":17},{"engine":"vllm-pr57250","name":"mmastrac diffgemma (26B-A4B)","url":"https://github.com/vllm-project/vllm/pull/57250","kind":"inference technique","technique":"diffusion","org":null,"pub":36.15,"priv":24.41,"full":30.28,"base":null,"caveat":null,"bench":{"CV-Bench":{"pub":47.38,"private":18.93},"BLINK":{"pub":25.09,"private":10.67},"RealWorldQA":{"pub":30.08,"private":17.33},"CharXiv":{"pub":25.31,"private":12.33},"InfographicVQA":{"pub":53.55,"private":50.0},"Mind2Web":{"pub":24.77,"private":15.33},"Winoground":{"pub":25.75,"private":35.0},"KIE (CORD+FUNSD)":{"pub":76.42,"private":39.09},"Moderation (Hateful Memes)":{"pub":28.94,"private":21.0},"R-Bench-M":{"pub":24.2,"private":null},"MMMU-Pro vision":{"pub":12.09,"private":null}},"params":25823778864,"active":"A4B","params_estimated":false,"cal":{"acc":0.5521,"conf":0.7132,"ece":0.1611,"over95":0.043,"n":13695,"benchmarks":11},"ms_row":84.8,"gpu_h":0.454,"rank_full":19,"rank_pub":19},{"engine":"intern-decision-4b","name":"Intern-Decision-4B","url":"https://huggingface.co/internlm/Intern-Decision-4B","kind":"full fine-tune","technique":"autoregressive","org":{"id":"internlm","name":"Intern Large Models","logo":"data/orgs/internlm.png"},"pub":61.83,"priv":51.95,"full":56.89,"base":{"name":"Qwen/Qwen3.5-4B-Base","pub":56.89,"priv":41.26},"caveat":"The vision patch embed was swapped for the identical linear map to run at normal speed; outputs agree to bf16 precision.","bench":{"CV-Bench":{"pub":73.85,"private":36.68},"BLINK":{"pub":53.6,"private":36.0},"RealWorldQA":{"pub":60.32,"private":34.67},"CharXiv":{"pub":57.14,"private":42.5},"InfographicVQA":{"pub":85.25,"private":67.33},"Mind2Web":{"pub":65.88,"private":43.33},"Winoground":{"pub":75.25,"private":87.0},"KIE (CORD+FUNSD)":{"pub":87.98,"private":78.07},"Moderation (Hateful Memes)":{"pub":26.63,"private":42.0},"R-Bench-M":{"pub":19.15,"private":null},"MMMU-Pro vision":{"pub":29.33,"private":null}},"params":4659865088,"active":null,"params_estimated":false,"cal":{"acc":0.6994,"conf":0.6899,"ece":0.0134,"over95":0.0021,"n":13695,"benchmarks":11},"ms_row":142.4,"gpu_h":0.762,"rank_full":8,"rank_pub":9},{"engine":"intern-decision-2b","name":"Intern-Decision-2B","url":"https://huggingface.co/internlm/Intern-Decision-2B","kind":"full fine-tune","technique":"autoregressive","org":{"id":"internlm","name":"Intern Large Models","logo":"data/orgs/internlm.png"},"pub":53.29,"priv":43.42,"full":48.36,"base":{"name":"Qwen/Qwen3.5-2B-Base","pub":43.77,"priv":34.45},"caveat":"The vision patch embed was swapped for the identical linear map to run at normal speed; outputs agree to bf16 precision.","bench":{"CV-Bench":{"pub":65.35,"private":36.35},"BLINK":{"pub":43.13,"private":26.0},"RealWorldQA":{"pub":49.78,"private":42.67},"CharXiv":{"pub":58.15,"private":25.53},"InfographicVQA":{"pub":81.42,"private":58.0},"Mind2Web":{"pub":49.98,"private":42.0},"Winoground":{"pub":63.75,"private":77.0},"KIE (CORD+FUNSD)":{"pub":84.57,"private":75.23},"Moderation (Hateful Memes)":{"pub":13.93,"private":8.0},"R-Bench-M":{"pub":10.48,"private":null},"MMMU-Pro vision":{"pub":17.19,"private":null}},"params":2274069824,"active":null,"params_estimated":false,"cal":{"acc":0.641,"conf":0.6175,"ece":0.0339,"over95":0.0015,"n":13695,"benchmarks":11},"ms_row":106.8,"gpu_h":0.572,"rank_full":11,"rank_pub":11},{"engine":"intern-decision-0.8b","name":"Intern-Decision-0.8B","url":"https://huggingface.co/internlm/Intern-Decision-0.8B","kind":"full fine-tune","technique":"autoregressive","org":{"id":"internlm","name":"Intern Large Models","logo":"data/orgs/internlm.png"},"pub":38.78,"priv":34.95,"full":36.86,"base":{"name":"Qwen/Qwen3.5-0.8B-Base","pub":29.58,"priv":26.81},"caveat":"The vision patch embed was swapped for the identical linear map to run at normal speed; outputs agree to bf16 precision.","bench":{"CV-Bench":{"pub":51.92,"private":20.6},"BLINK":{"pub":25.41,"private":20.0},"RealWorldQA":{"pub":37.57,"private":28.67},"CharXiv":{"pub":48.37,"private":21.76},"InfographicVQA":{"pub":70.49,"private":59.33},"Mind2Web":{"pub":21.07,"private":19.33},"Winoground":{"pub":43.25,"private":71.0},"KIE (CORD+FUNSD)":{"pub":62.6,"private":61.83},"Moderation (Hateful Memes)":{"pub":12.45,"private":12.0},"R-Bench-M":{"pub":5.79,"private":null},"MMMU-Pro vision":{"pub":8.18,"private":null}},"params":873438784,"active":null,"params_estimated":false,"cal":{"acc":0.5517,"conf":0.5003,"ece":0.0595,"over95":0.0002,"n":13695,"benchmarks":11},"ms_row":1618.4,"gpu_h":8.663,"rank_full":17,"rank_pub":18},{"engine":"pplx-decider-v1.1-27b","name":"Perplexity Decider v1.1 (27B)","url":"https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b","kind":"full fine-tune","technique":"autoregressive","org":{"id":"perplexity-ai","name":"Perplexity","logo":"data/orgs/perplexity-ai.png"},"pub":67.0,"priv":59.4,"full":63.2,"base":{"name":"Qwen/Qwen3.8-27B","pub":61.3,"priv":51.68},"caveat":"Ran through the checkpoint's own decision model, which accepts images.","bench":{"CV-Bench":{"pub":74.74,"private":41.71},"BLINK":{"pub":54.73,"private":62.0},"RealWorldQA":{"pub":57.83,"private":26.67},"CharXiv":{"pub":77.19,"private":56.64},"InfographicVQA":{"pub":79.78,"private":93.33},"Mind2Web":{"pub":68.76,"private":50.0},"Winoground":{"pub":84.5,"private":84.0},"KIE (CORD+FUNSD)":{"pub":91.32,"private":75.23},"Moderation (Hateful Memes)":{"pub":42.79,"private":45.0},"R-Bench-M":{"pub":29.25,"private":null},"MMMU-Pro vision":{"pub":28.0,"private":null}},"params":27800000000,"active":null,"params_estimated":false,"cal":{"acc":0.7345,"conf":0.784,"ece":0.0496,"over95":0.0157,"n":13695,"benchmarks":11},"ms_row":146.8,"gpu_h":0.786,"rank_full":3,"rank_pub":3},{"engine":"rsi-jev-v5.0-vl-3b","name":"RSI-Jev v5.0 VL 3B","url":"https://huggingface.co/shgao/rsi-jev-v5.0-vl-3b","kind":"full fine-tune","technique":"autoregressive","org":null,"pub":61.85,"priv":51.22,"full":56.54,"base":{"name":"Qwen/Qwen3.5-4B-Base","pub":29.06,"priv":23.1},"caveat":"Its base was read through the plain vision-language path, where a pretrained base answers letters poorly, so the gain over the base is overstated.","bench":{"CV-Bench":{"pub":72.96,"private":35.68},"BLINK":{"pub":47.64,"private":46.67},"RealWorldQA":{"pub":55.33,"private":23.33},"CharXiv":{"pub":66.67,"private":33.07},"InfographicVQA":{"pub":87.98,"private":66.67},"Mind2Web":{"pub":56.56,"private":40.67},"Winoground":{"pub":75.75,"private":88.0},"KIE (CORD+FUNSD)":{"pub":85.91,"private":80.91},"Moderation (Hateful Memes)":{"pub":42.46,"private":46.0},"R-Bench-M":{"pub":11.93,"private":null},"MMMU-Pro vision":{"pub":23.36,"private":null}},"params":4660000000,"active":null,"params_estimated":false,"cal":{"acc":0.6914,"conf":0.698,"ece":0.0252,"over95":0.0108,"n":13695,"benchmarks":11},"ms_row":64.9,"gpu_h":0.347,"rank_full":9,"rank_pub":8}],"refs":[{"engine":"ref-qwen35-397b","name":"Qwen3.5-397B (VLM)","kind":"stock · API","url":"https://huggingface.co/Qwen/Qwen3.5-397B-A17B","note":"Stock model via API, read from its generated letter rather than letter probabilities.","label":null,"params":397000000000.0,"active":"A17B","params_estimated":null,"pub":68.52,"priv":58.37,"full":63.44,"bench":{"CV-Bench":{"pub":78.31,"private":46.73},"BLINK":{"pub":50.54,"private":4.67},"RealWorldQA":{"pub":67.82,"private":41.33},"CharXiv":{"pub":62.16,"private":51.92},"InfographicVQA":{"pub":95.08,"private":99.33},"Mind2Web":{"pub":76.57,"private":57.33},"Winoground":{"pub":82.25,"private":87.0},"KIE (CORD+FUNSD)":{"pub":94.06,"private":84.97},"Moderation (Hateful Memes)":{"pub":49.22,"private":52.0},"R-Bench-M":{"pub":0.38,"private":null},"MMMU-Pro vision":{"pub":47.43,"private":null}},"ref":true,"cal":null,"ms_row":null},{"engine":"ref-qwen3vl-8b","name":"Qwen3-VL-8B (VLM)","kind":"stock","url":"https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct","note":"Stock vision-language model, no fine-tune.","label":null,"params":8800000000.0,"active":null,"params_estimated":null,"pub":62.47,"priv":44.84,"full":53.66,"bench":{"CV-Bench":{"pub":72.96,"private":34.0},"BLINK":{"pub":51.03,"private":20.0},"RealWorldQA":{"pub":54.77,"private":22.0},"CharXiv":{"pub":58.9,"private":16.1},"InfographicVQA":{"pub":85.25,"private":70.67},"Mind2Web":{"pub":71.22,"private":50.67},"Winoground":{"pub":73.75,"private":80.0},"KIE (CORD+FUNSD)":{"pub":89.05,"private":82.13},"Moderation (Hateful Memes)":{"pub":34.21,"private":28.0},"R-Bench-M":{"pub":21.85,"private":null},"MMMU-Pro vision":{"pub":28.13,"private":null}},"ref":true,"cal":{"acc":0.7032,"conf":0.8984,"ece":0.1952,"over95":0.1085,"n":13695,"benchmarks":11},"ms_row":119.4},{"engine":"ref-siglip2","name":"SigLIP 2 (contrastive)","kind":"contrastive floor","url":"https://huggingface.co/google/siglip2-so400m-patch14-384","note":"Image-text similarity only, no language model: the floor.","label":null,"params":1100000000.0,"active":null,"params_estimated":true,"pub":6.16,"priv":6.17,"full":6.17,"bench":{"CV-Bench":{"pub":-5.39,"private":4.86},"BLINK":{"pub":-0.85,"private":-10.0},"RealWorldQA":{"pub":5.39,"private":-6.0},"CharXiv":{"pub":4.01,"private":3.85},"InfographicVQA":{"pub":7.65,"private":-3.33},"Mind2Web":{"pub":8.46,"private":25.33},"Winoground":{"pub":33.5,"private":19.0},"KIE (CORD+FUNSD)":{"pub":-2.25,"private":0.51},"Moderation (Hateful Memes)":{"pub":-2.23,"private":2.0},"R-Bench-M":{"pub":0.74,"private":null},"MMMU-Pro vision":{"pub":2.68,"private":null}},"ref":true,"cal":null,"ms_row":null}],"excluded":[{"name":"reflex-27b-v2","why":"its official path takes no images"},{"name":"gevva-e4b","why":"its official path takes no images"},{"name":"gevva-e2b","why":"its official path takes no images"},{"name":"matilda-jev","why":"its base is undisclosed and its results return 404"}],"bench_info":{"CV-Bench":{"what":"Basic spatial vision: count objects, say which is left or right, and judge which is closer to the camera.","example":"How many chairs are in the photo, or is the lamp closer to the camera than the sofa?","why":"Many decisions start with getting simple facts about a scene right.","tech":"Counting, 2D relation and distance items; 2 to 6 options."},"BLINK":{"what":"Perception puzzles that people solve at a glance but models often miss, sometimes across two or more images.","example":"Given two photos of the same room, how did the camera move between them?","why":"Tests whether a model really looks at the image instead of guessing from the wording.","tech":"Seven subtasks of the validation split; 2 to 4 options over 1 to 4 images."},"RealWorldQA":{"what":"Questions about everyday real-world photos, many taken from a car or a phone.","example":"From this dashboard photo, which lane should the car be in to turn right?","why":"Real scenes are where a decision model meets the physical world.","tech":"Multiple-choice and yes/no items parsed from the prompt."},"CharXiv":{"what":"Reasoning over charts from scientific papers: read values, compare trends and find the right series.","example":"In this plot, which method has the highest score at the largest dataset size?","why":"Decisions often hinge on reading a chart correctly.","tech":"Reasoning questions converted to 4-way multiple choice with distractors from the same figure."},"InfographicVQA":{"what":"Answering questions about infographics that mix text, numbers, icons and layout.","example":"According to this infographic, which age group spends the most time online?","why":"Combines reading and layout understanding on dense documents.","tech":"Short answers converted to multiple choice."},"Mind2Web":{"what":"Web navigation: given a task and a screenshot of a real website, pick the element to click or type into next.","example":"To book a one-way flight to Denver, which of these four marked buttons comes next?","why":"Choosing the next action on a screen is a decision in the most literal sense.","tech":"Four marked boxes per screenshot: the right element plus three real distractors from the page."},"Winoground":{"what":"Matching captions to images when the words are the same but their order changes the meaning.","example":"Which caption fits: 'a dog bites a person' or 'a person bites a dog'?","why":"Tests whether a model understands who does what to whom.","tech":"Caption choice for each of two images; pairs of images and captions."},"KIE (CORD+FUNSD)":{"what":"Reading receipts and forms: pull the right field (total, date, item) out of a scanned document.","example":"On this receipt, what is the total after tax?","why":"Document extraction feeds many routing and approval decisions.","tech":"Receipt fields and form-field presence questions, multiple choice."},"Moderation (Hateful Memes)":{"what":"Deciding whether a meme is hateful, where the meaning often needs both the image and its text.","example":"Is this meme, its picture and caption together, hateful or not?","why":"A content-moderation call is a binary decision with real consequences.","tech":"Binary hateful / not hateful; a caption-only control is reported separately."},"R-Bench-M":{"what":"Expert-level multi-discipline questions with an image, from science, engineering and medicine.","example":"Given this circuit diagram, which option gives the current through the resistor?","why":"Shows headroom: even strong models score low here.","tech":"Six options A to F, one image; low weight in the index."},"MMMU-Pro vision":{"what":"College-level exam questions where the whole question and its options are shown inside the image.","example":"A photographed exam page asks which of ten options explains the graph.","why":"The model has to read and reason from the image alone.","tech":"Ten options inside the image; a quarter weight in the index."}},"bench_rows":{"CV-Bench":{"public":2038},"BLINK":{"public":961},"RealWorldQA":{"public":589},"CharXiv":{"public":532},"InfographicVQA":{"public":244},"Mind2Web":{"public":973},"Winoground":{"public":800},"KIE (CORD+FUNSD)":{"public":2465},"Moderation (Hateful Memes)":{"public":1213},"R-Bench-M":{"public":665},"MMMU-Pro vision":{"public":1730}},"stock_drop":{"name":"Qwen3-VL-8B (VLM)","drop":17.6}}

methodology.html

@@ -3,14 +3,14 @@
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
-<title>Methodology · Decision Index 0.2.1</title>
+<title>Methodology · Decision Index 0.3</title>
 <meta name="description" content="How the Decision Index is built: the frozen suite, every benchmark's subset rule and random baseline, the index formula, the rules every entrant plays by, how entrants are classified, and how latency is measured.">
 <link rel="icon" href="huggies/decision-favicon.png">
 <meta property="og:type" content="website">
-<meta property="og:title" content="Methodology · Decision Index 0.2.1">
+<meta property="og:title" content="Methodology · Decision Index 0.3">
 <meta property="og:description" content="Everything needed to audit the Decision Index: suite, benchmarks, formula, rules, classification, latency.">
 <meta property="og:url" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/methodology.html">
-<meta property="og:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png">
+<meta property="og:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png?v=0.3">
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500&family=Source+Sans+3:ital,wght@0,400;0,600;0,700;1,400&family=Chewy&display=swap" rel="stylesheet">
@@ -53,6 +53,8 @@ h3{font-size:17px;margin:0 0 6px}
 .panel.pad p{margin:0 0 12px;font-size:15.5px}
 .panel.pad p:last-child{margin-bottom:0}
 .prose{max-width:820px}
+.vsgrid{display:grid;grid-template-columns:minmax(0,1.5fr) minmax(0,1fr);gap:26px;align-items:start}.vsfig{margin:0;min-width:0;background:#F3F0E8;border:1.5px solid var(--line);border-radius:12px;padding:12px 14px 6px}.vsfig figcaption{font-family:var(--mono);font-size:11px;text-transform:uppercase;letter-spacing:.08em;color:var(--soft);margin-bottom:6px}.vsfig svg{width:100%;height:auto;display:block}.vsnote{font-family:var(--mono);font-size:10.5px;color:var(--soft);margin:4px 0 0}.mvax{font-family:var(--mono);font-size:11px;fill:var(--soft)}.mvlab{font-size:11.5px;fill:var(--ink);paint-order:stroke;stroke:#F3F0E8;stroke-width:3.5px}
+@media (max-width:900px){.vsgrid{grid-template-columns:1fr}}
 .prose ul,.prose ol{margin:6px 0 12px;padding-left:22px;font-size:15px}
 .prose li{margin:4px 0}
 .note{font-size:14px;color:var(--soft);line-height:1.6;margin:10px 0}
@@ -276,14 +278,34 @@ function gapsSection(){
 function render(){
  const s=D.suite,i=D.index,e=D.edition||{};
  const hw=s.hardware&&s.hardware.label||'';
- const label=e.label||s.label||'Decision Index 0.1',num=(label.match(/\d+(?:\.\d+)+/)||["0.1"])[0],note=s.note||'',subset=!!s.base_edition;
+ const pubLabel=e.label||s.label||'Decision Index 0.1',label='Decision Index 0.3',num=(label.match(/\d+(?:\.\d+)+/)||["0.1"])[0],note=s.note||'',subset=!!s.base_edition;
  document.title=`Methodology · ${label}`;
  const og=document.querySelector('meta[property="og:title"]');if(og)og.content=document.title;
  $('#app').innerHTML=`
  <div class="hero"><h1>Methodology · <mark>Index ${esc(num)}</mark></h1>
   <p>Everything needed to audit the Decision Index: the frozen suite, every benchmark's subset rule and random baseline, the formula, the rules every entrant plays by, how entrants are classified, and how latency is measured.</p>
-  <div class="meta">edition ${esc(e.id)} · ${esc(e.date)} · panel ${esc(e.panel_id)} · Jev ${esc(e.jev_version)} · reproductions on ${esc(hw)}</div>
-  <nav class="toc" aria-label="Sections"><a href="#what">What it is</a><a href="#suite">The suite</a><a href="#benchmarks">Benchmarks</a><a href="#formula">Formula</a><a href="#rules">Rules</a><a href="#entrants">Entrants</a><a href="#latency">Latency</a><a href="#gaps">Unanswered</a><a href="#changelog">Changelog</a>${D.notes?'<a href="#notes">Edition notes</a>':''}<a href="#reproduce">Reproduce</a></nav></div>
+  <div class="meta">${esc(e.label)} · ${esc(e.date)} · panel ${esc(e.panel_id)} · Jev ${esc(e.jev_version)} · reproductions on ${esc(hw)}</div>
+  <nav class="toc" aria-label="Sections"><a href="#v03">New in 0.3</a><a href="#what">What it is</a><a href="#suite">The suite</a><a href="#benchmarks">Benchmarks</a><a href="#formula">Formula</a><a href="#rules">Rules</a><a href="#entrants">Entrants</a><a href="#latency">Latency</a><a href="#gaps">Unanswered</a><a href="#changelog">Changelog</a>${D.notes?'<a href="#notes">Edition notes</a>':''}<a href="#reproduce">Reproduce</a></nav></div>
+
+ <div class="head" id="v03"><h2>New in 0.3: private tests</h2></div>
+ <div class="panel pad prose">
+  <p>Version 0.3 adds private tests. Most public tasks have a private counterpart that tests an equivalent competence, built from fresh data and kept private, so no model can have trained on it. A further group of private tasks from new decision domains checks that a model's skill reaches beyond the task types on the public board.</p>
+  <p>The public index is rebuilt for 0.3: GSM8K is rebuilt, ForecastBench is retired and WinoGrande now counts under Knowledge &amp; Reasoning. Every entrant and Jev is run on the private tests and the new-domain private tasks.</p>
+  <code class="note" style="display:block">Full score = 0.20 × public + 0.50 × private set for same skill + 0.30 × new-domain private tasks, each part on the public scale</code>
+  <p><b>Why these weights.</b> Half the score comes from private tests of the same skills, because that is the part nobody can train on. Almost a third comes from new-domain private tasks, which reward skill that carries beyond the board. The public benchmarks keep a fifth: they are the shared reference everyone can inspect.</p>
+  <p><b>One scale.</b> The three parts spread models out differently, so their raw points are not the same unit. Each part is mapped onto the public scale using the untuned stock models we run as bases: a model one spread above the stock average on the private tests gets the score one spread above the stock average on the public benchmarks. The mapping is the same for every model and never changes the order within a part.</p>
+  <p>Most of the score comes from tests nobody can train on, so a public-only advantage counts for little.</p>
+  <p><b>Ties.</b> Models within 0.9 points of the first model in their group share a rank, shown as =n. That is the gap a fresh sample of the same kinds of questions could produce, 95% of the time, for two neighbouring models on this board.</p>
+  <p>The "Public only" switch re-sorts the board by the public index.</p>
+  <p><b>Vision board</b> (the Vision switch on the index): Full score = 0.5 × public vision benchmarks + 0.5 × the private set for same skill, as chance-corrected skill × 100. The private vision tasks are harder, and stronger models drop more; a stock VLM, Qwen3-VL-8B, drops more than any fine-tune, so a drop alone does not show overfitting. SigLIP 2, an image-text similarity model with no language model, scores near zero, which shows the tasks need a VLM.</p>
+  <p>The sections below describe the public suite that the public index is built on.</p>
+ </div>
+ <div class="head" id="vision-score"><h2>Vision score — public + private benchmarks</h2></div>
+ <div class="panel pad"><div class="vsgrid"><figure class="vsfig"><figcaption>Public vs private set for same skill</figcaption><div id="mvscatter"></div><p class="vsnote">Each dot is a model. On the dashed line, it scores the same on both. References are darker.</p></figure>
+  <div class="prose"><p>Every vision task with a "+" has a private task that measures an equivalent competence. The Full score is half the public vision benchmarks and half the private set.</p>
+  <p>Every model, stock or fine-tuned, scores lower on the private set: the private tasks are harder, and stronger models drop more. <a href="index.html?mode=vision#learn">Learn more on the vision board →</a></p></div></div></div>
+ <div class="head" id="vision-runs"><h2>Vision run notes</h2><p>One line per model where its run needed an adaptation or a reading note.</p></div>
+ <div class="panel pad prose"><ul id="mvnotes"></ul></div>
 
  <div class="head" id="what"><h2>What the Decision Index is</h2></div>
  <div class="panel pad prose">
@@ -361,7 +383,22 @@ function wire(){
   if(tr&&!e.target.closest('a')){const d=t.querySelector(`tr.detail[data-for="${tr.dataset.id}"]`);if(d)d.hidden=!d.hidden}
  });
 }
-fetch('data/methodology.json?v='+Math.floor(Date.now()/6e5)).then(r=>{if(!r.ok)throw new Error(r.status);return r.json()}).then(d=>{D=d;render();if(location.hash){const el=document.querySelector(location.hash);if(el)el.scrollIntoView()}}).catch(err=>{$('#app').innerHTML=`<p class="empty">Could not load data/methodology.json (${esc(err.message)}).</p>`});
+const MVK={'full fine-tune':'#A855F7','LoRA':'#3E8E5E','LoRA + head':'#3E8E5E','head / adapter':'#FF9D00','inference technique':'#3B7CF6'};
+const MVR={'ref-qwen35-397b':'#232323','ref-qwen3vl-8b':'#55524B','ref-siglip2':'#8C8C8C'};
+function drawMV(V){
+ const host=document.getElementById('mvscatter');if(!host||!V)return;
+ const rows=V.entrants.concat(V.refs),W=600,H=420,L=48,R=16,T=12,B=44,hi=Math.ceil(Math.max(...rows.map(r=>Math.max(r.pub,r.priv)))/10)*10;
+ const px=v=>L+v/hi*(W-L-R),py=v=>T+(hi-v)/hi*(H-T-B),e=x=>String(x).replace(/[&<>"]/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
+ let g='';for(let v=0;v<=hi;v+=10)g+=`<line x1="${L}" x2="${W-R}" y1="${py(v)}" y2="${py(v)}" stroke="#E1DCD0"/><text x="${L-8}" y="${py(v)+4}" class="mvax" text-anchor="end">${v}</text><text x="${px(v)}" y="${H-B+16}" class="mvax" text-anchor="middle">${v}</text>`;
+ g+=`<line x1="${px(0)}" y1="${py(0)}" x2="${px(hi)}" y2="${py(hi)}" stroke="#232323" stroke-width="1.4" stroke-dasharray="5 4"/>`;
+ const lab=[];
+ for(const r of rows.slice().sort((a,b)=>(a.ref?1:0)-(b.ref?1:0))){const cx=px(r.pub),cy=py(r.priv),c=r.ref?MVR[r.engine]:(MVK[r.kind]||'#bbb');
+  g+=`<circle cx="${cx}" cy="${cy}" r="${r.ref?7:5}" fill="${c}" stroke="#232323" stroke-width="1.5"><title>${e(r.name)} · public ${r.pub.toFixed(1)} · private ${r.priv.toFixed(1)}</title></circle>`;
+  if(r.ref||r.rank_full===1)lab.push(`<text x="${cx+(r.engine==='ref-siglip2'?10:-10)}" y="${cy+(r.ref?16:-10)}" class="mvlab" text-anchor="${r.engine==='ref-siglip2'?'start':'end'}">${e(r.name)}</text>`)}
+ const nt=document.getElementById('mvnotes');if(nt)nt.innerHTML=V.entrants.filter(r=>r.caveat).map(r=>`<li><b>${e(r.name)}</b>: ${e(r.caveat)}</li>`).join('')+`<li><b>Not on the vision board</b>: ${V.excluded.map(x=>e(x.name)+' ('+e(x.why)+')').join('; ')}.</li>`;
+ host.innerHTML=`<svg viewBox="0 0 ${W} ${H}" role="img" aria-label="Public vision score against private set score">${g}${lab.join('')}<text x="${L}" y="${H-4}" class="mvax">public vision score →</text><text class="mvax" text-anchor="middle" transform="translate(12,${(H-B+T)/2}) rotate(-90)">private set for same skill ↑</text></svg>`;
+}
+fetch('data/methodology.json?v='+Math.floor(Date.now()/6e5)).then(r=>{if(!r.ok)throw new Error(r.status);return r.json()}).then(d=>{D=d;render();fetch('data/vision.json?v='+Math.floor(Date.now()/6e5)).then(r=>r.ok?r.json():null).then(drawMV).catch(()=>{});if(location.hash){const el=document.querySelector(location.hash);if(el)el.scrollIntoView()}}).catch(err=>{$('#app').innerHTML=`<p class="empty">Could not load data/methodology.json (${esc(err.message)}).</p>`});
 </script>
 </body>
 </html>

news.html

@@ -10,13 +10,13 @@
 <meta property="og:title" content="Jev Reproductions Tracker">
 <meta property="og:description" content="Who is rebuilding TypeSafe's Jev in the open? Parallel constrained decoders, trained decision heads, prior-art claims, architecture teardowns and explainers, in one tracker.">
 <meta property="og:url" content="https://huggingface.co/spaces/multimodalart/jev-decision-index">
-<meta property="og:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png">
+<meta property="og:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png?v=0.3">
 <meta property="og:image:width" content="2400">
 <meta property="og:image:height" content="1260">
 <meta name="twitter:card" content="summary_large_image">
 <meta name="twitter:title" content="Jev Reproductions Tracker">
 <meta name="twitter:description" content="Open reproductions of TypeSafe's Jev: decoders, trained heads, prior art, teardowns and explainers.">
-<meta name="twitter:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png">
+<meta name="twitter:image" content="https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/og.png?v=0.3">
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500&family=Source+Sans+3:ital,wght@0,400;0,600;0,700;1,400&family=Chewy&display=swap" rel="stylesheet">

og-index.html

@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html lang="en"><head><meta charset="UTF-8"><title>Decision Index 0.2.1 · social card</title>
+<html lang="en"><head><meta charset="UTF-8"><title>Decision Index 0.3 · social card</title>
 <link href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;500&family=Source+Sans+3:wght@400;600;700&family=Chewy&display=swap" rel="stylesheet">
 <style>
 :root{--bg:#E9E6DD;--surface:#FBF8F1;--ink:#232323;--soft:#55524B;--yellow:#FFD21E;--display:'Chewy',cursive;--body:'Source Sans 3',sans-serif;--mono:'IBM Plex Mono',monospace}
@@ -21,9 +21,9 @@ svg{width:100%;height:auto;display:block;overflow:visible}
 .bar{stroke:var(--ink);stroke-width:1.6}
 .foot{position:absolute;left:40px;right:40px;bottom:12px;font-family:var(--mono);font-size:12px;color:var(--soft);display:flex;justify-content:space-between}
 </style></head><body><div id="stage">
-<div class="lockup"><img src="huggies/decision.png" alt=""><h1>Decision <mark id="ed">Index 0.2.1</mark></h1></div>
+<div class="lockup"><img src="huggies/decision.png" alt=""><h1>Decision <mark id="ed">Index 0.3</mark></h1></div>
 <div class="sub" id="sub"></div>
-<div class="panel"><div class="cap"><span>Decision Index · open reproductions of Jev</span><span class="legend" id="legend"></span></div><div id="bars"></div></div>
+<div class="panel"><div class="cap"><span>Full score · open reproductions of Jev</span><span class="legend" id="legend"></span></div><div id="bars"></div></div>
 <div class="foot"><span id="foot"></span><span>huggingface.co/spaces/multimodalart/jev-decision-index</span></div>
 </div>
 <script>