Zero-Shot Text Classification in 2026: NLI, Embeddings, Rerankers, or LLMs?

TL;DR

We benchmark 35 zero-shot text classification models across 22 datasets, spanning NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs.

• 🔀 Modern rerankers perform best overall. Qwen3-Reranker-8B reaches a macro F1 of 0.72 and tops the benchmark.
• ⚡ Embedding models offer the strongest accuracy–efficiency trade-off. Models such as GTE-large-en-v1.5 reach performance comparable to strong NLI cross-encoders while being considerably faster at inference.
• 🤖 LLMs become highly competitive at moderate-to-large scale. Models in the 4–12B range reach up to 0.67 macro F1 and are particularly strong on topic classification.
• 📈 Scaling behaves very differently across architectures. Rerankers and LLMs improve substantially with scale, while embedding models tend to plateau much earlier.

🤗 Live Leaderboard · 📄 ICLR 2026 Paper · 💻 GitHub · 📊 Dataset

Zero-shot text classification used to have a fairly obvious default.

Take a model fine-tuned on Natural Language Inference (NLI), turn each candidate label into a hypothesis, and select the label with the highest entailment score. Models such as facebook/bart-large-mnli popularized this approach and it remains the basis of the familiar zero-shot classification pipeline in 🤗 Transformers.

But the model landscape has changed considerably.

Modern embedding models can compare documents and label descriptions directly in a shared representation space. Rerankers can score every document–label pair for relevance. And instruction-tuned LLMs can treat classification as a multiple-choice problem.

All three model families have improved rapidly, making the choice of a zero-shot classifier much less obvious than it used to be.

This leaves a simple practical question:

If you want to do zero-shot text classification today, should you use an NLI model, an embedding model, a reranker, or an LLM?

That is the question we built BTZSC — Benchmarking Text Zero-Shot Classification — to investigate.

The zero-shot classification landscape has changed

Despite the growing number of ways to perform zero-shot classification, comparing them is surprisingly difficult.

The four model families solve essentially the same problem in quite different ways:

• NLI cross-encoders score whether a candidate label is entailed by the input text.
• Embedding models independently encode the text and label descriptions and compare them using similarity.
• Rerankers treat the text as a query and candidate labels as documents to be ranked by relevance.
• Instruction-tuned LLMs receive the text and candidate labels directly and select among them using next-token probabilities.

These differences matter not only for accuracy, but also for inference speed and scaling behavior. An embedding model can encode a document once and compare it against many candidate labels, whereas a cross-encoder or reranker generally processes document–label pairs jointly. Autoregressive LLMs introduce yet another computational profile.

At the same time, existing benchmarks do not give a clean answer to which approach works best under the same zero-shot conditions. MTEB's classification setup, for example, has traditionally relied on supervised linear probes on top of frozen embeddings. Other zero-shot evaluations have tended to focus on one model family, a narrow set of tasks, or a relatively small number of datasets.

So rather than asking whether a particular model is good at zero-shot classification, we wanted to ask a broader question:

What happens when the major model families are evaluated on exactly the same classification tasks, using the same label descriptions and without task-specific labeled examples?

BTZSC provides that comparison.

BTZSC: putting four model families on equal footing

BTZSC contains 22 English-language datasets spanning four common classification settings:

• Sentiment — from product reviews and financial text to movie reviews
• Topic — including news, political text, Wikipedia content, and social media
• Intent — ranging from banking queries to assistant-style commands
• Emotion — including both short social-media text and longer conversational data

The datasets vary substantially in difficulty. Some tasks are binary, while others contain dozens of possible labels — such as Banking77 with 77 intents and MASSIVE with 59. Documents range from very short utterances to texts hundreds of tokens long.

The ICLR study evaluates 35 zero-shot classifiers, together with three untuned base-encoder controls, for 38 models in total. Our focus is on publicly available models that can realistically be deployed locally and at high throughput, rather than frontier proprietary APIs or very large language models.

The zero-shot models cover four families:

NLI cross-encoders → Embedding models → Rerankers → Instruction-tuned LLMs

The important part is that every family is evaluated on the same underlying task: match an input document to a set of human-readable label descriptions without using labeled examples from the target task for fine-tuning or model selection.

For example, a sentiment label such as positive is not presented simply as a single word. It is verbalized in context:

The overall sentiment within the Amazon product review is positive.

How document–label compatibility is computed then depends on the architecture. NLI models use entailment scores, embedding models use cosine similarity, rerankers score document–label relevance, and LLMs select among the candidate labels using next-token probabilities.

To compare datasets with very different numbers of classes and class distributions, we use macro F1 as the primary metric, giving equal weight to every class.

This gives us a common test bed for asking not only which model performs best, but also how architecture, scale, task type, and inference speed affect zero-shot classification.

Rerankers come out on top

The clearest result from BTZSC is that modern rerankers are extremely strong zero-shot classifiers.

Historically, rerankers have been designed for information retrieval: given a query and a set of candidate documents, they assign a relevance score to each candidate.

But zero-shot classification has essentially the same structure if we reinterpret the input text as the query and the candidate label descriptions as the documents to be ranked.

That simple reframing works surprisingly well.

The strongest model in BTZSC, Qwen3-Reranker-8B, reaches an average macro F1 of 0.72, making it the best-performing model in the benchmark.

For comparison:

• the strongest NLI cross-encoder reaches 0.60
• the strongest embedding model reaches 0.62
• the strongest instruction-tuned LLM reaches 0.67

What is particularly interesting is that this is not purely a scale effect.

The much smaller Qwen3-Reranker-0.6B already reaches 0.61 macro F1, slightly outperforming every NLI cross-encoder in the benchmark.

Not every reranker is strong. The older ms-marco-MiniLM-L6-v2, for example, reaches only 0.42 F1. The gains therefore appear to reflect advances in modern reranker training rather than the reranking architecture alone.

Still, the overall pattern is striking: models trained to answer something close to

How relevant is this candidate description to this text?

appear to transfer extremely well to zero-shot classification.

Scaling helps some architectures much more than others

The results become even more interesting when we look at model size.

Rerankers show a clear improvement with scale. Moving from Qwen3-Reranker-0.6B to Qwen3-Reranker-8B increases macro F1 from 0.61 to 0.72.

Instruction-tuned LLMs also benefit strongly from scale. Very small models struggle: Gemma-3-270M reaches only 0.28 F1, while Gemma-3-1B reaches 0.36. Performance improves substantially in the multi-billion-parameter regime, with Qwen3-4B at 0.65, Qwen3-8B at 0.66, and Mistral-Nemo-12B at 0.67.

Embedding models follow a different pattern.

Modern embeddings improve rapidly at smaller scales, but then largely plateau around 0.60–0.62 F1. Scaling Qwen3-Embedding-0.6B to Qwen3-Embedding-8B, for example, changes average F1 only from 0.58 to 0.59.

Simply asking “How large is the model?” therefore misses an important part of the story.

Model family matters at least as much as model size.

An 8B embedding model does not automatically outperform an embedding model with a few hundred million parameters. A 0.6B reranker can outperform considerably larger LLMs. And even within the LLM family, parameter count alone is not enough: Qwen3-4B reaches 0.65 F1, while similarly sized Llama-3.2-3B-Instruct and Phi-4-mini remain around 0.43.

The scaling curve depends heavily on what the model has actually been trained to do.

But no model wins everywhere

The overall leaderboard hides another important result: the best architecture depends on the type of classification task.

Rerankers dominate sentiment and intent classification. Qwen3-Reranker-8B reaches 0.92 F1 on sentiment and 0.70 on intent, the strongest results in both categories.

LLMs are particularly strong on topic classification. Mistral-Nemo-Instruct-2407 achieves the best topic score in the benchmark at 0.69 F1, followed closely by Qwen3-Reranker-8B at 0.66 and Qwen3-8B at 0.65.

Emotion classification tells a slightly different story again. Here, the strongest embedding models are highly competitive: Qwen3-Embedding-8B reaches 0.51 F1, narrowly ahead of e5-mistral-7b-instruct at 0.50 and Qwen3-Reranker-8B at 0.49.

This task dependence is easy to miss when looking only at an average benchmark score.

There is also substantial variation within model families. Qwen3-4B, for example, performs dramatically better than similarly sized Llama-3.2-3B-Instruct and Phi-4-mini. Likewise, newer rerankers such as Qwen3 are far stronger than older retrieval models such as ms-marco-MiniLM-L6-v2.

The practical takeaway is therefore not that one architecture has replaced all the others.

Rather:

Model family provides a strong prior for what will work well, but task type and training recipe still matter.

Are these rankings specific to BTZSC?

One natural concern with any benchmark is whether the conclusions depend heavily on its particular dataset mix.

As a robustness check, we re-evaluated the models on the eight English classification datasets from MTEB v2, using the same zero-shot setup.

The resulting model rankings are strongly aligned with BTZSC: the Kendall rank correlation is τ = 0.69. Qwen3-Reranker-8B is the strongest individual model on both suites, and the broad family-level picture remains similar.

That does not make any benchmark universal, but it gives us some confidence that the main results are not simply an artifact of the particular 22 datasets selected for BTZSC.

Embeddings hit the accuracy–efficiency sweet spot

Accuracy is only part of the story. In practice, a zero-shot classifier also needs to be fast enough to use at scale.

When we compare BTZSC performance with inference speed, a clear pattern emerges: embedding models tend to offer the best balance between accuracy and speed.

This follows naturally from how they operate. An embedding model encodes the input text and each candidate label independently, after which classification reduces to a similarity computation. When many documents are repeatedly classified against the same label set, the label embeddings can also be computed once and reused.

Strong models such as GTE-large-en-v1.5 reach 0.62 macro F1, essentially matching the strongest NLI cross-encoders while occupying a much more favorable part of the performance–latency space.

Rerankers make a different trade-off. They generally achieve higher peak accuracy, but document–label pairs need to be scored jointly. The strongest reranker, Qwen3-Reranker-8B, therefore sits well above the embedding models in accuracy but at substantially lower inference speed.

One notable exception is gte-reranker-modernbert-base, which combines competitive performance with relatively strong efficiency.

Large instruction-tuned LLMs occupy yet another regime. Models such as Qwen3-8B and Mistral-Nemo are accurate, but in our setup they are considerably slower than the strongest embedding alternatives.

The resulting picture is less about identifying one universally optimal model and more about identifying a Pareto frontier.

If maximum zero-shot accuracy is the priority, a modern reranker is currently the strongest choice in BTZSC.

If throughput matters, embeddings become much harder to beat.

The model with the highest benchmark score is not necessarily the model you would want to deploy.

So which model should you actually use?

The results suggest a useful set of rules of thumb.

Choose a modern reranker if accuracy is the main objective.
Qwen3-Reranker-8B is the strongest model in BTZSC overall and performs particularly well on sentiment and intent classification.

Choose an embedding model if you need strong performance at high throughput.
Models such as GTE-large-en-v1.5 come close to the strongest non-generative alternatives while offering a much better inference-speed trade-off. They are particularly attractive when the candidate label set can be reused across many documents.

Consider an instruction-tuned LLM when topic classification is central or you are already operating in an LLM-based pipeline.
The strongest 4–12B models are highly competitive, and LLMs perform particularly well on topic classification, although at substantially higher inference cost than strong embeddings.

Keep NLI cross-encoders as strong, established baselines.
They remain competitive, especially relative to older embedding and reranking models, but the results suggest more limited gains from additional scale than we see for rerankers and LLMs.

The broader point is that architecture should be treated as part of the model-selection problem.

If your zero-shot pipeline still defaults to an NLI model simply because that has historically been the standard approach, it is worth revisiting that choice.

And if you are unsure:

Benchmark candidate models on the task distribution that actually matters to you.

Benchmark your own model

BTZSC is not just a static benchmark from the paper. The full evaluation harness is available as a Python package, so you can run the same evaluation on your own Hugging Face model.

Install it with:

pip install btzsc

For example, evaluating intfloat/e5-base-v2 is only a few lines of code:

from btzsc import BTZSCBenchmark

benchmark = BTZSCBenchmark(tasks=["sentiment", "topic"])

results = benchmark.evaluate(
    model="intfloat/e5-base-v2",
    model_type="embedding",
    batch_size=64,
)

print(results.summary())

BTZSC currently provides adapters for all four model families used in the benchmark:

embedding · nli · reranker · llm

You can evaluate a subset of task families while experimenting, or run the complete benchmark across all 22 datasets.

For a quick smoke test, you can also limit the number of examples:

results = benchmark.evaluate(
    model="intfloat/e5-base-v2",
    model_type="embedding",
    max_samples=200,
)

Once you are happy with the setup, results can be exported directly in the format expected by the public leaderboard:

results.to_json("results/embedding/e5-base-v2.json")

And validated locally before submission:

btzsc validate-result results/embedding/e5-base-v2.json

The package also supports custom model adapters, so models that do not yet have a built-in BTZSC integration can still be evaluated under the same protocol.

For the full API, CLI, and submission instructions, see the BTZSC GitHub repository.

A living leaderboard

The results in the ICLR paper provide a snapshot of the model landscape at the time of evaluation.

But zero-shot models are evolving quickly, so we also maintain a live BTZSC leaderboard on Hugging Face.

👉 Explore the live BTZSC leaderboard

New models can be evaluated using the same package and submitted to the btzsc/btzsc-results repository. Valid submissions are then incorporated into the public leaderboard.

This is where we hope BTZSC becomes more useful over time.

Rather than treating the 35 zero-shot models in the paper as a final ranking, the goal is to provide a common evaluation framework as new embedding models, rerankers, NLI models, and LLMs are released.

Think we are missing an important zero-shot classifier? Benchmark it with BTZSC and add it to the leaderboard.

Limitations and what's next

BTZSC is intended as a broad comparison of modern zero-shot classifiers, but there are several important boundaries to the benchmark.

First, BTZSC currently focuses on English, single-label classification. Extending the benchmark to multilingual and multi-label settings is therefore a natural next step.

Second, as with most benchmarks built from public datasets, we cannot guarantee that every evaluation example is completely absent from every model's pretraining data. We exclude models that explicitly report training on BTZSC datasets as supervised targets and inspect publicly documented training data where possible, but undocumented overlap may still exist.

The current study deliberately focuses on models that remain practical for local, high-throughput deployment. For instruction-tuned LLMs, we therefore cap the comparison at 12B parameters rather than evaluating frontier proprietary APIs or much larger open-weight models.

Finally, the way labels are expressed in natural language is itself an important part of zero-shot classification. BTZSC uses fixed, contextualized label descriptions across models to make comparisons consistent, but improved verbalizations and prompting strategies are an obvious direction for future work.

Going forward, we would like BTZSC to evolve along with the model landscape: adding new architectures and checkpoints to the leaderboard, extending beyond English, and studying how label descriptions and prompts interact with the different model families.

Conclusion

Zero-shot text classification no longer has an obvious default architecture.

BTZSC shows that modern rerankers can outperform the traditional NLI approach by a substantial margin, while embedding models offer an especially attractive option when inference speed matters. LLMs are increasingly competitive as they scale, but the results also make clear that larger does not automatically mean better.

More broadly, the benchmark suggests that how a model is trained to compare texts matters at least as much as how large it is.

The leaderboard is open, the evaluation harness is public, and the model landscape will keep changing.

Have a model that should be on the leaderboard? Run it on BTZSC and submit your results.

🤗 Live Leaderboard · 📄 ICLR 2026 Paper · 💻 GitHub · 📊 Dataset