Natural Language Inference Explained: Entailment in NLP

I used to treat natural language inference as a GLUE checkbox. SNLI, MNLI, a 3-class accuracy, move on. Then I sat down and wrote three hypotheses for one Flickr caption the way Bowman et al. asked crowd workers to in 2015, and I felt the cheat in my own hands. For contradiction I reached for sleeping and nobody. For entailment I stripped the caption until the new sentence was almost a subset of the old one. The label leaked into the wording.

That leak is the plot. NLI is supposed to test whether a model can decide if a hypothesis follows from a premise. A lot of the famous numbers measure something cheaper: whether the hypothesis looks like the kind of sentence a worker writes for that label.

NLI is still the classification task I reach for when I want to see if a representation can do more than bag-of-words overlap. The job is a pair of sentences in, one of three labels out, and cross-entropy on the three logits. Same softmax as any classifier. The interesting part is the pair, and how easy it is to score the pair without actually comparing it.

flowchart LR
  P["premise p"] --> ENC["encoder"]
  H["hypothesis h"] --> ENC
  ENC --> Z["3 logits"]
  Z --> SM["softmax"]
  SM --> LAB["entailment / contradiction / neutral"]

What is natural language inference?

You get two texts.

Premise $p$: the context you are allowed to trust.
Hypothesis $h$: a claim about that context.

The model must pick one label:

Label	Crowd instruction (SNLI)	What I check
Entailment	$h$ is definitely true given $p$	$p$ licenses $h$
Contradiction	$h$ is definitely not true given $p$	$p$ rules $h$ out
Neutral	$h$ might be true given $p$	$p$ leaves $h$ open

These are the three-way labels used in SNLI and MultiNLI.¹ Older Recognizing Textual Entailment (RTE) datasets were often two-way: entails vs does not.² Three-way is stricter. “Not entailed” splits into “false” and “not enough information,” and that split is where models and humans both get sloppy.

[!NOTE] “A woman is playing a guitar on a sidewalk” does not prove “someone is performing music” in first-order logic. Crowd instructions ask whether a typical reader of $p$ would say $h$ is definitely true. Informal entailment, not a proof assistant.

The examples that made the task click for me are the ones on the SNLI project page, each with five Mechanical Turk labels:³

Premise	Hypothesis	Gold
A soccer game with multiple males playing.	Some men are playing a sport.	Entailment (E E E E E)
A man inspects the uniform of a figure in some East Asian country.	The man is sleeping.	Contradiction (C C C C C)
A smiling costumed woman is holding an umbrella.	A happy woman in a fairy costume holds an umbrella.	Neutral (N N E C N)

The soccer pair is easy for a reason: males $\to$ men, soccer $\to$ sport, playing stays playing. The sleeping pair shares almost no content with the premise; contradiction here is “this cannot be the same scene.” The umbrella pair is the honest one. Three workers said neutral, one said entailment, one said contradiction. Smiling vs happy is close. Fairy costume is an extra detail the premise never gave you. Neutral is the correct majority, and it is also the class with the worst agreement in SNLI (Fleiss’ $\kappa$ $0.60$ vs $0.77$ for contradiction).¹

If I cannot explain those three rows, I do not understand the task yet.

How was NLI labeled at scale?

RTE-style sets were tiny. The SNLI paper’s comparison point is “fewer than a thousand examples each.”¹ You cannot train a neural encoder on that.

Bowman’s protocol, 2015:

Take a Flickr30k image caption as $p$.
Ask a crowd worker to write three new sentences: one definitely true given $p$, one definitely false, one neither.
On a validation slice, collect four extra labels so each pair has five votes. Gold is a 3-out-of-5 consensus.

That produced the Stanford Natural Language Inference corpus.

Split	Pairs
Train	550,152
Dev	10,000
Test	10,000
Total	570,152

Premises average 14.1 tokens; hypotheses 8.3. About 57k pairs were validated: 98% got a three-annotator consensus, 58% were unanimous, overall Fleiss’ $\kappa = 0.70$.¹ A lexicalized classifier hit 78.2% test accuracy. A 100-d LSTM encoder hit 77.6%. Humans writing short captions plus humans writing short hypotheses is a lot of signal, including the wrong kind.

MultiNLI (Williams, Nangia, Bowman, 2018) copies the protocol and changes the text.⁴ 433k pairs, 392,702 in train, ten genres of written and spoken English. Dev and test are 20k each. Five genres appear in training (matched). Five are held out (mismatched). Premise length jumps to 22.3 tokens on average. ESIM, which posted 86.7% on SNLI, drops to 72.3% / 72.1% on MultiNLI matched / mismatched when trained on MultiNLI. Caption English was doing a lot of work.

ANLI (Nie et al., 2020) flips the collection loop.⁵ A human writes a hypothesis meant to fool the current model. If the model is wrong and two verifiers agree with the writer, the pair is kept. Then they train a stronger model and do it again.

Round	Train pairs	What the humans were beating
A1	16,946	BERT-Large trained on SNLI+MNLI
A2	45,460	a stronger model after A1
A3	100,459	another round
Total train	162,865

BERT trained on SNLI+MNLI scores 0.0 on A1 test. That is not a bug. A1 was collected to be the errors of that model. After throwing SNLI+MNLI+FEVER-NLI+ANLI at BERT-Large, test accuracy is 57.4 / 48.3 / 43.5 on A1 / A2 / A3. RoBERTa-Large gets to 73.8 / 48.9 / 44.4. MultiNLI at 86% did not mean the three-way decision was solved.

flowchart LR
  RTE["RTE 2005, hundreds of pairs"] --> SNLI["SNLI 2015, 570k captions"]
  SNLI --> MNLI["MultiNLI 2018, 10 genres"]
  MNLI --> ANLI["ANLI 2020, humans vs the model"]

How does a model score a pair?

Two families. I keep them separate because they fail in different ways.

Sentence-vector models encode $p$ and $h$ independently, then mix the two vectors. InferSent (Conneau et al., 2017) is the version I still sketch on paper.⁶ A shared BiLSTM-max encoder produces $u$ from the premise and $v$ from the hypothesis. Then three comparisons get concatenated:

$$ m = \big[\, u;\; v;\; \lvert u - v \rvert;\; u \odot v \,\big] $$

$u$ and $v$ are the two sentence vectors. $\lvert u - v \rvert$ is a cheap “where do they disagree.” $u \odot v$ is a cheap “where do they fire together.” Feed $m$ to a small MLP, get three logits, softmax, cross-entropy. InferSent’s BiLSTM-max hits 85.0% SNLI dev and 84.5% test. The encoder then transferred to other sentence tasks, which is why people treated SNLI as ImageNet-for-sentences for a couple of years.

Independent encoding cannot let a word in $h$ attend to a word in $p$. “The man is sleeping” never gets to look at “inspects the uniform” inside the encoder. All the comparison happens after both sentences have already been crushed into one vector each.

Cross-attention / BERT-style models put both sentences in one sequence and let every token see the other side.

BERT’s pair format:⁷

$$ [\mathrm{CLS}]\; p_1 \ldots p_{n}\; [\mathrm{SEP}]\; h_1 \ldots h_{m}\; [\mathrm{SEP}] $$

Token-type IDs are 0 on the premise side and 1 on the hypothesis side. After the transformer, you take the $[\mathrm{CLS}]$ hidden state $h_{[\mathrm{CLS}]} \in \mathbb{R}^{H}$ and map it to three logits:

$$ z = W h_{[\mathrm{CLS}]} + b, \qquad W \in \mathbb{R}^{3 \times H} $$

For BERT-base, $H = 768$, so $W$ is a $3 \times 768$ matrix. That is the entire NLI head. Fine-tuning updates $W$, $b$, and the pretrained stack. BERT-base reaches 84.6 / 83.4 on MultiNLI matched / mismatched. BERT-large reaches 86.7 / 85.9.⁷

flowchart LR
  SEQ["CLS + premise + SEP + hypothesis + SEP"] --> BERT["transformer"]
  BERT --> CLS["h_CLS"]
  CLS --> LIN["linear 3 x H"]
  LIN --> Z["z_E, z_C, z_N"]
  Z --> SM["softmax"]

Cross-attention can align soccer with sport token-by-token. It can also align not in the hypothesis with nothing in the premise and take that as a contradiction cue. Extra mixing gives the cheat a better surface unless the data punishes it.

What does a 3-class softmax look like on one example?

Same softmax + cross-entropy as any classifier. Walk one pair.

True label: entailment. Logits I will pretend a head just emitted:

$$ z = [z_E,\, z_C,\, z_N] = [2.0,\, 0.5,\, -0.5] $$

Subtract the max so the exponentials stay small:

$$ z - 2.0 = [0,\, -1.5,\, -2.5] $$

Exponentiate, one at a time:

$$ e^{0} = 1 $$

$$ e^{-1.5} \approx 0.2231 $$

$$ e^{-2.5} \approx 0.0821 $$

Sum:

$$ S = 1 + 0.2231 + 0.0821 = 1.3052 $$

Probabilities:

$$ p_E = \frac{1}{1.3052} \approx 0.7662 $$

$$ p_C = \frac{0.2231}{1.3052} \approx 0.1710 $$

$$ p_N = \frac{0.0821}{1.3052} \approx 0.0629 $$

Loss is only the true class:

$$ L = -\log p_E = -\log 0.7662 \approx 0.2664 $$

Gradient of softmax-cross-entropy is $p - t$. One-hot target is $[1, 0, 0]$, so

$$ \frac{\partial L}{\partial z} = [0.7662 - 1,\; 0.1710,\; 0.0629] = [-0.2338,\; 0.1710,\; 0.0629] $$

The entailment logit gets pulled up. The other two get pushed down, in proportion to the probability they currently hold. If the model had put $p_E = 0.05$ instead, that first component would be $0.05 - 1 = -0.95$, four times steeper. Confidence on the wrong classes is the thing being taxed.

import numpy as np

z = np.array([2.0, 0.5, -0.5])
z = z - z.max()
p = np.exp(z) / np.exp(z).sum()
y = 0
loss = -np.log(p[y])
grad = p.copy()
grad[y] -= 1.0

I still run this block on random 3-d logits before I trust a training loop. The numbers have to match F.cross_entropy on the same $z$ and $y$. If they do not, the bug is in my head, not in PyTorch.

Why can a model ignore the premise?

This is the part I wish I had learned before I memorized SNLI accuracies.

Gururangan et al. (NAACL 2018) trained a classifier on the hypothesis alone. No premise.⁸ fastText gets 67.0% on SNLI vs a 34.3% majority baseline, and 53.9% / 52.3% on MultiNLI matched / mismatched vs ~35% majority. A model that cannot see $p$ is not doing inference. It is doing stylometry on $h$.

The giveaways are visible in PMI(word, class):

Class	SNLI giveaways	MultiNLI giveaways
Entailment	outdoors (8.0% of class), animal, outside	some, something, yes
Neutral	tall, sad, favorite, first	because (4.1%), also, many
Contradiction	sleeping (3.2%), nobody, no	never (5.0%), no (7.6%), nothing

Workers generating a contradiction insert negation. Workers generating an entailment generalize (dog $\to$ animal, scene $\to$ outdoors) and drop specifics. Workers generating a neutral add a purpose clause or an adjective the caption never licensed. 8.8% of SNLI entailment hypotheses are a bag-of-words subset of their premise. 0.2% of neutrals and contradictions are.

[!GOTCHA] When I wrote my three hypotheses for one caption, I reproduced this table without trying. “A dog is running outdoors” is a legal entailment of a park photo caption, and it is also a sentence that means entailment even if you hide the caption.

They re-scored strong models on the subset where the hypothesis-only classifier failed (the “Hard” split). ESIM goes from 85.8% on full SNLI test to 71.3% on Hard, while Easy sits at 92.6%.⁸ The leaderboard was averaging a style detector with a harder remainder.

LLM-written NLI is not a free escape hatch. Proebsting and Poliak (2024) rebuilt a slice of SNLI with GPT-4 / Llama-2 / Mistral writing the hypotheses from the original worker instructions. BERT hypothesis-only accuracy landed between 86% and 96% on those sets. GPT-4 reused swimming in a pool in more than 10,000 contradictions.⁹ Same protocol, faster annotator, louder artifact.

So the eval I actually trust is:

Full pair accuracy.
Hypothesis-only accuracy on the same split.
The gap. If (1) is 84 and (2) is 67, the extra 17 points is the part that had to look at $p$. That is the number I quote, not 84.

What did later datasets change?

MultiNLI made the sentences longer and the genres messier. The artifact remains, just weaker (53% hypothesis-only, not 67%). ANLI made the label expensive: a pair only enters the set if a strong model already failed it. Hypothesis-only on ANLI-only training still lands in the 40s, so the cheat is reduced, not gone.⁵

Two other descendants I keep on the same shelf:

XNLI (Conneau et al., 2018): MultiNLI-style pairs translated into 15 languages. Same three labels, now a transfer test. If an English NLI head collapses on Swahili, the “understanding” was lexical English, not the decision procedure.
e-SNLI (Camburu et al., 2018): SNLI plus a written explanation and highlighted words. Accuracy stays in the same band; you get a second sequence to supervise. I treat the explanation as another string the model can overfit, not as a proof.

NLI also leaked into GLUE as QNLI: SQuAD converted to “does this paragraph sentence contain the answer?” That is binary entailment with a retrieval flavor. Same head shape, different collection story, same need for a no-context baseline.

Where I actually use this outside a benchmark

Fact-checking and RAG-style citation are the same decision with extra steps. Evidence is $p$, claim is $h$, and I want contradiction to fire when the cited paragraph does not support the sentence I just generated. Dagan et al. already framed QA and summarization as entailment problems in the RTE papers. The failure mode is identical: a model that sees the word not in $h$ and votes contradiction without reading $p$ will look calibrated until the first negation that is quoting the source.

Where does this sit next to language modeling?

Next-token prediction asks $p(x_t \mid x_{<t})$. NLI asks $p(y \mid p, h)$ with $y \in {E, C, N}$. Different $y$. Same ingredients.

Token embeddings turn each side into vectors.
Attention is how BERT lets $h$ tokens read $p$ tokens in one sequence.
Cross-entropy is the loss on the three logits. There is no special NLI loss.

I got this backwards for a while. I thought NLI was a semantic capability you inspect after pretraining, like a vibe. It is a supervised 3-way classifier. The capability claim lives or dies on whether the classifier used both sentences. Hypothesis-only accuracy is the control experiment. I run it on day one now, before I get attached to a number on SNLI.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. EMNLP. 570,152 pairs; train/dev/test 550,152 / 10,000 / 10,000; lexicalized classifier 78.2%; 100-d LSTM 77.6%; Fleiss’ $\kappa$ 0.70. ↩ ↩² ↩³ ↩⁴
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL Recognising Textual Entailment challenges (from 2005). Two-way entailment, hundreds of pairs per year. SNLI’s scale argument is aimed at this line of datasets. ↩
The SNLI corpus. The soccer / sleeping / umbrella triples are from their development examples. ↩
Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. NAACL. 433k pairs; 392,702 train; ESIM 72.3% / 72.1% matched / mismatched. ↩
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI. ACL. Train sizes from their label-split table; BERT-Large S+M scores 0.0 on A1; S+M+F+ANLI test 57.4 / 48.3 / 43.5. ↩ ↩²
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. EMNLP. Matching vector $[u; v; \lvert u-v \rvert; u \odot v]$; BiLSTM-max 84.5% SNLI test. ↩
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT. NAACL. GLUE Table 1: BERT-base MNLI 84.6 / 83.4, BERT-large 86.7 / 85.9. ↩ ↩²
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. NAACL. Hypothesis-only fastText 67.0% SNLI, 53.9% / 52.3% MultiNLI; ESIM SNLI Hard 71.3% vs 85.8% full. ↩ ↩²
Grace Proebsting and Adam Poliak. 2024. Hypothesis-only biases in large language model-elicited natural language inference. BERT hypothesis-only 86–96% on LLM-written SNLI-style data; GPT-4 swimming in a pool repetition is their count. ↩