--- layout: post title: "Natural Language Inference Explained: Entailment in NLP" date: 2026-08-13 15:30:00 +0530 tags: ["NLP", "NLI", "Classification", "BERT", "SNLI", "Transformers"] category: "ai" description: "What is natural language inference (NLI)? How models label a premise and hypothesis as entailment, contradiction, or neutral on SNLI, MNLI, and BERT." keywords: "natural language inference, NLI explained, textual entailment, SNLI dataset, MultiNLI MNLI, hypothesis-only artifacts, BERT NLI, premise hypothesis classification" image: /assets/images/og-natural-language-inference.png order: 16 --- I used to treat natural language inference as a GLUE checkbox. SNLI, MNLI, a 3-class accuracy, move on. Then I sat down and wrote three hypotheses for one Flickr caption the way Bowman et al. asked crowd workers to in 2015, and I felt the cheat in my own hands. For contradiction I reached for *sleeping* and *nobody*. For entailment I stripped the caption until the new sentence was almost a subset of the old one. The label leaked into the wording. That leak is the plot. NLI is supposed to test whether a model can decide if a hypothesis follows from a premise. A lot of the famous numbers measure something cheaper: whether the hypothesis looks like the kind of sentence a worker writes for that label. NLI is still the classification task I reach for when I want to see if a representation can do more than bag-of-words overlap. The job is a pair of sentences in, one of three labels out, and [cross-entropy](/blog/cross-entropy-loss-explained/) on the three logits. Same softmax as any classifier. The interesting part is the pair, and how easy it is to score the pair without actually comparing it. ```mermaid flowchart LR P["premise p"] --> ENC["encoder"] H["hypothesis h"] --> ENC ENC --> Z["3 logits"] Z --> SM["softmax"] SM --> LAB["entailment / contradiction / neutral"] ``` --- ## What is natural language inference? You get two texts. - **Premise** $p$: the context you are allowed to trust. - **Hypothesis** $h$: a claim about that context. The model must pick one label: | Label | Crowd instruction (SNLI) | What I check | |---|---|---| | Entailment | $h$ is definitely true given $p$ | $p$ licenses $h$ | | Contradiction | $h$ is definitely not true given $p$ | $p$ rules $h$ out | | Neutral | $h$ might be true given $p$ | $p$ leaves $h$ open | These are the three-way labels used in SNLI and MultiNLI.[^snli] Older Recognizing Textual Entailment (RTE) datasets were often two-way: entails vs does not.[^rte] Three-way is stricter. "Not entailed" splits into "false" and "not enough information," and that split is where models and humans both get sloppy. > [!NOTE] > "A woman is playing a guitar on a sidewalk" does not prove "someone is performing music" in first-order logic. Crowd instructions ask whether a typical reader of $p$ would say $h$ is definitely true. Informal entailment, not a proof assistant. The examples that made the task click for me are the ones on the SNLI project page, each with five Mechanical Turk labels:[^snli-page] | Premise | Hypothesis | Gold | |---|---|---| | A soccer game with multiple males playing. | Some men are playing a sport. | Entailment (E E E E E) | | A man inspects the uniform of a figure in some East Asian country. | The man is sleeping. | Contradiction (C C C C C) | | A smiling costumed woman is holding an umbrella. | A happy woman in a fairy costume holds an umbrella. | Neutral (N N E C N) | The soccer pair is easy for a reason: *males* $\to$ *men*, *soccer* $\to$ *sport*, *playing* stays *playing*. The sleeping pair shares almost no content with the premise; contradiction here is "this cannot be the same scene." The umbrella pair is the honest one. Three workers said neutral, one said entailment, one said contradiction. *Smiling* vs *happy* is close. *Fairy costume* is an extra detail the premise never gave you. Neutral is the correct majority, and it is also the class with the worst agreement in SNLI (Fleiss' $\kappa$ $0.60$ vs $0.77$ for contradiction).[^snli] If I cannot explain those three rows, I do not understand the task yet. --- ## How was NLI labeled at scale? RTE-style sets were tiny. The SNLI paper's comparison point is "fewer than a thousand examples each."[^snli] You cannot train a neural encoder on that. Bowman's protocol, 2015: 1. Take a Flickr30k image caption as $p$. 2. Ask a crowd worker to write three new sentences: one definitely true given $p$, one definitely false, one neither. 3. On a validation slice, collect four extra labels so each pair has five votes. Gold is a 3-out-of-5 consensus. That produced the Stanford Natural Language Inference corpus. | Split | Pairs | |---:|---:| | Train | 550,152 | | Dev | 10,000 | | Test | 10,000 | | **Total** | **570,152** | Premises average 14.1 tokens; hypotheses 8.3. About 57k pairs were validated: 98% got a three-annotator consensus, 58% were unanimous, overall Fleiss' $\kappa = 0.70$.[^snli] A lexicalized classifier hit **78.2%** test accuracy. A 100-d LSTM encoder hit **77.6%**. Humans writing short captions plus humans writing short hypotheses is a lot of signal, including the wrong kind. MultiNLI (Williams, Nangia, Bowman, 2018) copies the protocol and changes the text.[^mnli] 433k pairs, 392,702 in train, ten genres of written and spoken English. Dev and test are 20k each. Five genres appear in training (**matched**). Five are held out (**mismatched**). Premise length jumps to 22.3 tokens on average. ESIM, which posted 86.7% on SNLI, drops to **72.3% / 72.1%** on MultiNLI matched / mismatched when trained on MultiNLI. Caption English was doing a lot of work. ANLI (Nie et al., 2020) flips the collection loop.[^anli] A human writes a hypothesis meant to fool the current model. If the model is wrong and two verifiers agree with the writer, the pair is kept. Then they train a stronger model and do it again. | Round | Train pairs | What the humans were beating | |---|---:|---| | A1 | 16,946 | BERT-Large trained on SNLI+MNLI | | A2 | 45,460 | a stronger model after A1 | | A3 | 100,459 | another round | | **Total train** | **162,865** | | BERT trained on SNLI+MNLI scores **0.0** on A1 test. That is not a bug. A1 was collected to be the errors of that model. After throwing SNLI+MNLI+FEVER-NLI+ANLI at BERT-Large, test accuracy is 57.4 / 48.3 / 43.5 on A1 / A2 / A3. RoBERTa-Large gets to 73.8 / 48.9 / 44.4. MultiNLI at 86% did not mean the three-way decision was solved. ```mermaid flowchart LR RTE["RTE 2005, hundreds of pairs"] --> SNLI["SNLI 2015, 570k captions"] SNLI --> MNLI["MultiNLI 2018, 10 genres"] MNLI --> ANLI["ANLI 2020, humans vs the model"] ``` --- ## How does a model score a pair? Two families. I keep them separate because they fail in different ways. **Sentence-vector models** encode $p$ and $h$ independently, then mix the two vectors. InferSent (Conneau et al., 2017) is the version I still sketch on paper.[^infersent] A shared BiLSTM-max encoder produces $u$ from the premise and $v$ from the hypothesis. Then three comparisons get concatenated: $$ m = \big[\, u;\; v;\; \lvert u - v \rvert;\; u \odot v \,\big] $$ $u$ and $v$ are the two sentence vectors. $\lvert u - v \rvert$ is a cheap "where do they disagree." $u \odot v$ is a cheap "where do they fire together." Feed $m$ to a small MLP, get three logits, softmax, [cross-entropy](/blog/cross-entropy-loss-explained/). InferSent's BiLSTM-max hits **85.0%** SNLI dev and **84.5%** test. The encoder then transferred to other sentence tasks, which is why people treated SNLI as ImageNet-for-sentences for a couple of years. Independent encoding cannot let a word in $h$ attend to a word in $p$. "The man is sleeping" never gets to look at "inspects the uniform" inside the encoder. All the comparison happens after both sentences have already been crushed into one vector each. **Cross-attention / BERT-style models** put both sentences in one sequence and let every token see the other side. BERT's pair format:[^bert] $$ [\mathrm{CLS}]\; p_1 \ldots p_{n}\; [\mathrm{SEP}]\; h_1 \ldots h_{m}\; [\mathrm{SEP}] $$ Token-type IDs are 0 on the premise side and 1 on the hypothesis side. After the transformer, you take the $[\mathrm{CLS}]$ hidden state $h_{[\mathrm{CLS}]} \in \mathbb{R}^{H}$ and map it to three logits: $$ z = W h_{[\mathrm{CLS}]} + b, \qquad W \in \mathbb{R}^{3 \times H} $$ For BERT-base, $H = 768$, so $W$ is a $3 \times 768$ matrix. That is the entire NLI head. Fine-tuning updates $W$, $b$, and the pretrained stack. BERT-base reaches **84.6 / 83.4** on MultiNLI matched / mismatched. BERT-large reaches **86.7 / 85.9**.[^bert] ```mermaid flowchart LR SEQ["CLS + premise + SEP + hypothesis + SEP"] --> BERT["transformer"] BERT --> CLS["h_CLS"] CLS --> LIN["linear 3 x H"] LIN --> Z["z_E, z_C, z_N"] Z --> SM["softmax"] ``` Cross-attention can align *soccer* with *sport* token-by-token. It can also align *not* in the hypothesis with nothing in the premise and take that as a contradiction cue. Extra mixing gives the cheat a better surface unless the data punishes it. --- ## What does a 3-class softmax look like on one example? Same [softmax + cross-entropy](/blog/cross-entropy-loss-explained/) as any classifier. Walk one pair. True label: entailment. Logits I will pretend a head just emitted: $$ z = [z_E,\, z_C,\, z_N] = [2.0,\, 0.5,\, -0.5] $$ Subtract the max so the exponentials stay small: $$ z - 2.0 = [0,\, -1.5,\, -2.5] $$ Exponentiate, one at a time: $$ e^{0} = 1 $$ $$ e^{-1.5} \approx 0.2231 $$ $$ e^{-2.5} \approx 0.0821 $$ Sum: $$ S = 1 + 0.2231 + 0.0821 = 1.3052 $$ Probabilities: $$ p_E = \frac{1}{1.3052} \approx 0.7662 $$ $$ p_C = \frac{0.2231}{1.3052} \approx 0.1710 $$ $$ p_N = \frac{0.0821}{1.3052} \approx 0.0629 $$ Loss is only the true class: $$ L = -\log p_E = -\log 0.7662 \approx 0.2664 $$ Gradient of softmax-cross-entropy is $p - t$. One-hot target is $[1, 0, 0]$, so $$ \frac{\partial L}{\partial z} = [0.7662 - 1,\; 0.1710,\; 0.0629] = [-0.2338,\; 0.1710,\; 0.0629] $$ The entailment logit gets pulled up. The other two get pushed down, in proportion to the probability they currently hold. If the model had put $p_E = 0.05$ instead, that first component would be $0.05 - 1 = -0.95$, four times steeper. Confidence on the wrong classes is the thing being taxed. ```python import numpy as np z = np.array([2.0, 0.5, -0.5]) z = z - z.max() p = np.exp(z) / np.exp(z).sum() y = 0 loss = -np.log(p[y]) grad = p.copy() grad[y] -= 1.0 ``` {: data-title="nli_head.py" data-lineno="true" data-lines="4-9"} I still run this block on random 3-d logits before I trust a training loop. The numbers have to match `F.cross_entropy` on the same $z$ and $y$. If they do not, the bug is in my head, not in PyTorch. --- ## Why can a model ignore the premise? This is the part I wish I had learned before I memorized SNLI accuracies. Gururangan et al. (NAACL 2018) trained a classifier on the hypothesis alone. No premise.[^artifacts] fastText gets **67.0%** on SNLI vs a 34.3% majority baseline, and **53.9% / 52.3%** on MultiNLI matched / mismatched vs ~35% majority. A model that cannot see $p$ is not doing inference. It is doing stylometry on $h$. The giveaways are visible in PMI(word, class): | Class | SNLI giveaways | MultiNLI giveaways | |---|---|---| | Entailment | *outdoors* (8.0% of class), *animal*, *outside* | *some*, *something*, *yes* | | Neutral | *tall*, *sad*, *favorite*, *first* | *because* (4.1%), *also*, *many* | | Contradiction | *sleeping* (3.2%), *nobody*, *no* | *never* (5.0%), *no* (7.6%), *nothing* | Workers generating a contradiction insert negation. Workers generating an entailment generalize (*dog* $\to$ *animal*, scene $\to$ *outdoors*) and drop specifics. Workers generating a neutral add a purpose clause or an adjective the caption never licensed. 8.8% of SNLI entailment hypotheses are a bag-of-words subset of their premise. 0.2% of neutrals and contradictions are. > [!GOTCHA] > When I wrote my three hypotheses for one caption, I reproduced this table without trying. "A dog is running outdoors" is a legal entailment of a park photo caption, and it is also a sentence that *means entailment* even if you hide the caption. They re-scored strong models on the subset where the hypothesis-only classifier failed (the "Hard" split). ESIM goes from **85.8%** on full SNLI test to **71.3%** on Hard, while Easy sits at 92.6%.[^artifacts] The leaderboard was averaging a style detector with a harder remainder. LLM-written NLI is not a free escape hatch. Proebsting and Poliak (2024) rebuilt a slice of SNLI with GPT-4 / Llama-2 / Mistral writing the hypotheses from the original worker instructions. BERT hypothesis-only accuracy landed between 86% and 96% on those sets. GPT-4 reused *swimming in a pool* in more than 10,000 contradictions.[^llm-nli] Same protocol, faster annotator, louder artifact. So the eval I actually trust is: 1. Full pair accuracy. 2. Hypothesis-only accuracy on the same split. 3. The gap. If (1) is 84 and (2) is 67, the extra 17 points is the part that had to look at $p$. That is the number I quote, not 84. --- ## What did later datasets change? MultiNLI made the sentences longer and the genres messier. The artifact remains, just weaker (53% hypothesis-only, not 67%). ANLI made the label expensive: a pair only enters the set if a strong model already failed it. Hypothesis-only on ANLI-only training still lands in the 40s, so the cheat is reduced, not gone.[^anli] Two other descendants I keep on the same shelf: - **XNLI** (Conneau et al., 2018): MultiNLI-style pairs translated into 15 languages. Same three labels, now a transfer test. If an English NLI head collapses on Swahili, the "understanding" was lexical English, not the decision procedure. - **e-SNLI** (Camburu et al., 2018): SNLI plus a written explanation and highlighted words. Accuracy stays in the same band; you get a second sequence to supervise. I treat the explanation as another string the model can overfit, not as a proof. NLI also leaked into GLUE as **QNLI**: SQuAD converted to "does this paragraph sentence contain the answer?" That is binary entailment with a retrieval flavor. Same head shape, different collection story, same need for a no-context baseline.
Where I actually use this outside a benchmark Fact-checking and RAG-style citation are the same decision with extra steps. Evidence is $p$, claim is $h$, and I want contradiction to fire when the cited paragraph does not support the sentence I just generated. Dagan et al. already framed QA and summarization as entailment problems in the RTE papers. The failure mode is identical: a model that sees the word *not* in $h$ and votes contradiction without reading $p$ will look calibrated until the first negation that is quoting the source.
--- ## Where does this sit next to language modeling? Next-token prediction asks $p(x_t \mid x_{