NLP Interview Questions (2026): Most Asked, With Answers

NLP is the branch of machine learning concerned with getting a model to read, generate, or reason over human language, text or speech, rather than numbers in fixed columns. Text is harder to work with for a few concrete reasons: it's variable length, the same idea can be phrased a hundred different ways, meaning depends heavily on context and word order, and a model has to turn discrete symbols into continuous numeric vectors before any of the usual math applies.

A tabular model can assume a row means the same thing regardless of column order. A language model can't assume that, "dog bites man" and "man bites dog" use the exact same words and the exact same bag-of-words counts, but they mean opposite things.

Tokenization splits raw text into the smaller units, tokens, that a model or algorithm actually operates on. Whitespace splitting breaks down almost immediately: "don't" needs to become "do" and "n't" for most downstream tasks to make sense of the negation, punctuation glued onto a word needs separating, and languages like Chinese or Japanese don't use spaces between words at all.

import re

text = "I can't believe it's already 2026!"
whitespace_split = text.split()
# ["I", "can't", "believe", "it's", "already", "2026!"]

# a real tokenizer separates punctuation and contractions properly
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp(text)
[t.text for t in doc]
# ["I", "ca", "n't", "believe", "it", "'s", "already", "2026", "!"]

Both reduce a word to a base form, but stemming does it by chopping off suffixes with fixed rules, without knowing or caring whether the result is a real word. Lemmatization looks the word up against a vocabulary and grammar rules to return the actual dictionary form, its lemma, based on part of speech.

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

stemmer.stem("studies")   # "studi", not a real word
lemmatizer.lemmatize("studies") # "study", a real word

stemmer.stem("better")    # "better", unchanged
lemmatizer.lemmatize("better", pos="a") # "good"

Stemming is faster and simpler, and good enough for something like search indexing where you just need matches to collapse. Lemmatization costs more compute but produces output that's actually readable, which matters if a human ever sees the result.

Stop words are the high-frequency, low-information words a language uses constantly, "the", "is", "and", "of". Removing them used to be a default step for bag-of-words models, since they add noise and dimensionality without adding much signal for tasks like topic classification.

They should stay in for anything where word order or grammatical structure carries meaning: sentiment analysis, where "not good" and "good" differ entirely by a stop word, machine translation, question answering, or any transformer-based pipeline, where the model's own attention mechanism decides what to weight rather than a hand-written stop list.

One-hot encoding represents each word as a vector with a single 1 and zeros everywhere else, as long as the vocabulary. Every word is equally distant from every other word, "cat" and "dog" are exactly as different as "cat" and "spreadsheet". No relationship between words is captured at all.

An embedding represents each word as a dense, low-dimensional vector of real numbers, typically 100 to 1000 dimensions, learned so that words used in similar contexts end up close together in that vector space. "Cat" and "dog" end up near each other. "Cat" and "spreadsheet" don't.

# one-hot: 50,000-dimensional vector, mostly zeros, no notion of similarity
cat_onehot = [0, 0, 1, 0, 0,...] # length = vocab size

# embedding: 300-dimensional dense vector, learned from context
cat_embedding = [0.21, -0.15, 0.88,...] # length = 300

Both learn embeddings by training a shallow neural network to predict words from their context, then throwing away the prediction task and keeping the learned weight matrix as the embeddings. CBOW, continuous bag-of-words, predicts a target word from the surrounding context words. Skip-gram flips that: it predicts the surrounding context words from a single target word.

# sentence: "the quick brown fox jumps"
# CBOW:   context ["the", "quick", "fox", "jumps"] -> predict "brown"
# skip-gram: target "brown" -> predict each of ["the", "quick", "fox", "jumps"]

Skip-gram tends to do better on rare words since it generates more training pairs from each occurrence, while CBOW trains faster and does slightly better on frequent words. Mikolov's original word2vec paper describes both variants side by side.

A recurrent neural network processes a sequence one token at a time, carrying a hidden state forward from each step to the next, so by the time it reaches the last word it's theoretically accumulated information about everything before it. That matched how language actually works, order matters, and it was a natural fit before better alternatives existed.

hidden = init_hidden()
for token in tokens:
  hidden = rnn_cell(token, hidden) # each step depends on the previous hidden state
# hidden now (in theory) summarizes the whole sequence

BERT uses only the encoder half of the transformer, and it's trained bidirectionally, meaning every token can attend to every other token in the sequence, both before and after it. GPT uses only the decoder half, and it's trained with causal masking, meaning a token can only attend to tokens that came before it, never ahead.

# BERT: bidirectional attention, sees the whole sentence at once
"The [MASK] sat on the mat" # can use "sat", "on", "the", "mat" to predict [MASK]

# GPT: causal attention, only sees what came before
"The cat sat on the" # can only use these words to predict the next one, "mat"

That difference in training direction is why BERT is naturally suited to tasks that need full-sentence understanding, classification, extraction, while GPT is naturally suited to generation, since generation is inherently a left-to-right process anyway.

Masked language modeling randomly hides a percentage of tokens in the input, 15 percent in BERT's original setup, and trains the model to predict the hidden tokens using context from both directions. Causal language modeling trains the model to predict the next token given only everything before it, one token at a time, left to right, never looking ahead.

# masked LM (BERT): 15% of tokens replaced with [MASK], model predicts them using
# context from both sides
input: "the [MASK] sat on the [MASK]"
target: predict "cat" and "mat" using surrounding context

# causal LM (GPT): predict each next token using only prior tokens
input: "the cat sat on the"
target: predict "mat"

The practical consequence: BERT can't generate text token by token the way GPT does, since it was never trained to predict a sequence left to right. GPT can't natively do the kind of bidirectional fill-in-the-blank BERT was built for.

Part-of-speech tagging labels each individual word with its grammatical role: noun, verb, adjective, and so on. Named entity recognition labels spans of one or more words with a real-world category: person, organization, location, date.

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Marie Curie won the Nobel Prize in 1903")

[(t.text, t.pos_) for t in doc]
# [("Marie", "PROPN"), ("Curie", "PROPN"), ("won", "VERB"),...]

[(ent.text, ent.label_) for ent in doc.ents]
# [("Marie Curie", "PERSON"), ("Nobel Prize", "WORK_OF_ART"), ("1903", "DATE")]

Accuracy, the fraction of predictions that are correct, becomes misleading fast when classes are imbalanced. A spam filter where 95 percent of emails are legitimate can hit 95 percent accuracy just by predicting "not spam" every time, while catching zero actual spam.

Precision, recall, and F1 give a fuller picture: precision is how many predicted-positive cases were actually positive, recall is how many actual-positive cases got caught, and F1 is their harmonic mean. For a spam filter you'd usually care most about recall on the spam class, missing spam is worse than the occasional false positive, depending on the product.

RAG retrieves relevant documents from an external knowledge source at query time and feeds them into the model's prompt alongside the user's question, so the model generates its answer grounded in that retrieved text rather than relying purely on what it memorized during training.

Fine-tuning bakes knowledge into the model's weights at training time, which means updating that knowledge later requires retraining, and the model still has no reliable way to say where a fact came from. RAG solves both: update the knowledge base and the model's answers update immediately with no retraining, and the retrieved source documents can be cited directly.

Prompt engineering changes the model's behavior purely by changing the input text, instructions, examples, formatting, with no weight updates at all. Fine-tuning changes the model's actual parameters through additional training, which requires labeled data, compute, and a training pipeline.

Prompt engineering is the right first move for almost anything: it's fast, cheap, and reversible instantly. Fine-tuning earns its cost when you need a consistent output format or style across thousands of calls that prompting alone can't reliably hold, when the task needs domain knowledge the base model doesn't have, or when you need lower latency from a smaller fine-tuned model instead of a large general one carrying a long instruction prompt every call.

A multilingual model is trained on text from many languages at once, usually sharing a single vocabulary and set of weights across all of them, so it learns some structure that transfers between languages. A single-language model's vocabulary, tokenizer, and learned patterns are built entirely from one language's data.

The first thing that breaks with an English-only model on other languages is the tokenizer itself. Its subword vocabulary was built from English text, so non-English words get chopped into far more, far less meaningful pieces, sometimes down to individual characters or bytes, which hurts every downstream task before the model's actual language understanding even gets a chance to help.

Bag-of-words represents a document as raw counts of each word, ignoring order entirely. The problem: common words like "the" or "said" get huge counts in every document and drown out words that actually distinguish one document from another.

TF-IDF, term frequency times inverse document frequency, down-weights a word the more documents it appears in across the whole corpus. A word that shows up in every single document gets an IDF near zero, killing its weight, while a word that's common in one document but rare across the corpus gets weighted up.

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["the cat sat on the mat", "the dog barked at the cat"]
vec = TfidfVectorizer()
matrix = vec.fit_transform(docs)
# "the" gets a low weight since it appears in both documents
# "barked" gets a higher weight since it's specific to one document

A unigram model throws away word order completely, "not good" and "good not" produce identical counts. An n-gram model groups n consecutive words into a single token, so a bigram model treats "not good" as one unit distinct from "good" and "not" on their own, capturing at least local word order and short phrases.

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer(ngram_range=(1, 2))
matrix = vec.fit_transform(["this movie is not good"])
vec.get_feature_names_out()
# includes both unigrams ("good", "not") and bigrams ("not good", "is not")

The tradeoff is dimensionality. Bigrams roughly square the vocabulary size compared to unigrams, and trigrams grow it further, so most real systems cap n at 2 or 3 before the sparsity gets out of hand.

Each document becomes a vector as long as the entire vocabulary, tens of thousands of dimensions for a real corpus, and any single document only uses a tiny fraction of those words. Almost every entry in the vector is zero. That's what "sparse" means here.

Sparsity and high dimensionality together cause two concrete problems: the model needs far more training examples to reliably estimate a weight for every rarely-seen word, and distance metrics like cosine similarity behave strangely in very high-dimensional spaces where most points end up roughly equidistant from each other. It's part of why embeddings, dense vectors of a few hundred dimensions that pack in more information per dimension, replaced bag-of-words for most modern tasks.

Word2vec is a local, predictive method: it slides a context window across the corpus and trains a neural network to predict nearby words, one training example at a time. GloVe, Global Vectors, is a count-based method: it first builds a global word-word co-occurrence matrix across the entire corpus, then factorizes that matrix so the dot product of two word vectors approximates the log of how often those words co-occur.

In practice the resulting embeddings capture similar relationships and perform comparably on most downstream tasks. The real difference is training approach, not output quality: word2vec never explicitly builds a global co-occurrence matrix, GloVe is built directly from one.

Because the training objective pushes words that appear in similar contexts toward similar vector directions, relationships between words end up encoded as consistent vector offsets, not just proximity. The offset from "man" to "king" (royalty) turns out to be roughly the same direction and magnitude as the offset from "woman" to "queen", so subtracting and adding those vectors lands near the right answer.

king - man + woman ≈ queen
paris - france + italy ≈ rome

It tells you the embedding space isn't just clustering similar words together, it's encoding relationships as linear structure. That property doesn't hold perfectly for every analogy, and it degrades for less common word pairs, but it was one of the results that convinced the field embeddings were capturing something real about meaning, not just co-occurrence statistics.

Cosine similarity measures the angle between two vectors, ignoring their magnitude, and ranges from -1 (opposite direction) to 1 (identical direction). Two embeddings that point the same way but have different lengths still score a perfect 1.

import numpy as np

def cosine_sim(a, b):
  return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Embedding vector length often correlates with how frequently a word appeared during training rather than with meaning, so two semantically similar words trained at different frequencies can end up different lengths but pointing the same direction. Euclidean distance would penalize that magnitude difference; cosine similarity ignores it and focuses purely on direction, which is closer to what "similar meaning" actually means in this space.

Backpropagation through an RNN multiplies gradients through every time step going backward, and if those gradient values are consistently less than 1, the product shrinks toward zero the further back you go. The result: early tokens in a long sequence get almost no gradient signal, so the network effectively can't learn long-range dependencies.

An LSTM addresses this with a separate cell state that flows through the sequence with mostly additive updates, gated by learned forget and input gates, rather than being repeatedly multiplied and squashed through a nonlinearity at every step the way a plain RNN's hidden state is. That additive path lets gradients flow back much further before vanishing.

The forget gate outputs a value between 0 and 1 for each dimension of the cell state, and that value gets multiplied against the existing cell state before anything new is added. A value near 0 tells the cell to drop that piece of information; a value near 1 tells it to keep it.

forget_gate = sigmoid(W_f @ [hidden_prev, input] + b_f)
cell_state = forget_gate * cell_state_prev + input_gate * candidate_value

In practice this is how an LSTM learns to hold onto a subject mentioned ten words earlier while discarding irrelevant detail from the sentence in between, something a plain RNN's single hidden state struggles to do.

An early encoder-decoder model compressed an entire input sentence into a single fixed-length vector, then asked the decoder to generate the whole output from that one vector alone. For a short sentence that's fine. For a long sentence, cramming everything into one vector loses detail, and quality degrades noticeably as input length grows.

Attention lets the decoder look back at every encoder hidden state at each generation step, computing a weighted combination that changes depending on what it's currently generating, instead of relying on a single bottleneck vector for the whole sequence. Translation quality on long sentences improved sharply once this was introduced, which is part of why it became the foundation for what came next.

Every token gets projected into three vectors using three separate learned weight matrices: a query, a key, and a value. The query represents what this token is looking for, the key represents what each token (including itself) has to offer, and the value is the actual content that gets passed along once attention decides how much to weight it.

Q = X @ W_q  # what am I looking for
K = X @ W_k  # what does each token offer
V = X @ W_v  # the actual content to mix together

scores = (Q @ K.T) / sqrt(d_k)  # how much should each token attend to every other token
weights = softmax(scores, axis=-1)
output = weights @ V       # weighted mix of values

The dot product between a query and every key produces a relevance score, softmax turns those scores into weights that sum to 1, and the output for each token is a weighted sum of every value vector. That's the entire mechanism: every token gets to look at every other token and decide how much attention to pay to each one.

Pretraining trains a model from scratch (or from randomly initialized weights) on a huge, general corpus using a generic objective like masked or causal language modeling, with no task-specific labels required. It's expensive, often costing millions of dollars in compute, and it's what teaches the model grammar, facts, and general language patterns.

Fine-tuning starts from those already-trained weights and continues training on a much smaller, labeled dataset for a specific task, sentiment classification, question answering, a customer support chatbot. It's dramatically cheaper because the model already understands language; fine-tuning just adjusts it toward a narrower goal.

A word-level tokenizer needs every possible word in its vocabulary ahead of time, and anything not in that list becomes an unknown token, losing all information. Subword tokenization instead builds a vocabulary of frequently occurring character sequences, learned by repeatedly merging the most common adjacent pairs in a training corpus, so any word, even one never seen during training, can be broken down into pieces the tokenizer does recognize.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.tokenize("unhappiness")
# ["un", "##happiness"] or similar subword split, not a single unknown token

tok.tokenize("supercalifragilisticexpialidocious")
# breaks into known subword pieces instead of one giant [UNK]

That's the entire point: a made-up word or a rare proper noun still gets represented as a combination of known pieces instead of a single opaque unknown token that throws away all its meaning.

[CLS] is a special token prepended to the start of every input sequence during BERT's pretraining. Because self-attention lets every token attend to every other token, by the final layer, [CLS]'s hidden state has effectively aggregated information from the entire sequence, giving you a fixed-size vector that represents the whole input regardless of how long it was.

outputs = model(input_ids)
sentence_vector = outputs.last_hidden_state[:, 0, :] # the [CLS] token's vector
logits = classification_head(sentence_vector)

For a classification task you need one fixed-size vector per input regardless of sentence length, and [CLS] is the token BERT was specifically pretrained to use for that purpose, via the next-sentence-prediction objective in the original paper.

Encoder-only (BERT-style) fits tasks where you need to understand and classify existing text: sentiment analysis, named entity recognition, extractive question answering. Decoder-only (GPT-style) fits open-ended generation: chatbots, code completion, free-form writing, where you're producing new text one token at a time with no fixed target length.

Encoder-decoder (T5, the original transformer, BART) fits tasks that transform one sequence into a genuinely different one: translation, where the encoder reads the full source sentence and the decoder generates the target sentence conditioned on it, or abstractive summarization, where the output is a compressed rewrite rather than an extension of the input.

Perplexity measures how "surprised" a language model is by a held-out piece of text, based on the probability it assigns to each next token. Mathematically it's the exponentiated average negative log-likelihood of the test set. Lower perplexity means the model assigned higher probability to the actual next tokens, meaning it modeled that text distribution better.

import torch

# perplexity = exp(average negative log-likelihood over the sequence)
perplexity = torch.exp(loss) # loss is typically cross-entropy loss per token

Perplexity is useful for comparing language models on the exact same test distribution, but it doesn't directly measure whether generated text is factually correct, coherent, or useful for a downstream task. A model can have low perplexity on a domain and still produce fluent, confidently wrong output.

BLEU, introduced for machine translation evaluation, measures n-gram overlap between a generated output and one or more human reference translations, with a penalty for outputs that are too short (Papineni et al., 2002, ACL Anthology). A higher BLEU score means more of the generated n-grams matched a reference.

The biggest weakness: it only rewards matching the exact reference wording, and language allows for many equally correct phrasings. A translation that's perfectly accurate but phrased differently from the reference gets penalized the same as a genuinely wrong translation, which is why BLEU correlates with human judgment less reliably for creative or open-ended generation than it does for narrower translation tasks.

ROUGE measures overlap between a generated summary and a reference summary, but it's recall-oriented by default rather than precision-oriented like BLEU. ROUGE-N measures n-gram overlap, ROUGE-L measures the longest common subsequence, capturing sentence-level structure rather than requiring an exact contiguous n-gram match.

Summarization tends to prioritize recall, did the summary capture the key points from the reference, over precision, which fits ROUGE's design better than BLEU's, which was built for translation where getting every word right in order matters more. Neither metric is a substitute for human judgment on quality, coherence, or factual accuracy, but ROUGE is the more standard automatic metric reported in summarization papers.

Report per-class precision, recall, and F1 rather than a single overall accuracy number, and look at a confusion matrix to see specifically which classes get confused with which. Macro-averaged F1, which averages F1 across classes equally regardless of how many examples each class has, is usually more informative than a single micro-averaged number that lets a large majority class dominate the score.

from sklearn.metrics import classification_report

print(classification_report(y_true, y_pred, target_names=["negative", "neutral", "positive"]))
# shows precision, recall, and F1 per class, plus macro and weighted averages

For a genuinely rare class, like a small "very negative" category buried in mostly neutral reviews, it's worth checking that class's recall specifically, since a model can hit a respectable overall accuracy while essentially never predicting that class correctly.

Documents get split into chunks and each chunk is converted into an embedding vector, stored in a vector database ahead of time. At query time, the user's question is embedded with the same model, and the vector database returns the chunks whose embeddings are most similar to the query embedding. Those retrieved chunks get inserted into the prompt alongside the original question, and the language model generates its answer conditioned on both.

# offline: index the knowledge base
chunks = split_into_chunks(documents)
chunk_embeddings = embed(chunks)
vector_db.add(chunk_embeddings, chunks)

# online: answer a query
query_embedding = embed(user_question)
top_chunks = vector_db.search(query_embedding, k=5)
prompt = build_prompt(user_question, top_chunks)
answer = llm.generate(prompt)

A vector database stores dense embedding vectors and is built to answer nearest-neighbor queries efficiently, given a query vector, find the k stored vectors closest to it by cosine similarity or a similar metric. That's a fundamentally different query pattern than a keyword index, which matches exact terms or their variants.

A keyword search for "car problems" won't reliably return a document about "vehicle issues" since the words don't match. A vector search will, because the embeddings for those two phrases land near each other in vector space even though they share zero words. That's the whole point of semantic search, matching meaning instead of matching exact terms, and it requires an index structure, usually approximate nearest neighbor algorithms like HNSW, built specifically for high-dimensional vector similarity at scale.

Hallucination is when a model generates text that sounds fluent and confident but is factually wrong or entirely made up, a citation that doesn't exist, a statistic that was never in any training data. It happens because a language model is fundamentally predicting plausible next tokens, not looking up verified facts, so a confident-sounding wrong answer and a confident-sounding right answer look identical to the model's training objective.

RAG reduces this by grounding generation in retrieved source text, which gives the model actual facts to draw from instead of relying purely on parametric memory. It doesn't eliminate hallucination because the model can still ignore the retrieved context, misread it, or blend it incorrectly with what it already believes, and if the retrieval step itself returns irrelevant or wrong documents, the model has no way to know that and will confidently generate from bad source material anyway.

Few-shot prompting includes a handful of example input-output pairs directly in the prompt before the actual task, letting the model infer the pattern it's supposed to follow purely from those examples, with no gradient update or retraining involved.

prompt = """
Classify sentiment as positive or negative.

Review: "This product changed my life." Sentiment: positive
Review: "Complete waste of money." Sentiment: negative
Review: "Works exactly as described." Sentiment:
"""
# model completes with "positive", inferred purely from the two examples

Large pretrained models saw an enormous variety of patterns during training, including plenty of examples that look like "here's a task format, now continue it." Few-shot prompting works because it's activating a pattern-completion behavior the model already learned during pretraining, not teaching it something genuinely new.

Embeddings learn statistical patterns purely from the text they're trained on, and if that text reflects real-world social biases, the embeddings encode those same biases as geometric relationships, regardless of whether anyone intended that. This isn't a bug in the training algorithm, it's the algorithm working exactly as designed on biased input data.

A widely cited example: analyses of word2vec embeddings trained on Google News text found the analogy "man is to computer programmer as woman is to homemaker" emerging directly from the vector arithmetic, reflecting gender associations present in the training corpus rather than anything true about the concepts themselves. Downstream systems, resume screening, search ranking, that use these embeddings without correction can quietly propagate that same bias into real decisions.

Most implementations simply truncate the input to fit the context window, silently dropping tokens from the beginning or end, which means the model never sees the dropped portion at all. Some architectures will error out instead of silently truncating, depending on how positional encodings are implemented, since positions beyond the trained range may not have a meaningful learned (or even defined) encoding to fall back on.

Either way, the practical failure mode is the same: information near the edges of a long document quietly disappears from the model's view, and unless a pipeline explicitly checks input length against the model's context limit, the failure can be silent, the model still returns a confident-sounding answer, just one based on incomplete input.

The tokenizer falls back to breaking the unfamiliar string into smaller and smaller known subword pieces until every piece is something in its vocabulary, down to individual characters or bytes if necessary. That's the entire reason subword tokenization exists over word-level tokenization: there's no unknown-token fallback needed for typos or invented words, since the vocabulary is built from small enough units that almost anything can be assembled from pieces it already knows.

tok.tokenize("gooooood")
# something like ["go", "##oo", "##oo", "##od"], broken into recognizable pieces

tok.tokenize("xkqz") # a nonsense string
# falls back toward individual characters or bytes if no larger piece matches

The quality of that breakdown still varies. A typo that happens to align with real subword boundaries gets handled gracefully; a typo that scrambles the word into an unusual character sequence can end up represented in a way that carries very little useful signal for the model, even though technically no token was "unknown."

A static embedding gives "bank" exactly one vector, a blend of every sense it's ever used in during training, river bank and financial bank averaged into the same point in space. That breaks polysemy: a model can't tell "I sat by the bank of the river" from "I deposited a check at the bank" using the word vector alone, since it's identical in both sentences.

# static (word2vec/GloVe): one fixed vector for "bank" regardless of sentence
bank_vector = embeddings["bank"] # same vector every time

# contextual (BERT): a different vector each time, computed from surrounding words
outputs_river = model("I sat by the bank of the river")
outputs_money = model("I deposited a check at the bank")
# the vector for "bank" differs between these two, based on context

Contextual embeddings, from models like BERT or ELMo, compute a word's vector on the fly from its surrounding sentence using self-attention, so the same word gets a different representation depending on what's around it. That's the single biggest reason transformer-based models outperformed static embeddings on almost every task where word sense actually matters.

Every token computes an attention score against every other token, so for a sequence of length n, the attention matrix has n by n entries. Double the input length and the compute and memory required for that attention step quadruple, not double.

# attention matrix shape: (sequence_length, sequence_length)
# 512 tokens -> 262,144 scores
# 2048 tokens -> 4,194,304 scores (16x more, not 4x)

This is exactly why context window length has historically been such an expensive thing to extend, and why techniques like sparse attention, sliding window attention, and FlashAttention (which reduces memory reads and writes rather than the underlying compute) exist specifically to make longer context windows practical without a full quadratic memory blowup.

An RNN processes tokens strictly in order, one at a time, so position is baked into the architecture itself, the model literally sees word 3 after word 2. Self-attention has no such built-in order. It looks at every token simultaneously and computes attention scores based purely on content, so without extra information, "the dog bit the man" and "the man bit the dog" would produce the exact same set of attention scores between matching tokens.

# without positional info, attention treats input as an unordered set of tokens
# positional encoding adds order-dependent values before the first attention layer
pos_encoding[pos, 2*i]  = sin(pos / 10000 ** (2*i / d_model))
pos_encoding[pos, 2*i+1] = cos(pos / 10000 ** (2*i / d_model))
embeddings = token_embeddings + pos_encoding

The original transformer paper added a fixed sinusoidal pattern to each token's embedding based on its position before the first attention layer, giving the model a way to distinguish position without giving up the parallelism that made attention fast in the first place. Later models like BERT use learned positional embeddings instead of the fixed sinusoidal version, and newer architectures use relative positional schemes like RoPE, but the underlying need, giving order-blind attention some sense of sequence, hasn't gone away.

Catastrophic forgetting happens when fine-tuning on a narrow task overwrites weights the model needed for its original, broader capabilities. A model fine-tuned aggressively on customer support tickets can get noticeably worse at general reasoning or writing tasks it handled fine before, because the fine-tuning gradient updates pulled its weights toward the new narrow distribution and away from the broad pretrained one.

Common mitigations: use a low learning rate so updates stay small relative to the pretrained weights, mix a slice of general-purpose data back into the fine-tuning set so the model keeps practicing its broader skills, freeze most of the network and only train a small number of added parameters (the idea behind LoRA and adapters), or use early stopping so training doesn't run long enough to drift too far from the base model.

Full fine-tuning updates every single weight in the model, which for a modern large language model can mean billions of parameters, requiring enormous GPU memory just to store optimizer states for each one. LoRA, low-rank adaptation, freezes the original pretrained weights entirely and instead trains two small low-rank matrices that get added to specific weight matrices at inference time, reducing the trainable parameter count by orders of magnitude.

# full fine-tuning: update the entire weight matrix W (e.g. 4096 x 4096)
W_finetuned = W_pretrained + delta_W # delta_W is the same huge size as W

# LoRA: approximate delta_W as a product of two small matrices
delta_W ≈ A @ B # A: 4096 x 8, B: 8 x 4096, far fewer trainable parameters

The industry moved toward LoRA and similar methods mainly for cost and practicality: it needs far less GPU memory, trains faster, and lets you swap in different task-specific adapters on top of the same frozen base model instead of storing a full separate copy of the model per task. The tradeoff is that full fine-tuning can still edge out LoRA on tasks that need deep, broad changes to the model's behavior, though the gap has narrowed a lot as the technique has matured.

Both metrics measure surface-level overlap with a reference text, not meaning, factual correctness, or coherence. A generated summary can reuse a lot of the reference's exact words and phrases, scoring reasonably on ROUGE, while stitching them together into something that misrepresents the source, contradicts itself, or reads awkwardly. None of that shows up in an n-gram overlap score.

This is exactly why production teams evaluating generative NLP systems increasingly pair automatic metrics with human evaluation, or with a separate model used as a judge, for qualities like factual consistency and fluency that word-overlap metrics were never designed to capture in the first place.

Quantization reduces the numeric precision used to store a model's weights, from 16-bit floating point down to 8-bit or even 4-bit integers, shrinking the model's memory footprint and often speeding up inference, since lower-precision math is cheaper on most hardware.

# rough memory footprint for a 7B parameter model
# FP16: ~14 GB
# INT8: ~7 GB
# INT4: ~3.5 GB

The tradeoff is accuracy loss, since each weight is represented with fewer possible values, and the model's outputs can degrade, more noticeably at INT4 than INT8, and more noticeably on tasks that need fine-grained numerical or logical precision than on general text generation. Modern quantization techniques like GPTQ and AWQ try to minimize that loss by choosing quantization scales carefully per layer rather than applying one blunt scale across the whole model, but the tradeoff between size and fidelity doesn't disappear, it just gets managed better.

Chunk size controls how much text gets embedded and retrieved as a single unit. Too small, and a chunk gets cut off mid-thought, losing the surrounding context needed to understand it correctly, a single sentence pulled out of a paragraph can read as ambiguous or misleading on its own. Too large, and a chunk's embedding becomes an average over many different ideas, diluting the specific signal the retrieval step needs to match against a focused query.

# too small: a chunk like "It increased by 40%." loses what "it" refers to
# too large: a 2,000-word chunk embeds one vector representing five different topics,
# so a query about any single one of them matches weakly against all of them

In practice, most production RAG pipelines land somewhere around a few hundred tokens per chunk with some overlap between adjacent chunks, so a fact sitting near a chunk boundary still appears intact in at least one chunk. The right size still depends heavily on the source material, dense technical documentation and conversational transcripts don't chunk the same way.

Exact-match scores a generated answer as correct only if it's character-for-character identical to a reference answer, which fails immediately for any task where multiple phrasings are equally valid. "The meeting is at 3pm" and "3 PM" and "fifteen hundred hours" can all be correct answers to the same question, and exact-match would mark two of them wrong.

It also breaks down when the reference answer itself is incomplete or when a question has multiple valid correct answers, common in real-world question answering, but rare in the clean benchmark datasets exact-match metrics were originally designed to evaluate. Most modern evaluation setups for generative tasks use a mix of softer metrics (F1 over tokens rather than the full string, semantic similarity scoring, human judgment, or a separate model acting as a judge) precisely because exact-match punishes correct answers that simply weren't phrased exactly like the reference.

Extractive summarization selects existing sentences or phrases directly from the source document and stitches them together, so every word in the output actually appeared somewhere in the input. Abstractive summarization generates new sentences that convey the source's meaning, potentially using words and phrasing that never appeared in the original text at all, closer to how a person would actually summarize something.

Abstractive summarization is harder to evaluate for a specific reason: since the output isn't constrained to reuse the source's exact words, a correct summary can look very different, word-for-word, from any single reference summary a human wrote, which is exactly the scenario where n-gram overlap metrics like ROUGE are weakest. Extractive summaries are easier to grade automatically precisely because they're built from the source's own sentences, so overlap with a reference (built the same way) tends to be a more reliable signal.

Distribution shift shows up constantly with text specifically: training data scraped from formal writing, product descriptions, news articles, encountering real user input full of typos, slang, code-switching between languages, or domain-specific jargon the model never saw. A sentiment model trained on movie reviews often falls apart on short, sarcastic social media posts, a completely different writing style and length distribution.

A second common cause is test set leakage that's easy to miss with text data specifically: near-duplicate examples between train and test sets, common when a dataset is built by scraping overlapping sources, inflate test scores without reflecting genuine generalization. A third: the tokenizer's vocabulary was built from the training domain and handles production text worse than the test set suggested, because the test set happened to share the training domain's vocabulary distribution while real production traffic doesn't. Any of these can produce a strong test score that quietly doesn't transfer, without a single line of the model code actually being wrong.