NLP · Advanced interview

Self-attention lets every token attend to every other token, computing context-aware representations via query/key/value dot products and softmax weighting, with multi-head attention capturing different relationship types. This models long-range dependencies directly and processes all positions in parallel, unlike sequential RNNs. Stacked attention plus feed-forward layers, with residuals and normalization, is the transformer block underlying modern NLP.

BERT is an encoder-only model trained with masked language modeling (predict masked tokens using bidirectional context), great for understanding tasks like classification and NER where you encode text. GPT is a decoder-only model trained autoregressively (predict the next token left-to-right), great for generation. The pretraining objective and directionality differ: BERT understands, GPT generates; encoder-decoder models (T5) do both for seq2seq.

Standard self-attention is O(n²) in sequence length for compute and memory, so long contexts are expensive, and models also degrade at using information far back. Approaches include efficient/sparse attention (Longformer, FlashAttention for speed/memory), better positional schemes (RoPE, ALiBi) for length generalization, and retrieval to bring in only relevant context rather than extending the window indefinitely. Long context is a key area of ongoing model improvement.

Prompting (in-context learning) is fastest and cheapest — use it when a capable model can do the task with instructions/examples. RAG adds external, up-to-date, or proprietary knowledge by retrieving relevant documents into the prompt — use it when the model needs facts it wasn't trained on or that change. Fine-tuning adapts the model's weights — use it to teach a specific style, format, or skill, or to improve a smaller model, when you have quality data and prompting/RAG aren't enough. They're often combined.

First, self-supervised pretraining on massive text via next-token prediction, learning broad language and world knowledge. Then alignment: supervised fine-tuning on instruction-response examples, and preference optimization (RLHF or methods like DPO) to make outputs helpful, honest, and safe per human preferences. Pretraining gives capability; alignment shapes behavior. Compute, data scale/quality, and the alignment stage all drive quality.

LLMs generate the most probable continuation, not verified truth — they have no built-in grounding, so they confidently produce plausible-but-wrong content, especially for facts outside their training or beyond their knowledge cutoff. You reduce it with retrieval-augmented generation (ground answers in retrieved sources and cite them), constraining the model to provided context, asking it to express uncertainty or abstain, verification/tool use, and good prompting. You can't eliminate it, so design for verification.

RAG augments an LLM with external knowledge: you chunk and embed a document corpus into a vector store; at query time you embed the question, retrieve the most similar chunks (semantic search, often with reranking), and insert them into the prompt as context for the LLM to answer from, ideally with citations. It grounds responses in current/proprietary data without retraining, reducing hallucination — quality hinges on chunking, retrieval relevance, and prompt construction.

You encode text into dense embedding vectors where semantic similarity corresponds to vector proximity, store them in a vector database, and retrieve by nearest-neighbor search (cosine/dot product) — finding meaning-based matches rather than keyword overlap. Approximate nearest-neighbor indexes (HNSW, IVF) make this fast at scale. It's the retrieval backbone of RAG and semantic search, often combined with keyword search (hybrid) and a reranker for precision.

Reference-overlap metrics (BLEU/ROUGE) are weak for open-ended generation, so you combine: task-specific automatic checks (exact match/F1 for QA, factuality/groundedness checks against sources for RAG), LLM-as-a-judge for scaled qualitative scoring (with care about its biases), and human evaluation for quality, helpfulness, and safety. You also build held-out test sets and monitor live signals. Evaluation is a hard, multi-pronged problem, not a single metric.

Levers include choosing a right-sized model for the task, distillation/quantization to shrink models, caching (responses, embeddings, KV cache), batching requests, streaming tokens for perceived latency, limiting output length, and routing easy queries to cheaper models. For self-hosted models, optimized serving (vLLM, tensor parallelism) helps. You balance quality against latency and per-token cost, often using a smaller model with RAG over a giant model.

Prompt injection is when untrusted input (user text or retrieved/web content) contains instructions that hijack the model — overriding system prompts, exfiltrating data, or misusing tools. Defenses: separate and clearly delimit untrusted content from instructions, never grant the model unchecked tool/data access, validate and constrain outputs, apply least privilege to any actions, use guardrails/filters, and treat all model output and external content as untrusted. It's an unsolved, evolving security problem requiring defense in depth.

Start with prompting plus RAG over domain documents — often enough and cheapest. If you need deeper adaptation, fine-tune (full or parameter-efficient like LoRA) on curated domain data for terminology, style, or tasks, and consider continued pretraining on a domain corpus for big shifts. You also build domain-specific evaluation sets and retrieval. The progression is prompt → RAG → fine-tune, escalating only as needed against cost and data availability.