NLP · Intermediate interview

Word2Vec learns word vectors by predicting context: in skip-gram, given a word it predicts surrounding words; in CBOW, it predicts a word from its context. Training (with tricks like negative sampling) adjusts vectors so words appearing in similar contexts get similar vectors — the distributional hypothesis that meaning comes from usage. The result is static embeddings capturing semantic and analogical relationships.

Static embeddings (Word2Vec, GloVe) assign one fixed vector per word regardless of context, so 'bank' has the same vector in 'river bank' and 'bank account'. Contextual embeddings (from models like BERT/ELMo) produce a different vector for a word depending on the sentence, resolving polysemy and capturing meaning in context. Contextual embeddings dramatically improved most NLP tasks but cost more compute.

Subword tokenization (Byte-Pair Encoding, WordPiece, SentencePiece) splits words into frequently-occurring subword units, so common words stay whole while rare words break into pieces. It gives a fixed, manageable vocabulary, eliminates the out-of-vocabulary problem (any word is representable from its parts), and handles morphology and typos gracefully. It's the standard tokenization for transformer models.

RNNs/LSTMs process text token by token, maintaining a hidden state that accumulates context, making them suited to sequence tasks — classification (using the final state), sequence labeling (output per token), and language modeling. LSTMs handle longer dependencies than vanilla RNNs via gating. They were the workhorses of NLP before transformers, which now dominate due to parallelism and better long-range modeling.

A sequence-to-sequence model maps an input sequence to an output sequence of possibly different length, used for translation, summarization, and question answering. An encoder reads the input into a representation, and a decoder generates the output token by token from it. Early seq2seq used RNNs with a single context vector (a bottleneck), which attention then fixed; transformers are the modern encoder-decoder.

In encoder-decoder models, compressing a whole input into one fixed vector loses information, especially for long sequences. Attention lets the decoder look back at all encoder states and weight the most relevant ones for each output token, dramatically improving translation and long-sequence tasks. It also provides interpretability (which inputs the model focused on) and became the core idea behind transformers.

NER identifies and classifies spans of text into categories like person, organization, location, or date. It's typically framed as sequence labeling, tagging each token with a scheme like BIO (Begin/Inside/Outside an entity), and solved with sequence models (BiLSTM-CRF historically, transformers now). It's foundational for information extraction, search, and knowledge graphs.

Classic approaches use TF-IDF features with a linear model (logistic regression, SVM) or naive Bayes — strong, fast baselines. Neural approaches use embeddings with CNNs/LSTMs, and modern approaches fine-tune a pretrained transformer (BERT) or use its embeddings, usually giving the best accuracy. You choose based on data size, latency needs, and whether a simple baseline suffices.

Options: chunk the document into overlapping windows and aggregate (pooling, voting, or a second model over chunk representations); use a hierarchical model (encode sentences, then combine); use long-context architectures designed for efficiency (Longformer, sparse attention); or, for retrieval/QA, retrieve only the relevant passages instead of feeding everything. The right choice depends on the task and how much of the document truly matters.

BLEU measures n-gram precision overlap between generated and reference text, common in translation. ROUGE measures recall-oriented overlap, common in summarization. Perplexity measures how well a language model predicts text (lower is better) — the exponentiated average negative log-likelihood. BLEU/ROUGE evaluate generation against references (imperfectly); perplexity evaluates a language model intrinsically without references.

With precision, recall, and F1 computed over entity spans, not just tokens — a prediction counts as correct only if the span boundaries and type both match (exact-match), and you average across entity types (micro or macro). This penalizes partial or mislabeled spans appropriately. You use a held-out test set and standard scorers (e.g. seqeval) to compare systems fairly.

Many text tasks are imbalanced (most reviews positive, few entities per token). You use metrics beyond accuracy (F1, PR-AUC), apply class weights in the loss, oversample/augment minority classes (back-translation, synonym replacement, paraphrasing), and possibly adjust thresholds. For rare entities/intents, gathering more labeled examples or using pretrained models that need less data often helps most.