NLP · Beginner interview

Natural Language Processing is the field of getting computers to understand, interpret, and generate human language. It's hard because language is ambiguous (words have multiple meanings), context-dependent, full of irregularity, sarcasm, and idioms, and varies across people and domains. Unlike structured data, text is unstructured and rich in nuance, which is why NLP needs specialized representations and models.

Tokenization splits text into smaller units (tokens) — words, subwords, or characters — that a model can process. For example, a sentence becomes a list of word tokens. It's the first step in most NLP pipelines, and the choice of granularity matters: word tokens are intuitive but have huge vocabularies and out-of-vocabulary issues, which subword tokenization addresses.

Typical steps include lowercasing, removing punctuation/special characters, tokenization, removing stop words (common words like 'the'), and normalizing words via stemming (chopping to a root, crudely) or lemmatization (mapping to a dictionary base form, more accurate). The goal is to reduce noise and variation so the representation focuses on meaningful content — though modern transformer pipelines do less of this.

Bag-of-words represents a document as counts of each word, ignoring order — simple but loses context and produces large sparse vectors. TF-IDF weights those counts by how informative a word is: term frequency times inverse document frequency, so common words across all documents are down-weighted and distinctive words up-weighted. Both are classic, interpretable baselines for text classification and search.

Word embeddings are dense, low-dimensional vectors representing words, learned so that words used in similar contexts have similar vectors — capturing meaning and relationships (famously, king − man + woman ≈ queen). Unlike sparse one-hot or bag-of-words vectors, they encode semantic similarity and generalize better. Word2Vec and GloVe are classic examples.

One-hot vectors are huge (one dimension per vocabulary word), sparse, and treat every word as equally different — there's no notion that 'cat' and 'dog' are related. Embeddings are compact and place similar words close together, so models can generalize from one word to related ones and need fewer parameters. This semantic structure is why embeddings dramatically improved NLP.

Text classification (sentiment analysis, spam detection, topic labeling), named entity recognition (finding people/places/orgs), part-of-speech tagging, machine translation, question answering, summarization, and text generation. Many are framed as classification or sequence labeling. Knowing which task your problem is helps choose the right model and metric.

A language model assigns probabilities to sequences of words — essentially predicting how likely a piece of text is, or what word comes next given the previous ones. Classic n-gram models estimate this from word co-occurrence counts; modern neural language models (and LLMs) learn it with networks. Next-word prediction is the foundation of text generation and much of modern NLP.

Stop words are very common words (the, is, and) that carry little distinguishing meaning, so classic pipelines remove them to reduce noise and dimensionality for tasks like bag-of-words classification or search. But you shouldn't always remove them — they matter for tasks like sentiment ('not good'), translation, and anything where syntax matters, and transformer models handle them natively. It's task-dependent.

Out-of-vocabulary (OOV) refers to words a model hasn't seen in training — with a fixed word vocabulary, these can't be represented and are often mapped to an 'unknown' token, losing information. It's common with rare words, typos, names, and new terms. Subword tokenization (BPE, WordPiece) largely solves it by breaking unknown words into known pieces, so any word can be represented.

For classification tasks, use accuracy, precision, recall, and F1 (with attention to class imbalance). For generation/translation, automatic metrics like BLEU and ROUGE compare overlap with reference texts, though they're imperfect proxies for quality. You hold out a test set and, for generation, often add human evaluation since automatic metrics miss meaning and fluency.

NLTK and spaCy for classic preprocessing, tokenization, POS tagging, and NER (spaCy is fast and production-friendly). scikit-learn for traditional ML on text features. Hugging Face Transformers is the standard for modern pretrained models (BERT, GPT-style) and fine-tuning. Gensim for topic modeling and word vectors. You pick based on whether you need classic pipelines or transformer models.