Foundations

Natural Language Processing

NLP

The field of AI concerned with enabling computers to understand, interpret and produce human language, in text and in speech.

THREE ERAS OF NATURAL LANGUAGE PROCESSINGRuleshand-written grammarsStatistics and MLa separate model per taskLLMsone model, every language taskTranslation, summarisation, sentiment analysis, question answering: once separate systems, now abilities of one model.

swipe to see the whole diagram →

MEmehmeterkek.com/glossary/natural-language-processing

In plain terms

Human language is untidy: one word means several things, one thing is said in a hundred ways, and meaning depends on context. NLP is the discipline that has worked for seventy years on making computers cope with that. Spell-checkers, search engines, translation, voice assistants and spam filters are all NLP. Language models are its latest and most successful chapter.

Why it matters

The term now turns up mostly in older systems, in job titles and in supplier descriptions. It helps to know that “uses NLP” says very little: it can mean a keyword list from 2010 or a current language model. Classic NLP techniques still have a place. For narrow, high-volume tasks such as routing documents or detecting a language, they are fast, cheap and predictable.

Example

In 2016 a bank analysed complaint letters with a classic NLP pipeline: one component to detect the language, one for the topic, one for sentiment and one to pull out account numbers, each built and maintained separately. Today a single language model does all four from one instruction, and drafts the reply as well. The bank has kept the old language detector: it costs almost nothing and is almost never wrong.

Most often confused with

NLP vs. Large language model (LLM)

NLPThe field: every method for handling language
Large language model (LLM)One technology within it, now the dominant one

NLP is the problem area; an LLM is one solution. For decades NLP meant a toolbox of separate techniques, one for each task. LLMs replaced most of that toolbox with a single general model. Saying that a product “uses NLP” and saying that it “uses an LLM” are different claims, and the second is the more specific.

Under the hood

Classic tasks: tokenisation, part-of-speech tagging, parsing, named-entity recognition, sentiment analysis, text classification, machine translation, summarisation, question answering, speech recognition and speech synthesis. Three eras: rule-based systems (hand-written grammars and dictionaries); statistical and machine-learning methods (n-grams, TF-IDF, then word embeddings such as word2vec, and recurrent networks); and pre-trained transformers (BERT from 2018, then generative LLMs). Sub-fields: natural language understanding (NLU) and natural language generation (NLG). Classic methods remain sensible for very high volumes at low cost, strict latency limits, tasks with stable and narrow definitions, and settings that require fully deterministic behaviour. Metrics from the field, such as precision, recall, F1, BLEU and ROUGE, are still in use.

Written by Mehmet Erkek · Last updated: