Natural Language Processing: Where AI Meets Language

Natural language processing is the field of getting computers to work with human language, and it long predates the current generation of large models. That history matters for a practical reason: most text problems reduce to a handful of task families that were named decades ago and still have specialised, cheap, reliable solutions. Reaching for a general language model without recognising the family is how teams end up paying per token for something a small classifier would do better.

Five families cover most real problems

Classification assigns a label to a text: spam or not, which department, positive or negative, urgent or routine. It is the most common text problem in business software by a wide margin, it is well understood, and small models are excellent at it.

Extraction pulls structured pieces out of unstructured text — names, dates, amounts, identifiers, relations. Also called named entity recognition and information extraction. The output is structured, which means you can validate it.

Similarity and retrieval finds related text. This is what embeddings do, and it underpins search, deduplication, clustering, and every retrieval-augmented pipeline.

Transformation rewrites text into other text with a determinate goal: translation, summarisation, style change. There is a right-ish answer, though not a unique one.

Open generation produces text where the space of acceptable answers is wide. This is the family general models are uniquely good at, and the hardest to evaluate because there is no reference to compare against.

Naming the family first tells you the model class, the metric, and roughly the cost. A team that says "we need AI for our support tickets" usually needs classification plus extraction, both of which are solved problems, and neither of which needs a frontier model.

Small specialised model or general LLM?

Both work for classification and extraction. They differ on everything else.

A fine-tuned small model is orders of magnitude cheaper per call, runs in milliseconds, can run on your own hardware, gives you a stable interface, and produces calibrated probabilities you can threshold. It costs you labelled data and a training pipeline.

A general model needs no training data, handles the long tail of unusual inputs better, adapts to a new label by editing a prompt, and does the harder families well. It costs more per call, adds latency, changes underneath you when the provider updates it, and does not naturally give you a confidence score you can trust.

The pattern that works well in practice is to use both: a general model to bootstrap labels and handle the tail, a small model to serve the high-volume common case. Route by confidence, and send anything the small model is unsure about to the larger one.

# The common two-tier pattern: cheap model first, escalate on low confidence.
label, confidence = small_classifier(text)

if confidence < 0.85:
    label = llm_classify(text, labels=LABELS)  # slower, costlier, better on the tail

The decision should be measured, not assumed. Build a labelled evaluation set of a few hundred real examples before choosing, because the answer differs by domain and the intuition "the bigger model must be better" is wrong often enough to matter — general models frequently underperform a tuned classifier on a narrow, high-volume task.

Ambiguity is a data problem before it is a model problem

Language is ambiguous in ways that make evaluation harder than the metric suggests. The same sentence means different things by context; sentiment depends on who is speaking; and for many real labelling tasks, two careful humans disagree a meaningful fraction of the time.

That disagreement rate is your ceiling. If annotators agree only 80 percent of the time on a label, a model scoring 85 percent against one annotator's judgements is not necessarily better than the humans — it may just be fitting one person's idiosyncrasies. Measure inter-annotator agreement before you measure the model, and where agreement is low, the fix is a clearer label definition rather than a bigger model.

Two more evaluation traps recur. Class imbalance: a classifier that always predicts the majority class can score 95 percent accuracy and be useless, which is why precision, recall, and the confusion matrix matter more than accuracy. And distribution shift: language changes, products get renamed, and users adopt new phrasings, so a model that was accurate at launch drifts.

What no longer applies

Some habits from earlier NLP are now actively harmful with transformer models. Aggressive stopword removal, stemming, and lowercasing destroy information these models use. Subword tokenisation handles vocabulary you have not seen. Manual feature engineering on text is rarely worth the effort.

What does still apply: cleaning genuine noise such as boilerplate and markup, splitting long documents thoughtfully, handling multiple languages deliberately, and normalising the things that are genuinely equivalent in your domain.

# Still useful: strip real noise, keep the signal the model relies on.
text = strip_html(raw)
text = collapse_whitespace(text)
# Don't: lowercase(), remove_stopwords(), stem() -- transformers use all of it.

Common mistakes

Skipping the task family. It determines the model class, the metric, and the cost before any product is compared.

Assuming the general model wins. On a narrow, high-volume task a tuned small model is frequently better and always cheaper.

Reporting accuracy on imbalanced data. It hides a model that has learned to predict the majority class.

Applying pre-transformer preprocessing. Stemming and stopword removal now remove signal.