Inference

⚡ Inference is using a trained model to make predictions on new data. Training happens once — the model learns. Inference happens continuously — the model applies what it learned. Every time you type a prompt into ChatGPT, ask Siri a question, or have your face recognised by your phone, inference is running. It is the production phase of AI — and optimising it for speed and cost is one of the most important engineering challenges.

Category: Foundational Concepts · Difficulty: Beginner · Last updated: 15 May 2026 · 4 min read

---

Inference — How a Trained AI Model Goes From Lab to Production

What is Inference ?

A medical student spends four years in school learning medicine. After graduation, they spend decades applying that knowledge to patients. The four years of learning are training. Every patient encounter for the rest of their career is inference — applying trained knowledge to new situations.

AI works the same way. A model trains once (or periodically) on large amounts of data — an expensive, time-consuming process requiring powerful hardware. Then it is deployed and runs inference continuously — receiving new inputs and producing predictions, millions of times per day, in response to real-world requests.

Training a GPT-scale language model costs tens of millions of dollars and takes months. Serving it at inference — responding to 100 million daily users — is an entirely different engineering problem: how do you make predictions fast enough, cheap enough, and reliably enough for production at scale?

TRAINING VS INFERENCE

HOW INFERENCE IS OPTIMISED

Quantisation: reduce weight precision from 32-bit float to 8-bit or 4-bit integers. A 70B parameter model at full precision requires 140GB of GPU memory. At 4-bit quantisation it fits in 35GB — runnable on two consumer GPUs — with minimal accuracy loss.

Batching: instead of processing one request at a time, group multiple requests and process them together. GPU utilisation improves dramatically. Trade-off: individual requests wait slightly longer to form a batch.

Model distillation: train a smaller student model to mimic the outputs of a larger teacher model. The student runs inference much faster with modestly lower accuracy.

Specialised hardware: NVIDIA Triton Inference Server, AWS Inferentia, Google Cloud TPUs, and Apple Neural Engine are all optimised for inference workloads rather than general-purpose GPU compute.

Real-world examples

Not theory — what real teams actually shipped using this technique.

• OpenAI serves ChatGPT inference to over 100 million daily users. Each query triggers inference across GPT-4’s billions of parameters — returning a response in seconds. The engineering cost of this inference infrastructure is estimated at tens of millions of dollars per day.
• Spotify’s recommendation model runs inference every time you open the app — predicting which songs and podcasts to surface from a catalogue of 100 million tracks in under 100 milliseconds.
• Tesla’s FSD (Full Self-Driving) chip runs inference on 8 camera feeds simultaneously at 36 frames per second — 2,000 frames per second of inference throughput — with a power budget of 72 watts. All on a custom inference chip designed to prioritise throughput and power efficiency over training flexibility.

Common pitfalls

• Training-inference gap — a model that performs well in training may degrade in production due to distribution shift (real-world data differs from training data), staleness (the world changes after training), or edge cases not seen in training.
• Cold start latency — loading a large model into GPU memory takes time. Keeping models warm (loaded and ready) increases cost; cold loading increases latency. Production systems must balance both.
• Cost at scale — inference costs scale linearly with usage. A model that costs $0.001 per request seems cheap until it handles 100 million requests per day ($100,000 daily). Model optimisation and hardware selection have enormous financial impact.
• Monitoring — inference in production requires continuous monitoring. Data distribution drift, model degradation, and unexpected edge cases must be detected and addressed. Inference without monitoring is flying blind.

---

Frequently asked questions

QUESTION 1 What is inference in machine learning?

ANSWER 1 Using a trained model to make predictions on new data. Training is learning; inference is applying. Every ChatGPT response, face unlock, and spam filter decision is inference.

QUESTION 2 What is the difference between training and inference?

ANSWER 2 Training: model learns by updating weights — expensive, infrequent, requires large GPU clusters. Inference: model applies fixed weights to new input — cheaper per prediction, continuous, scalable to smaller hardware.

QUESTION 3 Why does inference latency matter?

ANSWER 3 Users expect sub-second responses. Self-driving cars need millisecond decisions. Fraud detection must complete before transactions clear. Latency is a product quality and safety requirement.

QUESTION 4 How is inference optimised?

ANSWER 4 Quantisation (reduce weight precision), batching (group requests), distillation (smaller model mimics larger), and specialised inference hardware (TPUs, Inferentia, Neural Engine).

---