What is AI inference? A plain-English guide
Erwin Berkouwer
AI inference is the phase in which an already trained AI model is applied to new data to immediately generate a prediction, answer, or decision, as opposed to training, where the model itself learns. For SMEs, inference is the part of AI they actually pay for and experience day to day, through API calls and token usage, and it determines the speed of their AI tools. The choice between real-time and batch inference is primarily a trade-off between speed and cost.
Inference is the phase where a trained AI model actually gets to work: it processes new input and immediately delivers an answer, prediction, or decision.
AI inference is the moment a trained model is put to work on new, previously unseen data to immediately produce an answer, prediction, or decision. Where training is about teaching the model something, inference is about the model applying what it already learned. Every time you ask a chatbot a question, scan a document, or get a recommendation, that is inference happening in real time.
Inference is the model in action: no new learning, just applying what it already knows.
For most small business owners, inference is exactly the part of AI they use every day and pay for, even if they have never heard the term. Almost no SME trains its own models, that work sits with the large tech companies. Inference happens every time you use an off-the-shelf AI tool.
How it works
The difference between training and inference is easiest to understand as two separate stages in a model's life cycle.
- Training: the model is shown huge amounts of data and learns patterns from it. This is computationally heavy, can take hours to weeks, and runs on powerful GPU or TPU clusters.
- Inference: the already-trained model processes one new input at a time and returns a result for it. This is much lighter than training, but often needs to happen instantly and at large scale.
Within inference, the same three-step process repeats every time:
- Input preparation: new data (a question, photo, document) is put into the format the model expects.
- Forward pass: the model analyzes the input using the knowledge it built up during training. Nothing new is learned, only applied. This step is called a forward pass.
- Output generation: the result (an answer, score, classification) is returned to the application and shown to the user.
The speed of this process is called latency, the delay between sending your request and getting an answer back. For a chatbot, latency of a few seconds is usually acceptable; for fraud checks in a payment flow, it needs to happen within milliseconds. Every inference call also costs money: most AI services charge per API call or per amount of tokens processed (the chunks of text a model breaks language into). More usage literally means more cost, even without any new training involved.
Why it matters for SMEs
As a small business owner, you almost never train an AI model yourself. You use existing, ready-made models (through a tool, an API, or a chat interface) and pay for the inference behind it. That means the cost and speed you experience are not about how well the model was trained, but about how efficiently it responds every time it is used.
This has practical consequences:
- Response speed shapes user experience. A customer service chatbot that takes three seconds per answer feels slower than a competitor that replies within a second.
- Costs scale with usage. The more questions, documents, or customer requests your AI tool processes, the higher the bill. This differs from traditional software, where costs are often fixed.
- Not every process needs speed. Some tasks (like categorizing invoices overnight) work fine in batch, and that is often cheaper than processing everything in real time.
Understanding what inference costs and how to manage it helps you choose and configure AI tools that fit your budget and your expectations around speed.
A concrete example
Say a small business deploys an AI chatbot for customer support on its website. Every time a visitor types a question, the following happens:
- The question (input) is sent to the language model.
- The model runs a forward pass: it recognizes the intent of the question based on everything it learned about language and customer queries during training.
- The model generates an answer (output), which appears in the chat within a few seconds.
This entire process, from question to answer, is one inference call. At a hundred visitors a day, that is a hundred (or more, in a multi-message conversation) separate inference calls, each with its own latency and cost. The same applies to a document scanner that reads invoices automatically: every page scanned is a separate inference step where the model recognizes text and amounts.
When to use it, when not
Not every application needs real-time inference. The choice between real-time and batch is mostly a cost trade-off.
| Situation | Approach | Why |
|---|---|---|
| Customer waits for an immediate answer (chat, search) | Real-time inference | Low latency is essential for user experience |
| Overnight processing of invoices or documents | Batch inference | Cheaper, speed is not urgent |
| Fraud or risk detection on transactions | Real-time inference | Decision must happen within milliseconds |
| Monthly reporting or trend analysis | Batch inference | Large data volumes, no time pressure |
| Sensitive data that cannot leave a device | Inference on the device itself (edge) | Privacy and lower bandwidth costs |
As a rule of thumb: only use real-time inference where a user or process is genuinely waiting on the result. For background tasks, batch processing is often just as effective and considerably cheaper.
Related concepts
- Training: the phase where a model learns from data, prior to inference.
- Foundation model: a large, pre-trained model (like GPT or Gemini) that serves as the basis for inference tasks without you having to train it yourself.
- Latency: the delay between a request and the model's response.
- Tokens: the units text is broken into and processed as, the basis for most AI pricing models.
- API calls: the technical requests your software makes to ask a model for an inference.
Curious how inference cost and speed play out in your business? An AI scan shows where AI already works for you and where the cost of real-time processing does not outweigh the benefit. Considering a chatbot or automation that runs inference continuously, such as an AI agent for customer service or scheduling? It pays to estimate expected volumes upfront. At AI consultancy, we help SMEs find the right balance between real-time convenience and manageable cost.
Veelgestelde vragen
Korte, heldere antwoorden die je helpen sneller beslissen.
What is the difference between training and inference?
Training is the phase where an AI model learns from large amounts of data, which is computationally heavy and can take hours to weeks. Inference is the phase after that, where the already-trained model is used to quickly produce an answer or prediction on new input. A small business almost always only deals with inference, not training.
Why does inference matter if I do not build my own AI models?
Every time you use an AI tool, such as a chatbot or document scanner, an inference call runs in the background. The speed (latency) and cost you experience as a user are a direct result of how efficiently that inference runs, even though you are not training anything yourself.
Roughly what does inference cost?
Most AI services charge per API call or per amount of tokens processed. The exact price varies significantly by provider and model, so check with each tool how usage is billed before scaling up to many users.
When should I choose batch inference instead of real-time?
Choose batch inference when nobody needs the result immediately, for example when categorizing invoices overnight or generating a monthly report. This is usually cheaper than processing everything in real time. Choose real-time inference when a customer or process needs an answer right away, such as with a chatbot or fraud detection.
