Skip to main content
Bluecoders
← Tech glossary

Inference

TermConcept

Inference = using the model

Inference is the stage where an already trained AI model is used to produce a result from new data: a prediction, a classification, an LLM answer. It stands in contrast to training, which is used to build the model.

For an LLM, every request is an inference: the model receives a prompt and generates its answer token by token. Three indicators matter: latency (the time to the first token), throughput (tokens produced per second) and cost, which depends on the hardware, usually GPUs.

Training is a one-off effort, while inference repeats with every use. At scale, it often becomes the largest cost item of an AI product.

Hence an engineering field dedicated to optimizing it: quantization, grouping requests (batching), caching, speculative decoding. It relies on specialized engines such as vLLM, TensorRT-LLM or llama.cpp, and on inference platforms that rent out this service.

Why it matters when hiring

Inference optimization is a rare skill, at the crossroads of ML and systems engineering: it is found among ML Engineers, MLOps engineers or GPU-specialized engineers. The good signal: a candidate who cites measured gains (latency, cost per request) and explains the trade-offs made on quality. Do not confuse an AI Engineer who calls an API with a profile who can serve a model themselves.

A recruitment need?

Describe what you need. A recruiter from your sector will call you back within 24 business hours.