Inference is the stage where an already trained AI model is used to produce a result from new data: a prediction, a classification, an LLM answer. It stands in contrast to training, which is used to build the model.
For an LLM, every request is an inference: the model receives a prompt and generates its answer token by token. Three indicators matter: latency (the time to the first token), throughput (tokens produced per second) and cost, which depends on the hardware, usually GPUs.
Training is a one-off effort, while inference repeats with every use. At scale, it often becomes the largest cost item of an AI product.
Hence an engineering field dedicated to optimizing it: quantization, grouping requests (batching), caching, speculative decoding. It relies on specialized engines such as vLLM, TensorRT-LLM or llama.cpp, and on inference platforms that rent out this service.
Why it matters when hiring
Inference optimization is a rare skill, at the crossroads of ML and systems engineering: it is found among ML Engineers, MLOps engineers or GPU-specialized engineers. The good signal: a candidate who cites measured gains (latency, cost per request) and explains the trade-offs made on quality. Do not confuse an AI Engineer who calls an API with a profile who can serve a model themselves.
