From LLM or Agent Pilot to Production LLM Infrastructure: Capacity, Isolation and Operational Changes

An LLM or agent pilot needs production LLM infrastructure once a second person or service depends on the model. A restart then cuts off each person waiting on an answer, because the model runs in one desktop program such as Ollama or LM Studio.

A container restarts the model after a crash. A serving engine such as vLLM answers many requests at once, which cuts the wait for the slowest 1% of requests, a figure engineers call P99 latency.

A cluster of several engine copies also needs four operations systems: safe releases, failure handling, model tracking and hardware cost control.

Agent pilots hit each stage sooner than chat pilots. Anthropic describes an agent as an LLM that calls tools in a loop, so one task sends many model requests.

Three Stages From LLM Pilot to Production Cluster

We sort LLM deployments into three stages by who depends on the model, because each stage needs a different fix:

1. Solo prototype: one person or script calls the model, so a crash costs one session.
2. Shared single-node server: a team or several services call one containerized process on one machine, so one failure interrupts each of them.
3. Multi-replica cluster: several copies of the model, called replicas, handle traffic behind a router that spreads requests among the copies.

The table pairs each stage's warning sign with its fix.

Stage 1: The Solo LLM Prototype

Stage 1 tools are built for one machine and one operator. Ollama runs models such as Llama, Mistral and Phi from the command line. The computer's memory caps the model size.

LM Studio and the llama.cpp runtime are also built for one computer, so a crash in that program ends the caller's session.

Stage 1 ends when a second user, service or agent calls the same model. Requests then wait in line in one program, and a restart drops the other caller's unfinished response.

Agent pilots also need their tool calls kept away from live systems. Anthropic's guide recommends testing agents in sandboxed environments with guardrails, because errors can compound across an agent's many tool calls.

Run three checks before a second caller arrives:

1. Licence family: read the model card and its licence file to place the model in one of the four families below. The family decides whether selling, changing or sharing the model comes with conditions, such as crediting the maker or passing on its usage rules.
2. EU AI Act duties: ask legal whether the EU AI Act, the EU law that regulates AI models, makes the team a provider of a general-purpose AI model in the EU. The European Commission's guidelines treat further training that uses more than one third of the original model's training compute as a sign the team may count as a provider.
3. Model reliability on real prompts: test the model with the prompts the product will send, at their real length, on the target hardware. Keep these test cases with expected answers, because Stage 3 release pipelines reuse them to check each new model version.

Four Open-Weight LLM Licence Families to Check in Stage 1

Open-weight model makers publish the trained weights for download, and each release carries its own licence. VDF AI's licensing guide groups those licences into four families:

1. Permissive: Apache-2.0 and MIT releases allow commercial use, modification and redistribution with credit to the maker. They don't come with an acceptable-use policy, a list of forbidden uses.
2. Custom community licences: these allow most commercial use under the publisher's own conditions. Meta's licence for Llama models requires a "Built with Llama" notice, model names that start with "Llama" and a copyright notice. It also binds users to Meta's acceptable-use policy.
3. Open weights with mandatory flow-down: Google's Gemma Terms of Use allow internal commercial use. A company that redistributes the model must write Google's use restrictions into its own contract with each recipient.
4. Research or non-commercial: these models are released for evaluation or research, and commercial deployment needs a separate paid agreement.

The licence attaches to the exact model version. VDF AI notes that terms can change between releases in the same family, so record the licence for each version deployed.

Stage 2: The Shared Single-Node LLM Server

Stage 2 starts by moving the model into a container. Ollama ships an official image for Docker, a widely used container tool.

Docker then restarts the container after a crash, keeps its logs and gives it its own network port. Ollama also accepts requests in the format of OpenAI's API, so other tools can call the model over that port.

One machine now serves several callers from one program.

Memory then has to cover several requests at once. Memory use depends on quantization (storing weights at lower precision), context length, the number of simultaneous requests and the share of the model held in GPU memory.

Our guide to sizing memory for open-weight LLMs does that math for a shared server or a cluster.

Track two measures separately on a shared server:

1. Throughput: the tokens per second the server produces, which shows one machine's capacity.
2. P99 latency: the wait for the slowest 1% of requests, which climbs first when requests form a queue.

Stage 2 ends when P99 latency rises with the number of simultaneous requests, or when one process restart drops requests from several callers.

Stage 3: Serving Engines for a Production LLM Cluster

Stage 3 starts with a switch to a serving engine. Red Hat compared vLLM and Ollama in a benchmark on one NVIDIA A100 running Llama 3.1 8B.

vLLM peaked at 793 tokens per second, and Ollama, tuned to 32 parallel requests, peaked at 41. At those peaks, the P99 wait before a response started was 80 ms on vLLM and 673 ms on Ollama.

Ollama's 673 ms shows the queueing that ends Stage 2. These figures come from one A100 and one 8-billion-parameter model, so rerun the test with the product's prompts before sizing hardware.

vLLM batches many requests together. It stores each request's working memory, known as the KV cache, in small pages through a method called PagedAttention, so more requests fit in memory at once.

Our comparison of inference serving engines weighs vLLM against TensorRT-LLM, SGLang and TGI.

Four Operations Systems for a Production LLM Cluster

The engine switch doesn't control releases, failures or hardware spending. Modular's LLM Inference Handbook groups InferenceOps, the practices for deploying and updating models at scale, into four areas:

1. Standardized deployment workflows: automated release pipelines (CI/CD) test each new model version against saved prompts such as the Stage 1 test cases. Canary releases send a small share of traffic to a new version first. Blue-green releases keep the old version running until the new one takes the traffic.
2. Safe updates and fault tolerance: rolling updates replace replicas one at a time and roll back automatically on failure. Retries, timeouts and circuit breakers stop one failing model from taking down the application. A circuit breaker cuts off calls to a replica that keeps failing.
3. Central model registry: a registry records which models are deployed, where and by whom, with version history and ownership. The licence for each version, recorded in Stage 1, belongs in the same record.
4. Cost control and access logs: automated cleanup shuts down idle hardware. Access control and audit logs record who changed what in production.

A cluster that lacks these four can lose its replicas to one bad release.

Where to Check First When Diagnosing an LLM Deployment Stage

Check in this order: the number of people, services and agents that depend on the model, then P99 latency at rising numbers of simultaneous requests, then the release process.

A team hosting the cluster in its own building also needs space, power and network plans, covered in our on-premise LLM deployment guide.

zWARE is Zettabyte's software for operating GPU clusters. It schedules workloads, tracks hardware, power and system health with automated recovery, and manages several clusters across sites.