Self Hosted LLM Infrastructure: Five Requirements Before Buying GPUs

We treat a self-hosted LLM, a language model run on GPUs its operator controls, as feasible when five requirements hold: GPU memory, serving, storage and security, staffing, and token volume.

Check them in that order, because a failure near the top of the list costs less to find. Alpacked's self-hosting guide puts break-even against API pricing near 2 million tokens per day.

DataNorth's guide calls the comparison close until about 50 million. The API price explains most of that spread: a self-hosted model matches OpenAI's GPT-4o pricing at a far lower volume than it needs to match the cheaper GPT-4o mini.

The Five Self-Hosted LLM Requirements

Each requirement covers a different part of the build:

1. GPU memory: the GPUs' memory (VRAM) holds the model weights and the KV cache, the working memory each request uses as the model writes its answer.
2. Serving engine: a production serving engine keeps response times steady when many requests arrive at once.
3. Storage, network and security: the items a GPU quote leaves out get their own budget line.
4. Staffing: a named person owns drivers, monitoring and model updates.
5. Token volume: daily tokens processed cover the fixed GPU and staff cost at the API's per-token price.

Starting Points by Self-Hosted LLM Deployment Type

Three common deployment types begin with different requirements:

1. Single-GPU prototype: a workstation build starts with GPU memory, then load-tests the serving engine to measure latency with several users at once.
2. Owned or leased facility: building or leasing a facility adds power, cooling and network fabric to the storage and security work.
3. Sovereignty mandate: a data sovereignty rule, which keeps data inside one country, adds a choice between dedicated GPUs and GPUs shared with other customers.

GPU Memory Requirements for a Self-Hosted LLM

Model weights set the starting memory figure. Alpacked's guide uses about 0.5 GB of VRAM per billion parameters with 4-bit quantization, which cuts each weight from 16 bits to 4.

PremAI's guide gives the 16-bit case: a 70-billion-parameter (70B) model needs about 140 GB at 16-bit precision and about 35 GB at 4-bit. Quantization decides whether that model fits on one 80 GB NVIDIA H100 or needs two GPUs.

The KV cache grows with context length and with the number of simultaneous requests. Alpacked reports that doubling the context window from 8K to 16K tokens can add 15 to 20% to memory use.

DataNorth's infrastructure guide puts the cache at 1 to 2 GB for 4K to 8K contexts at modest batch sizes, and 8 GB or more at 32K to 128K tokens.

Each simultaneous request holds its own cache, so a GPU that fits the weights for one user can run out of memory at production load.

The LLM GPU memory requirements breakdown gives VRAM and KV-cache figures for common models.

LLM Serving Engines Under Concurrent Load

A serving engine queues incoming requests and runs them on the GPU. Its design decides how long users wait when traffic is heavy.

Red Hat's benchmark ran two open-source serving engines, vLLM and Ollama, on one NVIDIA A100 40 GB GPU with Meta's Llama 3.1 8B model. vLLM peaked at 793 output tokens per second and Ollama at 41, a gap of about 19 times.

At peak throughput, 99% of requests got their first token within 80 ms on vLLM, against 673 ms on Ollama. Ollama handles four requests at a time by default.

Red Hat raised that limit to 32, which still left Ollama's throughput below vLLM's at each tested concurrency level. A test with one user misses this gap.

The figures come from one GPU and one 8B model, so they work as an order of magnitude for other setups.

The path from local prototype to production shows what breaks at each stage.

Self-Hosted LLM Storage, Network and Security

A quantized Llama 3 70B checkpoint, the saved model file, takes 35 GB or more. PremAI calls NVMe solid-state storage essential, because slower disks stretch load times.

Rollback copies, several model versions and embedding stores multiply that figure across servers. Alpacked lists five security items to put in place before production:

1. Network isolation: the inference server runs in a private subnet, reachable through an API gateway or VPN.
2. Access control: API keys and gateway rules decide who can call the model, since a regulated firm may need each request on record for a compliance report.
3. Retrieval permissions: retrieval-augmented generation (RAG), which feeds company documents to the model at query time, filters results by each user's access rights.
4. Output validation: filters, guardrail layers and fixed refusal rules screen responses for sensitive data before they reach users.
5. Model weight protection: weight files get encryption at rest, filesystem access controls and regular backups.

Alpacked calls a shared, unfiltered document index a common mistake, because one user's query can surface documents meant for another user. The RAG permission mapping needs a named owner for that reason.

A build in an owned or leased facility adds power, cooling and network fabric to this list. The on-premise LLM deployment guide covers that facility work.

A rule that keeps data in one country, or isolates it from other customers, adds a tenancy decision. The dedicated versus shared private LLM hosting comparison sets out both options.

Staffing a Self-Hosted LLM Deployment

Self-hosting adds an operations job that lasts as long as the deployment. PremAI and Alpacked name the same tasks: GPU drivers, monitoring, model updates and troubleshooting CUDA, NVIDIA's GPU software layer.

Alpacked puts the cost of an MLOps engineer, who keeps models running in production, above the hardware itself. Its production setup runs the stack on Kubernetes, which schedules the containers, with Prometheus and Grafana for monitoring.

DataNorth budgets between half an engineer and one engineer for the first year. It calls that staff time the largest cost for most teams, ahead of the GPUs.

New open-weight model releases bring rounds of retesting and repeat quantization. Capacity monitoring has to catch KV-cache exhaustion and queue delays before users see failures.

A named owner stops these tasks piling up unnoticed until an outage.

When a Self-Hosted LLM Costs Less Than an API

A self-hosted LLM costs less than an API when its daily token volume, at steady load, covers the fixed GPU and staff cost at the API's per-token price. Published estimates of that volume vary by a factor of 25.

Alpacked and PremAI put break-even near 2 million tokens per day. Alpacked puts payback at 6 to 12 months for a steady load of 10 million tokens or more a day.

DataNorth's European figures show an API cheaper in most cases below 5 million tokens per day and a close call up to 50 million. Above that, self-hosting pays when utilization stays high and steady.

PremAI prices a 7B model on one H100 spot instance (a discounted cloud GPU the provider can reclaim) at about $1.65 per hour. That comes to about $10,000 a year at 70% utilization, or roughly $27 a day.

Our calculation divides that daily cost by the API's output-token price. At GPT-4o mini's $0.60 per million output tokens, the GPU pays off above about 46 million tokens a day.

At GPT-4o's $10 per million output tokens, the same GPU pays off above about 2.7 million tokens a day. The 2 million rule of thumb fits a workload that would otherwise run on a GPT-4o-class model.

These thresholds count each token at the output rate and leave out staff costs, so the real break-even volume runs higher. Input tokens cost less than output tokens on both OpenAI models.

Alpacked adds two effects that lower the threshold: one GPU server can serve several applications at once, and fine-tuning runs on hardware already paid for.

A hybrid setup runs steady workloads on self-hosted GPUs and sends spiky work, or tasks that need the largest proprietary models, to APIs. Alpacked and PremAI both report 40 to 70% savings against a setup that sends each request to an API.

One fintech case in both guides fell from $47,000 to $8,000 a month. Neither guide names the company or links a primary source, so the figure can't forecast savings for another workload.

Checking a Self-Hosted LLM Plan Before Buying Hardware

Run the five checks in order for the workload in question, and stop at the first failure:

1. GPU memory fits the model at peak load. Size the weights at the chosen quantization, then add the KV cache for the target context length and peak concurrency. DataNorth expects real usage to run 10 to 20% above that bare calculation. If the sum exceeds the VRAM available to buy or rent, change the model size, quantization or GPU count before going further.
2. The plan names a production serving engine. Load-test the engine with the real model at the target concurrency. Record P99 time to first token, the longest first-token wait for 99% of requests, alongside throughput. A plan built on single-user tooling carries latency risk at production traffic levels.
3. Storage, network and security each carry a budget line. Count NVMe capacity for each checkpoint, rollback copy and embedding store. Assign an owner to each of the five security items, starting with the RAG permission mapping. Add facility costs for an owned or leased site and a tenancy decision for a residency rule.
4. A named owner runs the platform. Record the person responsible for drivers, monitoring, model updates and CUDA troubleshooting, and the share of their time the platform gets. Define the escalation path, including who rolls back a failed model update and who answers a latency alert out of hours.
5. Daily token volume covers the GPU cost at the right API price. Measure average and peak daily tokens over a representative stretch of pilot or production traffic. Divide the daily GPU and staff cost by the per-token price of the API model the workload would otherwise use, and compare the result with the measured average. The peak sets the concurrency target for check 1. Below that volume, price an API or hybrid setup for the same workload.

A workload that clears the five checks makes a feasible self-hosted LLM candidate.

Self-Hosted LLM FAQ

Is a self-hosted LLM the same as an on-premise deployment?

Self-hosting covers more than an on-site facility. A self-hosted LLM can run on dedicated cloud GPUs or on racks in an owned or leased facility.

Can a self-hosted LLM run without GPUs?

A quantized model can run on server CPUs through llama.cpp, an open-source inference engine, at far lower throughput than a GPU. DataNorth suggests CPU inference for overnight batch jobs and document classification, where users aren't waiting on each response.