Self-Hosting LLMs on a Dedicated Server: Hardware Sizing Guide

To self-host an LLM on a dedicated server, size the hardware for the model, context length, active requests, and response-time target. Budget for model weights, request memory, runtime overhead, system RAM, CPU work, and storage. A model that loads successfully has passed a memory check, not a production-capacity test.

The gap becomes clear during a busy hour. One short request works. Several users submit long documents, the request queue grows, and the same server runs out of memory or takes too long to respond.

Avoid that problem by planning in this order:

1. Define the exact model and request profile.
2. Calculate weight memory and cache memory separately.
3. Budget for the complete server and supporting services.
4. Test realistic traffic against response-time and quality targets.
5. Compare complete costs for the configurations that pass.

Start with the model and request profile

Server sizing starts with what the application must do. “Run an LLM for our team” is too broad to produce a useful hardware recommendation.

Record these inputs before comparing servers:

Input	What to record
Model	Exact name, revision, architecture, and license
Weight format	FP16, BF16, or the specific quantized artifact
Context	Normal and maximum prompt-plus-output token counts
Traffic	Active generations, arrival rate, and burst behavior
Performance	Time to first token and acceptable completion time
Quality	Tasks the model must complete correctly
Other services	Embeddings, reranking, vector search, APIs, and monitoring
Operations	Update process, recovery needs, and growth reserve

Context includes system instructions, chat history, retrieved text, user input, and generated output. A short question can produce a long model input if the application adds several documents.

Also separate users from active requests. A service with 100 registered users may have only a few generations running at once. An agent used by one person may issue several requests during a single task.

For a definition of the serving process, see what AI inference means.

Keep inference separate from training

This guide covers inference: using an already trained model to produce an answer. Training and fine-tuning can require additional memory for gradients, optimizer state, and activations, depending on the method.

Do not use an inference sizing table as a training recommendation. If the same machine must serve users and run fine-tuning jobs, test the combined schedule or separate those workloads.

Calculate model-weight memory first

Model weights are the stored numerical values used during inference. Their memory footprint provides the starting point for a hardware budget.

Idealized weight memory in bytes = parameter count × bits per weight ÷ 8

For a nominal eight-billion-parameter model at 16 bits:

8,000,000,000 × 16 ÷ 8 = 16,000,000,000 bytes, or 16 GB

At an idealized four bits per weight, the same calculation gives 4 GB.

Nominal model size	FP16/BF16 weight floor	8-bit weight floor	4-bit weight floor
8B	16 GB	8 GB	4 GB
14B	28 GB	14 GB	7 GB
32B	64 GB	32 GB	16 GB
70B	140 GB	70 GB	35 GB

These figures cover idealized weights only. They exclude request cache, working buffers, quantization metadata, and other runtime allocations. The exact model may also have more parameters than its rounded name suggests.

Why a 4-bit file needs more than the raw calculation

A quantized artifact can contain scale values, metadata, and tensors stored at other precisions. Its downloaded size and loaded memory footprint can differ.

Record the exact artifact, then inspect memory after loading it in the intended runtime. Do not treat every format labeled “4-bit” as interchangeable.

Reducing weight precision also does not automatically reduce the precision of the request cache. Those are separate settings with separate compatibility and quality checks.

Use total parameters for MoE weight planning

A mixture-of-experts model, or MoE model, activates only part of its expert network for each token. That can reduce computation relative to using every expert, but the other weights still need a placement plan.

For full residency, budget for the stored model, not just its active parameter count. Expert offload and distributed placement can change where memory is used, but they introduce their own transfer and execution costs.

Keep GB and GiB consistent

One GB is one billion bytes. One GiB is 1,073,741,824 bytes. A 4 GB payload is about 3.73 GiB.

Convert values to the same unit before subtracting model memory from a device budget. Use the actual available memory reported for the deployment rather than mixing a product label with a differently reported measurement.

Add KV cache, context length, and concurrent requests

The key-value cache, usually called the KV cache, stores attention data used while the model generates text. Model weights can be shared across requests within one model instance, but active sequences also need request state.

That is why longer contexts and more active requests can consume memory even when the loaded model does not change.

For a conventional decoder with the same full-attention layout in each layer, estimate logical cache payload as:

KV bytes ≈ 2 × layers × KV heads × head dimension × bytes per cached value × total cached tokens

The factor of two accounts for keys and values. Total cached tokens means the sum across active sequences. If all sequences have the same length, multiply that length by the number of active sequences.

This is not a universal formula for every model. Sliding-window attention, other cache architectures, cache quantization, and runtime allocation rules can change the calculation. Hugging Face documents cache strategies with different memory behavior, including limits for sliding-window and chunked-attention layers. 

A worked cache example

Meta’s Llama 3.1 8B configuration uses 32 layers, 32 attention heads, eight KV heads, and a hidden dimension of 4,096. The head dimension is 4,096 ÷ 32 = 128.

Assume a two-byte KV cache, full attention, and no prefix sharing:

2 × 32 × 8 × 128 × 2 = 131,072 bytes per cached token per sequence

That equals 128 KiB. 

Cached tokens per sequence	Active sequences	Calculated logical KV payload
8,192	1	1 GiB
8,192	4	4 GiB
32,768	4	16 GiB
32,768	8	32 GiB

The model is unchanged, yet request memory changes sharply. This is why adding a fixed percentage to model size cannot account for every context and concurrency target.

These are calculated payloads, not measured GPU allocations. A runtime may reserve cache blocks in advance or keep allocations for reuse. Memory reported by a monitoring tool may therefore differ from the live token payload.

Build a complete GPU memory budget

For a fully GPU-resident deployment, start with:

GPU memory budget = loaded weights + KV allocation + runtime working memory + operating reserve

Use measured runtime values where possible. Avoid counting the same reserved cache pool twice.

Example: a short-context assistant versus a shared service

The following is a teaching scenario, not an Atal benchmark. Assume:

• Loaded quantized weights: 5 GiB.
• Other runtime working memory: 2 GiB.
• Operating reserve: 2 GiB.
• Cache layout from the Llama 3.1 8B example above.

Memory item	Short-context assistant	Longer-context shared service
Active sequences	1	4
Cached tokens per sequence	8,192	32,768
Loaded weights, assumed	5 GiB	5 GiB
Logical KV payload, calculated	1 GiB	16 GiB
Other working memory, assumed	2 GiB	2 GiB
Reserve, assumed	2 GiB	2 GiB
Combined planning total	10 GiB	25 GiB

Under these assumptions, the first profile clears a 16 GiB memory-budget check. The second exceeds a 24 GiB budget. Neither result proves throughput or response time.

The reserve and overhead figures are inputs for this example, not general recommendations. Replace them with measurements from the exact runtime and settings before selecting hardware.

What about a 70B model?

A nominal 70B model has an idealized four-bit weight floor of 35 GB. It cannot be fully GPU-resident within 24 GB of available memory, even before cache and overhead.

Possible paths include a larger-memory GPU, supported multi-GPU placement, CPU offload, or a smaller model that passes the task-quality test. A 48 GB device is not a blanket recommendation for every 70B workload. Context, active requests, format overhead, and runtime behavior still matter.

Choose GPU hardware by more than VRAM

Enough memory makes a configuration possible. It does not make the configuration fast enough.

Compare candidates using the same workload:

Hardware check	What to confirm
Available VRAM	Actual allocation after partitions and other processes
Runtime support	GPU architecture, driver, backend, and model support
Weight and cache formats	Supported kernels and memory behavior
Memory bandwidth	Performance under the selected model and request mix
Interconnect	Device topology for the chosen parallel method
Power and cooling	Sustained operating limits in the server
Service terms	Dedicated or shared resources, support, and replacement process

Avoid selecting a GPU from a generic users-per-card table. Ten short requests and ten long-document requests are different workloads.

Keep the shortlisted model revision, runtime version, prompt/output limits, and active-request target beside each hardware option. Otherwise, the comparison is difficult to reproduce.

Size system RAM, CPU, storage, and networking

System RAM: budget for loading and host services

System RAM supports the operating system, model loading, APIs, tokenization, monitoring, and any CPU-resident model or cache state. Embedding models, rerankers, and vector databases add further demand if they share the host.

Measure peak RAM during startup as well as steady operation. The loading method can use temporary copies or buffers that are absent after startup.

There is no single RAM-to-VRAM ratio that fits every inference stack. Build a host budget from the selected runtime, offload plan, and supporting services. A model that fits on the GPU can still fail during loading because host memory is insufficient.

CPU: match cores and memory bandwidth to the task

On a GPU-backed server, the CPU still handles request processing, tokenization, networking, and application work. CPU-only inference also makes processor execution and system-memory behavior central to performance.

Test the intended thread count. More cores do not automatically produce a matching increase in token rate. For multi-socket systems, inspect memory locality and the placement of GPUs and network devices.

A CPU-only dedicated server can be a candidate for batch jobs, experiments, or low-frequency tasks. Whether it is suitable for interactive use depends on the model and measured response time.

NVMe storage: include versions and recovery

Budget storage for more than one downloaded model:

Storage budget = active model files + rollback files + runtime artifacts + application data + retained logs + free working space

Add vector indexes and document data when retrieval shares the server. Also consider temporary downloads and conversion files.

Local NVMe can reduce model-loading and data-access delays. It does not guarantee a higher generation rate after all required weights are resident in memory. Measure cold-start time separately from warm request performance.

Networking: separate client traffic from model traffic

A public network port handles client connections, downloads, and transfers. GPU-to-GPU links and the network between distributed inference nodes serve different purposes.

Do not use a public 10 Gbps port specification as proof of multi-node inference performance. Check the actual topology, communication method, and runtime requirements.

For voice applications, measure the full turn: speech input, transcription, model response, speech generation, and delivery. LLM time to the first token covers only part of that path.

Match the runtime and multi-GPU design to the workload

Choose the runtime before making a final hardware decision. It controls model support, placement, batching, and memory management.

Runtime	Useful evaluation path	Check before committing
Ollama	Model trials and a simpler serving workflow	Hardware support, context, parallel requests, and loaded-model limits
llama.cpp	GGUF models, CPU inference, and partial GPU offload	Build/backend, artifact support, offloaded layers, and measured speed
vLLM	Batched serving and supported multi-GPU deployments	Model/format compatibility, memory controls, topology, and latency under load

Ollama documents increased memory needs with parallel requests and longer context. llama.cpp supports CPU/GPU hybrid inference. These are options to test, not guarantees of a specific token rate. 

Split one model or run separate replicas?

Splitting a model across GPUs can help when it does not fit on one device. Tensor parallelism divides supported tensor operations; pipeline parallelism divides model stages or layers. MoE deployments may also use expert parallelism.

Replicas serve a different purpose. Each copy holds its own weights and can handle a separate set of requests. Replicas can add serving capacity when one copy already fits, but they do not combine their memory to hold one larger model.

Two GPUs do not become one automatic memory pool. Per-device allocation, communication, and runtime support still constrain the layout. vLLM documents multiple parallel methods and topology considerations.

NVLink is not a universal requirement. Compare the actual supported layout and benchmark it. Neither adding a second GPU nor doubling total VRAM guarantees twice the useful throughput.

Benchmark the server before production

Test the application target, not the highest number in a benchmark screenshot.

Metric	What it tells you
Time to first token, or TTFT	How long the client waits for initial model output
Inter-token latency	How smoothly output arrives after generation starts
Per-request output rate	Generation speed experienced by one request
Aggregate output rate	Total output across the server
Request completion time	How long the full response takes
Errors and queue growth	Whether demand exceeds usable capacity

The definitions and measurement boundaries matter. NVIDIA’s benchmark guidance distinguishes user-level token rates, system throughput, and response latency. Keep the selected tool’s definitions with the results. 

Use a repeatable acceptance test

1. Fix the configuration. Record model revision, artifact, tokenizer, runtime, driver, GPU, CPU, RAM, storage, and power limits.
2. Test one request. Measure cold and warm behavior with representative input and output lengths.
3. Increase concurrency. Check normal and long requests at several active-request levels.
4. Test arrival bursts. Send requests at the expected rate so queues and overload behavior become visible.
5. Check quality. Repeat task tests after changing precision, context, or serving settings.
6. Record the pass conditions. Include latency, successful throughput, memory, quality, and required reserve.

Use several measured repetitions after consistent warm-up. Report failures, sample counts, and the spread of results. A p99 figure based on too few requests can mislead.

If prefix caching is enabled, test both repeated and new prefixes. If embeddings or reranking share the machine, include them in the load test. A synthetic short-prompt result cannot establish long-document capacity.

Use current tools and record their versions. NVIDIA’s current GenAI-Perf documentation says it is being phased out and directs new benchmarking work to AIPerf. 

When memory is the limiting resource

First identify whether the problem is model loading, cache reservation, context growth, or other services. Then change one factor and test again.

Possible options include a smaller model, a supported quantized artifact, shorter context limits, fewer active sequences, or another placement plan. These choices can change quality, queueing, and throughput.

vLLM documents context-length and sequence limits as memory controls, and notes that CUDA graph capture uses additional GPU memory. Treat settings as part of the measured configuration. 

Compare self-hosting cost with APIs and cloud GPUs

Compare only configurations that meet the required task quality and response time. A cheap server that produces unusable answers is not a saving.

For a rented dedicated server, include:

• Rental and setup charges over the planned term.
• Storage, transfer, software, and support charges.
• Operating work for updates, monitoring, and incidents.
• Capacity needed for maintenance or failure handling.

Do not add power again when the rental charge already includes it. For owned hardware, use a separate ownership model covering purchase, energy, facilities, and support.

Cost per accepted request = complete cost during a period ÷ accepted requests during that period

For example, a fictional complete monthly cost of $1,200 divided by 600,000 accepted requests gives $0.002 per accepted request. This is a teaching calculation, not an Atal quote or measured serving capacity.

Do not assume ten times the demand fits the same server at the same response time. The cost changes when capacity, staffing, or reserve needs change.

For API comparisons, price input tokens, output tokens, caching, and other billed categories separately. For cloud GPUs, include running hours, attached storage, transfers, and required supporting services.

There is no universal token volume at which self-hosting becomes cheaper. Utilization, model quality, request length, hardware, and operating responsibility determine the result.

Check deployment limits before ordering

Hardware fit is one part of readiness. Confirm the exact model license and permitted use. Check driver, runtime, and artifact compatibility.

Control access to the inference endpoint. Set request-size, output, queue, and rate limits. Define logging and retention rules. Review outbound connections, external embeddings, tools, and fallback APIs if data must remain inside a specific boundary.

A rented dedicated server is not automatically on premises or air-gapped. Self-hosting gives you control over deployment choices; it does not prove that every prompt stays private under every configuration.

Plan model updates, rollback files, monitoring, and recovery. If service must continue during a host failure, the architecture needs spare capacity outside that host. Multiple GPUs inside one server do not provide a second host.

Frequently asked questions

Can I self-host an LLM without a GPU?

Yes. Supported runtimes can run models on CPUs, and some can divide work between CPU and GPU. Start with a model and format that fit system RAM, then test response time. CPU-only inference can suit batch or low-frequency tasks, but interactive suitability must be measured.

Is 24 GB of VRAM enough for an LLM server?

It depends on the artifact, context, active requests, and runtime. Several compact quantized models may fit, but memory fit does not establish production capacity. A nominal 70B model’s idealized four-bit weights alone require 35 GB, which exceeds 24 GB of available VRAM.

How much memory does a 70B model need?

The idealized weight floors are 140 GB at 16 bits, 70 GB at eight bits, and 35 GB at four bits. Add format overhead, cache, working memory, and reserve. Placement across GPU and host memory depends on the runtime.

Can two 24 GB GPUs replace one larger GPU?

Sometimes, if the model and runtime support a suitable split. They are not automatically equivalent to one larger device. Communication, per-device overhead, cache placement, and server topology affect both fit and speed.

Why does the model load but fail with several requests?

Loading checks whether the initial model allocation fits. Additional requests can increase cache and working-memory demand. Measure the exact failure point and check context, active-sequence limits, and other loaded services before changing hardware.

How much system RAM does an inference server need?

Budget for startup peaks, the operating system, runtime, API services, CPU offload, and any retrieval components. Measure both loading and steady operation. There is no fixed system-RAM requirement that follows from parameter count or GPU memory alone.

Is a dedicated server cheaper than an API?

It can be, but compare complete costs at equivalent task quality and response time. Low or irregular demand may leave rented hardware idle. Sustained demand can improve utilization, provided the selected server can handle it within the service target.

Recommended Next Steps

Choose hardware from a documented request profile, not parameter count alone. Calculate weights and cache separately, add the full server budget, and test the configurations that remain.

Before committing:

1. Fix the model revision, artifact, context, and active-request target.
2. Calculate the memory budget using consistent units.
3. Confirm runtime support and per-device placement.
4. Test latency, throughput, quality, and failure behavior.
5. Compare complete costs over the same period.

To discuss a suitable configuration, review Atal Networks dedicated servers and share your model, request profile, memory estimate, location, and software requirements. Confirm GPU availability and supported options for the specific offer before ordering.

Sources and calculation notes

Sources were checked on October 28, 2026. Model configurations, runtime support, and software behavior can change, so record the version used for a deployment.

• Meta Llama model configurations.
• Hugging Face cache strategies.
• vLLM memory controls.
• vLLM parallelism and scaling.
• Ollama FAQ.
• llama.cpp project.
• NVIDIA inference benchmarking concepts.
• NVIDIA benchmark-tool status.

Weight tables use idealized bit counts and decimal GB. KV-cache figures use binary GiB and the stated full-attention assumptions. The 5 GiB weight allocation, 2 GiB working-memory input, 2 GiB reserve, and dollar example are fictional teaching inputs. They are not Atal measurements, prices, or capacity claims.