{"id":25139,"date":"2026-10-28T06:25:00","date_gmt":"2026-10-28T06:25:00","guid":{"rendered":"https:\/\/atalnetworks.com\/?p=25139"},"modified":"2026-10-28T06:28:33","modified_gmt":"2026-10-28T06:28:33","slug":"llm-inference-server-sizing","status":"publish","type":"post","link":"https:\/\/atalnetworks.com\/ar\/llm-inference-server-sizing\/","title":{"rendered":"Self-Hosting LLMs on a Dedicated Server: Hardware Sizing Guide"},"content":{"rendered":"

To self-host an LLM on a dedicated server, size the hardware for the model, context length, active requests, and response-time target.<\/b> Budget for model weights, request memory, runtime overhead, system RAM, CPU work, and storage. A model that loads successfully has passed a memory check, not a production-capacity test.<\/span><\/p>\n

The gap becomes clear during a busy hour. One short request works. Several users submit long documents, the request queue grows, and the same server runs out of memory or takes too long to respond.<\/span><\/p>\n

Avoid that problem by planning in this order:<\/span><\/p>\n

    \n
  1. Define the exact model and request profile.<\/span><\/li>\n
  2. Calculate weight memory and cache memory separately.<\/span><\/li>\n
  3. Budget for the complete server and supporting services.<\/span><\/li>\n
  4. Test realistic traffic against response-time and quality targets.<\/span><\/li>\n
  5. Compare complete costs for the configurations that pass.<\/span><\/li>\n<\/ol>\n

    Start with the model and request profile<\/span><\/h2>\n

    Server sizing starts with what the application must do. \u201cRun an LLM for our team\u201d is too broad to produce a useful hardware recommendation.<\/span><\/p>\n

    Record these inputs before comparing servers:<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
    Input<\/b><\/th>\nWhat to record<\/b><\/th>\n<\/tr>\n<\/thead>\n
    Model<\/span><\/td>\nExact name, revision, architecture, and license<\/span><\/td>\n<\/tr>\n
    Weight format<\/span><\/td>\nFP16, BF16, or the specific quantized artifact<\/span><\/td>\n<\/tr>\n
    Context<\/span><\/td>\nNormal and maximum prompt-plus-output token counts<\/span><\/td>\n<\/tr>\n
    Traffic<\/span><\/td>\nActive generations, arrival rate, and burst behavior<\/span><\/td>\n<\/tr>\n
    \u0627\u0644\u0623\u062f\u0627\u0621<\/span><\/td>\nTime to first token and acceptable completion time<\/span><\/td>\n<\/tr>\n
    Quality<\/span><\/td>\nTasks the model must complete correctly<\/span><\/td>\n<\/tr>\n
    Other services<\/span><\/td>\nEmbeddings, reranking, vector search, APIs, and monitoring<\/span><\/td>\n<\/tr>\n
    Operations<\/span><\/td>\nUpdate process, recovery needs, and growth reserve<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Context includes system instructions, chat history, retrieved text, user input, and generated output. A short question can produce a long model input if the application adds several documents.<\/span><\/p>\n

    Also separate users from active requests. A service with 100 registered users may have only a few generations running at once. An agent used by one person may issue several requests during a single task.<\/span><\/p>\n

    For a definition of the serving process, see <\/span>what AI inference means<\/span><\/a>.<\/span><\/p>\n

    Keep inference separate from training<\/span><\/h3>\n

    This guide covers inference: using an already trained model to produce an answer. Training and fine-tuning can require additional memory for gradients, optimizer state, and activations, depending on the method.<\/span><\/p>\n

    Do not use an inference sizing table as a training recommendation. If the same machine must serve users and run fine-tuning jobs, test the combined schedule or separate those workloads.<\/span><\/p>\n

    Calculate model-weight memory first<\/span><\/h2>\n

    Model weights are the stored numerical values used during inference. Their memory footprint provides the starting point for a hardware budget.<\/span><\/p>\n

    Idealized weight memory in bytes = parameter count \u00d7 bits per weight \u00f7 8<\/b><\/p>\n

    For a nominal eight-billion-parameter model at 16 bits:<\/span><\/p>\n

    8,000,000,000 \u00d7 16 \u00f7 8 = 16,000,000,000 bytes, or 16 GB<\/b><\/p>\n

    At an idealized four bits per weight, the same calculation gives 4 GB.<\/span><\/p>\n\n\n\n\n\n\n\n\n
    Nominal model size<\/b><\/th>\nFP16\/BF16 weight floor<\/b><\/th>\n8-bit weight floor<\/b><\/th>\n4-bit weight floor<\/b><\/th>\n<\/tr>\n<\/thead>\n
    8B<\/span><\/td>\n16 GB<\/span><\/td>\n8 GB<\/span><\/td>\n4 GB<\/span><\/td>\n<\/tr>\n
    14B<\/span><\/td>\n28 GB<\/span><\/td>\n14 GB<\/span><\/td>\n7 GB<\/span><\/td>\n<\/tr>\n
    32B<\/span><\/td>\n64 GB<\/span><\/td>\n32 GB<\/span><\/td>\n16 GB<\/span><\/td>\n<\/tr>\n
    70B<\/span><\/td>\n140 GB<\/span><\/td>\n70 GB<\/span><\/td>\n35 GB<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    These figures cover idealized weights only.<\/b> They exclude request cache, working buffers, quantization metadata, and other runtime allocations. The exact model may also have more parameters than its rounded name suggests.<\/span><\/p>\n

    Why a 4-bit file needs more than the raw calculation<\/span><\/h3>\n

    A quantized artifact can contain scale values, metadata, and tensors stored at other precisions. Its downloaded size and loaded memory footprint can differ.<\/span><\/p>\n

    Record the exact artifact, then inspect memory after loading it in the intended runtime. Do not treat every format labeled \u201c4-bit\u201d as interchangeable.<\/span><\/p>\n

    Reducing weight precision also does not automatically reduce the precision of the request cache. Those are separate settings with separate compatibility and quality checks.<\/span><\/p>\n

    Use total parameters for MoE weight planning<\/span><\/h3>\n

    A mixture-of-experts model, or MoE model, activates only part of its expert network for each token. That can reduce computation relative to using every expert, but the other weights still need a placement plan.<\/span><\/p>\n

    For full residency, budget for the stored model, not just its active parameter count. Expert offload and distributed placement can change where memory is used, but they introduce their own transfer and execution costs.<\/span><\/p>\n

    Keep GB and GiB consistent<\/span><\/h3>\n

    One GB is one billion bytes. One GiB is 1,073,741,824 bytes. A 4 GB payload is about 3.73 GiB.<\/span><\/p>\n

    Convert values to the same unit before subtracting model memory from a device budget. Use the actual available memory reported for the deployment rather than mixing a product label with a differently reported measurement.<\/span><\/p>\n

    Add KV cache, context length, and concurrent requests<\/span><\/h2>\n

    The key-value cache, usually called the <\/span>KV cache<\/b>, stores attention data used while the model generates text. Model weights can be shared across requests within one model instance, but active sequences also need request state.<\/span><\/p>\n

    That is why longer contexts and more active requests can consume memory even when the loaded model does not change.<\/span><\/p>\n

    For a conventional decoder with the same full-attention layout in each layer, estimate logical cache payload as:<\/span><\/p>\n

    KV bytes \u2248 2 \u00d7 layers \u00d7 KV heads \u00d7 head dimension \u00d7 bytes per cached value \u00d7 total cached tokens<\/b><\/p>\n

    The factor of two accounts for keys and values. Total cached tokens means the sum across active sequences. If all sequences have the same length, multiply that length by the number of active sequences.<\/span><\/p>\n

    This is not a universal formula for every model. Sliding-window attention, other cache architectures, cache quantization, and runtime allocation rules can change the calculation. Hugging Face documents cache strategies with different memory behavior, including limits for sliding-window and chunked-attention layers.\u00a0<\/span><\/p>\n

    A worked cache example<\/span><\/h3>\n

    Meta’s Llama 3.1 8B configuration uses 32 layers, 32 attention heads, eight KV heads, and a hidden dimension of 4,096. The head dimension is 4,096 \u00f7 32 = 128.<\/span><\/p>\n

    Assume a two-byte KV cache, full attention, and no prefix sharing:<\/span><\/p>\n

    2 \u00d7 32 \u00d7 8 \u00d7 128 \u00d7 2 = 131,072 bytes per cached token per sequence<\/b><\/p>\n

    That equals 128 KiB.\u00a0<\/span><\/p>\n\n\n\n\n\n\n\n\n
    Cached tokens per sequence<\/b><\/th>\nActive sequences<\/b><\/th>\nCalculated logical KV payload<\/b><\/th>\n<\/tr>\n<\/thead>\n
    8,192<\/span><\/td>\n1<\/span><\/td>\n1 GiB<\/span><\/td>\n<\/tr>\n
    8,192<\/span><\/td>\n4<\/span><\/td>\n4 GiB<\/span><\/td>\n<\/tr>\n
    32,768<\/span><\/td>\n4<\/span><\/td>\n16 GiB<\/span><\/td>\n<\/tr>\n
    32,768<\/span><\/td>\n8<\/span><\/td>\n32 GiB<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The model is unchanged, yet request memory changes sharply. This is why adding a fixed percentage to model size cannot account for every context and concurrency target.<\/span><\/p>\n

    These are calculated payloads, not measured GPU allocations. A runtime may reserve cache blocks in advance or keep allocations for reuse. Memory reported by a monitoring tool may therefore differ from the live token payload.<\/span><\/p>\n

    Build a complete GPU memory budget<\/span><\/h2>\n

    For a fully GPU-resident deployment, start with:<\/span><\/p>\n

    GPU memory budget = loaded weights + KV allocation + runtime working memory + operating reserve<\/b><\/p>\n

    Use measured runtime values where possible. Avoid counting the same reserved cache pool twice.<\/span><\/p>\n

    Example: a short-context assistant versus a shared service<\/span><\/h3>\n

    The following is a <\/span>teaching scenario<\/b>, not an Atal benchmark. Assume:<\/span><\/p>\n