Linux Network Tuning for 10Gbps Dedicated Servers: Measure Before You Change

Linux Network Tuning for 10Gbps banner

Linux network tuning for 10Gbps starts with throughput, latency, packet-drop, and CPU measurements. A useful change addresses a measured limit, improves the real workload, and can be reversed. Larger buffers, more queues, or a different TCP algorithm can help some transfers and make other traffic worse.

 

We use a repeatable process in this guide: record the starting state, locate the likely limit, test one change, and compare the result. The goal is a server that meets its workload targets without extra delay, wasted memory, or fragile configuration.

What should you measure before tuning a 10Gbps Linux server?

Measure useful throughput and the resources consumed to deliver it. Port speed alone does not show whether the limit sits in the host, application, remote endpoint, or network path.

 

If you have not located the likely layer, start with our guide to diagnosing a slow 10Gbps server. Continue here when the evidence points to Linux host processing or TCP behavior.

Set a goal for the real workload

Write down what needs to improve. For a backup server, that could be completion time. For an API server, request latency during busy periods may matter more than peak transfer speed.

 

측정 기록할 내용 중요한 이유
Receiver throughput Delivered Gbit/s or application MB/s Measures useful delivery
Idle and loaded latency RTT and application response times Shows delay added during traffic
Retransmissions Change during each TCP test Helps identify delivery problems
Interface counters Errors and drops before and after Helps locate device or link trouble
CPU per core Busy cores and softirq activity Exposes limits hidden by average CPU use
Memory Socket use and host memory pressure Checks the cost of buffer changes
Application result Completion time or response-time distribution Shows whether the change helped users

 

Set acceptable limits before testing. We recommend recording the median and spread across at least three comparable trials. This is a practical comparison method, not a claim that three runs describe every traffic condition.

Record the host and path

The examples use Bash and common Linux tools: iproute2, ethtool, procps, sysstat, and iperf3. Install missing tools through your distribution’s package manager. Flags, counters, and supported settings depend on the installed versions and driver.

 

Run these read-only checks on each endpoint:

 

cat /etc/os-release

uname -r

iperf3 – 버전

ethtool –version

ip -br address

 

In the commands below, replace the example values before use. 192.0.2.20 is a documentation address, not a public test server.

 

peer=192.0.2.20

iface=enp1s0

 

ip route get “$peer”

ethtool “$iface”

ethtool -i “$iface”

 

Check that the route uses the expected interface and source address. A bond, bridge, VLAN, or tunnel may require inspection of both the logical interface and the underlying physical port.

 

Record NIC model, firmware, negotiated link speed, and the peer’s capacity. Check the NIC’s PCIe connection against its hardware requirements if the host remains suspect. Our 10Gbps 서버 선택 가이드 covers the wider hardware decision.

Save state before making changes

Create a private working directory in the same Bash terminal:

 

umask 077

run_dir=$(mktemp -d “$PWD/network-tuning.XXXXXX”)

printf ‘Test records: %s\n’ “$run_dir”

date -Is > “$run_dir/start-time.txt”

 

Keep its absolute path in your change record. Shell variables disappear when the session ends.

 

Use console or out-of-band access before changing an interface that carries SSH. Confirm who manages the settings: a network profile, TuneD, a boot service, or a deployment tool may reapply them. Schedule tests so that they do not disrupt customer traffic.

Run single-stream, parallel-stream, and reverse tests

Test one TCP stream, several streams, and both directions before changing Linux network settings. Each result answers a different question.

 

Use two separate, capable hosts. Start the listener on the intended receiving endpoint and restrict TCP port 5201 to the test peer through your normal firewall controls:

 

iperf3 -s

 

On the client, run these tests one at a time:

 

iperf3 -c “$peer” -P 1 -T 30 -J > “$run_dir/forward-one.json”

iperf3 -c “$peer” -P 4 -T 30 -J > “$run_dir/forward-four.json”

iperf3 -c “$peer” -P 1 -T 30 -R -J > “$run_dir/reverse-one.json”

iperf3 -c “$peer” -P 4 -T 30 -R -J > “$run_dir/reverse-four.json”

 

-P selects the stream count. -r makes the server send to the client. -J saves JSON output. Use a new filename for each repeat.

 

Beginning with iperf3 3.16, each test stream runs in its own thread. Record the version on both endpoints because tool behavior can affect CPU use and results.

 

If four streams outperform one, investigate the single-flow limit. Do not treat the larger total as proof that one download can achieve the same rate.

 

Keep the peer, duration, direction, stream count, and background load as consistent as possible. A 30-second interval at a constant 10Gbps represents 37.5 GB at the raw-rate calculation, before protocol overhead. Actual transferred and billable volume depends on the test and service terms.

Measure while the connection is active

Use another terminal on each host. Set its peer 과 iface variables for that host:

 

mpstat -p 모두 1 10

ss -tinm dst “$peer”

 

SS can display TCP information and socket memory. Inspect fields such as RTT, retransmissions, congestion window, and send or receive limits when your version exposes them. A field is evidence to interpret, not a diagnosis by itself.

 

Compare ping before and during the load as a rough path-delay check. Also measure application latency. ICMP replies do not replace HTTP, database, or game-server response measurements.

 

For the complete test setup and output interpretation, use our 10Gbps iperf3 testing guide.

Match the symptom to the next check

Choose a tuning test only after the measurements point to a likely limit. The same throughput symptom can have several causes.

 

Measured symptom Check next Possible action
One stream is slow; several streams are faster RTT, socket state, one busy core, peer limits Test the specific flow limit
One receive-processing core stays busy Queue activity, IRQ distribution, CPU topology Assess queue or CPU placement
NIC receive-drop counters rise Driver counter definitions, ring depth, CPU drain rate Test a supported ring change
Kernel backlog-drop counters rise Per-core processing, backlog, processing budgets Test the matching software limit
Latency rises sharply under load Local and path queues, active qdisc Investigate queueing and pacing
Physical error counters rise Cable, optic, port, driver, switch evidence Investigate the link
iperf3 is healthy; file transfers remain slow Storage, encryption, application workers, remote storage Test the application path
단 하나의 목적지만 느립니다 Remote host and network path Compare another capable endpoint

 

Collect absolute counters before each test:

 

ip -s link show dev “$iface” > “$run_dir/link-before.txt”

ethtool -S “$iface” > “$run_dir/nic-before.txt”

nstat -asz > “$run_dir/nstat-before.txt”

cat /proc/net/softnet_stat > “$run_dir/softnet-before.txt”

 

Repeat after the test with after filenames. nstat -asz prints absolute counters, includes zero values, and avoids updating its history file.

 

Calculate changes over the same interval. A nonzero lifetime counter may describe an old event. Host-wide counters may also include unrelated traffic, so compare them with the workload and device evidence.

Use RTT and BDP to assess TCP buffer needs

Use bandwidth-delay product, or BDP, to estimate the data that must be in flight at a target rate. Then inspect socket behavior before testing larger buffers.

 

The calculation is:

 

BDP in bytes = target rate in bits per second × RTT in seconds ÷ 8

 

At a target rate of 10,000,000,000 bits per second:

 

RTT 비행 중 계산된 데이터 Decimal MB
1 ms 1,250,000 bytes 1.25 MB
10ms 12,500,000 bytes 12.5MB
50 ms 62,500,000 bytes 62.5 MB
100ms 125,000,000 bytes 125MB

 

These are calculated values, not benchmark results or direct sysctl recommendations. TCP window behavior, memory accounting, loss, flow sharing, and endpoint capacity still matter. Red Hat uses BDP as part of its TCP buffer-sizing procedure and warns about memory and latency costs.

Understand the settings you are testing

Read the current values:

 

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem

sysctl net.ipv4.tcp_moderate_rcvbuf

sysctl net.core.rmem_max net.core.wmem_max

 

tcp_rmem 과 tcp_wmem contain minimum, initial, and maximum values. The maximum controls the automatically selected TCP buffer size. Receive autotuning uses tcp_moderate_rcvbuf.

 

An application that sets SO_RCVBUF 또는 SO_SNDBUF changes the sizing behavior for that socket. Inspect its configuration and applicable core limits before assuming the TCP autotuning ceiling controls the application. Linux also includes accounting overhead in socket-buffer handling.

 

Do not raise minimums and initial sizes merely because the NIC supports 10Gbps. A higher maximum is a ceiling, not that amount of memory allocated to every socket. Connection count and memory pressure still need review.

Example: test a receive-buffer ceiling

Use this example only when evidence points to a receive-buffer limit and the host has room for the trial. Suppose you have selected a 64 MiB ceiling for a controlled test. That is an example value, not a 10Gbps preset.

 

On the receiving host, save the original triplet:

 

sysctl net.ipv4.tcp_rmem > “$run_dir/tcp-rmem.before.conf”

read -r rx_min rx_initial rx_max < <(sysctl -n net.ipv4.tcp_rmem)

trial_rx_max=67108864

 

If the read succeeded and the current maximum is smaller, test only the maximum:

 

if [[ “$rx_min” =~ ^[0-9]+$ &&

      “$rx_initial” =~ ^[0-9]+$ &&

      “$rx_max” =~ ^[0-9]+$ ]] &&

   (( rx_max < trial_rx_max )); then

    sudo sysctl -w “net.ipv4.tcp_rmem=$rx_min $rx_initial $trial_rx_max”

else

    printf ‘%s\n’ ‘No change: inspect the current values.’

FI

 

sysctl net.ipv4.tcp_rmem

 

Repeat the same transfer and inspect the socket, throughput, latency, and memory result. Restore the original value after the trial unless the result justifies keeping it. First check that the live setting still belongs to your trial and no other administrator changed it:

 

cat “$run_dir/tcp-rmem.before.conf”

sysctl net.ipv4.tcp_rmem

sudo sysctl -p “$run_dir/tcp-rmem.before.conf”

sysctl net.ipv4.tcp_rmem

 

sysctl -p loads the named file. Use the saved file for this host and trial only. Keep the send-buffer test separate, on the sender, with its own saved tcp_wmem value.

Tune NIC rings and kernel backlogs only for matching drops

Increase queue capacity only when the matching counters and workload support that test. A larger queue cannot make a persistently overloaded CPU process packets faster.

Keep queue types separate

Queue or buffer What it handles Inspection
NIC RX/TX rings Packet descriptors used by the device and driver ethtool -g, driver statistics
Kernel input backlog Packets awaiting software processing where this path applies softnet_stat, relevant sysctl values
TCP socket buffers Data associated with a connection ss -tinm, TCP settings
Listen queues Pending connection work Listener state and application metrics
Egress qdisc Packet scheduling on transmit tc -s qdisc show

 

Do not use a larger listen backlog as a general remedy for a slow established transfer. First identify whether the problem concerns connection acceptance or data delivery.

Example: test RX ring depth

Inspect and save the current state:

 

ethtool -g “$iface” | tee “$run_dir/rings-before.txt”

ethtool -S “$iface” > “$run_dir/ring-counters-before.txt”

 

Suppose the current RX ring is 512 descriptors, the driver supports 1024, and its documented counters show receive-ring pressure during the test. In a maintenance window, a supported trial is:

 

sudo ethtool -G “$iface” rx 1024

ethtool -g “$iface”

 

Repeat the workload and compare drops, throughput, latency, and CPU use. If the trial does not meet the goal, and the current state still matches your trial, restore this example’s original value:

 

sudo ethtool -G “$iface” rx 512

ethtool -g “$iface”

 

Use the values from your own device. Some drivers can briefly interrupt traffic during ring changes. Recheck related device state after the operation.

Check software backlog and processing budgets

Read the relevant controls:

 

sysctl net.core.netdev_max_backlog

sysctl net.core.netdev_budget net.core.netdev_budget_usecs

 

netdev_max_backlog limits the input backlog. netdev_budget 과 netdev_budget_usecs limit work within a packet-processing polling cycle.

 

Compare per-CPU softnet_stat snapshots. On commonly documented layouts, the second field records backlog drops and the third records processing-budget pressure. Confirm the layout for the running kernel before parsing its hexadecimal fields.

 

If a matching counter rises, test one supported change at a time and save its old value first. More processing time can compete with application work. More backlog can add waiting time. Restore the setting if the trade-off fails your service targets.

Check RSS, IRQ distribution, and NUMA placement

Inspect CPU placement when traffic overloads one processing core or a small group of queues. Low total CPU use does not rule out a per-core limit.

 

Receive Side Scaling, or RSS, maps flows to hardware receive queues. A flow normally stays on one queue; adding queues does not split that flow across every CPU. RPS steers receive processing in software, RFS considers application placement, and XPS helps select transmit queues.

 

Read the queue and interrupt state:

 

ethtool -l “$iface”

ethtool -x “$iface”

cat /proc/interrupts

lscpu -e

systemctl status irqbalance

 

For a physical PCIe NIC, also inspect:

 

cat “/sys/class/net/$iface/device/numa_node”

 

A value of -1 means the node is unknown. Logical interfaces may not expose that path.

 

Use the following sequence:

 

  1. Compare per-core load with queue and IRQ activity during the test.
  2. Check whether irqbalance already manages the assignment.
  3. Compare NIC locality with the CPUs and application workers involved.
  4. Test one placement or supported channel change only if the evidence warrants it.
  5. Re-read queue and IRQ identities after the change.

 

Do not paste a fixed CPU mask from another machine. Save the original assignments and manager state before any manual trial. If you restore channel counts, check the resulting IRQ identifiers before restoring placement. irqbalance may overwrite manual placement.

Test congestion control and qdiscs as separate changes

Compare TCP algorithms only under matched conditions, and inspect packet scheduling as a separate part of the test. Changing both at once makes the result harder to explain.

 

Read the current state:

 

sysctl net.ipv4.tcp_available_congestion_control

sysctl net.ipv4.tcp_congestion_control

tc -s qdisc show dev “$iface”

 

The available list shows registered algorithms. The default does not establish which algorithm every existing socket uses.

 

For a forward iperf3 test, -C can request a sender-side algorithm without changing the host default. If both choices are available and permitted, compare:

 

iperf3 -c “$peer” -P 1 -T 30 -C cubic -J > “$run_dir/cubic-one.json”

iperf3 -c “$peer” -P 1 -T 30 -C bbr -J > “$run_dir/bbr-one.json”

 

Confirm the selected algorithm and inspect errors. Use repeated trials and record the installed kernel’s implementation. “BBR” alone does not describe every version or downstream build.

 

Keep the qdisc unchanged during this comparison. Assess receiver throughput, loaded latency, retransmissions, and other active traffic. Neither algorithm wins every workload.

 

A qdisc schedules egress packets. The fq scheduler supports per-flow pacing, but changing a default qdisc setting is not evidence that the active interface hierarchy changed.

 

Before any qdisc trial, prepare commands that can recreate the original hierarchy, including classes, filters, and options. A text listing is a record, not an automatic restore script. Do not replace production traffic policy with a generic tc qdisc replace command.

Check offloads and interrupt coalescing against the workload

Keep working offloads unless a specific workload, driver issue, or measurement gives a reason to test a change. Disabling every feature can add CPU work.

 

얼굴의 생김새 Main function
TSO Lets supported hardware segment TCP data
GSO Supports segmentation later in the transmit path
GRO Combines compatible received packets for processing
Checksum offload Moves supported checksum work to the device

 

Feature dependencies matter. For example, TCP segmentation offload depends on partial checksum support.

 

Read and save the full state:

 

ethtool -k “$iface” | tee “$run_dir/offloads-before.txt”

ethtool -c “$iface” | tee “$run_dir/coalesce-before.txt”

 

If you have a specific reason to test GRO, confirm that it is currently on and can be changed. A temporary off trial is:

 

sudo ethtool -K “$iface” gro off

ethtool -k “$iface”

 

Repeat the workload. Compare CPU cost and application latency as well as throughput. If this trial started with GRO on, restore it after checking the live state:

 

sudo ethtool -K “$iface” gro on

ethtool -k “$iface”

 

Use the full feature listing to check for dependent changes. Entries marked [fixed] cannot be changed through that control.

 

Interrupt coalescing reduces interrupt frequency by batching notifications. This can reduce processing cost but may add delay. Inspect adaptive settings before testing timing or frame thresholds. Save all changed values and repeat the latency test after restoration.

 

Local packet captures can also reflect offload processing rather than packets exactly as they appear on the wire. Check capture location and driver behavior before treating large captured packets or checksum reports as link faults.

Check path MTU before testing jumbo frames

Use a larger MTU only on a path that supports it. A 10Gbps port does not require an MTU of 9000.

 

The interface MTU limits packet size on that interface. Path MTU concerns the usable size across the route. TCP MSS describes TCP data size within a segment. Tunnels add headers, which can reduce the size available to inner traffic.

 

Read the current interface value:

 

ip link show dev “$iface”

cat “/sys/class/net/$iface/mtu”

 

For a controlled private network, confirm support on both endpoints, switches, logical interfaces, and any intermediate devices before a trial. Driver limits differ; the ixgbe documentation, for example, lists adapter-specific MTU and configuration details.

 

Use this order:

 

  1. Record the current MTU and persistent profile.
  2. Confirm the complete relevant path supports the trial size.
  3. Arrange console access and a maintenance window.
  4. Change only the intended path and verify connectivity.
  5. Repeat the workload and compare CPU use, throughput, and latency.
  6. Restore the saved value if the trial fails.

 

Do not change an internet-facing interface to 9000 because two local devices support it. Also, tcp_mtu_probing controls TCP packetization-layer probing; it does not turn an incompatible path into a jumbo-frame path.

Repeat the application test and keep only useful changes

Keep a tuning change only when the real workload improves within its resource and latency limits. A better iperf3 total is one part of that decision.

 

Consider a backup job that reads data, encrypts it, sends it, and writes it at the destination. Network-only throughput can improve while the job remains limited by either storage system or encryption work. Test the complete job before making another kernel change.

 

작업량 Result to check after tuning
Backup or replication Completion time, storage activity, other services’ latency
HTTPS downloads Delivered rate, TLS CPU use, application workers, client capacity
API or database service Request rate, errors, and high-percentile response time
VPN gateway Encrypted throughput, CPU use, tunnel MTU, forwarding behavior
Game or voice service Application delay, packet delivery, and CPU scheduling

 

For each change, complete a record:

 

분야 Entry
Goal Workload and acceptance limits
증거 Counter changes and observed limit
Saved state Host, interface, namespace, values, files, manager
Trial One setting and the reason for testing it
기준선 Median and spread across comparable runs
After change Throughput, latency, drops, CPU, memory, application result
Decision Keep, restore, or investigate another layer
Restore check Original state restored and normal traffic checked

 

Use a fresh record for a new hypothesis. A small gain within normal run-to-run variation is weak evidence for a permanent change.

Keep other traffic types in scope

TCP endpoint tests do not establish the performance of a forwarding gateway, a UDP service, or QUIC. Test packet size, packet rate, concurrency, encryption, and the actual application path for those workloads. TCP congestion-control settings do not select the congestion algorithm used by a QUIC implementation.

 

For containers or network namespaces, identify where the setting applies. Record whether a command reads the host, a namespace, or a virtual interface before comparing values.

Restore failed changes and make proven settings persistent

Restore the exact saved state for the setting you changed. Then check whether a persistent file or manager can reapply the trial.

 

Setting Save before testing Restore and check
sysctl Exact targeted value or triplet Load the saved value and re-read it
Rings Current and supported ring state Restore supported original values and inspect related state
Channels and RSS Counts, mapping, and related IRQ state Restore counts and mapping; recheck IRQ identities
IRQ placement Assignments and irqbalance state Restore the correct current IRQs and manager state
Offloads Full supported feature state Restore changed features and inspect dependencies
Coalescing Timing, thresholds, adaptive state Restore recorded settings and repeat latency checks
MTU Live value and network profile Restore both as needed and check connectivity
TCP algorithm Default and application overrides Restore the relevant setting and test new connections
Qdisc Hierarchy plus reconstruction commands Recreate the original policy and inspect it

 

If live values differ from your trial, stop and check ownership before restoring anything. Another administrator or configuration service may have changed them.

Persist only the accepted settings

For an accepted sysctl change, place the reviewed assignment in a dedicated file under /etc/sysctl.d/, using the site’s naming and configuration-management rules. Apply that file explicitly and re-read the value. Avoid reloading every sysctl file just to apply one test result.

 

On systemd-based systems, sysctl.d files can set values at boot, and file ordering affects which assignment wins. Keep the reason, baseline, accepted result, owner, and rollback value beside the change record.

 

Use your distribution’s supported network-profile mechanism for persistent ethtool or MTU settings. Check what happens after an interface restart, driver reload, and planned reboot.

 

Deleting a trial file alone does not restore a live sysctl value. Restore the runtime value as well, and remove or correct the entry that would reapply it.

자주 묻는 질문

Do modern Linux servers need manual tuning for 10Gbps?

Some workloads need a targeted change; others already meet their goals with current settings. Start with a repeatable test and live evidence. If throughput, latency, drops, CPU, and memory meet the service targets, keep the configuration and monitor it. Port speed alone is not a reason to replace Linux defaults.

Why is one TCP stream slower than several streams?

Multiple streams can use more processing resources and have separate TCP state. A single stream may face a per-core, window, congestion, application, or peer limit. Compare both results, then inspect the active socket and CPU distribution. The multi-stream total does not establish the speed available to one connection.

How much TCP buffer does a 10Gbps connection need?

There is no single value for every 10Gbps path. At 50 ms RTT, the calculated BDP at a 10Gbps target is 62.5 MB of data in flight. Actual buffer settings also depend on TCP behavior, accounting overhead, application options, memory pressure, and flow count. Use the calculation to guide investigation, then test.

Should we use BBR or keep CUBIC?

Keep the current algorithm until a matched comparison supports a change. Test the available implementation with the same peer, traffic pattern, and qdisc. Include loaded latency and competing traffic in the result. Record kernel and tool versions so that the comparison can be repeated after an update.

Should we disable GRO and TSO?

Do not disable them as a general 10Gbps tuning step. These features reduce or move packet-processing work. Test a specific feature only when the workload or a documented issue gives a reason. Record the starting state, inspect dependent features, and compare CPU cost and latency before keeping the change.

Do 10Gbps dedicated servers need jumbo frames?

No. Jumbo frames are a separate option for compatible paths. They may reduce packet-processing work in a suitable environment, but require end-to-end planning. Confirm endpoints, switches, logical interfaces, and tunnel overhead before testing a larger MTU. Restore the saved value if connectivity or application performance worsens.

Can larger queues make latency worse?

Yes. Extra queue capacity can let packets wait longer under load. A queue that handles short bursts better may still hurt a service with strict response-time limits. Measure application latency during the same traffic test. If extra depth only postpones drops while processing remains overloaded, investigate processing capacity or traffic demand.

How do we undo Linux network tuning changes?

Restore the exact saved value for each changed setting, inspect the live result, and check normal traffic. Then remove or correct the persistent file or profile that could reapply the trial. For qdiscs, use a prepared reconstruction procedure. For channel changes, recheck IRQ identities before restoring placement.

위로 스크롤