Performance Tuning on Linux — TCP
Tune TCP
We want to improve the performance of TCP applications. Some of what we do here can help our public services, predominantly HTTP/HTTPS, although performance across the Internet is limited by latency and congestion completely outside our control. But we do have complete control within our data center, and we must optimize TCP in order to optimize NFS and other internal services.
We will first provide adequate memory for TCP buffers. Then we will disable two options that are helpful on WAN links with high latency and variable bandwidth but degrade performance on fast LANs. Finally, we will increase the initial congestion window sizes.
Memory Management for TCP
Kernel parameters for TCP are set in
/proc/sys/net/core/[rw]mem* and
/proc/sys/net/ipv4/tcp*,
those apply for both IPv4 and IPv6.
As with all the categories we've seen so far, only a few
of the available kernel parameters let us make useful changes.
Leave the rest alone.
Parameters core/rmem_default and
core/wmem_default
are the default receive and send tcp buffer sizes, while
core/rmem_max and core/wmem_max
are the maximum receive and send
buffer sizes that can be set using
setsockopt(), in bytes.
Parameters ipv4/tcp_rmem and
ipv4/tcp_wmem
are the amount of memory in bytes for
read (receive) and write (transmit) buffers per open socket.
Each contains three numbers: the minimum, default, and maximum
values.
Parameter tcp_mem is the amount of memory
in 4096-byte pages totaled across all TCP applications.
It contains three numbers: the minimum, pressure, and maximum.
The pressure is the threshold at which TCP will start to
reclaim buffer memory to move memory use down toward the
minimum.
You want to avoid hitting that threshold.
# grep . /proc/sys/net/ipv4/tcp*mem /proc/sys/net/ipv4/tcp_mem:181419 241895 362838 /proc/sys/net/ipv4/tcp_rmem:4096 87380 6291456 /proc/sys/net/ipv4/tcp_wmem:4096 16384 4194304
Increase the default and maximum for
tcp_rmem and tcp_wmem
on servers and clients when they are on either
a 10 Gbps LAN with latency under 1 millisecond,
or communicating over high-latency low-speed WANs.
In those cases their TCP buffers may
fill and limit throughput, because
the TCP window size can't be made large enough to handle
the delay in receiving ACK's from the other end.
IBM's
High Performance Computing page
recommends 4096 87380 16777216.
Then, for tcp_mem, set it to twice
the maximum value for tcp_[rw]mem
multiplied by the maximum number of running network
applications divided by 4096 bytes per page.
Increase rmem_max and wmem_max
so they are at least as large as the third values of
tcp_rmem and tcp_wmem.
Calculate the bandwidth delay product, the total amount of data in transit on the wire, as the product of the bandwidth in bytes per second multiplied by the round-trip delay time in seconds. A 1 Gbps LAN with 2 millisecond round-trip delay means 125 Mbytes per second times 0.002 seconds or 250 kbytes.
If you don't have buffers this large on the hosts, senders have to stop sending and wait for an acknowledgement, meaning that the network pipe isn't kept full and we're not using the full bandwidth. Increase the buffer sizes as the bandwidth delay product increases. However, be careful. The bandwidth delay product is the ideal, although you can't really measure how it fluctuates. If you provide buffers significantly larger than the bandwidth delay product for connections outbound from your network edge, you are just contributing to buffer congestion across the Internet without making things any faster for yourself.
Internet congestion will get worse as Windows XP machines are retired. XP did no window scaling, so it contributed far less to WAN buffer saturation.
TCP Options
TCP Selective Acknowledgement (TCP SACK), controlled by
the boolean tcp_sack,
allows the receiving side to give the sender more detail
about lost segments, reducing volume of retransmissions.
This is useful on high latency networks, but disable this
to improve throughput on high-speed LANs.
Also disable tcp_dsack, if you aren't sending
SACK you certainly don't want to send duplicates!
Forward Acknowledgement works on top of SACK and will be
disabled if SACK is.
There is an option
tcp_slow_start_after_idle
which causes communication to start or resume gradually.
This is helpful if you are on a
variable speed WAN like 3G or 4G (LTE) mobile network.
But on a LAN or across a fixed-bandwidth WAN you want
the connection to start out going as fast as it can.
There are several TCP congestion control algorithms,
they are loaded as modules and
/proc/sys/net/ipv4/tcp_available_congestion_control
will list the currently loaded modules.
They are designed to quickly recover from packet loss
on high-speed WANs, so this may or may not be of interest
to you.
Reno is the TCP congestion control algorithm used
by most operating systems.
To learn more about some of the other choices:
BIC
CUBIC
High Speed TCP
Scalable TCP
TCP Low Priority
TCP Veno
YeAH TCP
TCP Illinois
DataCenter TCP (DCTCP)
Use modprobe to load the desired modules and then
echo or sysctl to place the desired
option into tcp_congestion_control.
Setting Kernel Parameters
The following file /etc/sysctl.d/02-netIO.conf
will be used by sysctl at boot time.
### /etc/sysctl.d/02-netIO.conf ### Kernel settings for TCP # Provide adequate buffer memory. # rmem_max and wmem_max are TCP max buffer size # settable with setsockopt(), in bytes # tcp_rmem and tcp_wmem are per socket in bytes. # tcp_mem is for all TCP streams, in 4096-byte pages. # The following are suggested on IBM's # High Performance Computing page net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 16777216 net.core.wmem_default = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 87380 16777216 # This server might have 200 clients simultaneously, so: # max(tcp_wmem) * 2 * 200 / 4096 net.ipv4.tcp_mem = 1638400 1638400 1638400 # Disable TCP SACK (TCP Selective Acknowledgement), # DSACK (duplicate TCP SACK), and FACK (Forward Acknowledgement) net.ipv4.tcp_sack = 0 net.ipv4.tcp_dsack = 0 net.ipv4.tcp_fack = 0 # Disable the gradual speed increase that's useful # on variable-speed WANs but not for us net.ipv4.tcp_slow_start_after_idle = 0
Tune Initial Window Size
The header on every TCP segment contains a 16-bit field advertising its current receive window size, 0 through 65,535, and a 32-bit field reporting the acknowledgement number. This means that everything up to the acknowledgement position in the stream has been received, and the receiver has enough TCP buffer memory for the sender to transmit up to the window size in bytes beyond the latest acknowledgement.
The idea is to more efficiently stream data by using buffers along the way to store segments and acknowledgements "in flight" in either direction. Across a fast switched LAN the improvement comes from allowing the receiver to keep filling the buffer while doing something else, putting off an acknowledgement packet until the receive buffer starts to fill.
Increases in the initial TCP window parameters can significantly improve performance. The initial receive window size or initrwnd specifies the receive window size advertised at the beginning of the connection, in segments, or zero to force Slow Start.
The initial congestion window size or initcwnd limits the number of TCP segments the server is willing to send at the beginning of the connection before receiving the first ACK from the client, regardless of the window size the client advertise. The server might be overly cautious, only sending a few kbytes and then waiting for an ACK because its initial congestion window is too small. It will send the smaller of the receiver's window size and the server's initcwnd. We have to put up with whatever the client says, but we can crank up the initcwnd value on the server and usually make for a much faster start.
Researchers at Google studied this. Browsers routinely open many connections to load a single page and its components, each of which will otherwise start slow. They recommend increasing initcwnd to at least 10.
CDN Planet has an interesting and much simpler article showing that increasing initcwnd to 10 cuts the total load time for a 14 kbyte file to about half the original time. They also found that many content delivery networks use an initcwnd of 10 and some set it even higher.
The initrwnd and initcwnd are specified in the routing table, so you can tune each route individually. If specified, they apply to all TCP connections made via that route.
First, look at the routing table. Let's use this simple example. This server has an Internet-facing connection on enp0s2 and is connected to an internal LAN through enp0s3. Let's say it's an HTTP/HTTPS server on the Internet side, and an client of NFSv4 over TCP on the internal side.
# ip route show default via 24.13.158.1 dev enp0s2 10.1.1.0/24 dev enp0s3 proto kernel scope link src 10.1.1.100 24.13.158.0/23 dev enp0s2 proto kernel scope link src 24.13.159.33
Now we will modify the routes to specify both initcwnd and initrwnd of 10 segments:
# ip route change default via 24.13.158.1 dev enp0s2 initcwnd 10 initrwnd 10 # ip route change 10.1.1.0/24 dev enp0s3 proto kernel scope link src 10.1.1.100 initcwnd 10 initrwnd 10 # ip route change 24.13.158.0/23 dev enp0s2 proto kernel scope link src 24.13.159.33 initcwnd 10 initrwnd 10 # ip route show default via 24.13.158.1 dev enp0s2 initcwnd 10 initrwnd 10 10.1.1.0/24 dev enp0s3 proto kernel scope link src 10.1.1.100 initcwnd 10 initrwnd 10 24.13.158.0/23 dev enp0s2 proto kernel scope link src 24.13.159.33 initcwnd 10 initrwnd 10
The appropriate commands could be added to
/etc/rc.d/rc.local for application
at boot time.
Changes to the TCP window size also affect UDP buffering.
On nets faster than 1 Gbps make sure that your applications
use setsockopt() to request larger
SO_SNDBUF and SO_RCVBUF.
Going Further
Calomel has a great page on performance tuning, although it is specific to FreeBSD. But read through their descriptions of how to tune BSD kernel parameters, and apply what you can to an analysis and tuning of your server.
And next...
Now that we have tuned the networking protocols, we can tune NFS file service running on top of TCP.