TCP/IP Buffer Sizes in Linux

TCP along with managing the basic data transfer process, also ensures that data is sent reliably. It manage the flow of data between devices to ensure that data is transferred efficiently without either device sending data faster than the other can receive it.

Managing Retransmission for reliability

To ensure the delivery of data packets, TCP has its own mechanism of retransmission.
The TCP system works according to the following specific sequence:

Placement On Retransmission Queue: As soon as a segment containing data is transmitted, a copy of the segment is placed in a data structure called the retransmission queue. A retransmission timer is started for the segment when it is placed on the queue. The queue is kept sorted by the time remaining in the retransmission timer, so the TCP software can keep track of which timers have the least time remaining before they expire.

Acknowledgment Processing: If an acknowledgment is received for a segment before its timer expires, the segment is removed from the retransmission queue.

Retransmission Timeout: If an acknowledgment is not received before the timer for a segment expires, a retransmission timeout occurs, and the segment is automatically retransmitted.

What does TCP Window mean?

Client's Receive window: In a client server connection, the client tells the server the number of bytes it can receive at a time from the server. This is client's receive limit at a time and this becomes server's send window.

Server's Receive Window: Similarly, he server tells the client how many bytes of data it can take from the client at one time; this is the server's receive window and the client's send window.

TCP Window Size

TCP/IP related parameters in Linux

All TCP/IP tunning parameters are located under /proc/sys/net/...  Here is a list of the most important tunning parameters, along with short description of their meaning:

/proc/sys/net/core/rmem_max - Maximum read buffer size for all protocols /proc/sys/net/core/wmem_max - Maximum write buffer size for all protocols

/proc/sys/net/ipv4/tcp_rmem - TCP read buffer size /proc/sys/net/ipv4/tcp_wmem - TCP write buffer size

/proc/sys/net/ipv4/tcp_timestamps - timestamps add 12 bytes to the TCP header /proc/sys/net/ipv4/tcp_sack - tcp selective acknowledgements. /proc/sys/net/ipv4/tcp_window_scaling - support for large TCP Windows. Needs to be set to 1 if the Max TCP Window is over 65535.

Keep in mind everything under /proc is volatile, so any changes you make are lost after reboot.

Linux Tune Network Buffers for improved performance

The Linux network stack is not configured for high speed network transfers. This is done to preserve the kernel memory resources. The developers can easily tune these network stack parameters to meet their needs at server/client side.

You can easily find the actual values of network buffer parameters using the cat command as below:

$ cat /proc/sys/net/ipv4/tcp_rmem                 // read the tcp buffer size

To tune the parameters, make the changes to /etc/sysctl.conf

1. Tune the core buffer sizes

echo 'net.core.wmem_max = 12582912'  >>  /etc/sysctl.conf echo 'net.core.rmem_max = 12582912' >> /etc/sysctl.conf

2. Tune the TCP receive and send buffers

echo 'net.ipv4.tcp_rmem = 10240 87380 12582912'  >>  /etc/sysctl.conf echo 'net.ipv4.tcp_wmem = 10240 87380 12582912'  >>  /etc/sysctl.conf

3. Turn on window scaling to enlarge the transfer window:

echo 'net.ipv4.tcp_window_scaling = 1'  >>  /etc/sysctl.conf

4. Enable timestamps and selective acknowledgements:

echo 'net.ipv4.tcp_timestamps = 1'  >>  /etc/sysctl.conf echo 'net.ipv4.tcp_sack = 1'  >>  /etc/sysctl.conf

5. Now reload the changes:

sysctl -p

Use Wireshark to monitor TCP Window

There are a few ways to monitor the tcp window with Wireshark. By default, the TCP window of the packet sender is displayed in the info summary view for each ACK packet.

Another way to show this is by using the I/O Graphs looking for the TCP Window Size to drop. To do this, use the tcp.analysis.window_update filter. Try it with the sample trace file using these settings. 

TCP Window on Wireshark

This graph shows the full size TCP Window dropping to nothing several times. While the window is down near zero, data is halted while the sender waits for the receive buffer to clear. Watch for these dips during large data transfers. The I/O graph makes them easier to see than combing through packet by packet!

If there are any questions about the sample trace file, or more about the function of the TCP Window, feel free to email or comment.