How TCP/IP Works: Key Protocols, Optimization Strategies, and Limitations

The core of the Internet is the TCP/IP internet protocol suite. TCP/IP has two crucial protocols that make communication on the Internet happen, TCP and IP.

Introduction

The internet runs on TCP/IP—but what makes it reliable yet slow? In this article, we will discuss the inner workings of TCP/IP, and the different algorithms such as flow control, slow start, congestion control, and congestion avoidance. We will talk about how TCP/IP provides reliability and data integrity. We will discuss the limitations of TCP/IP, and finally, we will talk about optimizing TCP/IP for performance.

Let’s start by talking briefly about the important protocols:

IP (Internet Protocol)

The IP, or Internet Protocol, is responsible for routing and addressing data packets, including the source and the destination IP addresses for host-to-host communication. IP has two major protocols (IPv4 and IPv6) for providing the addressing scheme.

TCP (Transmission Control Protocol)

TCP (Transmission Control Protocol) is a reliable network abstraction over an unreliable channel. TCP abstracts out all the complexity of handling lost packets, in-order delivery, congestion control, congestion avoidance, and data integrity. Working with TCP protocol means guaranteed delivery of identical information as shared by the sender at the recipient end.

UDP (User Datagram Protocol)

UDP like TCP is a transmission protocol, but this doesn’t have any congestion control, congestion avoidance, or error-checking overhead. UDP is faster than TCP since it doesn’t provide reliability and data integrity checks as compared to TCP. It just receives a data packet from the IP and forwards the packet to the Application layer for this reason it is commonly referred to as a null protocol.

A Three-Way Handshake

Every TCP connection starts with a three-way handshake. A three-way handshake is where the client and the server agree to establish a connection before sharing any data. Both the client and the server randomly select packet sequence numbers.

SYN

The client randomly selects a sequence number X and sends a SYN packet. This packet can also contain other connection-specific TCP option flags.

SYN ACK

The server gets the SYN request from the client and increases the client’s sequence number X by one. The server randomly picks its own sequence number Y, adds a set of flags and options, and dispatches the response.

ACK

The client receives the response from the server, increases both X and Y sequence numbers by one, and sends the last ACK packet to the server. This marks the three-way handshake complete.

After the client and the server complete the three-way handshake, both now can start sending actual request data.

sequenceDiagram
	Client --> Server: 0ms
	Client ->> Server: SYN X=2300
	Client --> Server: 32ms
	Server ->> Client: SYN ACK X=2301, Y=3000
	Server --> Client: 64ms
	Client ->> Server: ACK X=2302, Y=3001
	Client --> Server: 96ms

This process is followed for every new connection. This implies that every TCP connection has a performance overhead. Every new TCP connection will have a full RTT of latency without sending any actual data packet. This limitation of TCP introduces an optimization scope to reuse the existing TCP connection whenever possible.

Assuming one-way latency is 32ms, a total of 96ms is taken by TCP just to complete the handshake. Now consider how much does the TCP add latency to each request. A simple request to get a CSS, image, etc. would have an extra latency just for handshake.

TCP Fast Open

TFO or TCP Fast Open is an attempt to minimize this latency overhead of a three-way handshake. TFO allows data transfer within the SYN packet so that at least we can transmit some data during this three-way handshake process. TFO is not suitable for large data transfer, only a small amount of data can be sent in the initial SYN packet, which typically ranges between 4-16 KB.

There is also a compatibility issue with TFO since both the client and server should be able to support it. Apart from this, there are other technical limitations to using TFO. But it is suitable for small resources and where support for client and server is enabled.

Congestion Collapse & Control

Congestion control is the one of the serious problem that TCP/IP addresses with various mechanisms. When a host sends a packet, and it doesn’t get ACK of the packets sent within the maximum retransmission interval, that host will begin to send more and more copies of the same packet. UDP datagrams do not have congestion control built-in, hosts will keep sending the datagrams without worrying about the router’s status on the network.

This becomes a serious trouble. Eventually, this situation will cause the available buffer in the switching nodes to be full causing queue delays, and the nodes will have to drop packets in order to serve other traffic. Hosts are sending each packet many times and at some moment each packet will arrive at its destination multiple times.

In 1986 NSFNet, an early internet routers, collapsed under TCP retransmission, reducing throughput to near zero.

To handle this situation of higher traffic causing serious network degradation, TCP introduces various congestion control algorithms:

Flow Control

Flow control is the mechanism to protect the receiver from overwhelmingly large traffic than it can ideally handle. A receiver could be busy processing available packets, it can be under a heavy load, or may not be willing to allocate a bigger buffer.

To protect this condition each side in the TCP connection advertises a receive window called rwnd. A rwnd tells the other side the maximum buffer size for the data it can accept.

sequenceDiagram
	Client --> Server: 0ms
	Client ->> Server: SYN X=2300
	Client --> Server: 32ms
	Server ->> Client: SYN ACK X=2301, Y=3000
	Server --> Client: 64ms
	Client ->> Server: ACK X=2302, Y=3001, rwnd=8192
	Client --> Server: 96ms

The above diagram shows the usage of rwnd variable in ACK.

In practice, a client or a server can keep sharing the rwnd to advertise the state of each other in each ACK and would cause the other party to adjust the data flow rate. If a client becomes congested, it can advertise zero as the rwnd which would signal the server that it cannot receive more packets until the client can process its buffer.

rwnd becomes a limiting factor for both parties. Even if we have higher bandwidth available on the physical layer, the other party cannot send data more than the shared rwnd size, it is a waste of leftover bandwidth.

Slow-start

Flow control provides a mechanism to prevent the receiver from having buffer bloat, but it does not prevent the network condition from causing the congestion collapse. In flow control, we only advertise the buffer size of the end client (receiver) but what about if a router in the network path is slow or the network link’s bandwidth and latency is slower than the receiver? The host will continue to send the data at a rate that the receiver has advertised, but the intermediate network wouldn’t be able to cope with the data flow rate.

Slow-start deals with such conditions by introducing a congestion window cwnd. The only way to estimate the available capacity at the network links is to start with a small congestion window size and exponentially increase the size by observing the network conditions.

cwnd is not advertised to the receiver but is kept private by the server. The server initializes the cwnd with a small packet based on settings and keeps increasing the cwnd size exponentially every RTT until a failure is detected or ssthresh size is reached. Once a failure is detected, that is a packet loss is reported, it multiplicatively decreases the cwnd size and then switches to additive increase. This mechanism is known as AIMD

Slow-start featuring congestion avoidance, fast retransmit, and recovery. It avoids congestion by starting slow i.e., by starting with the smaller packet size. If 3 duplicate ACK arrive, it assumes that a packet was lost and quickly retransmits a.k.a. fast retransmit. By doing a multiplicative decrease, of the packet size in case of packet lost it starts the recovery phase.

This works well for large data transfers because eventually they will reach at the near max capacity and utilize the available bandwidth. But other side is that, it will do multiple RTT by starting slow and exponentially reaching the available capacity.

For small data transfers, the connection would end even before it reaches the full capacity of the network.

If a connection becomes idle for some time, it can cause SSR that is slow to start reset. This will reset the cwnd size to the initial smaller size.

These issues add to the latency time and is not efficient at utilizing the whole available bandwidth.

Slow Start vs. Flow Control: What’s the Difference?

Head-of-Line Blocking

TCP provides in-order delivery of all the packets, which means when TCP hands over the data received to the Application layer, the data would be in order. This is done by utilizing the sequence numbers. Once TCP gets all the packets, it uses the sequence number to arrange the data packets in order. It will keep the data contained to TCP only until it receives all the lost packets, another feature, which guarantees data integrity and reliability.

The application layer wouldn’t receive any data until TCP receives all the packets and does all kinds of data integrity checks. This means the Application layer would keep waiting for the data in the meantime. This causes a delay for the Application layer to receive the data, and this is known as Head-of-Line Blocking.

Performance Optimization for TCP

In this article, we have learned how TCP provides reliability by providing flow control, slow start, congestion control and avoidance, fast retransmission, recovery, etc. But we have also learned that this adds latency overheads and wastes bandwidth.

Now, that we know the limitations of the TCP, we can follow some best practices to get the maximum performance from it.

1. We cannot make the latency faster because there is a hard limit on the fastest available channel speed. But we can bring the parties closer to reduce the distance between them. This will help us achieve low latency. Recalling the slow-start increases cwnd Size exponentially on every RTT, by reducing the distance between client and server we can decrease the latency. This will help us quickly reach the maximum capacity of the network bandwidth.
2. Use the updated server kernel to the latest version.
3. We can set the initial cwnd size to 10 instead of the default 4.
4. We can disable the SSR after idle time.
5. Eliminate redundant data transfers. That is, if we have already fetched a resource e.g., an image from a server, we should avoid re-fetching it from the server instead we should utilize a cache.
6. Compressing the data before transferring would reduce the number of bits to be transmitted on the network.
7. Reuse the TCP connection to avoid a three-way handshake for every new connection. HTTP/1.1 has keep-alive enabled by default. HTTP/2 has even better performance.

In summary, we can say that TCP/IP is the backbone of the Internet. It starts with a three-way handshake to maintain connection reliability but adds extra latency. TCP Fast Open reduces the initial handshake delay by including data in the initial SYN request. Flow control supports congestion control and slow start enhances it by adding congestion avoidance.

While TCP/IP powers today’s internet, new technologies will address its shortcomings. For example, QUIC and HTTP/3 improve latency by replacing TCP with UDP to eliminate handshakes and head-of-line blocking and built-in encryption.