Unit 1 · Chapter 2 · Book pp. 31–66

The Transport Layer: TCP, UDP and SCTP

The densest chapter in Unit 1, and the one that pays best. Two ten-mark questions on last year's paper came from here — the SCTP association handshake and the TCP state transition diagram — and both are drawing questions, so the diagrams matter as much as the prose.

Q3(a) [10] SCTP association establishment and termination, with a neat diagram. OR Q3(b) [10] Draw the TCP state transition diagram and explain it. Both are worked in the solved paper; the diagrams on this page are the same ones, made steppable.

Three transports, one socket API

The transport layer is where your program's data first becomes someone else's problem. You get to pick which of three protocols carries it, and the choice decides what guarantees you receive — and therefore how much work you have to do yourself.

	UDP	TCP	SCTP
Connection	Connectionless	Connection-oriented	Connection-oriented (an association)
Reliability	None	Full	Full
Ordering	Not preserved	Preserved	Preserved, per stream
Duplicates	Possible	Discarded	Discarded
Flow control	No	Yes	Yes
Congestion control	No	Yes	Yes
Boundaries	Message-oriented — length preserved	Byte stream — no boundaries	Message-oriented
Endpoints	One address each	One address each	Multiple addresses each (multihoming)
Data unit	User datagram	Segment	Chunk
Socket type	`SOCK_DGRAM`	`SOCK_STREAM`	`SOCK_STREAM` or `SOCK_SEQPACKET`

UDP — User Datagram Protocol

Described in RFC 768. The application writes a message to a UDP socket; it is encapsulated in a UDP datagram, which is encapsulated in an IP datagram, which is sent. That is the whole story.

There is no guarantee that a UDP datagram ever reaches its destination, that order is preserved, or that datagrams arrive only once.
On error detection or drop, there is no automatic retransmission.
Each datagram has a length, and that length is passed to the receiving application along with the data. Boundaries are preserved.
It is connectionless: there need be no long-term relationship. A client can create one socket, send a datagram to one server, and immediately send another to a different server on the same socket. A server can receive datagrams from many clients on a single socket.

UDP is a postcard. You write it, drop it in the box, and that is the last you know about it. It might arrive, it might not, and if you send three they might arrive in any order. What you gain is that there is no phone call to set up first — for a single short question, that saves a great deal.

TCP — Transmission Control Protocol

A TCP client establishes a connection with a server, exchanges data across it, and then terminates the connection. In exchange for that setup cost you get a long list of guarantees:

Reliability: When TCP sends data it requires an acknowledgment in return. If none arrives, TCP retransmits automatically and waits a longer amount of time before trying again.
Round-trip time estimation: TCP dynamically estimates the RTT between client and server, so it knows how long to wait for an acknowledgment. This is not a fixed timeout — a satellite link and a LAN get different answers.
Sequencing: TCP associates a sequence number with every byte it sends. If segments arrive out of order the receiving TCP reorders them before passing data up; if duplicates arrive, it discards them.
Flow control: TCP tells its peer exactly how many bytes it is willing to accept — the advertised window. This stops a fast sender from overrunning a slow receiver's buffer.
Full-duplex: A TCP connection carries data in both directions independently, which is exactly why closing it takes four segments and not two.

TCP is a byte stream with no record boundaries. If the sender does two writes of 100 bytes, the receiver may see one read of 200, or 60 then 140, or any other split. Anything the receiver needs to know about structure must be put into the bytes — a length prefix, a delimiter, a fixed-size record. This is the single most commonly examined property of TCP.

SCTP — Stream Control Transmission Protocol

Designed to serve newer applications, particularly telephony signalling and multimedia. It provides reliability, sequencing, flow control and full-duplex transfer just like TCP — and then adds two features TCP does not have.

Association, not connection: SCTP says association rather than connection deliberately. A connection is between two IP addresses. An association is between two endpoints, each of which may have several IP addresses, so the word "connection" would be too narrow.
Multistreaming: An association carries multiple independent streams, each with its own reliable sequenced delivery. A message lost in one stream does not block delivery in the others. Contrast TCP: a loss anywhere in the single byte stream blocks all subsequent data until it is repaired. This is called head-of-line blocking, and multistreaming is the cure.
Multihoming: A single SCTP endpoint may support multiple IP addresses, typically on different networks with different paths to the Internet. If one network or path fails, SCTP can work around it by switching to another address already part of the association. This gives increased robustness against network failure, with no help from the application.
Message-oriented: SCTP provides sequenced delivery of individual records, so message boundaries are preserved — the useful part of UDP, kept alongside the reliability of TCP.

A web page is HTML plus twenty images. Over one TCP connection, if the packet carrying image 3 is lost, images 4 through 20 sit in the receiver's buffer undeliverable until image 3 is retransmitted — even though they arrived safely and nothing depends on image 3. Over SCTP with twenty streams, image 3's loss delays only image 3.

TCP connection establishment

Understanding these two exchanges is what lets you make sense of connect, accept and close, and what lets you read netstat output when something goes wrong. Step the figure below; the endpoint states change as each segment lands.

The four things that happen, in the book's own numbering:

The server must be prepared to accept an incoming connection — normally socket, bind, listen. This is a passive open.
The client issues an active open by calling connect. Its TCP sends a SYN segment telling the server the client's initial sequence number J. Normally no data is sent with the SYN — just an IP header, a TCP header and options.
The server must acknowledge the client's SYN and send its own SYN carrying its initial sequence number K. It sends both in a single segment.
The client must acknowledge the server's SYN.

Three segments minimum — hence three-way handshake.

An acknowledgment number always means "the next byte I expect". A SYN occupies one byte of sequence number space even though it carries no data, so having received SYN J the server next expects byte J+1, and it writes ACK J+1. The same rule applies to a FIN, which is why the ACK of FIN M is M+1.

TCP options carried on the SYN

Each SYN can carry options. Three are commonly used.

MSS option — maximum segment size: The TCP sending the SYN announces the maximum amount of data it is willing to accept in each segment on this connection. The sending TCP then uses the receiver's announced MSS as the maximum size of the segments it sends. Fetched and set with the TCP_MAXSEG socket option (§7.9).
Window scale option: The advertised window field in the TCP header is 16 bits, so the largest window that can be advertised is 65,535. High-speed connections (45 Mbits/sec and faster) and long-delay paths (satellite links) need more to reach full throughput. This option says the advertised window must be left-shifted by 0–14 bits, giving a maximum window of almost one gigabyte (65,535 × 2¹⁴). Both ends must support it. Influenced by the SO_RCVBUF socket option.
Timestamp option: Needed on high-speed connections to prevent data corruption from old, delayed or duplicated segments. Negotiated like the window scale option. As a network programmer there is nothing you need to do about it.

The last two are called the RFC 1323 options, or the long fat pipe options — a network with either high bandwidth or long delay is called a long fat pipe. §2.6 · pp.38–39

A TCP may send the window scale option with its SYN on an active open, but it may only scale its windows if the other end also sends the option with its SYN. Likewise the server's TCP may send the option only if it received one with the client's SYN. The logic assumes implementations ignore options they do not understand — required and common, but not guaranteed.

TCP connection termination

Three segments to set up, but four to tear down. The reason is that a TCP connection is full-duplex: it is really two independent byte streams, and each must be shut down separately.

One application calls close first — that end performs the active close. Its TCP sends a FIN, meaning "I have finished sending data".
The other end performs the passive close. Its TCP acknowledges the FIN, and the receipt of the FIN is passed to the application as end-of-file, after any data already queued. This is why read returns 0.
Sometime later that application closes its own socket, causing its TCP to send a FIN.
The TCP at the active-close end acknowledges that final FIN.

"Normally four", not "always four". The FIN in step 1 may be sent along with data, and the segments in steps 2 and 3 are both from the passive-close end and could be combined into one.

Half-close: Between steps 2 and 3, data can still flow from the passive-close end to the active-close end. One direction is shut, the other is not. The shutdown function (§6.6) exists to produce this deliberately.

Either end may perform the active close. It is often the client, but with some protocols — notably HTTP/1.0 — the server does. Do not write "the client always closes first" in an answer.

Each FIN is sent when a socket is closed — but a socket is closed not only by calling close. When a Unix process terminates, voluntarily (exit, or main returning) or involuntarily (a signal that kills it), all open descriptors are closed, which sends a FIN on every still-open TCP connection. This is exactly what Chapter 5 relies on when the server host is shut down.

The TCP state transition diagram

Q3(b), 10 marks: "Draw the TCP state transition diagram and explain the same." Eleven states, and the marks are in naming them correctly, labelling each arrow with both its trigger and what is sent, and marking the normal client and server paths.

Connection establishment and termination together can be specified as a state machine. There are 11 states, and the rules of TCP dictate the transitions between them based on the current state and the segment received in it.

appl: means the transition is taken because the application did something — called connect, called close. recv: means a segment arrived. send: says what TCP transmits as a result. So "recv: SYN / send: SYN, ACK" reads as: a SYN arrived, so we reply with a SYN and an ACK, and move along this arrow.

The eleven states

State	Meaning	Whose
`CLOSED`	No connection. The starting and ending point.	both
`LISTEN`	Passive open done; waiting to be contacted.	server
`SYN_SENT`	Active open done; SYN sent, waiting for the reply.	client
`SYN_RCVD`	A SYN arrived, SYN+ACK sent, waiting for the third segment.	server
`ESTABLISHED`	The connection is open. Where all data transfer happens.	both
`FIN_WAIT_1`	Active close: our FIN is sent, not yet acknowledged.	active closer
`FIN_WAIT_2`	Our FIN is acknowledged; waiting for theirs. Half-closed.	active closer
`CLOSE_WAIT`	Passive close: their FIN received and acknowledged; waiting for our application to call close.	passive closer
`CLOSING`	Simultaneous close — both FINs crossed. Rare.	both
`LAST_ACK`	Passive close: our FIN sent, waiting for the final ACK.	passive closer
`TIME_WAIT`	Active close complete; waiting 2MSL before releasing the socket pair.	active closer

This is the pair examiners look for. If the application calls close before receiving a FIN, that is an active close and the transition is to FIN_WAIT_1. If a FIN is received while in ESTABLISHED, that is a passive close and the transition is to CLOSE_WAIT. One state, two exits, and which one you take is decided by who moved first.

Two transitions the book flags as possible but rare: a simultaneous open, when both ends send SYNs at about the same time and they cross in the network, and a simultaneous close, when both send FINs at the same time. The path buttons on the diagram walk both. A practical reason for knowing all eleven names: they are exactly what netstat prints.

A connection is stuck in CLOSE_WAIT for minutes. Whose bug is it?

The local application's. CLOSE_WAIT means the peer's FIN arrived and was acknowledged — TCP has done everything it can — and the connection is now waiting for your application to call close. TCP cannot move to LAST_ACK until it does.

A pile of sockets in CLOSE_WAIT is the classic signature of a server that reads until end-of-file and then forgets to close the descriptor. Compare TIME_WAIT, which is TCP working correctly and needs no fix.

The TIME_WAIT state

The book calls this "undoubtedly one of the most misunderstood aspects of TCP", and it is a favourite exam question because it has a crisp two-part answer.

§2.7 · p.43 The end that performs the active close goes through TIME_WAIT. The duration is twice the maximum segment lifetime — 2MSL.

MSL — maximum segment lifetime: The maximum time any given IP datagram can live in a network. RFC 1122 recommends 2 minutes; Berkeley-derived implementations have traditionally used 30 seconds. So TIME_WAIT lasts somewhere between 1 and 4 minutes.
Why the lifetime is bounded at all: Every datagram carries an 8-bit hop limit — the IPv4 TTL field, the IPv6 hop limit field — with a maximum value of 255. Strictly this is a hop limit, not a time limit; the assumption is that a packet with the maximum hop limit of 255 cannot survive in a network for more than MSL seconds.
Lost duplicate / wandering duplicate: A packet caught in a routing loop. A router crashes or a link goes down, and the routing protocols take seconds or minutes to stabilise. During that window a loop can form (A sends to B, B sends back to A) and a packet circulates. Meanwhile the sending TCP times out, retransmits, and the retransmission arrives by another path. Later the loop clears and the original packet finally arrives. TCP must handle these.

The two reasons — learn these exactly

1 · Reliable full-duplex close

Suppose the final ACK is lost. The peer will time out and resend its FIN, so this end must keep enough state to resend the final ACK. If it had thrown that state away, it would answer with an RST instead — which the peer would read as an error, turning a clean shutdown into a failure.

This is also why it is the active-close end that waits: that is the end which might have to retransmit the final ACK.

2 · Let old duplicates expire

Suppose a connection between 12.106.32.254:1500 and 206.168.112.219:21 closes, and shortly after, a new connection opens between the same four values. The book calls the new one an incarnation of the previous connection.

TCP must stop an old duplicate from a previous incarnation being mistaken for data on the new one. So it refuses to create a new incarnation while the old one is in TIME_WAIT. At 2MSL, that allows MSL seconds for a packet in one direction to die and another MSL for its reply.

2MSL, not MSL — and the reason for the doubling is worth one mark on its own: one MSL for a packet travelling out, one MSL for the reply coming back.

Why does a server that has just crashed often fail to restart with "Address already in use", while a client never has this problem?

Because the server's socket is in TIME_WAIT on its well-known port, and TCP will not create a new incarnation of that socket pair until 2MSL has passed. The server must bind that exact port — it has no alternative — so the bind fails.

A client binds an ephemeral port, so on restart it is simply given a different one and nothing collides. The standard fix for the server is the SO_REUSEADDR socket option — see Chapter 7.

SCTP association establishment and termination

Q3(a), 10 marks: "Discuss in detail about SCTP Association Establishment and Termination procedures with a neat diagram." Both figures are below, and both are worked into a full answer in the solved paper.

SCTP is connection-oriented like TCP, so it also has establishment and termination handshakes. Establishment takes four packets, not three.

The server must be prepared to accept an incoming association — socket, bind, listen. A passive open.
The client issues an active open by calling connect, or by sending a message which opens the association implicitly. Its SCTP sends an INIT telling the server: the client's list of IP addresses, its initial sequence number, an initiation tag identifying all packets in this association, the number of outbound streams it is requesting, and the number of inbound streams it can support.
The server acknowledges with INIT-ACK, containing the server's list of IP addresses, initial sequence number, initiation tag, stream counts, and a state cookie. The cookie contains all the state the server needs to ensure the association is valid, and is digitally signed to ensure its validity.
The client echoes the cookie back in a COOKIE-ECHO. This message may also carry user data bundled in the same packet.
The server acknowledges that the cookie was correct and the association is established, with COOKIE-ACK. This message may also carry user data.

Five numbered steps, but the first is preparation — the minimum number of packets is four, hence four-way handshake.

In TCP, the moment a server receives a SYN it must allocate memory to remember the half-open connection. An attacker who sends thousands of SYNs from forged addresses and never completes the handshake exhausts that memory — the classic SYN flood.

SCTP's server stores nothing after INIT. It packs all the state it would have kept into the cookie, signs it so it cannot be forged, and posts it to the client. Only when the client echoes a valid cookie back — proving it really is at the address it claimed — does the server allocate anything. The cookie is a cloakroom ticket: the server hands you your own coat to hold, and only takes it back when you return with a ticket it can verify.

The INIT carries a verification tag Ta and an initial sequence number J. Ta must be present in every packet sent by the peer for the life of the association. The peer likewise chooses its own tag Tz, which must appear in each of its packets. J is the starting sequence number for DATA messages, which SCTP calls DATA chunks. At the end of the handshake each side chooses a primary destination address, used by default in the absence of network failure. §2.8 · pp.45–46

Association termination

Termination takes three packets, and SCTP has no TIME_WAIT.

SCTP does not support a half-close. When one end shuts down the association, the other stops accepting new data from its application — but both ends still deliver everything already queued. TCP's half-close lets one direction keep sending indefinitely; SCTP deliberately does not. It also has no TIME_WAIT state: the verification tags already prevent an old packet being mistaken for part of a new association, which is precisely the job TIME_WAIT does for TCP.

	TCP	SCTP
Set-up packets	3 (three-way handshake)	4 (four-way handshake)
Teardown packets	4	3
Set-up messages	SYN, SYN+ACK, ACK	INIT, INIT-ACK, COOKIE-ECHO, COOKIE-ACK
Teardown messages	FIN, ACK, FIN, ACK	SHUTDOWN, SHUTDOWN-ACK, SHUTDOWN-COMPLETE
Server state on first packet	Allocated immediately	None — held in a signed cookie
Half-close	Supported	Not supported
TIME_WAIT	Yes, 2MSL at the active-close end	No — verification tags do that job
Data on handshake	Not normally	Yes, on COOKIE-ECHO and COOKIE-ACK

Port numbers

At any moment many processes may be using a given transport. All three transports use 16-bit port numbers to tell them apart. Sixteen bits gives 65,535 usable ports, and the IANA — Internet Assigned Numbers Authority — divides them into three ranges.

Range	Name	Controlled by	Notes
`0 – 1023`	Well-known ports	Assigned and controlled by IANA	FTP is 21, a web server is 80, daytime is 13. Where possible the same number is assigned for TCP, UDP and SCTP. On Unix these are also the reserved ports — binding one requires superuser privilege.
`1024 – 49151`	Registered ports	Not controlled by IANA; registered and listed as a convenience	X Window servers use 6000–6063. The upper limit of 49151 was introduced to leave room for ephemeral ports; RFC 1700 had listed it as 65535.
`49152 – 65535`	Dynamic or private ports	IANA says nothing	These are the ephemeral ports. 49152 is three-quarters of 65536.

Well-known port: A fixed number a server binds so that clients can find it without being told. Every TCP/IP implementation that supports FTP assigns port 21 to the FTP server; TFTP servers get UDP port 69.
Ephemeral port: A short-lived port, assigned automatically by the transport protocol to a client. The client does not care what the number is — it only needs it to be unique on the client host, and the protocol code guarantees that.

An IP address gets a packet to the right machine. The port number gets it to the right program on that machine. Address is the building, port is the flat number. The server's must be published in advance; the client's is whichever one happens to be free.

The socket pair

§2.9 · p.52 The socket pair for a TCP connection is the four-tuple that defines the two endpoints of the connection:

( local IP address , local port , foreign IP address , foreign port )

A socket pair uniquely identifies every TCP connection on a network.

The two values identifying one endpoint — an IP address and a port number — are together often called a socket.

For SCTP: An association is identified by a set of local IP addresses, a local port, a set of foreign IP addresses, and a foreign port. Where neither endpoint is multihomed this reduces to the same four-tuple as TCP. But when either endpoint is multihomed, multiple four-tuple sets — different IP addresses, the same port numbers — may identify the same association.
For UDP: The concept extends even though UDP is connectionless. bind lets the application specify the local IP address and local port for TCP, UDP and SCTP sockets alike.

Twenty browser tabs all connect to the same web server: the same foreign address and the same foreign port 80, and from the same local address. If a connection were identified by fewer than four values they would be indistinguishable. It is the local port — a different ephemeral port per tab — that separates them. This is also exactly how a concurrent server keeps hundreds of clients apart while every one of them is talking to port 9877.

Buffer sizes and limitations

A list of hard numbers. They are easy marks in a short-answer question, and they explain why a "write" of 4,000 bytes does not appear on the wire as one thing.

Quantity	Value	Because
Maximum IPv4 datagram	65,535 bytes, including the IPv4 header	The 16-bit total length field
Maximum IPv6 datagram	65,575 bytes, including the 40-byte IPv6 header	The 16-bit payload length field, which excludes the header — hence 65,535 + 40
Ethernet MTU	1,500 bytes	Dictated by the hardware
Minimum link MTU, IPv4	68 bytes	Room for a maximum IPv4 header (20 fixed + 30 options) and a minimum fragment
Minimum link MTU, IPv6	1,280 bytes	RFC 2460. IPv6 can run over smaller links, but needs link-specific fragmentation to make them look like 1,280
Minimum reassembly buffer, IPv4	576 bytes	The smallest datagram any implementation is guaranteed to accept
Minimum reassembly buffer, IPv6	1,500 bytes	Raised from IPv4's 576

65,535 for IPv4, 65,575 for IPv6. IPv4's total length field includes the header; IPv6's payload length field excludes it, so the 40-byte IPv6 header is added on top. That 40-byte difference is exactly the point of the question.

MTU — maximum transmission unit: The largest frame a given link can carry. Often fixed by hardware; PPP links are configurable.
Path MTU: The smallest MTU on the path between two hosts. Today the Ethernet MTU of 1,500 is often the path MTU. It need not be the same in both directions, because Internet routing is often asymmetric — the route from A to B can differ from the route from B to A.
MSS — maximum segment size: The maximum amount of TCP data the peer can send per segment. Announced in the MSS option on each SYN. Typically the path MTU minus the IP and TCP headers.
Fragmentation point (SCTP): SCTP keeps a fragmentation point based on the smallest path MTU found to all of the peer's addresses — a consequence of multihoming, since different addresses may have different paths.

Fragmentation

When an IP datagram is to be sent out an interface and its size exceeds the link MTU, fragmentation is performed. Fragments are not normally reassembled until they reach the final destination. Who is allowed to fragment differs between the two IP versions, and this is a favourite one-mark distinction:

	IPv4	IPv6
Hosts fragment datagrams they generate	Yes	Yes
Routers fragment datagrams they forward	Yes	No
Where the fields live	In the fixed IPv4 header	In a fragmentation extension header — since fragmentation is the exception, not the rule
Datagram too big for the outgoing link	If the DF (don't fragment) bit is set, the router returns ICMPv4 "destination unreachable, fragmentation needed but DF bit set"	There is an implied DF bit on every IPv6 datagram, so the router returns ICMPv6 "packet too big"

Both errors are what makes path MTU discovery work: send with DF set, and if an ICMP error comes back, reduce the amount of data per datagram and try again. Note also that firewalls acting as routers sometimes reassemble fragments so the whole packet can be inspected — at the cost of complexity, and of requiring the firewall to sit on the only path, which reduces redundancy.

Your application writes some bytes. TCP chops them into segments no larger than the peer's announced MSS. Each segment becomes an IP datagram. If that datagram is bigger than the path MTU, something must fragment it — or, with DF set, refuse and report back. The whole chain exists so that the application never has to think about the 1,500-byte cable at the far end.

The cost of TCP, counted

A useful thing to be able to say in an answer. If the entire purpose of a connection is to send a one-segment request and receive a one-segment reply, TCP costs eight segments of overhead: three to set up, four to tear down, plus the acknowledgment. With UDP only two packets are exchanged — the request and the reply.

But switching to UDP removes all the reliability TCP provides, pushing those details up into the application — including congestion control, which is the one people forget. Many applications are nevertheless built on UDP precisely because they exchange small amounts of data and want to avoid the connection setup and teardown cost. DNS is the canonical example.

In the full packet trace (Figure 2.5), the acknowledgment of the client's request is sent together with the server's reply rather than on its own. This is called piggybacking, and normally happens when the server takes less than about 200 ms to produce its reply. Take a second and you would see the ACK first and the reply later.

The client sends 2,000 bytes in one write() over Ethernet. How many TCP segments appear on the wire, and why?

Normally two. The Ethernet MTU is 1,500 bytes; subtracting a 20-byte IP header and a 20-byte TCP header leaves an MSS of 1,460. TCP therefore sends 1,460 bytes in the first segment and 540 in the second.

Note what did not happen: IP did not fragment anything. TCP sized its own segments to fit the path MTU, which is the entire purpose of the MSS option being exchanged on the SYNs.

And the receiver has no way to know it was one write. It may see one read of 2,000, two reads of 1,460 and 540, or any other split — the byte stream has no record boundaries.

Give three differences between TCP's connection establishment and SCTP's.

Three packets versus four. TCP: SYN, SYN+ACK, ACK. SCTP: INIT, INIT-ACK, COOKIE-ECHO, COOKIE-ACK.
State allocation. A TCP server allocates state the moment a SYN arrives, which is what makes SYN flooding possible. An SCTP server allocates nothing until a valid signed state cookie is echoed back, so the equivalent attack does not work.
Addresses. TCP's handshake establishes exactly one address at each end. SCTP's INIT and INIT-ACK each carry a list of addresses, so the association is multihomed from the first packet, and each side then picks a primary destination address.

A fourth, if you need it: SCTP's COOKIE-ECHO and COOKIE-ACK may carry user data, so the application's first message can travel during the handshake.