Unit 2 · Chapter 7 · Book pp. 191–238

Socket Options

Every socket has knobs. This chapter is the catalogue — what they are called, which layer owns each one, what type of value it takes, and which of them you must set before the connection exists because afterwards is too late.

Q2(c) [5]: "Give the syntax of the socket options listed below, analyze and state the category of every option: IP_TTL, SO_BROADCAST, TCP_MAXSEG, IPV6_DONTFRAG, SO_LINGER." The five are answered together further down, and again in the solved paper. Note what the question wants: syntax and category — so a bare description scores nothing.

Three ways to affect a socket

The getsockopt and setsockopt functions — the subject of this chapter.
The fcntl function — the POSIX way to set a socket non-blocking, or to set the owner for signal-driven I/O.
The ioctl function.

getsockopt and setsockopt

§7.2 · p.192

#include <sys/socket.h>

int getsockopt(int sockfd, int level, int optname,
               void *optval, socklen_t *optlen);

int setsockopt(int sockfd, int level, int optname,
               const void *optval, socklen_t optlen);
                                Both return: 0 if OK, -1 on error

sockfd: Must refer to an open socket descriptor.
level: Specifies the code in the system that interprets the option — the general socket code (SOL_SOCKET) or some protocol-specific code (IPPROTO_IP, IPPROTO_IPV6, IPPROTO_TCP, IPPROTO_SCTP, IPPROTO_ICMPV6).
optname: Which option. The constants are unique per level.
optval: A pointer to a variable from which the new value is fetched by setsockopt, or into which the current value is stored by getsockopt.
optlen: The size of that variable. A value for setsockopt, and a value-result for getsockopt — which is why the two prototypes differ by one asterisk, and why getsockopt is on the list of value-result cases in §3.3.

setsockopt takes socklen_t optlen. getsockopt takes socklen_t *optlen. Getting that asterisk wrong is a compile error at best and a silent stack overwrite at worst — and noticing it is worth a mark, because it demonstrates you know which direction the data is travelling.

The two categories

§7.2 · p.193 There are two basic types of options: binary options that enable or disable a certain feature (flags), and options that fetch and return specific values that we can either set or examine (values).

Flag options

A switch. *optval is an int. getsockopt returns zero if the option is disabled and non-zero if it is enabled. setsockopt requires a non-zero *optval to turn it on and zero to turn it off.

Marked with a • in the "Flag" column of the book's table.

SO_BROADCAST SO_KEEPALIVE SO_REUSEADDR TCP_NODELAY IPV6_DONTFRAG

Value options

Carries an actual value of the datatype in the table — sometimes an int, sometimes a structure. If the "Flag" column does not contain a •, the option is used to pass a value of the specified datatype between the user process and the system.

SO_LINGER SO_RCVBUF IP_TTL TCP_MAXSEG SO_RCVTIMEO

"Flag or value" is the whole question of category. Ask yourself: does this option answer yes or no, or does it answer how much? SO_BROADCAST is "may I broadcast — yes or no". IP_TTL is "how many hops — 64". The first is a flag, the second is a value.

The book writes a datatype with two braces to indicate a structure — so linger{} in the table means struct linger.

Every option, filterable

The whole of Figures 7.1 and 7.2. Filter by level, or by whether an option is a flag or a value; click any row for its syntax and what it actually does.

Socket states and inheritance

Some options have timing considerations — it matters when you set them relative to the state of the socket. There is one rule to remember and it is a very common exam point.

§7.4 · p.198 The following socket options are inherited by a connected TCP socket from the listening socket:

SO_DEBUG, SO_DONTROUTE, SO_KEEPALIVE, SO_LINGER, SO_OOBINLINE, SO_RCVBUF, SO_RCVLOWAT, SO_SNDBUF, SO_SNDLOWAT, TCP_MAXSEG and TCP_NODELAY.

To ensure that one of these options is set on the connected socket when the three-way handshake completes, you must set that option on the listening socket.

Why: the connected socket is created by the kernel during the handshake, before accept ever returns. By the time you hold connfd it is too late for anything that had to be negotiated on the SYN — the window scale derived from SO_RCVBUF, for instance. Setting it on listenfd means every socket the kernel creates from it starts out correct.

The five options from Q2(c)

Pinned here together, since the exam asked for exactly these. Click each for the full syntax.

Option	Level	Category	Datatype
`IP_TTL`	`IPPROTO_IP`	value	`int`
`SO_BROADCAST`	`SOL_SOCKET`	flag	`int`
`TCP_MAXSEG`	`IPPROTO_TCP`	value	`int`
`IPV6_DONTFRAG`	`IPPROTO_IPV6`	flag	`int`
`SO_LINGER`	`SOL_SOCKET`	value	`struct linger{}`

Note the trap in that set. Four of the five take an int, so the syntax lines look nearly identical — and then SO_LINGER takes a structure. If you write all five the same way you lose the mark on the one that is different. And IP_TTL looks like it ought to be a flag but is not: you pass a hop count, not on/off.

The options worth knowing properly

SO_LINGER

Specifies how close operates for a connection-oriented protocol — so TCP and SCTP, but not UDP. By default close returns immediately, and if data still remains in the socket send buffer the system will try to deliver it in the background.

§7.5 · p.202 · <sys/socket.h>

struct linger {
    int  l_onoff;    /* 0 = off, nonzero = on */
    int  l_linger;   /* linger time, POSIX specifies units as seconds */
};

Three scenarios, depending on the two members:

`l_onoff`	`l_linger`	What `close` does
0	ignored	The default. The option is off; `close` returns immediately and the data is delivered in the background.
non-zero	0	TCP aborts the connection. It discards any data still in the send buffer and sends an RST to the peer, instead of the normal four-packet termination. This avoids TIME_WAIT — and by doing so leaves open the possibility of another incarnation being created within 2MSL and receiving old duplicates. SCTP does an abortive close too, by sending an ABORT chunk.
non-zero	non-zero	The kernel lingers. If data remains in the send buffer, the process is put to sleep until either (i) all the data is sent and acknowledged by the peer TCP, or (ii) the linger time expires.

1. If the socket is non-blocking, close will not wait for completion, even with a non-zero linger time.

2. You must check the return value of close. If the linger time expires before the remaining data is sent and acknowledged, close returns EWOULDBLOCK and any remaining data in the send buffer is discarded. Silently, if you did not look.

A recurring bad suggestion is to set l_onoff non-zero and l_linger zero purely to avoid TIME_WAIT so a listening server can restart. This should not be done and could lead to data corruption (RFC 1337). Use SO_REUSEADDR before bind instead. As the book puts it: the TIME_WAIT state is our friend — it exists to let old duplicates expire, and the right response is to understand it rather than defeat it.

There are legitimate uses for an abortive close. The book's example is an RS-232 terminal server, which might otherwise hang forever in CLOSE_WAIT trying to deliver data to a stuck port, but would properly reset that port on receiving an RST.

close — "I'm done; deliver what you can, I'm not waiting."
SO_LINGER with a time — "I'm done; tell me when it has actually arrived."
SO_LINGER with zero — "Stop. Throw it away and send an RST."
shutdown — "I've finished sending; keep receiving." (§6.6)

SO_REUSEADDR and SO_REUSEPORT

SO_REUSEADDR serves four distinct purposes.

The one that matters daily. It allows a listening server to bind its well-known port even if a previously existing connection using that port still exists in TIME_WAIT. This is the fix for "Address already in use" when restarting a server. It must be set between the calls to socket and bind.
It allows multiple servers on the same port with different local addresses, which is common for a multihomed host.
It allows a single process to bind the same port to multiple sockets, as long as each binds a different local IP address.
It allows a completely duplicate binding — same address and same port — but normally only for UDP sockets, and typically for multicast.

int  on = 1;

sockfd = Socket(AF_INET, SOCK_STREAM, 0);

Setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));

Bind(sockfd, (SA *) &servaddr, sizeof(servaddr));

§7.5 · p.198 All TCP servers should specify SO_REUSEADDR to allow the server to be restarted.

SO_REUSEPORT: Allows a completely duplicate binding — the same IP address and the same port — but only if every socket that wants the binding sets the option, including the very first one. Normally supported only for UDP sockets, and only when the address being bound is a multicast address. It is not present on all systems, which is why the book's checkopts program wraps it in an #ifdef.

SO_KEEPALIVE

When set on a TCP socket, if no data has been exchanged in either direction for two hours, TCP automatically sends a keep-alive probe to the peer. There are three possible outcomes:

The peer	Result
Responds with the expected ACK	All is well. The application is not told anything; the timer resets for another two hours.
Responds with an RST	The host crashed and rebooted. The socket's pending error is set to `ECONNRESET` and the socket is closed.
Does not respond at all	After further probes and no reply, TCP gives up. The error is `ETIMEDOUT` — or `EHOSTUNREACH` if an ICMP error came back.

This is the answer to the question Chapter 5 leaves open: how does an idle client notice the server host has crashed? Without keep-alive it does not, because the crash sends nothing and the client sends nothing. With it, TCP probes on your behalf. Two hours is a long time, and the interval is usually a system-wide setting rather than a per-socket one.

SO_RCVBUF and SO_SNDBUF, and the bandwidth-delay product

Every socket has a send buffer and a receive buffer. These two options change their sizes. The timing rule is strict:

For a TCP socket, SO_RCVBUF must be set before the connection is established, because the receive buffer is what the window scale option is derived from, and that option is exchanged on the SYNs. For a client, set it before connect. For a server, set it on the listening socket — since it is one of the inherited options, the connected socket will pick it up.

Bandwidth-delay product: The capacity of the pipe: bandwidth (bytes per second) × round-trip time (seconds) = bytes in flight. The TCP send buffer must hold data until it is acknowledged, so if it is smaller than this product the connection cannot keep the pipe full and throughput is limited by the buffer rather than by the link.

A 100 Mbit/s link with a 60 ms round trip.
100,000,000 bits/s ÷ 8 = 12,500,000 bytes/s. × 0.06 s = 750,000 bytes in flight.

With a default 64 KB buffer you would use less than one-tenth of that link, and no amount of extra bandwidth would help. This is exactly why the window scale option had to be invented: without it the advertised window cannot exceed 65,535, full stop.

TCP_NODELAY and the Nagle algorithm

The Nagle algorithm: Its purpose is to reduce the number of small packets on a WAN. It says: if a connection has outstanding data that has not yet been acknowledged, small amounts of new data are held back until that acknowledgment arrives. So a connection may have at most one small unacknowledged segment in flight at any time.
TCP_NODELAY: Setting it disables the Nagle algorithm. A flag option.

Nagle interacts badly with delayed ACKs — the receiver's habit of waiting up to 200 ms before acknowledging, hoping to piggyback the ACK on a reply. Combine the two and a client that writes a header and then a body as two separate writes stalls for the delayed-ACK interval between them, on every single request. The correct answer is usually to write once rather than to disable Nagle, but for a request-response protocol that genuinely cannot, TCP_NODELAY is the escape hatch.

IP_TTL, and how it connects back to TIME_WAIT

IP_TTL sets and fetches the default time-to-live the kernel places in the IP header of every unicast packet sent on this socket. TTL is a hop limit: each router that forwards the datagram decrements it, and the datagram is discarded when it reaches zero. Defaults are 64 for TCP and UDP, and 255 for raw sockets.

This is the field that makes MSL a meaningful idea. The maximum TTL is 255, and the assumption is that a packet with the maximum hop limit cannot survive in a network for more than MSL seconds. That assumption is what lets TCP say "after 2MSL, every old duplicate is definitely gone" — which is the whole justification for TIME_WAIT. A one-byte header field, and a four-minute wait derived from it.

The IPv6 equivalent option is IPV6_UNICAST_HOPS.

IPV6_DONTFRAG

Turning this on disables automatic fragmentation for datagram or raw sockets. A packet too large for the path MTU is dropped rather than fragmented.

No error is returned to the process. The packet is discarded silently. An application that needs to know must also set IPV6_RECVPATHMTU to learn about path MTU changes. Remember from §2.11 that IPv6 routers never fragment forwarded datagrams anyway — there is an implied DF bit on every IPv6 datagram — so this option is about what the sending host does with its own oversized packets.

SO_ERROR

When an error occurs on a socket, the kernel sets a variable named so_error for that socket. SO_ERROR can be fetched but not set, and fetching it clears the value to 0.

A read-once mailbox. It is how you find out whether a non-blocking connect actually succeeded: select tells you the descriptor is ready, and SO_ERROR tells you whether "ready" meant "connected" or "failed, and here is why".

The fcntl function

§7.11 · p.233

#include <fcntl.h>

int fcntl(int fd, int cmd, ... /* int arg */ );
       Returns: depends on cmd if OK, -1 on error

fcntl — "file control" — performs several socket operations that setsockopt does not cover. The three that matter here:

Operation	How
Non-blocking I/O	Set the `O_NONBLOCK` file status flag with `F_SETFL`.
Signal-driven I/O	Set the `O_ASYNC` file status flag with `F_SETFL`, which causes `SIGIO` to be sent when the socket status changes.
Socket ownership	Set the process ID or process group ID to receive `SIGIO` and `SIGURG` signals, with `F_SETOWN`.

Always fetch the current flags, OR in the bit you want, then set them back. Assigning the flag directly wipes every other flag on the descriptor:

int flags;

if ( (flags = fcntl(fd, F_GETFL, 0)) < 0)
    err_sys("F_GETFL error");
flags |= O_NONBLOCK;
if (fcntl(fd, F_SETFL, flags) < 0)
    err_sys("F_SETFL error");

Your server crashes and will not restart — bind fails with EADDRINUSE. Give the correct fix and the tempting wrong one.

Correct: set SO_REUSEADDR on the socket between socket and bind. It permits the bind even though a previous incarnation of that connection is still in TIME_WAIT. Every TCP server should do this as a matter of course.

Tempting and wrong: setting SO_LINGER with l_onoff non-zero and l_linger zero, so that close sends an RST and skips TIME_WAIT entirely. That does make the port free immediately — and it discards unsent data and reopens the door to old duplicates being delivered to a new incarnation. RFC 1337 spells out the corruption. TIME_WAIT is not the problem.

Why must SO_RCVBUF be set on the listening socket rather than on the connected one?

Because the connected socket does not exist yet at the moment the decision is made. The receive buffer size determines the window scale option, and that option is exchanged on the SYNs — during the three-way handshake, which the kernel performs before accept returns.

By the time you hold connfd the SYNs are long gone. Setting it on listenfd works because SO_RCVBUF is one of the eleven options a connected socket inherits from the listening socket.