$$ c = k \oplus m. $$

To decrypt this ciphertext, we simply apply the key again:

$$ m = k \oplus m. $$

To understand why this is secure, we need to think like an attacker. Suppose we had access to \(c\). What can we say about \(m\)? We might try a few possible keys and see whether we recover anything interesting, but the a problem is immediate: with a length \(L\) message, there are \(2^L\) keys. And since the key is selected at random, we have no reason to suspect one more than other. This is like finding a needle in the haystack.

Intuitively we know there is no way we can expect to glean any information about the message from the ciphertext. But how can we formalize this? Claude Shannon captured this idea with the notion of perfect secrecy. An attacker might start with some initial impressions of what messages are likely or unlikely. Perfect secrecy demands that they have no reason to change these impressions based on intercepting a ciphertext alone. Mathematically, we might write this like

$$ P(m) = P(m \mid c), $$

where \(P(m)\) is the prior probability of a given message, and \(P(m \mid c)\) is the conditional probability of a message given the ciphertext.

We can use Bayes’ theorem to find an equivalent reformulation. Recall that this tells us that

$$ P(m \mid c) = \frac{P(c \mid m) P(m)}{P(c)}. $$

Combining the two equations we see that perfect secrecy is equivalent to

$$ P(c) = P(c \mid m), $$

or in English: the ciphertext must be independent of the message.

It’s easy to verify this for the one-time pad. Recalling that we choose the key at random, we have

$$ P(c \mid m) = P( c = m \oplus k) = \frac{1}{2^L}. $$

Here we see that we have more than independence: the conditional probability is uniform. Thus, we’ve shown that the one-time pad has perfect secrecy.

If you look at a book like Boneh and Shoup you’ll see this idea of perfect secrecy formulated in a slightly more general context. Here they say a Shannon cipher involves a key space \(\mathcal{K}\), a message space \(\mathcal{M}\), and a cihpertext space \(\mathcal{C}\), together with an encoder \(E : \mathcal{K} \times \mathcal{M} \to \mathcal{C}\) and decoder \(D : \mathcal{K} \times C \to \mathcal{M}\). To say that decoding “works”, mathematically, is to say that for all \(k \in \mathcal{K}, m \in \mathcal{M}\), we have

$$ D(k, E(k, m)) = m. $$

In this context, the requirement of perfect security demands that, for all \(m_0, m_1 \in \mathcal{M}\) and \(c \in \mathcal{C}\), if \(k\) is uniformly distributed over \(\mathcal{K}\), then

$$ P[ E(k, m_0) = c] = P[ E(k, m_1) = c]. $$

This is just a generalization of what we saw with the one-time pad!

Perfect secrecy is a lot to ask for. Although the one-time pad has this property, it comes at a significant cost: we need a key that is as large as our messages. This is typically not tractable, but the scheme is still important as a conceptual “gold standard”.

A basic requirement of any routing protocol is that it cope with routing loops. Recall that under the model of destination-based forwarding, each router forwards a packet based on the packet’s destination IP address. The sequence of routers a packet visits as it is forwarded through the network defines a path. If at any point this packet returns to a router it has previously visited, this path has a loop. Loops are dangerous because a packet will never escape, ultimately exhausting the IP TTL. Note that a routing loop is not the same as a loop in the network. Many network topologies have loops for the sake of redundancy; a network loop is necessary for there to be more than one path between hosts. But it’s the routing protocol’s job to ensure that packets are forwarded in a loop-free fashion.

A naive distance-vector protocol will eventually converge to a loop-free state, given enough time. But this convergence takes time, and you see what is known as “counting to infinity”, where loops are only evicted after computed distances increase step-by-step until reaching a large enough value that they are effectively infinite. While there are various tricks to mitigate this, BGP takes a simpler approach.

Routers running BGP (also known as BGP “speakers”) are connected in peering relationships. Here it’s useful to distinguish between connections within a single autonomous system (iBGP connections) and connections across autonomous systems (eBGP connections). Both eBGP and iBGP connections run the same protocol – BGP – but they are configured slightly differently.

Whereas routers running a naive distance-vector algorithm will advertise distance information, BGP speakers advertise routes. In the context of BGP, a route pairs a set of destinations with several attributes of a path between those destinations. There are several path attributes, but I will mention two of the mandatory ones.

1. The AS_PATH attribute is a sequence of AS-path segments. In its simplest form this is a list of AS numbers, like [1,6,7], though the actual specification allows for unordered sets as well.
2. The NEXT_HOP is the IP address of the router that should be used as the next hop to the destination.

When a route is advertised on an eBGP connection, the sender adds its AS number to the AS_PATH. Thus, as information propagates from one AS to another, the route carries information about its history. This makes it straightforward to detect possible inter-AS routing loops. I’ll quote from the spec:

If the AS_PATH attribute of a BGP route contains an AS loop, the BGP route should be excluded from the Phase 2 decision function. AS loop detection is done by scanning the full AS path (as specified in the AS_PATH attribute), and checking that the autonomous system number of the local system does not appear in the AS path. Operations of a BGP speaker that is configured to accept routes with its own autonomous system number in the AS path are outside the scope of this document.

Note that this “Phase 2 decision function” is just jargon for the rule that determines what advertised information actually makes it into the local forwarding table of a BGP speaker (and hence could be subsequently advertised to peers).

The AS_PATH information is enough to prevent AS loops. But what happens within a single AS?

Consider an AS with three routers A, B, and C, as shown below.

                   +-------+        +-------+
             eBGP  |       |  iBGP  |       |
            -------|   A   |--------|   B   |
                   |       |        |       |
                   +-------+        +-------+
                         \            /
                     iBGP \          / iBGP
                           \        /
                            +-------+
                            |       |
                            |  C    |
                            |       |
                            +-------+

Suppose A advertises a route to B, and B advertises this to C, and then C advertises this to A. The AS_PATH of this route does not reveal the loop, because it is internal to the current AS. If A accepted this advertisement, it would be a problem.

The BGP answer to this issue of intra-AS loops is another simple mechanism: speakers should not re-advertise routes received via iBGP. In our above example, this means A will advertise to B and C directly, and neither B nor C will send anything back to A. No loops are possible with this restriction.

While this solution is straightforward, it introduces a scalability problem. Since routes learned via iBGP are not re-advertised, we need to have iBGP connections between every pair of routers running BGP (this is sometimes called “full mesh” connectivity). In the above diagram that’s OK, because there are only three routers. But for N routers, there are N*(N-1)/2 pairs, and the amount of BGP updates flowing around the network and stored within the various routing-information bases can quickly get out of hand.

Fortunately, there is a BGP extension in RFC 4456 allowing a BGP speaker to be designated as a “route reflector”. Route reflectors can advertise iBGP learned routes to configured iBGP peers. Here’s a diagram from RFC 4456:

                 / - - - - - - - - - - - - -  -
                 |           Cluster           |
                   +-------+        +-------+
                 | |       |        |       |  |
                   | RTR-A |        | RTR-B |
                 | |Client |        |Client |  |
                   +-------+        +-------+
                 |       \           /         |
                    IBGP  \         / IBGP
                 |         \       /           |
                           +-------+
                 |         |       |           |
                           | RTR-C |
                 |         |  RR   |           |
                           +-------+
                 |           /   \             |
                  - - - - - /- - -\- - - - - - /
                     IBGP  /       \ IBGP
                  +-------+         +-------+
                  | RTR-D |  IBGP   | RTR-E |
                  |  Non- |---------|  Non- |
                  |Client |         |Client |
                  +-------+         +-------+

The idea of route reflection is to subdivide the routers into “clusters”. Within each cluster we have a designated “route reflector” which is linked in a star topology to a set of “client routers”. Across clusters we have a full-mesh connectivity of the route reflectors. When a route reflector receives advertisements from another cluster, it can advertise this to its clients. If a route reflector receives an advertisement from a client, it can advertise this to all other iBGP peers. If we had k clusters of size M, using route reflectors requires O(k^2 + M) total iBGP connections. Assuming k is more or less a small, fixed constant, you can see that this has linear scaling (rather than the naive quadratic scaling of iBGP).

It’s worth noting that there’s another extension, RFC 5065, that allows several ASes to be treated as a “confederation” that appears as a single AS to external peers. Within each AS of the confederation one would still need full mesh iBGP, but because eBGP is used between ASes of the confederation the connectivity can be sparser. If your confederation had k ASes of size M, then you would need something like O(kM^2) total iBGP connections. In practice, it seems like route reflectors are the more popular option, though there’s a risk for misconfiguration.

In the jargon of computer networks, such entities are called autonomous systems. They may control routing within their domains, but how should routing happen across domains?

Some of these autonomous systems are “stubs”, i.e. organizations that just want to provide internet access to their own networks. Others, such as an ISP or network carrier, are in the business of selling connectivity or transit services. Autonomous systems can enter into various relationships with each other: a service provider might sell access to a stub, and two service providers might agree to transfer data between themselves (known as “peering”).

As noted, each AS has their own incentives and priorities. A transit AS may want to forward traffic for which they get paid but not traffic for which they don’t. A government may want to ensure that their data is never routed through a certain country. For the internet to even exist, such incentives have to be aligned. From this perspective, inter-AS (also called inter-domain) routing isn’t just about minimizing the number of hops or the network latency. Rather, it’s about providing autonomous systems with a sufficiently rich set of policy levers that they can conduct business.

Within the world of routing protocols, a distinction is often made between “distance-vector” and “link-state” protocols. A distance-vector protocol computes routes by distributing the computation. Routers publish the destinations they can reach along with the cost to reach them (hence the name: mathematically this looks like a vector of distances). When one router receives distance vectors from another, it re-calculates its own costs based on this updated information. This is a distributed form of the Bellman-Ford algorithm, and one of the oldest approaches to routing. One of the earliest implementations is the Routing Information Protocol (RIP).

The distance-vector idea is conceptually simple, but a naive implementation suffers from several performance issues. One notable problem is that convergence can be slow in the face of topology changes. Underlying this is the fact that stale information can circulate around network loops, and routers don’t have any way of knowing that they are acting on this stale information.

Over time a new paradigm emerged, the so-called link-state algorithms. Rather than distribute the path computation, link-state protocols distribute knowledge of the underlying network topology. Routers do this by advertising the links they have available (hence the name). They also maintain a database of the advertisements they have seen. Shortest paths through the network can be computed on a router directly (e.g. using Dijkstra’s algorithm). The internet standard here is the Open Shortest Path First protocol.

Link-state protocols typically have better convergence properties than distance-vector protocols, but they are not well suited for routing across autonomous systems. This goes back to issues of policy and incentive. Suppose an autonomous system is only willing to route traffic when they get paid to. This kind of constraint cannot be enforced in the network topology or from clever ways of assigning link costs. Nonetheless, evaluating whether a proposed route adheres to this constraint is straightforward.

This is where the internet’s Border Gateway Protocol (BGP) comes in. BGP takes the distance-vector paradigm but turns it on its head. What distance-vector gets right, from the perspective of inter-domain routing, is that it is centered on sharing reachability information rather than link information. But the emphasis on distances or costs as the bottom line is too narrow-minded. Instead, routers running BGP advertise paths and metadata about these paths. Administrators have control over what paths their AS advertises. They also control how their AS uses path information shared by domains.

BGP was first proposed in 1989 in RFC 1105. The initial RFC is surprisingly short. I was amused to learn that the original idea was developed in an IETF cafeteria as the authors scribbled their thoughts on a pair of napkins, leading BGP to be known as the “two-knapkin protocol”. BGP was not created in a vacuum. The problem of inter-domain routing had already been considered in RFC 827 but was showing its limits as the internet grew. So BGP was an attempt at a next-generation inter-domain routing protocol.

The latest version, BGP-4, takes 100 pages to specify in the core standard. Today, inter-domain routing runs on BGP-4. What the protocol has lost in elegance or brevity, it’s made up for by successfully scaling the internet.

Sections 1 & 2 (Introduction and TCP/IP Overview)

RFC 1180 starts with a discussion of the basic structure of TCP/IP. This has not fundamentally changed, but there are some recent competitors to TCP, such as QUIC, that should be noted. Likewise, while Ethernet is still in widespread use, many people rely on wireless link layer protocols.

Section 3 (Ethernet)

Ethernet started as a shared-bus protocol and relied heavily on carrier-sense multiple access to manage communication on broadcast channels. These days it’s mostly used with point-to-point full duplex links, connected via switches, so the emphasis has moved from medium access control techniques to packet switching.

Section 4 (ARP)

ARP solves the problem of determining link-layer MAC addresses given an IP address. RFC 1180 describes a “typical scenario” in which the destination IP address is used to directly look up the destination MAC address, so that an Ethernet frame can be constructed. There are two extensions of this discussion that I’d consider:

1. A typical case not mentioned is when an IP packet is destined for an address outside of the current subnet. In this case the IP packet needs to be sent to the next-hop router, and so it is this next-hop router’s MAC address that is needed. While this can be inferred from the discussion in the next section (on IP), I think it would have been worth an explicit call-out here.
2. By now, we recognize some of ARP’s weaknesses, such as ARP spoofing.
3. Although ARP is still used for IPv4, it is not used for IPv6. Instead, see Neighbor Discovery Protocol.

Section 5 (Internet Protocol)

Overall this section was clear, and I found the distinction between Direct Routing (where IP adds little beyond Ethernet) and Indirect Routing (where IP is used to connect multiple Ethernet networks) well motivated. The extended discussion is dated in a few ways:

1. This section predates CIDR. So the discussion on addressing, as well as the routing examples, are “classful”.
2. There’s also no mention of DHCP, since it didn’t exist at the time.
3. RFC 1180 discusses the role of hosts files in associating names to IP addresses. While these are still used today, DNS is the main mechanism for managing this association.
4. There’s no discussion of IPv6.

Section 6 & 7 (UDP and TCP)

As far as I can tell, UDP hasn’t really changed since 1991. Maybe the only thing I would add here is some additional context around which contemporary application protocols use UDP.

TCP, on the other hand, has seen a lot of development, even if the basic protocol has not changed. Just a few examples:

1. There has been a fair amount of work on TCP congestion control. This has ramifications at lower levels of the stack (e.g. router bufferbloat), and is probably worth mentioning if only briefly.
2. This tutorial is notably light on security. I think it should include a discussion of TLS.

Section 8 (Network Applications)

In terms of philosophy, this section is still well motivated. But the particular examples (TELNET, FTP, rsh, NFS, SNMP, X-Windows) could be revised. Here are a few protocols I’d mention:

1. HTTP and HTTPS are ubiquitous examples that must be mentioned.
2. ssh as a modern and widespread replacement for rsh.
3. A mail or messaging protocol, like SMTP.
4. Some RPC protocol like gRPC.
5. Perhaps a reference to an overlay protocol like Tor.

Concluding Thoughts

When writing a tutorial there’s a tension between what to include and what to leave out. On the whole I think the authors of RFC 1180 did an admirable job. These days, this kind of “overview” can be found in any networking textbook, but in 1991 such information was undoubtedly harder to come by. I imagine this RFC helped orient a generation of system administrators, software developers, and network operators to the organization and function of the TCP/IP stack.

The primary rationale for IPv6 is to go beyond the limits of IPv4’s address space. As you recall, IPv4 addresses are 32 bits, limiting the protocol to roughly 4 billion addresses. Over the past decade or so, various entities responsible for managing IPv4 addresses have allocated their final blocks. The IANA declared the ’top-level’ allocation exhausted in 2011, followed by various regional registries in subsequent years. Wikipedia has a more detailed discussion of this here.

IPv6 introduces a 128-bit address space. Many of the fundamentals of IPv4 are still here: datagrams, “best-effort delivery”, and so on. But in some ways IPv6 also smooths some of the rough edges of IPv4.

You can see this by looking at the IPv6 header:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version| Traffic Class |           Flow Label                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Payload Length        |  Next Header  |   Hop Limit   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +                                                               +
   |                                                               |
   +                         Source Address                        +
   |                                                               |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   +                                                               +
   |                                                               |
   +                      Destination Address                      +
   |                                                               |
   +                                                               +
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Let’s take a look at the fields in more detail:

• The Version field is a hardcoded 6 for IPv6.
• The Traffic Class field can be used for differentiated services and explicit congestion notification. This basically lets IPv6 routers do things like traffic shaping or signal congestion to transport-level protocols like TCP.
• The Payload Length specifies the size of the data following the header.
• The Next Header field indicates the type of header immediately following this one (e.g. ICMP, UDP, TCP). You can see the current values here.
• The Hop Limit is equivalent to the IPv4 time-to-live (TTL) field.
• The Source Address and Destination Address are as you would expect.

Compared to the IPv4 header (see here) this is remarkably streamlined. In particular, forwarding an IPv6 packet is simplified because the header has a fixed size (whereas IPv4 has “options”) and there is no checksum that has to be recomputed when the hop limit / TTL gets decremented.

IPv6 eliminates the IPv4 variable-length “Options” field in a simple and elegant way: it defers options to various “Extension Headers” that sit between the IPv6 header and transport-layer data. Each of these extension headers has a “Next Header” field, allowing them to be chained together. About a third of RFC 8200 is defining these extension headers, but a virtue of this approach is that the basic mechanism of datagram forwarding can be accomplished without needing any of these.

IPv6 also eliminates IPv4’s fragment offset field. The spec requires that any implementor have a minimum transmission unit (MTU) of 1280 bytes, and suggests using Path MTU Discovery.

There’s much more that could be said about IPv6 – I only briefly touched on the extension headers, and didn’t discuss IPv6 addressing in any detail. But I have been wondering: how many people actually use IPv6? The original specification is almost 30 years old, but adoption of IPv6 has been slowed by the ubiquity of IPv4 and the use of things such as network address translation to cope with the limited address space. Google has a running chart of IPv6 traffic to their services. At the time of writing this article, total IPv6 usage only accounts for 47% or so of Google’s requests. There’s still a long way to go!

But many network administrators disable ICMP, or at least certain types of ICMP traffic. This is sometimes for security reasons, as ICMP can be used to map out a network, or as part of a denial-of-service attack. So what if you need to ping a host but can’t?

If this host accepts TCP requests, one option is to try to open a TCP connection. A convenient tool for this is ncat, a part of the nmap suite. On macOS, installing this is as simple as brew install nmap.

Let’s do a quick comparison between ncat and ping. To open a TCP connection and immediately close it, we can run ncat -zv google.com 80. The -z is for “zero I/O” mode, and we want -v to see some output. I’m asking for port 80, since I know google.com is serving HTTP requests there. I see the following output:

Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: Connected to 142.251.40.46:80.
Ncat: 0 bytes sent, 0 bytes received in 0.09 seconds.

On the other hand, running ping 142.251.40.46 I get

PING 142.251.40.46 (142.251.40.46): 56 data bytes
64 bytes from 142.251.40.46: icmp_seq=0 ttl=116 time=16.572 ms
64 bytes from 142.251.40.46: icmp_seq=1 ttl=116 time=16.137 ms
64 bytes from 142.251.40.46: icmp_seq=2 ttl=116 time=16.583 ms
^C
--- 142.251.40.46 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss

A key difference is that ncat reports significantly higher latency than ping. How can we explain this?

When I ran the above, I also captured the network traffic with Wireshark. Below is what I see for the ncat call:

There are a few things to note here:

1. The initial SYN + SYN/ACK round trip (the first part of establishing the TCP connection) has a latency of about 16ms, which is consistent with what we see from ping.
2. ncat sets up and immediately closes the connection, so there are two round trips here.
3. Network latency doesn’t fully account for the duration that ncat reports.

I’m not quite sure how to make sense of this 90ms duration when the TCP connection establishment and teardown was less than 40ms. It’s likely that the ncat duration is computed across a few calls to a system timer, and also includes the user/kernel transitions for the underlying socket processing – so some additional latency beyond the network time-of-flight is expected. With that said, I re-ran the same command a few times and saw durations as low as 50ms and as high as 170ms. ¯\_(ツ)_/¯

Because there are a huge number of possible IP addresses, routers typically work with ranges of addresses, as in CIDR. In this convention, and assuming IPv4, a range is denoted a.b.c.d/x where x indicates the number of bits associated with the network. This leaves 32-x bits for hosts. So for example, 0.0.0.0/0 is a range that covers all IP addresses, whereas 192.168.1.0/24 covers just those addresses of the form 192.168.1.*.

Rather than require that the ranges in the forwarding table be disjoint, the convention is that ranges may overlap. Both 0.0.0.0/0 (a ‘default’ route) and 192.168.1.0/24 (a more specific override) could appear in a forwarding table. When resolving the output port for a given IP packet, it’s not enough to find an entry in the forwarding table that matches. Instead, we are interested in the longest prefix match.

It’s useful to distinguish between the “data plane” and “control plane” of a router. The data plane is the part of the router that actually forwards packets. The control plane is the part that manages the forwarding table and coordinates with other routers as part of a routing algorithm.

The data plane has stringent performance requirements and typically involves custom hardware. Here Kurose & Ross suggest considering a simplified picture with three components:

1. An input port receives incoming data. This is where the physical and link (e.g. Ethernet) layers end, and the IP processing begins. Each input port can separately resolve the output port based on a lookup to the forwarding table.
2. After the output port is resolved, a packet is sent through a switching fabric. This is basically a piece of digital logic (e.g. a bus or interconnection network) that physically implements the forwarding.
3. An output port receives packets from the switching fabric and does the necessary link and physical layer operations to send these out.

Each of these components has its own complexities. For example, while “forwarding table lookup” sounds innocent enough, the implementation can be highly nontrivial. This article goes into a few details, noting that forwarding tables in high-end routers can contain millions of entries and may exceed the capacity of on-chip SRAM. Meanwhile, the router’s line rate may require that a longest-prefix match lookup happens in as little as a single clock cycle. Routers may use (seemingly) esoteric elements such as “ternary content-addressable memories” to efficiently implement forwarding calculations.

Another issue is that an output port may not be able to keep up with all of the inbound packets (for example, if they are arriving from multiple inputs simultaneously). Routers typically have memory buffers that can hold a short backlog of packets, smoothing out a burst of activity by making some packets wait around. But ultimately, excess packets are dropped, and packets which are delayed too long may be considered failures by the transport or application layers. Determining the optimal size of these buffers, and an efficient policy for managing them, can be highly nontrivial.

While we won’t go into any further details here, I hope this gives some appreciation for some of what goes into a router. It’s not the sort of knowledge I find myself needing for typical software work, but it is still nice to have some perspective on this essential pillar of networking.

For what follows, we’ll assume a simple client & server interaction where the client makes two GET requests over a single, persistent HTTP/2 connection. At first I tried to do this in Python using hyper-h2 but after fussing around a bit I just gave up and switched to Go. The Go net/http package supports HTTP/2 and it was pretty easy to have ChatGPT write a simple client and server for me. The client makes two GET / requests, and the server responds with "hello world" to each. The only thing I had to tweak was the http.Transport settings used by the client, to force HTTP/2 over a single connection.

When I ran this, the resulting Wireshark capture looked like this:

To start with, we see the same sort of TLS negotiation that we discussed previously. After this we see the HTTP/2 messages.

The first few of these, indicated below, come before the actual GET request.

Since this was run on a loopback interface, distinguishing the source and destination of the packets can be tricky. The easiest way to identify these is to note that there is a TCP ACK (in the opposite direction) immediately following each. With that in mind, here’s what we see:

1. The server (running on port 8443) sends the client a segment with a single SETTINGS frame. The HTTP/2 spec requires that the server send this to confirm that the protocol is in use and establish initial settings for the HTTP/2 connection (this is known as the connection preface).
2. The client (running on port 54081) sends a segment that appears to have three frames: Magic, SETTINGS, and WINDOW_UPDATE. Magic indicates the string "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" which starts the client’s connection preface (this was chosen so that HTTP/1.1 intermediaries would not attempt to process further frames). Following this is a SETTINGS frame. The WINDOW_UPDATE frame is used for flow control.¹
3. Following this, both the client and the server respond with a SETTINGS frame to acknowledge that the previous settings have been received.

After this exchange, the connection is ready for the actual requests & responses. I indicate these below:

This looks like what we expect:

1. The first GET request is carried by a HEADERS frame from the client to the server.
2. The server responds. The response has two parts: a HEADERS frame (e.g. carrying the 200 OK code) and a DATA frame with the actual body of the response.
3. Since our script makes two responses back-to-back, we see another pair of HEADERS (from the client) followed by HEADERS+DATA from the server.

Looking closely, we can see HPACK in action. The size of the first HEADERS packet from the client is 145 bytes. But the second time around this is only 113 bytes. The actual header is the same, but the second time around the client can reference previously sent values, reducing the transmitted data size.

HTTP/2 is undoubtedly harder to work with than HTTP/1.1, but much of the friction will ease up as tooling matures. With first-class support for HTTP/2, Wireshark makes debugging and understanding the protocol a breeze.

POST /users HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 50

name=FirstName%20LastName&email=bsmth%40example.com

In the above example, the header is larger than the actual body of the request. Worse still, HTTP is stateless, so session metadata (e.g. cookies) must be replicated for every request.

HTTP/2 improves upon this situation by introducing a header compression format known as HPACK. Introduced in RFC 7541, HPACK manages key-value pairs through a shared table which can be updated during the lifespan of a persistent HTTP/2 connection. A client can mark that a key-value pair should be added to the table, and subsequently refer to either the key or the key & value by their index into this table.

HPACK provides a mechanism but leaves the compression strategy up to the implementation (several flavors of dictionary coding are possible). The sender has full control over when and how the table is managed; a receiver simply decodes messages according to the usage of the sender.

To illustrate HPACK in action, we can step through an example given in the appendix of RFC 7541. We’ll skip some of the bitwise minutiae, instead just following the high-level logic.

Suppose a client makes an initial request with the following headers:

:method: GET
:scheme: http
:path: /
:authority: www.example.com

The first three key-value pairs are so common that rather than require the client to introduce them explicitly, they correspond to entries 2, 6, and 4 in a pre-defined static table. On the wire they only require three bytes: 0x82 (0x80 + offset 2) for the first, 0x86 for the second, and 0x84 for the third.

Things get interesting with the next pair, :authority: www.example.com. Here we have a very common header key that appears in the static table (:authority: shows up with index 1), but the value (www.example.com) is new to this connection. In such cases, HPACK lets you refer to the key as a standalone entity and then provide the value directly. On the wire this looks like 0x41 (“just the key with index 1”), followed by 0x0f (indicating the value is a 15 byte literal), followed by the byte values for www.example.com.

Taken together, the header for this first message takes 20 bytes to represent with HPACK. Considering that the first :method: GET, followed by a carriage return and line feed, is 14 bytes under HTTP/1.1, I’d say HPACK is pretty effective in this example.

The real power of HPACK comes from the fact that it provides for a dynamic key-value table which persists across the lifetime of the HTTP connection. Initially this table is empty, and after this first request the dynamic table would have a single entry:

Note that the static table uses indices 1-61, so the dynamic table starts at 62.

Suppose now the client makes a second request with the following header:

:method: GET
:scheme: http
:path: /
:authority: www.example.com
cache-control: no-cache

As before, the first three entries are compressed as 0x82 86 84 due to their presence in the static table. But now the :authority www.example.com pair is present with index 62, so the client can refer to it with 0xbe (this is 0x80 + 62).

The last line, cache-control: no-cache, has a key that appears in the static table (index 24) but a value that does not. So it is represented as 0x58 (for the key) followed by 0x08 (designating an eight character string literal) and then the actual encoding of the string no-cache.

As we see, this entire header is represented in just 14 bytes under HPACK. After it is sent the dynamic table has a new entry for cache-control: no-cache. By convention, new entries are inserted at the head of this table (shifting the other entries by 1):

The specification requires that the entire static+dynamic table have a fixed size, to make decoding inexpensive. This is controllable via an HTTP/2 parameter SETTINGS_HEADER_TABLE_SIZE which defaults to 256. Concretely, this means that inserting new entries to the table can cause old ones to roll off.

As you can see, HPACK provides a simple but effective mechanism for header compression. A session cookie would only need to be sent once, after which it would occupy a single entry in the dynamic table and could be referred to by index. This keeps headers small, so that network resources may be more efficiently be used by application data.

Transport Layer Security, or TLS for short, is an approach to security which sits just above TCP (hence the name). At a high level, the protocol describes a way to open an encrypted channel on top of an underlying TCP stream. An initial handshake settles cryptographic conventions and identities, after which encrypted application data is sent over ordinary TCP segments.

There are a lot of cryptographic details that can obscure the high level structure of the protocol. To get a feel for how TLS works, we can instead just try it out with an HTTP request (so-called HTTPS) and look at some of the captured packets.

Last time, we used cURL to generate an HTTP request. Today, we’ll use Python. In the following code we use the ssl package to wrap an ordinary TCP socket with TLS:

 1import socket
 2import ssl
 3
 4host = 'google.com'
 5port = 443
 6
 7context = ssl.create_default_context()
 8# Log TLS key information for use with Wireshark
 9context.keylog_filename =  "sslkeylog.log"
10
11with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
12    sock.connect((host, port))
13    with context.wrap_socket(sock, server_hostname=host) as ssock:    
14        request = (
15            "GET / HTTP/1.1\r\n"
16            f"Host: {host}\r\n"
17            "Connection: close\r\n"
18            "\r\n"
19        )
20    
21        ssock.sendall(request.encode('ascii'))
22    
23        response_parts = []
24        while True:
25            data = ssock.recv(4096)
26            if not data:
27                break
28            response_parts.append(data)
29            
30        ssock.close()
31    
32        response = b''.join(response_parts).decode('utf-8', errors='replace')
33        print(response)

Note that in Python, we need an SSLContext in order to wrap a socket. In the above code we explicitly indicate that the context should save the session’s TLS keys. We’ll need these to decrypt the packet contents in Wireshark.

After running the above and catching the packets, we see the following:

Although we can’t follow every detail in this, we can at least understand the shape of the interaction:

1. There’s an ordinary TCP connection establishment.
2. Then, some back and forth between the client and server to set up the TLS session. Wireshark labels many of these packets with TLSv1.3 as the protocol, and in some cases gives suggestive names like “Server Hello, Change Cipher Spec”.
3. There are a number of TLSv1.3 segments that just get labeled “Application Data”. These are encrypted but presumably include the HTTP GET and response.
4. Finally, the TCP connection closes.

To further inspect these packets we need the keys we recorded during our request. These are ephemeral to the session so it’s safe to share the contents of sslkeylog.log :

# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET bb058323a16b445da4c69a211dd9fb3718a5574fef4a13e59a00c94eb46b5731 f431df302f018369fe4540bf3d148ea62faedeff5b518db06e6077267ca2807241c7934238702e288641759d473caaab
EXPORTER_SECRET bb058323a16b445da4c69a211dd9fb3718a5574fef4a13e59a00c94eb46b5731 b24fc91dddbe4c1fd5cd308d1cdd0db3c0dacfc3756ac9661d7c5a20d04a10c0759c55fcc3b8a38fa2793e191d7b8992
SERVER_TRAFFIC_SECRET_0 bb058323a16b445da4c69a211dd9fb3718a5574fef4a13e59a00c94eb46b5731 d8a62b42acffc95f5730618010de5cbdeec02c8d0e26a85d53da989e806cd6f0b297772b9c082bae2d2c86b6276848a6
CLIENT_HANDSHAKE_TRAFFIC_SECRET bb058323a16b445da4c69a211dd9fb3718a5574fef4a13e59a00c94eb46b5731 f58ff4ceb0cd5ff5dd668c7f6217896e17eb5a1b3a00a2af0a8a6118d0ab3bc57f9de4cf67a432285db4dcee7b43a1b9
CLIENT_TRAFFIC_SECRET_0 bb058323a16b445da4c69a211dd9fb3718a5574fef4a13e59a00c94eb46b5731 5312948edcc68e012acd1622c0f2191bb36b2ed1b26a4d76a7b6dd8df2a807239fcdae09da5e5581a5198bed25e4588e

In Wireshark you can provide this file (Preference -> Protocols -> TLS -> (Pre)-Master-Secret log filename) and then see something like this:

Here we see a few more details that weren’t visible before:

1. The HTTP request and response, as expected.
2. A TLSv1.3 message to provide a “New Session Ticket”. This allows a client to resume this session in a future connection (e.g. reducing the cost of a handshake).

This was just a quick “anatomical exercise”. I won’t dwell too much on TLS, but it’s ubiquitous and so worth getting familiar with. For a more detailed dive into how TLS drives the underlying TCP connection, see The Illustrated TLS 1.3 Connection.

When I run:

1curl -v http://google.com -H "Connection: close"

I see the following:

* Host google.com:80 was resolved.
* IPv6: (none)
* IPv4: 142.250.72.174
*   Trying 142.250.72.174:80...
* Connected to google.com (142.250.72.174) port 80
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/8.7.1
> Accept: */*
> Connection: close
> 
* Request completely sent off
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.com/
< Content-Type: text/html; charset=UTF-8
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-EB7GxjMVJsa5Mr_7PWB0yA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Date: Wed, 12 Feb 2025 01:43:25 GMT
< Expires: Fri, 14 Mar 2025 01:43:25 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 219
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< Connection: close
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
* Closing connection

Looking at Wireshark, I see the following capture:

The first few segments set up the TCP connection. You can see the “three-way handshake” in action:

With the TCP connection established we make the HTTP GET request. The server acknowledges this before sending the response:

In principle, the server could bundle the above ACK with the actual HTTP response, but doing so would introduce a delay if the response isn’t ready immediately. So it plays it safe: first, it sends an ACK, then it responds to the request.

The next segment has the actual HTTP response, which we acknowledge:

Finally, the TCP connection is torn down. I ran this experiment twice, and the next screenshot is from the second run, but the behavior and relative sequence numbers are the same. I was expecting to see a three-way handshake, but there are four segments here:

Recall that I asked curl to use the Connection: close header in my request. This tells Google that it is free to close the connection once it has served a response. What we see in the above screenshot is that both sides of the TCP connection initiate a close simultaneously. Because of this, each FIN is sent before receiving the other’s. Since a FIN gets its own sequence number, it requires a separate ACK. So we need an extra acknowledgment to fully close this connection.

The takeaway here is that we can see HTTP riding on TCP just by sniffing packets. This gives a solid sanity check for what we have already learned about these protocols.

The WWW was developed by Tim Berners-Lee and others at CERN. Tim describes the motivation in a 1991 newsgroup announcement:

The WWW project was started to allow high energy physicists to share data, news, and documentation.

To turn the dream of the WWW into a reality, Berners-Lee had to develop some key technologies:

1. A textual format, HTML, for hypertext documents. These documents include typesetting information as well as links to others.
2. A protocol, HTTP, to request and receive hypertext documents.
3. A server, which can host HTML files (or generate them on-the-fly).
4. A client, the earliest web browser, which can request and display HTML.

The earliest public version of HTTP is version 0.9. This is an extremely simple application protocol running on top of TCP. It defines a single type of request made by a client: get an HTML document at a given address. The server responds with the contents of the HTML document, and then closes the connection. If there is an error it is reported as a human-readable HTML message.

Even without a web browser, a person could manually issue a GET request directly in ASCII text using a telnet client. No doubt this ease of use helped drive early adoption. Over the next few years, HTTP incorporated additional features, including new request types and headers for communicating additional metadata. In 1996, the HTTP/1.0 specification was proposed in RFC 1945. If you’re curious about the evolution of HTTP, I highly recommend this article from the Mozilla developer documentation.

RFC 793 provides a diagram of the TCP segment format, which I’ve reproduced below:

    0                   1                   2                   3   
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                            TCP Header Format

          Note that one tick mark represents one bit position.

                               Figure 3.

To better understand what TCP offers, let’s contrast it with UDP, a much simpler protocol. RFC 768 provides an analogous diagram for the UDP header:

                  0      7 8     15 16    23 24    31  
                 +--------+--------+--------+--------+ 
                 |     Source      |   Destination   | 
                 |      Port       |      Port       | 
                 +--------+--------+--------+--------+ 
                 |                 |                 | 
                 |     Length      |    Checksum     | 
                 +--------+--------+--------+--------+ 
                 |                                     
                 |          data octets ...            
                 +---------------- ...                 

                      User Datagram Header Format

There are a few features common to both:

• Ports are used for multiplexing. This lets applications address specific processes rather than just hosts.
• Checksums are used to detect corruption of individual segments/datagrams.

That’s about it! Let’s look at a few of the differences:

• The TCP header has a variable length, depending on whether options are used. Because of this, it needs a Data Offset field to indicate where the data actually starts.
• Unlike TCP, UDP explicitly includes a Length field to indicate the total size of the datagram. TCP does not require this field because the length can be inferred from the enclosing IP packet’s size. One possible reason for UDP’s design choice is that it comes from a time when alternatives to IP were still being considered. Additionally, including Length ensures that the UDP header has 4-byte alignment.
• TCP provides a reliable byte stream by explicitly communicating the position in this stream for data being sent (the Sequence Number). Since TCP is bi-directional, a segment also indicates the next byte expected by it sender (the Acknowledgement Number). The sequence number increases as data is sent, and the acknowledgement number increases as data is successfully received. If sent data is not acknowledged, TCP can re-send this data.
• To prevent scenarios where a sender is overwhelming a receiver with more data than it can keep up with, TCP provides a notion of flow control. This relies on the protocol managing an interval of sequence numbers, known as a “sliding window”. The window defines what sequence numbers the sender is allowed to send, and this window advances as earlier sequence numbers get acknowledged. Instead of using a fixed window size, the TCP specification allows for the receiver to set the size with the Window field of the header.
• The TCP header has various flags that can be used for things like establishing or closing a connection.
• The Urgent Pointer. In principle, when the Urgent flag is set, this field indicates an offset into the TCP segment’s data where the bytes are considered “urgent”. It’s not obvious from the RFC how this is used in practice.

This comparison offers an additional perspective on TCP that complements my previous discussion about the TCP state machine and sequence diagrams. While the state machine and sequence diagrams define TCP’s semantics, the segment format defines TCP’s syntax. To fully understand TCP, you need both.

Broadly speaking, a protocol is a set of rules describing how entities (e.g. people, machines, processes) interact in a given system. Almost inevitably these rules depend on whether certain circumstances are met, and typically the entire protocol can be described as a state machine where transitions are triggered by events and themselves imply certain actions (e.g. starting a timer).

There are many ways to describe such a state machine, but one of these most legible approaches is through a graphical diagram, where states are vertices and transitions are edges. These edges are labeled with a precondition (which must be met for the transition to occur) and an action (performed as part of the transition). For example, the TCP specification describes the connection state machine and summarizes it with a diagram. This is a full page of ASCII art – feast your eyes:

September 1981                                                          
                                           Transmission Control Protocol
                                                Functional Specification

                                    
                              +---------+ ---------\      active OPEN  
                              |  CLOSED |            \    -----------  
                              +---------+<---------\   \   create TCB  
                                |     ^              \   \  snd SYN    
                   passive OPEN |     |   CLOSE        \   \           
                   ------------ |     | ----------       \   \         
                    create TCB  |     | delete TCB         \   \       
                                V     |                      \   \     
                              +---------+            CLOSE    |    \   
                              |  LISTEN |          ---------- |     |  
                              +---------+          delete TCB |     |  
                   rcv SYN      |     |     SEND              |     |  
                  -----------   |     |    -------            |     V  
 +---------+      snd SYN,ACK  /       \   snd SYN          +---------+
 |         |<-----------------           ------------------>|         |
 |   SYN   |                    rcv SYN                     |   SYN   |
 |   RCVD  |<-----------------------------------------------|   SENT  |
 |         |                    snd ACK                     |         |
 |         |------------------           -------------------|         |
 +---------+   rcv ACK of SYN  \       /  rcv SYN,ACK       +---------+
   |           --------------   |     |   -----------                  
   |                  x         |     |     snd ACK                    
   |                            V     V                                
   |  CLOSE                   +---------+                              
   | -------                  |  ESTAB  |                              
   | snd FIN                  +---------+                              
   |                   CLOSE    |     |    rcv FIN                     
   V                  -------   |     |    -------                     
 +---------+          snd FIN  /       \   snd ACK          +---------+
 |  FIN    |<-----------------           ------------------>|  CLOSE  |
 | WAIT-1  |------------------                              |   WAIT  |
 +---------+          rcv FIN  \                            +---------+
   | rcv ACK of FIN   -------   |                            CLOSE  |  
   | --------------   snd ACK   |                           ------- |  
   V        x                   V                           snd FIN V  
 +---------+                  +---------+                   +---------+
 |FINWAIT-2|                  | CLOSING |                   | LAST-ACK|
 +---------+                  +---------+                   +---------+
   |                rcv ACK of FIN |                 rcv ACK of FIN |  
   |  rcv FIN       -------------- |    Timeout=2MSL -------------- |  
   |  -------              x       V    ------------        x       V  
    \ snd ACK                 +---------+delete TCB         +---------+
     ------------------------>|TIME WAIT|------------------>| CLOSED  |
                              +---------+                   +---------+

                      TCP Connection State Diagram
                               Figure 6.

                                                               [Page 23]

For all of its complexity, this diagram gives a single page summary of what would otherwise take many pages to describe in words. One analogy that comes to mind is with the configuration space of a physical system. The state diagram provides a complete map of possible states: at any given moment, a TCP implementation must be in one of these states.

If state diagrams summarize the structure of a system, sequence diagrams reveal its evolution over time. Typically these show how several actors, each running their own version of the state machine, interact via messages. TCP has several such diagrams in the specification. For example, on pg. 31 there is the following diagram for the “3-way handshake” used to establish a connection:

      TCP A                                                TCP B

  1.  CLOSED                                               LISTEN

  2.  SYN-SENT    --> <SEQ=100><CTL=SYN>               --> SYN-RECEIVED

  3.  ESTABLISHED <-- <SEQ=300><ACK=101><CTL=SYN,ACK>  <-- SYN-RECEIVED

  4.  ESTABLISHED --> <SEQ=101><ACK=301><CTL=ACK>       --> ESTABLISHED

  5.  ESTABLISHED --> <SEQ=101><ACK=301><CTL=ACK><DATA> --> ESTABLISHED

          Basic 3-Way Handshake for Connection Synchronization

                                Figure 7.          

Here we have two hosts running TCP, and hence each in their own TCP state. As time progresses from top to bottom, these hosts communicate with TCP messages and advance from state to state. The first host, TCP A, goes from CLOSED to SYN-SENT to ESTABLISHED, while the second host, TCP B, goes from LISTEN to SYN-RECEIVED to ESTABLISHED. Although this scenario was latent in the state diagram itself, it’s much more legible in a sequence diagram.

Whether it visually or in text, defining a protocol involves describing a state machine and explaining the sequences it can produce. While the above examples are presented informally, one can imagine formalizing such diagrams to prove the mathematical correctness of a protocol¹. But that’s a discussion for another time.

Rather than slog through RFC 793, we can start by looking at Python’s socket library. This is just a Pythonic wrapper around BSD sockets. These allow you to interact with network interfaces more-or-less as if they were files (e.g. with read and write, or their counterparts recv and send).

It’s pretty easy to write a simple UDP server in Python, and since there are many tutorials available on the web I will be brief in description. In the following example, the server accepts messages and then echoes them back to the sender:

1import socket
2
3def udp_server(host: str, port: int):
4    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
5    sock.bind((host, port))
6    while True:
7        msg, client = sock.recvfrom(2048)
8        sock.sendto(msg, client)

You can tell that this is UDP because the socket type is SOCK_DGRAM. A toy client might look like the following:

1def udp_client(addr: str, port: int):
2    dest = (addr, port)
3    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
4    msg = input('Please enter a message: ')
5    sock.sendto(msg.encode(), dest)
6    response, _ = sock.recvfrom(2048)
7    print(response.decode())

All this does is get a message from the user, send it to the server, and then print out the response.

When we move to TCP, things get a bit more complicated. A similar server (but now with type SOCK_STREAM) requires a bit more set-up:

1def tcp_server(host: str, port: int):
2    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
3    listener.bind((host, port))
4    listener.listen(1)
5    while True:
6        conn, _ = listener.accept()
7        msg = conn.recv(2048)
8        conn.send(msg)
9        conn.close()

Here we have a listener socket which is bound to the given (host, port) pair. It then notifies the operating system that it is ready to listen for incoming connections.

The TCP client now has to explicitly connect, as shown below:

1def tcp_client(addr: str, port: int):
2    dest = (addr, port)
3    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
4    sock.connect(dest)
5    msg = input('Please enter a message: ')
6    sock.send(msg.encode())
7    response = sock.recv(2048)
8    print(response.decode())

On the server side, the accept call responds to the client’s connect. This also produces a new socket (conn above), dedicated to that specific connection. This is a lot more complicated than the UDP counterpart – what’s going on here?

This connect/accept logic handles the TCP connection establishment. The first TCP segment (roughly, the TCP equivalent of a packet) sent from the client to the server has a special flag in the header to indicate that it wants a connection. This segment is sent to listener. Under the hood, there is a brief back-and-forth with the client (this is the “three way handshake” described in the Wikipedia article). After that, the accept call on the server returns a new socket. Subsequent TCP segments to and from the client carry normal data, and use this second socket.

Notice also that in the code above, the TCP server could have multiple connections (and so multiple sockets) associated with the same port. What we see here is TCP multiplexing. It can be helpful to contrast with UDP. When a UDP datagram arrives at a host, it is routed to the appropriate socket simply by looking at the pair (dest addr, dest port). This means that for a given UDP port, all incoming packets are routed to the same socket, regardless of sender. With TCP, each connection is uniquely identified by the tuple (source addr, source port, dest addr, dest port). This allows multiple connections on the same destination port, as long as they originate from different sources. But because the listener handles the communication before the TCP connection is established, we can only bind one socket to a given (dest adddr, dest port) pair.

Anyways, this post is something of a teaser – there’s still a lot more to say about TCP. Next time we’ll break down the structure of a TCP segment and explore how sequence numbers and acknowledgements are used to ensure reliable communication.

Whereas IP operates at the “network layer”, UDP sits at the “transport layer”. Really, UDP adds just one key feature on top of the basic IP datagram model: ports. A port lets you distinguish between multiple services behind a single network interface. This distinction is not important for routing across the network, but it’s a practical necessity.

The Kurose and Ross analogy is roughly this: think of hosts as households and these services as people. IP is like the postal service, delivering letters to a specific address. The port is the name of the intended recipient, but it’s up to whoever checks the mail (roughly “the operating system”) to deliver this.

That’s it for today. Next time, we’ll start looking at bigger problems that the transport layer solves.

Any conforming implementation of Internet Protocol must support its companion, the Internet Control Message Protocol. Defined in RFC 792, ICMP provides a very lightweight mechanism for error reporting.

In essence, ICMP defines messages that are sent back to the source of a packet that fails delivery. These messages use the standard IP header format but contain specific data fields that define their purpose. The spec describes a dozen or so message types, but several of these have been deprecated. Some of the most notable types in active use are:

1. Destination Unreachable (Type 3): Sent when a packet cannot reach its destination.
2. Echo Request (Type 8) and Echo Reply (Type 0): Used by tools like ping to check network connectivity.
3. Time Exceeded (Type 11): Indicates that the packet time-to-live (TTL) has expired.

A good example of how the Time Exceeded message can be used is traceroute, which sends packets with successively increasing TTL values in order to probe the route(s) taken to a host. For example, when I run:

1traceroute 8.8.8.8

I get the following output:

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  192.168.1.1 (192.168.1.1)  5.796 ms  4.522 ms  6.533 ms
 2  * * *
 3  lag-62-10.dtr02psdnca.netops.charter.com (96.34.102.37)  13.634 ms  13.846 ms  11.718 ms
 4  lag-30.crr03rvsdca.netops.charter.com (96.34.97.154)  18.296 ms  19.656 ms  26.183 ms
 5  lag-811.bbr01rvsdca.netops.charter.com (96.34.3.18)  15.703 ms  19.682 ms  16.463 ms
 6  lag-801.prr01lsanca.netops.charter.com (96.34.3.129)  16.699 ms  19.619 ms  16.894 ms
 7  sar02sghlga-eth-2-1.sghl.ga.charter.com (96.34.1.69)  18.305 ms
    72.14.219.250 (72.14.219.250)  17.759 ms  17.314 ms
 8  * * *
 9  dns.google (8.8.8.8)  20.087 ms  19.610 ms  17.278 ms

Here I see various intermediate nodes from my ISP all the way to Google’s DNS server. The asterisks here mean that there was no response. These nodes likely limit or disable ICMP responses.

Why has IP stood the test of time? My guess is that there are a few factors.

1. As mentioned in the preface, RFC 791 is actually based on six earlier versions of the ARPA Internet Protocol. So this is not creation ex nihilo but rather incremental refinement of some underlying ideas.
2. IP solves one problem and it solves it well. The authors are intentionally restrained, stating

The internet protocol is specifically limited in scope to provide the functions necessary to deliver a package of bits (an internet datagram) from a source to a destination over an interconnected system of networks. There are no mechanisms to augment end-to-end data reliability, flow control, sequencing, or other services commonly found in host-to-host protocols.

3. The “user interface” (i.e. the part of this protocol that systems programmers might interact with) is exceedingly simple. Although the details are OS dependent, this basically amounts to a pair of subroutines, send and receive.

With that said, the protocol still has more complexity than I expected. One challenge IP had to address is packet size differences across networks. Although the specification allows for packets up to 65kB, it only requires implementations to support packets as small as 68 bytes (60 bytes for the header and 8 for the data). This kind of flexibility was probably necessary, and to accommodate it IP supports something called fragmentation. Basically, IP describes how to break up or reassemble packets.

Fragmentation introduces significant complexity. The header has fields indicating how a fragment relates to its original packet. When a packet is fragmented, some fields of the original will copy to the first fragment but not the rest. Since IP uses checksums for error detection, fragmentation requires recomputing these. And there’s a whole page of the RFC detailing the complex (relative to the rest of the document) assembly procedure to recover a packet from its fragments.

In short, fragmentation is an engineering compromise to cope with networking realities of the time. The consensus seems to be that fragmentation is considered harmful, since if any of the fragments get dropped then the whole packet gets dropped. These days it looks like people try to avoid sending packets that exceed the maximum transmission unit (MTU) along the path, cf RFC 1191 and RFC 8899.

Anyways, I’ve said enough about fragmentation. To close this entry out, here’s something I didn’t know: the IP header has a 16 bit field for representation the security level of the packet, ranging from ‘Unclassified’ to ‘Top Secret’. There are eight additional levels reserved for future use – or so they claim….

There are really two things I hope to get out of this effort. The first is to take seriously the idea of writing-as-learning. In my present stage of life, with my workdays sliced as thin as salami and my home life focused on keeping up with a small child, I simply don’t have as much time to think deeply about things as I used to. I miss that. My hope is that by developing a writing habit, I can better leverage the time I do have. Gone are the days of spending a weekend hacking on a personal project or digesting a complex mathematical argument. But I can find an hour here and there to write.

The second thing I hope to achieve is to change how I relate to the broader internet. When I first experienced the internet in the late 90s, it felt like such a strange and wonderful place. Part of this was the novelty of it, but the early internet was also fundamentally open and personal. Although the internet of today is a much larger and, in some ways, more connected than ever, I’ve nonetheless grown increasingly disenchanted. These days, I mostly engage with the internet passively – ‘consuming content’ through a handful of link aggregators or social media sites. While I don’t expect to rekindle the original magic of the internet, blogging feels like a meaningful way to re-engage as an active participant.

I don’t know exactly what this blog will turn into, but I’m making a personal commitment: over the next year, I aim to write roughly one ’long form’ post a month. I’ll also aim to share shorter, less polished, posts more frequently.

Finally, I’ve switched from Hakyll to Hugo for this site. Hugo isn’t perfect, but tinkering with my old setup feels like too much work right now. (Heck, I don’t even feel like installing Haskell on this machine!) Hugo lets me get started quickly, and I can always tinker later. For now, it’s time to write.