WebSockets Protocol - Very Informative - 2025

This course has been structured in a way that will enable you to learn advanced concepts to do with WebSockets.

The content has been formulated to take your development skills to the NEXT LEVEL, by focusing on all aspects of a WebSocket connection, such as: 

• the TCP handshake process

• the HTTP upgrade process

• the actual WebSocket API itself (what interfaces does it rely on, what events does it emit, etc)

• how WebSocket messages are formatted into data frames

• how to scale WebSockets, what alternative technologies exist, etc

The internet is a global system that connects billions of computers across the world. It has pretty much always used a standard internet protocol suite known as TCP/IP to facilitate communication between devices and networks. This is important because as you'll see later, WebSockets were designed over TCP.

The HTTP protocol was created in 1989 which was pretty much the same time that the World Wide Web was created. HTTP (Hypertext Transfer Protocol) is the set of rules that allows web browsers and web servers to communicate and exchange information over the internet.

In other words, it's the protocol that makes the World Wide Web work.

However, the HTTP protocol was designed as a simple request/response model, which in today's world of financial markets and online gaming, just doesn't cut it.

TCP was created in the 1980's and stands for "Transmission Control Protocol". Today, it is a persistent protocol in that it establishes a reliable connection between the client and server, and it does not terminate the connection even after a request/response cycle has been completed. The initial connection is achieved through a three-way handshake.

There is a lot more to TCP than first meets the eye. TCP also employs mechanisms such as flow control, retransmission of lost packets, and sequencing to guarantee that data is delivered accurately and in the correct order.

TCP (Transmission Control Protocol) is a persistent protocol. This means that it establishes a reliable connection between the client and server, ensuring that data is transmitted in the correct order and without errors. This persistence is achieved through a three-way handshake for connection establishment.

Wireshark is a powerful network protocol analyzer that captures and displays real-time details of network traffic. It is widely used for troubleshooting network issues, analyzing network protocols, and ensuring network security. Wireshark can capture packets from a network connection and display them at a granular level, allowing users to filter and drill down into the data to identify specific issues.

HTTP is the most popular and most use network protocol in the world! It's been around since "pa fell off the bus" but did you know there are some interesting facts about HTTP that you might not have known.

Comet and AJAX are two web development concepts or terms used to describe how HTTP can be used to make the user experience seamless. In other words, they are words to describe ways in which a client can request a web page from a server, dynamically. However, although AJAX and comet have come a long way in helping us, they are not enough for today's high powered online gaming sites and financial markets. In other words, they don't offer true bi-directional communication.

You have a high level understanding of how the internet was created, and that it was built on the HTTP protocol. However, we have more modern protocols today that are better suited for real-time applications. And this brings me onto ... you guessed it ... WEBSOCKETS.

The WebSocket protocol has a rich history that dates back to 2008. Initially, it was referred to as TCPConnection in the HTML5 specification as a placeholder for a TCP-based socket API.

The name "WebSocket" was coined, and the protocol was standardized in 2011 via RFC 6455. Google Chrome was the first browser to ship full support for WebSockets in December 2009, with other browsers following suit over the next few years. Today, almost all modern browsers support WebSockets, which have become a preferred choice for building interactive, real-time digital experiences that provide delightful user experiences.

The RFC process can be confusing at first, so this lecture explains the following: 

1. Not all RFCs are standards. In fact, most RFCs do not get into the Standards Track and hold as much weight as a feather

2. Only the IETF can approve an RFC to the point of being included in the Standards Track.

3. If an RFC reaches the Standards Track, it can be in (1) proposed state or (2) Internet Standard.

4. It is counter-intuitive, because even if an RFC is in proposed state, it is good enough to be authoritative and most RFCs stay in this state forever.

Some fun facts about WebSockets. WebSockets are a great protocol if you want to send updated data from a server to a client constantly.

WebSockets are governed by an RFC and also a living standard.

WebSockets are used by various organizations and applications today. This proves it is a powerful protocol that is worth learning about.

You already know that WebSockets is a protocol, allowing two endpoints to communicate over the internet. But what exactly does this mean? 

"Packets" are the basic units of data that are transmitted over the internet. Each packet contains both a header (containing routing information) and a payload (the actual data being transmitted).

Packets are used to break down large messages into smaller, manageable chunks, allowing them to be transmitted efficiently over the internet.

While web browsers like Chrome, Firefox, and Safari are the most common way to interact with the HTTP protocol, they are not the only tools available. cURL, a command-line tool, is another popular way to view and interact with the HTTP protocol.

Using cURL, you can inspect the HTTP protocol in detail, which can be useful for debugging, testing, and understanding how web applications communicate with servers. This is particularly helpful when working with APIs or troubleshooting issues that may be related to the underlying HTTP protocol.

HTTP: HTTP and WebSockets are both communication protocols used in web applications, but they serve different purposes. HTTP operates on a request-response model, where the client sends a request to the server and waits for a response, making it suitable for retrieving static content and non-real-time data exchanges.

WebSockets: In contrast, WebSockets establish a persistent, bidirectional connection that allows for real-time communication, enabling the server to push updates to the client without the need for repeated requests.

Nothing in web development works in isolation. There are many parts (protocols) that have to work together.

WebSockets use TCP to establish a connection between the client and server. This connection is maintained throughout the duration of the WebSocket session, allowing for continuous communication.

The Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS) are two related but distinct concepts. As I mentioned in the previous video, networks will typically not allow a file size of 20,000 bytes to be contained within one packet.

But wait a second, the IP protocol allows packet sizes to be represented by 16 bits, which can represent a maximum value of 65,535 bytes.

This is where MTU comes into the picture.

The Maximum Segment Size is set by the TCP layer, which represents the max size of one segment, including the payload and headers.

WebSockets work by establishing a persistent connection between a client and a server.

You know this.

But WebSockets can't send data over the internet by itself. In order to do so, other protocols need to be used such as the Transmission Control Protocol (TCP) and Internet Protocol (IP).

A brief summary of what we've covered.

IP datagrams contain all the information needed to route the data from the source to the destination on an IP network.

When an IP datagram needs to be transmitted, it is broken down into smaller IP packets that can be sent over the network.

Congratulations! You’ve crushed the first section!

Before diving into the nitty-gritty about WebSockets, you have to master the fundamentals.

WebSockets have truly transformed the way we connect and communicate online.

Why, you ask?

Well, traditional HTTP operates on a request/response cycle, which can feel a bit like waiting for a bus that never arrives - especially for the dynamic web applications we love today!

For applications like geo-location tracking, high-speed stock trading, and those real-time chat groups, WebSockets are like a turbocharged rocket ship, leaving HTTP in the dust!

Now that you’re equipped with some high-level knowledge about TCP, HTTP, and the magic of WebSockets, it’s time to put that brainpower to the test!

See you in the next section.

WebSockets were built using TCP.

Later in the course I'll explain how HTTP/3 changes this.

But for now, understand that TCP uses a 3-way handshake to establish a reliable connection between two devices.

The handshake involves three steps:

1. the client sends a SYN (synchronize) message to the server

2. the server responds with a SYN-ACK (synchronize-acknowledge) message, and

3. the client acknowledges the server's response with an ACK (acknowledge) message.

This process ensures that both devices synchronize their sequence numbers and establish the necessary parameters for data transmission. In other words, it helps ensure a reliable connection.

In simple terms, a persistent TCP connection is like keeping a phone line open between two people, even after they've finished their initial conversation, so they can easily talk to each other again without having to dial the number every time. This allows for faster and more efficient communication, as the connection is already established and ready to be used whenever needed, without the overhead of setting up a new connection each time.

A few interesting facts about TCP

I want to show you the practicality of seeing the 3-way TCP handshake in action. I will show you the SYN - SYN/ACK - ACK in action.

The TCP/IP model is a four-layered communication model that breaks down network communication into manageable chunks. This model consists of the Application Layer, Transport Layer, Internet Layer, and Network Access Layer. Each layer performs a specific function to ensure efficient and reliable data transfer over the internet.

The TCP/IP model is a four-layered communication model that breaks down network communication into manageable chunks. This model is widely used in computer networking and is the basis for the internet.

The application layer is the very top layer of the TCP/IP model. It is the closest thing that us developers work with on a daily basis and is where 99% of time is spent by web developers.

This layer is responsible for providing interfaces and protocols needed by users and applications to access the internet. Common application layer protocols include HTTP, WebSockets, SMTP and FTP.

A quick summary of what we've covered so far.

Once a WebSocket connection has been established, the next step is that the message has to be sent across the wire. In order for this to be done, the WebSocket application layer has to hand control of the message to the Transport Layer.

It is inside of the Transport Layer that source and destination port numbers are used to identify which process to send the data to.

The main difference between TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) lies in their approach to data transmission.

Ports help in directing and managing data by assigning unique identifiers to different applications or services running on a device. For example, in web apps, ports like 80 for HTTP and 443 for HTTPS are commonly used to handle incoming and outgoing data requests.

The cURL command provides the --local-port option that allows you to specify a single port number or a range of port numbers to use as the local (source) port for the connection.

This gives you the ability to control the source port used by cURL when making a request.

Before I finish talking about the Transport Layer, you need to know that the 'package' of data it creates is called a segment. I also want to show you the two most important headers that the transport layer creates, and briefly discuss the "window" property.

The Internet Layer is the third layer of the TCP/IP model, responsible for routing and addressing data packets between devices.

It provides logical routing functions, ensuring that data packets are delivered efficiently and reliably across the network.

IP is more generic, and the port identifies where exactly you want the data processed.

Although we are talking about networking, don't lose sight of the fact that data is ultimately always transferred over the wire as electrical inputs.

How is it possible that two tabs open can make a request to the same server at the same time? 

The Network Layer in the TCP/IP model is responsible for managing data at the data link level. It is also known as the Data Link Layer. This layer is responsible for framing. The Network Layer is responsible for ensuring that data is transmitted correctly over the physical medium, such as a network cable or wireless link.

A few interesting facts about the MAC address.

In summary, while an IP address identifies a device for routing purposes across networks, the MAC address is essential for local communication within a network segment. Both addresses work together to ensure reliable data transmission, with MAC addresses facilitating local delivery and IP addresses managing broader network routing.

"Tracert" is short for traceroute - it provides a map of how data on the internet travels from its source to its destination. Let' shave some fun and see how data travels in a simple HTTP GET request.

Some key takeaways from this section.

Mostly, developers open a WebSocket connection and send data across the wire. Developers don't typically understand the deeper process involved of that particular data packet from travelling over the wire. As you now know, the data has to be 'packaged' into a certain format, devices have to be recognized, and the data has to be converted to binary before it can reach its destination.

This is a summary lecture of what you've learnt thus far.

enjoy.

Sockets are not the same as WebSockets.

Sockets are a general-purpose network communication endpoint that uses TCP or UDP protocols to establish a "door" or "socket" or "connection" between two processes.

WebSockets, on the other hand, are a specific protocol built on top of TCP.

WebSockets and sockets are not the same.

Buckle up, my dear student!

We have just tackled a heavy-duty networking section, but don't worry, I've got your back. You see, there's a method to my madness. It's all about getting you to remember two key things:

#ONE: WebSockets are an application-layer protocol

#TWO: Even WebSockets need to physically get their data across the internet. And that's where lower-level transport protocols come into the picture.

By understanding the nitty-gritty of how data travels through the network layers, you'll be able to truly appreciate the power and elegance of WebSockets.

It's like learning the secret handshake to join the coolest club in town.

Now, get ready for some fun and punchy questions that'll have you feeling like a networking ninja in no time.

This is where the real party starts, my friend!

See you in the next section.

WebSockets are a game-changing technology that you need in your repertoire.

By understanding WebSockets, you'll not only become a more proficient developer but also gain the knowledge to explore other cutting-edge protocols like WebRTC and WebTransport.

To ensure you get the most out of this course, I encourage you to approach it with dedication and focus. Take your time watching the videos, carefully consider the concepts you're learning, and tackle the practice tests. These exercises will solidify your understanding and prepare you for real-world applications.

Stay focused, stay curious, and enjoy the process!

In the previous section you've learnt how the TCP connection is established. This is important because this first has to be done in order to switch over to WebSockets.

Before we start looking at how to send data across a WebSocket connection, you first need to understand how to open a WebSocket connection between two socket endpoints.

This video explains that when using HTTP/1.1, there is an HTTP handshake that needs to be performed in order for a valid WebSocket connection to be established.

Navigating the complexities of WebSocket support across different protocols and platforms can be daunting. Let me help you clear things up.

I want to show you how to analyze and see the handshake process in action, by making a simple WebSocket request in the browser's console.

WebSockets is an event-driven API that enables real-time, bidirectional communication between a client and a server over a single, persistent connection. Unlike traditional HTTP, which requires the client to initiate a new request for each interaction, WebSockets allows both the client and server to send messages at any time, without the need for polling or long-polling techniques. This event-driven nature of WebSockets makes it ideal for applications that require instant updates, such as chat applications, real-time collaboration tools, and multiplayer games, as it reduces latency and improves overall responsiveness compared to traditional HTTP-based communication.

End-to-end HTTP headers are those that must be forwarded unchanged by any intermediaries (such as proxies) between the client and server, and are relevant to the entire end-to-end request/response.

In contrast, hop-by-hop headers only apply to a single connection between two sockets, and can be modified or deleted by intermediaries. Examples include Connection, Keep-Alive, and yes, the Upgrade header.

In simple terms, a reverse proxy is an intermediary server that sits between clients (like web browsers) and the backend web servers that host a website or application. When a client sends a request, it doesn't go directly to the target web server, but instead goes through the reverse proxy first.

The Upgrade and Connection headers are used to facilitate the transition from one protocol to another within an existing HTTP connection.The Upgrade header is used by the client to indicate the protocols it would like to switch to, in order of preference. This allows the server to choose one of the proposed protocols and upgrade the connection accordingly.

The Connection header is mandatory when the Upgrade header is present. It signals that the client wants to upgrade the connection, rather than just adding a new header.

The Upgrade header tells the server that the client wants to switch to the WebSocket protocol, while Connection: upgrade indicates that the client wants to upgrade the connection to a different protocol

some extra info on the Upgrade and Connection header

The Sec-WebSocket-*** headers are a set of HTTP headers used in the WebSocket opening handshake to negotiate and confirm the WebSocket connection.

These headers are critical for establishing a secure WebSocket connection and agreeing on any optional extensions or subprotocols to be used for the application-level protocol layered on top of WebSocket. The headers are only used during the initial handshake, after which the WebSocket data frames are exchanged using a different format.

I want to explain why the headers being with "Sec".

The Sec-WebSocket-Key header is used in the WebSocket opening handshake to provide part of the information used by the server to prove that it received a valid WebSocket opening handshake.

It's main purpose according to the RFC is to prevent accidental WebSocket upgrade requests: The Sec-WebSocket-Key header is randomly generated by the client and is unlikely to be used by non-WebSocket clients. This helps prevent servers from accepting WebSocket upgrade requests from clients that are not expecting WebSocket protocol

You cannot start a WebSocket connection using AJAX. The WebSocket connection must be established separately, typically using the native WebSocket API in the browser or a WebSocket library on the client-side. AJAX and WebSockets serve different purposes and are used in different scenarios within web applications.

In order to become a grandmaster at WebSockets, I'm teaching you the inner workings.

In this bonus lecture I'm going to show you how to calculate your base64 encoded nonce value yourself - without the use of an API.

Final words on Sec-WebSocket-Key header.

Some interesting facts about the Sec-WebSocket-*** headers

Base64 is a way to convert binary data (like images, documents, or other files) into a text format that can be easily transmitted over the internet. It works by taking the binary data and converting it into a string of characters that are easy to send and receive.

Base64 is commonly used when you need to send binary data, like attachments in emails or data in web applications, over text-based protocols like HTTP.

This is a continuation of the previous lecture, with and example showing you how to encode the word "DOG" into base64

By working with the UTF-8 hexadecimal values, where each hex pair corresponds to an 8-bit byte, you can then encode any character / data using Base64 without any issue.

A quick fun example of how one would go about encoding the rocket emoji into base64

The Sec-WebSocket-Accept header is an important part of the WebSocket handshake process. When a client initiates a WebSocket connection, it sends a request with a Sec-WebSocket-Key header containing a randomly generated value. The server then takes this Sec-WebSocket-Key value, concatenates it with a specific GUID string, computes the SHA-1 hash of the result, and includes the Base64-encoded hash in the Sec-WebSocket-Accept header of the response. This allows the client to verify that the server understood the WebSocket protocol and is responding to the correct handshake request, preventing accidental WebSocket upgrades.

Why does the server have to hash the KEY + GUID number?

It's easy to think that WebSockets sit at the application level in the newtork stack, and for the most part I agree. But there are unique characteristics of WebSockets that make it more of a lower-level layer.

If the client supports multiple sub-protocols, it includes a list of sub-protocol names in the Sec-WebSocket-Protocol header of the initial request. The server then selects one or more of these sub-protocols and includes the chosen sub-protocol name(s) in the Sec-WebSocket-Protocol header of the response, indicating which sub-protocol(s) will be used for the WebSocket communication.

Two last headers in the WebSocket handshake process that we need to cover are the Sec-WebSocket-Version and Sec-WebSocket-Extensions headers.

The Sec-WebSocket-Version header specifies the WebSocket protocol version that the client or server supports, ensuring compatibility between the communicating parties. This header is essential for establishing a successful WebSocket connection, as both the client and server must agree on the protocol version to be used.

The Sec-WebSocket-Extensions header, on the other hand, is used to negotiate any additional protocol extensions that the client and server may want to use during the WebSocket session. These extensions can provide additional functionality or optimizations to the WebSocket communication, such as data compression.

You'll often see Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits and so I explain what Deflate means, as well as the client_max_window_bits.

Enjoy :)

This section has been about the WebSocket headers and upgrade process. But if you ask 99% of developers about this, they will not know it.

This is because most developers will use the WebSocket API and frameworks, which handle this process for you.

You've come a long way in this section.

You are WELL equipped at knowing how the inner workings of a WebSocket connection are established.

Well done.

Well done for getting to this section.

In this section we will take a closer look at the WebSocket API and also touch on some server side code for WebSockets.

I can't wait.

The WebSocket API is built directly into the browser.

Up until this point you know that a WebSocket is a protocol that allows two devices / hosts to send data to each other.

In this section I want to focus on how a connection is established.

In this section I want to introduce you to the WebSocket API.

To learn better, I think its best we build a simple project together.

In the next few lectures, I'll be creating the HTML and CSS files necessary for the rest of this section. Of course, this means that the next few lectures are not WebSocket related. You may therefore want to skip the next 2 lectures and just download the HTML and CSS files that I provide.

Building the HTML of our WebSocket Demo project.

Building the CSS together to style up our HTML

The actual HTML and CSS is not crucial to understanding WebSockets, so you can skip the building of the HTML and CSS if you want to, in which case you can just download the HTML and CSS attached to this lecture.

In this lecture I will be turning our attention to building a WebSocket server. In fact this will take 3 lectures.

In this lecture I'll show you how to create an HTTP server in Node.js.

I'll be showing you how to use Node's inbuilt NPM to install a module, add it to the package.json file, and also add a node_modules folder. I can't wait.

It is now turn to mount our WebSocket server instance onto our HTTP server (remember that WebSockets are HTTP compatible). After this lecture, you will have a fully functional WebSocket echo server.

Both the client and server need to "speak" the WebSocket protocol for the real-time communication to work. The client initiates the connection, but the server has to understand and respond using the WebSocket protocol as well.