Commit 0407a60c authored by Caleb C. Sander's avatar Caleb C. Sander
Browse files

WebSockets notes and chat specs

parent 0e7e75ff
No related merge requests found
Showing with 657 additions and 1 deletion
+657 -1
......@@ -34,7 +34,7 @@ You can contact me via:
| 3 | [Promises](notes/promises/promises.md) | [`make`](specs/make/make.md) | 2020-04-24 |
| 4-5 | [Streams](notes/streams/streams.md) | [`grep`](specs/grep/grep.md) | 2020-05-08 |
| 6 | [HTTP](notes/http/http.md) | [Wiki Game](specs/wiki-game/wiki-game.md) | 2020-05-15 |
| 7 | WebSockets | Chat server **OR** Chat client | 2020-05-22 |
| 7 | [WebSockets](notes/websockets/websockets.md) | [Chat client](specs/chat/client.md) **OR** [Chat server](specs/chat/server.md) | 2020-05-22 |
| 8-9 | `async`-`await` | MiniVC | 2020-06-05 |
## JavaScript reference
......
<html>
<head>
<style>
#grid-canvas {
border-collapse: collapse;
}
#grid-canvas td {
width: 20px;
height: 20px;
padding: 0;
border: 1px solid black;
}
</style>
<script>
const GRID_WIDTH = 20
const GRID_HEIGHT = 20
// The <input type='color'> used to select the drawing color
let colorInput
// The <table> used to display the pixel art
let grid
// The WebSocket
let socket
// Rectangles are drawn by clicking and dragging.
// These store the row and column of the cell that was clicked.
let startRow
let startCol
// Sends a `set-color` message with the current color.
// Called when the page loads and whenever `colorInput` changes.
function sendColor() {
const message = {type: 'set-color', color: colorInput.value}
socket.send(JSON.stringify(message))
}
// Sends a `draw` message to create a rectangle
// from `startRow` and `startCol` to the given row and column
function sendRectangle(row, col) {
const message = {
type: 'draw',
// Using Math.min() and Math.abs()
// since the rectangle could have been drawn in any direction
row: Math.min(startRow, row),
col: Math.min(startCol, col),
rows: Math.abs(startRow - row) + 1,
cols: Math.abs(startCol - col) + 1
}
socket.send(JSON.stringify(message))
}
// Adds the table cell elements to `grid` and attaches
// `mousedown` and `mouseup` event handlers to them
function makeGrid() {
// Build a grid of <tr> and <td> elements, like in Minesweeper
for (let row = 0; row < GRID_HEIGHT; row++) {
const gridRow = document.createElement('tr')
for (let col = 0; col < GRID_WIDTH; col++) {
const cell = document.createElement('td')
// Handles the click of a click-and-drag
cell.onmousedown = () => {
startRow = row
startCol = col
}
// Handles the release of a click-and-drag
cell.onmouseup = () => sendRectangle(row, col)
gridRow.append(cell)
}
grid.append(gridRow)
}
}
// Handles a `draw` message from the server.
// `{color, ..., cols}` unpacks the message's fields.
function draw({color, row, col, rows, cols}) {
// For each cell in the rectangle, update its background color
for (let i = 0; i < rows; i++) {
const gridRow = grid.children[row + i]
for (let j = 0; j < cols; j++) {
const gridCell = gridRow.children[col + j]
gridCell.style.background = color
}
}
}
// We need to run code that accesses HTML elements after
// the rest of the HTML is parsed (signaled by the `load` event)
window.onload = () => {
// Retrieve HTML elements
colorInput = document.getElementById('color')
grid = document.getElementById('grid-canvas')
colorInput.onchange = sendColor
makeGrid()
// Create a WebSocket connection
socket = new WebSocket('ws://localhost')
// As soon as socket opens, send the initial color (black)
socket.onopen = sendColor
// Handles `draw` messages from the server.
// `event.data` is the JSON string received.
socket.onmessage = event => draw(JSON.parse(event.data))
}
</script>
</head>
<body>
<input type='color' id='color' />
<table id='grid-canvas'></table>
</body>
</html>
const WebSocket = require('ws')
// The currently connected WebSockets
const sockets = new Set()
// The array of `draw` messages sent out. When a new client connects,
// we replay these messages to initialize the pixel grid.
const drawJSONs = []
// Create a WebSocket server on port 80 (the default)
new WebSocket.Server({port: 80})
// The 'connection' event handler is called for new connections
.on('connection', socket => {
// Send all previously drawn rectangles to the client
for (const drawJSON of drawJSONs) socket.send(drawJSON)
// Register the WebSocket to receive updates to the drawing
sockets.add(socket)
// Each WebSocket has a `color` variable storing its current paint color
let color
socket
// The 'message' event handler receives each message from the client
.on('message', json => {
// `json` stores the (JSON string) contents of the message,
// which we turn back into an object
const message = JSON.parse(json)
if (message.type === 'set-color') {
// Handle `set-color` by updating the client's color
color = message.color
}
else if (message.type === 'draw') {
// Handle `draw` by constructing a `draw` message to send.
// We add the `color` field to the message from the client.
const drawMessage = {...message, color}
const drawJSON = JSON.stringify(drawMessage)
// Send the message to all connected WebSockets
for (const socket of sockets) socket.send(drawJSON)
// And add it to the history of `draw` messages
drawJSONs.push(drawJSON)
}
})
// The 'close' event handler is called when the client disconnects
.on('close', () => {
// Unregister the socket so it stops receiving updates
sockets.delete(socket)
})
})
# WebSockets
WebSocket is a modern protocol used on the Web in tandem with HTTP.
HTTP and WebSocket are both built on top of the same lower-level protocol, [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol).
However, WebSocket is a thinner abstraction over TCP and exposes more of TCP's functionality.
This set of notes discusses reasons to use WebSocket over HTTP and demonstrates how to use WebSockets in an application.
## Limitations of HTTP
The main drawback of HTTP in Web applications is its request-response nature.
As discussed [last time](../http/http.md), all HTTP connections to a server are initiated by the client.
Often this makes sense: a browser knows when its user wants to load a page or ask the server to do something.
But sometimes, a server has new data to send to the client.
For example, an online game must tell the browser when the game state changes so it can be displayed.
An email/messaging application wants to notify the browser of a new message.
With HTTP, this requires the browser to periodically request updates, which is both slow and wasteful.
For example, if the browser checks for updates each minute, it can take a whole minute to learn about new data, and most requests may return no new data.
WebSockets solve this by leveraging the *bidirectionality* of TCP.
A WebSocket connection (or just "WebSocket") allows the client and server to send messages to each other at any time.
There are no "requests" or "responses"; the server can send data without the client asking for it, and the server doesn't have to respond to data from the client.
WebSockets are also *persistent*: many messages can be sent across the same WebSocket.
(HTTP uses a new connection for every request.)
This means a server can remember which WebSocket belongs to which user.
WebSockets also avoid the HTTP headers required for every HTTP request, which reduces the amount of data sent and the time until it arrives (the "latency").
## Example: collaborative pixel art
We are going to build a WebSocket application that allows multiple users to paint a grid of pixels.
Each client can change its paint color and draw rectangles with its current color.
The first step in building a WebSocket application is designing the API used to communicate with the server.
(This is the same as with HTTP; see the [HTTP notes on JSON APIs](../http/http.md#apis) for a refresher.)
Our API has two types of JSON messages to the server:
- `set-color`: changes the client's painting color.
The JSON has the form
```json
{"type": "set-color", "color": "#rrggbb"}
```
- `draw`: fills a rectangle with the current color.
The client specifies the top-left row and column and the number of rows and columns in the rectangle.
The JSON has the form
```json
{
"type": "draw",
"row": number,
"col": number,
"rows": number,
"cols": number
}
```
And one message to the client:
- `draw`: tells the client that a rectangle was drawn.
The JSON has the form
```json
{
"type": "draw",
"color": "#rrggbb",
"row": number,
"col": number,
"rows": number,
"cols": number
}
```
Next, we explain how to build the client and server around this WebSocket API.
It's a lot of code to read, so I recommend focusing on the part (either the client or server) you plan to implement on this week's project.
### The client
The client opens the WebSocket to the server using the builtin `WebSocket` constructor.
The server will run on the same computer, so the client connects to `localhost`.
(WebSocket URLs use `ws://`, or `wss://` for secure WebSockets.)
```js
socket = new WebSocket('ws://localhost')
```
When the socket connects, the client sends a `set-color` message with the initial drawing color.
We also want to send `set-color` whenever a new color is selected.
The HTML uses an `<input type='color'>` which conveniently gives us a color in the `#rrggbb` format.
```js
function sendColor() {
// Build the message we want to send, then convert it to a JSON string
const message = {type: 'set-color', color: colorInput.value}
socket.send(JSON.stringify(message))
}
// When the socket opens, call `sendColor()`
socket.onopen = sendColor
// When `colorInput`'s value changes, call `sendColor()`
colorInput.onchange = sendColor
```
Rectangles can be drawn by clicking and dragging, and this should send a `draw` message.
We use the `mousedown` event to capture clicking on a table cell and `mouseup` to capture releasing on a different table cell.
```js
// During a drag, these variables store
// the row and column of the cell that was clicked
let startRow
let startCol
for (let row = 0; row < GRID_HEIGHT; row++) {
for (let col = 0; col < GRID_WIDTH; col++) {
const cell = document.createElement('td')
cell.onmousedown = () => {
startRow = row
startCol = col
}
cell.onmouseup = () => sendRectangle(row, col)
}
}
function sendRectangle(row, col) {
const message = {
type: 'draw',
// Using Math.min() and Math.abs()
// since the rectangle could have been drawn in any direction
row: Math.min(startRow, row),
col: Math.min(startCol, col),
rows: Math.abs(startRow - row) + 1,
cols: Math.abs(startCol - col) + 1
}
socket.send(JSON.stringify(message))
}
```
And finally, we need to handle messages *from* the server.
This requires setting the `message` event handler on the socket.
```js
// Handles `draw` messages from the server.
// `event.data` is the JSON string received.
socket.onmessage = event => draw(JSON.parse(event.data))
// The curly braces around the arguments are shorthand for
// extracting these fields from the message object
function draw({color, row, col, rows, cols}) {
// For each cell in the rectangle, update its background color
for (let i = 0; i < rows; i++) {
const gridRow = grid.children[row + i]
for (let j = 0; j < cols; j++) {
const gridCell = gridRow.children[col + j]
gridCell.style.background = color
}
}
}
```
### The server
In Node.js, you can use the npm package [`ws`](https://www.npmjs.com/package/ws) to create WebSocket servers.
You can install this package by running `npm install ws` in a terminal.
```js
const WebSocket = require('ws')
// Make a WebSocket server and listen on port 80 (the default)
new WebSocket.Server({port: 80})
// The 'connection' event handler is called for new connections
.on('connection', socket => {
// ... send and receive messages over the WebSocket
})
```
This `socket` is very similar to a `WebSocket` object in the browser.
The server needs to store the color selected by each `socket`, and update it whenever `set-color` is received:
```js
let color
socket.on('message', json => {
// `json` stores the (JSON string) contents of the message,
// which we turn back into an object
const message = JSON.parse(json)
if (message.type === 'set-color') {
color = message.color
}
})
```
When a `draw` message is received, we need to send `draw` messages to all connected sockets.
This requires us to store the set of connected WebSockets.
```js
const sockets = new Set()
new WebSocket.Server({port: 80})
.on('connection', socket => {
// Register the WebSocket to receive updates to the drawing
sockets.add(socket)
// When the client disconnects, stop sending it updates
socket.on('close', () => {
sockets.delete(socket)
})
// ...
})
```
Now we can handle `draw` messages, including sending new clients the previous `draw` messages:
```js
// The array of `draw` messages previously sent
const drawJSONs = []
new WebSocket.Server({port: 80})
.on('connection', socket => {
let color
socket.on('message', json => {
const message = JSON.parse(json)
if (message.type === 'draw') {
// Handle `draw` by constructing a `draw` message to send.
// We add the `color` field to the message from the client.
const drawMessage = {...message, color}
const drawJSON = JSON.stringify(drawMessage)
// Send the message to all connected WebSockets
for (const socket of sockets) socket.send(drawJSON)
// And add it to the history of `draw` messages
drawJSONs.push(drawJSON)
}
})
// When a client connects, send it all previously drawn rectangles
for (const drawJSON of drawJSONs) socket.send(drawJSON)
// ...
})
```
The full code for the client is in [`pixel-art.html`](pixel-art.html), and the server code is in [`pixel-art.js`](pixel-art.js).
I recommend playing around with it (try opening it in multiple tabs) and `console.log()`ing the messages received by the client/server to see what's being sent over the WebSockets!
## *Aside*: WebSocket requests *are* HTTP requests
Astute readers may have noticed that our WebSocket server listens on port 80, the same as the default port for HTTP.
That is because WebSocket is an extension of the HTTP protocol!
When a browser opens a WebSocket to `ws://some-url.com/some-path`, it actually makes a HTTP `GET` request to path `/some-path` on `some-url.com`.
The browser sends the [HTTP header](../http/http.md#aside-headers) `Upgrade: websocket` to request a WebSocket connection.
The server sends back a [status code](../http/http.md#aside-status-codes) of `101 Switching Protocols` and the headers `Connection: Upgrade` and `Upgrade: websocket` to accept the connection.
Then, unlike a normal HTTP request, the server leaves the connection open and both the client and server start communicating using the WebSocket protocol.
This allows the same server to serve both HTTP and WebSocket requests, which is very useful in practice.
Most websites using WebSockets rely primarily on HTTP for clients to request webpages and make API calls, but use WebSockets when they need to push data to clients.
# Chat API
The chat server and client communicate over a WebSocket connection.
When a user logs in to the chat application, the client opens a WebSocket to the server that lasts until the page is closed.
Messages can be sent in either direction over the WebSocket at any time.
Whether you choose to implement the client or the server, you will need to implement both message directions.
All messages are sent as JSON strings.
The field `type` indicates which type of message it is.
For simplicity, the API doesn't have any way to signal that a message was invalid (e.g. a `message` sent to a user that doesn't exist).
## Concepts
**Users**:
To log into the chat application, you provide your username.
(For simplicity, there is no password mechanism.)
The server stores all the usernames that have ever logged on.
You can chat with any user even if they aren't currently connected; they just won't see your messages until they connect.
**Chat messages**:
A chat message is a piece of text sent between two users, from the "sender" to the "recipient".
Each message has a string ID so read receipts can refer to it.
These IDs are picked by senders and should be approximately unique (a good choice would be the combination of the sender's username and the time it was sent—see [`Date.now()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/now)).
**Read receipts**:
When a chat message is read by the recipient, they respond with a "read receipt".
(A message is counted as read if the recipient selects the sender on the chat page after receiving the message.)
Before a read receipt has been received, the sender shows the message with a grey background; afterwards, it has a blue background.
## WebSocket messages from client to server
- `login`: this message tells the server the client's username.
This must be the first message sent on the WebSocket.
It is a JSON string with this format:
```json
{"type": "login", "user": string}
```
- `message`: this message asks the server to send a chat message to another user.
The client must specify the message ID, the recipient's username, and the string to send.
It is a JSON string with this format:
```json
{
"type": "message",
"message": {
"id": string,
"user": string,
"content": string
}
}
```
- `read`: this message asks the server to send a read receipt for a previous message.
The client must specify the message ID.
It is a JSON string with this format:
```json
{"type": "read", "id": string}
```
## WebSocket messages from server to client
- `users`: this message tells the client the usernames of all users who have ever logged on (including this client).
The usernames are in no particular order.
The server sends this message to each client as soon as the client sends a `user` message.
It is a JSON string with this format:
```json
{"type": "users", "users": string[]}
```
- `message`: this message tells the client that they have received a message.
The server includes the message ID, the sender's username, and the string that was sent.
If the recipient is logged in, this message is delivered as soon as the server receives `message`; otherwise, it is sent whenever the recipient logs in.
It is a JSON string with this format:
```json
{
"type": "message",
"message": {
"id": string,
"user": string,
"content": string
}
}
```
- `read`: this message is a read receipt and contains the message ID that was read.
If the sender is not logged in when the recipient responds with a read recipient, the sender doesn't receive the read receipt.
It is a JSON string with this format:
```json
{"type": "read", "id": string}
```
- `new-user`: this message is sent to all connected clients when a new user logs on for the first time.
The message contains the new user's username.
It is a JSON string with this format:
```json
{"type": "new-user", "user": string}
```
File added
specs/chat/chrome-websockets.png

1.04 MB

# Recommended JavaScript reading
Communicating with the server requires constructing and destructing JSON objects.
JavaScript has shorthand notation that you may find useful for packing variables into objects, and for "destructuring" objects into variables.
There are examples under "Objects" in the [builtin types](../../notes/js/js.md#types) notes and under "destructuring" in the [variables](../../notes/js/js.md#variables) section.
# Chat client
This week, you will implement a chat application built on WebSockets.
You should write **either** the [Node.js server](server.md) **or** the browser-side chat client.
You're welcome to write both (or find another student who implemented the other half) and put them together, but this is not required.
This [video](chat-client.mp4) shows 3 clients using the chat app.
The server waits half a second before processing each message to simulate a slow network.
## WebSockets
A chat app can benefit from both the bidirectionality and persistence of WebSockets:
- Since the chat server can send messages to the client without the client having to request them, new messages can be delivered as soon as possible
- Since the connections to the server stay open, there is less overhead in sending a message, which reduces latency and bandwidth use
For these reasons, the chat app communicates entirely over WebSockets.
## API
The JSON API that is used to communicate over the WebSockets is documented in the [Chat API writeup](api.md).
## Codebase
Like on Minesweeper, an `index.html` with the HTML layout of the chat client is provided.
It uses [Bootstrap](https://getbootstrap.com), a popular style framework for webpages.
You are welcome to change the page structure and style as long as it supports all the required functionality.
There is a small bit of additional CSS in `style.css`.
One change is that `client.js` runs *before* the elements are defined, which is typical for webpages.
This means, for example, that `document.getElementById('login')` won't work until the rest of the page is loaded.
Any code that depends on page elements should be put inside the `window.onload` callback, e.g.:
```js
window.onload = () => {
const loginButton = document.getElementById('login')
// ...
}
```
## Requirements
When the page is opened, the user should be asked to enter their username.
Once the user clicks the login button, the client should open a WebSocket to the server and send `login`.
The login screen should also be hidden and the users/messages screen shown instead.
(You can add/remove the class `hidden` to hide/show elements.)
Users should be shown in sorted order on the left side of the screen.
(Arrays have a `.sort()` method that sorts them.)
The user list is a [Boostrap "list group"](https://getbootstrap.com/docs/4.4/components/list-group); you can add an entry to it by appending a `<li class='list-group-item'>username</li>`.
(An element's text can be set using `element.innerText = username`.)
When `new-user` is received, the new username should be added to the user list.
Clicking on a user should select them in the chat screen, showing all the messages that have been sent to and received from them.
Only messages sent or received since the chat app was opened need to be shown.
When the send button is clicked, the message entered in the input should be sent to the selected user using a `message` API message.
The sent message should be shown in the chat.
The input should be cleared so a new message can be typed (you can set an input's value using `element.value = ...`).
The list of messages is also a list group, so you can display messages by adding list group items to it.
Received messages should appear on the left side and sent messages should appear on the right (use the Bootstrap class `text-right` to align text to the right.)
When a message is sent, it should initially show a grey background (use the class `list-group-item-secondary`).
Once a read receipt is received, the message should turn blue (use the class `active`).
To be able to refer to a message element, you can set its `id` to the message ID with `element.id = messageID` and use `document.getElementById(messageID)` to select it.
When `message` is received, if the sender is currently selected in the chat window, the received message should be shown and a `read` for the message should be sent.
If the sender is not selected, a count of the number of unread messages should be shown next to the sender's username.
You can use a [Bootstrap badge](https://getbootstrap.com/docs/4.4/components/badge) for this: `<span class='badge badge-danger'>unread_count</span>`.
When the sender is selected, `read` should be sent for all unread messages and the unread count should disappear.
When `read` is received, the corresponding message should turn blue.
Note that a `read` can be received for a message that was sent in a previous chat session; in this case, it can be ignored.
## Testing
My implementation of the chat server is running on my website so your client can connect to it.
It uses secure WebSockets; its URL is `wss://calebsander.com:8443`.
It's not hard to make it crash by sending invalid messages.
It will restart automatically if it crashes, but please don't flood it with bad messages; it is a shared resource for the entire class.
The server also restarts at the start of every hour so you can try it with fresh data.
Most browsers have a debugging window that shows what messages are being sent across the WebSocket.
For example, in Google Chrome:
![Google Chrome WebSockets](chrome-websockets.png)
## Going further
The chat client you've written allows people to use the chat app, but the WebSocket API can also be leveraged for chat bots.
You can write a chat bot that logs in with a fixed username and responds to any messages it receives.
For example, users could send the bot city names and it could use a weather API to send the current weather there.
# Recommended JavaScript reading
Communicating with the server requires constructing and destructing JSON objects.
JavaScript has shorthand notation that you may find useful for packing variables into objects, and for "destructuring" objects into variables.
There are examples under "Objects" in the [builtin types](../../notes/js/js.md#types) notes and under "destructuring" in the [variables](../../notes/js/js.md#variables) section.
You will have to install the [`ws` package](https://www.npmjs.com/package/ws) from npm this week to use WebSockets in Node.js.
The [npm section](../../notes/js/js.md#installing-npm-packages) of `js.md` has information on how to install npm packages.
You can also use `npm install --save` to automatically add the dependency to the `package.json` file.
# Chat server
This week, you will implement a chat application built on WebSockets.
You should write **either** the [browser-side chat client](client.md) **or** the Node.js server.
You're welcome to write both (or find another student who implemented the other half) and put them together, but this is not required.
There are standalone tests provided for the server's WebSocket interface, so you don't need a browser client to test it.
This [video](chat-client.mp4) shows 3 clients using the chat app.
The server waits half a second before processing each message to simulate a slow network.
## WebSockets
A chat app can benefit from both the bidirectionality and persistence of WebSockets:
- Since the chat server can send messages to the client without the client having to request them, new messages can be delivered as soon as possible
- Since the connections to the server stay open, there is less overhead in sending a message, which reduces latency and bandwidth use
For these reasons, the chat app communicates entirely over WebSockets.
The server will be run using the command `node server.js port`.
It should host a WebSocket server on the specified port and handle all incoming connections.
## API
The JSON API that is used to communicate over the WebSockets is documented in the [Chat API writeup](api.md).
For simplicity, you can assume that the messages sent from the clients to the server conform to the API.
A real server would need to handle any malformed request without crashing, corrupting data, etc.
## Requirements
The requirements of the server are mostly explained in the API document, but here is a complete list:
- When a client sends `login`, the server should send `users`, followed by `message` for each chat message sent to that user which has not yet been read.
If this user connected for the first time, the server should send `new-user` to all connected clients.
- When a client sends `message`, the server should send a corresponding `message` to the recipient if the recipient is currently connected.
Otherwise, the chat message should be delivered when the recipient next connects.
- When a client sends `read`, the server should send a corresponding `read` to the sender of the message if they are connected.
If the sender is no longer connected, no `read` needs to be sent.
- `users` should include all users which have ever logged in, not just the currently connected users
- All data on the server should be stored in memory (i.e. as JavaScript values).
This means that if the server crashes or needs to be restarted, all users and messages will be lost.
Real servers prevent this by storing data in files or databases; you'll do this on the MiniVC project.
- The server should only store the messages that are still unread.
As soon as a read receipt is received for the message, the server should discard the message to save memory.
- If a client sends `login` with the same username as an existing connection, the server should assume the old connection became disconnected and send all future messages to the new connection
## `ws`
Node.js doesn't have a builtin library for WebSockets.
Luckily, there's a package for that.
Node.js has one of the best package managers, [npm](https://www.npmjs.com)!
Most modern JavaScript projects (including browser-side ones) use some of the over 1 million open-source packages on npm.
You should install the [`ws` package](https://www.npmjs.com/package/ws) so that your server can `require('ws')`.
Make sure to include `ws` in the `dependencies` section of your `package.json` so that others can install the necessary dependencies for your project by running `npm install`.
This only scratches the surface of what npm can do; I think npm's super cool, so please ask me if you want to know more about it!
## Going further
This chat app supports the core functionality of a real chat app like Facebook Messenger.
One feature you could add is chat groups:
- Users should be able to create new groups, add other users to groups they belong to, and leave groups
- Clients should be able to send messages to a group or an individual user.
If the client sends to a group, all users in the group should receive the message and send read receipts when it's read.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment