01 Two services, a single link
The game itself is a Python application (FastAPI and Django): it's the one that knows the players, the friends, the games and the teams, in PostgreSQL. Real time is entrusted to a second service, written in Rust with axum and tokio: it keeps the WebSocket connections open, knows who's looking at which grid and distributes the messages. The two never call each other. Their only link is Redis.
This split comes down to one rule: Python stays the only source of truth. The Rust server reads no database, never calls the API and can't change anything about a game. It checks who is at the end of each connection and forwards messages, nothing more. The two services only share one secret, Django's SECRET_KEY, from which each derives the same HMAC keys.
This real time is used for three things:
- the row above the grid, which shows which friends have the same grid open right now, with their scores and game endings as they happen;
- quick reactions between friends (a clap, a flame, a rocket…);
- live team play: several friends play the same grid, each on their own phone, and see the others' moves and the cell each one has opened.
02 Signals, not data
Before the code, the principle that makes everything else simple: a real-time message is never the truth. It says "something has changed", and the page can always find the exact state again by reloading the grid from the API.
| Information | Where it lives | Survives a restart? |
|---|---|---|
| Games, answers, scores, teams, invitations | PostgreSQL | yes, it's the truth |
| Who has the grid open right now | a Redis sorted set, entries valid for 60 s | no, and that's fine |
| The cell a teammate has opened | nowhere: relayed then forgotten | no |
| The messages ("Alice scored") | nowhere: relayed then forgotten | no |
A lost message therefore costs a small delay, never a wrong grid. That's also why Redis runs with no persistence and no volume (--save "" --appendonly no): nothing that goes through it is worth keeping.
03 The signed ticket
To know who is at the end of a connection without querying the database, the Rust server receives a ticket. When your page loads the grid, the API attaches a small signed document to it:
{
"v": 2,
"grid": 42,
"player": "<your presence key>",
"friends": [["<key>", "ABCD2345EFGH"]],
"team": {"key": "<team key>", "members": [["<key>", "K7MP3QRS9TUV"]]},
"exp": 1762297200
}
playeris not your player identifier. That identifier is your only proof of identity on the API: whoever knows it plays in your place. The ticket carries a presence key instead, an HMAC-SHA256 of your identifier truncated to 32 characters: the same for you every time, impossible to trace back without the secret, useless on the API.friendslists the players you follow, each as a[presence key, friend code]pair. The server uses the key to recognize connections, and only sends the browser the friend code, the one you already exchange to add each other (12 characters taken from23456789ABCDEFGHJKMNPQRSTUVWXYZ, with no0,O,1,IorL).teamonly appears if you're playing in a team on this grid. Two connections are teammates when their tickets carry the same team key.exp: 12 hours after signing, or the closing of the weekly grid if that comes first.
The format is that of Django's salted_hmac: the compact JSON encoded in base64url, a dot, then the signature. Each use has its own salt, hence its own key: a value forged for one use can't pass for another.
ticket = payload_b64 "." signature
signature = base64url(HMAC-SHA256(key, payload_b64))
key = SHA-256("stationdoku.live.ticket" + SECRET_KEY)
The page sends the ticket in the first WebSocket message, never in the URL, where it would end up in a proxy's logs. On the Rust side, the signature is checked before reading a single byte of the content, and in constant time:
// ticket.rs: the signature first, the content after.
let (payload_b64, signature) = token.split_once('.').ok_or(TicketError::Shape)?;
let signature = URL_SAFE_NO_PAD
.decode(signature)
.map_err(|_| TicketError::Base64)?;
let mut mac = mac(salt, secret);
mac.update(payload_b64.as_bytes());
// Constant-time comparison: how long it takes reveals nothing about the right signature.
mac.verify_slice(&signature)
.map_err(|_| TicketError::Signature)?;
An ordinary comparison stops at the first wrong byte: by timing a lot of requests, an attacker could guess a signature byte by byte. verify_slice always takes the same time. Once the signature is validated, the content is still checked field by field (keys of 32 hexadecimal characters, codes of 12 characters from the alphabet), even though Python signed it: a bug on one side must not become weird behavior on the other.
Whoever steals a ticket can, until it expires, appear present on that grid under your name and see which of your friends are there. Nothing else: no identifier, no other grid, no action on a game.
Only players who have a reason to be live receive a ticket: those who follow at least one friend, who are followed, or who play in a team. Most players have no friends on the site: no ticket for them, hence no WebSocket and no Redis traffic. The server's load follows the social side of the game, not the number of visitors.
04 The life of a connection
A WebSocket isn't covered by CORS: without a precaution, any site could open one to /live/ws from a visitor's browser. The first check is therefore on the Origin header, compared to the list of allowed origins. Then:
- the server waits at most 10 seconds for a
{"type":"hello","ticket":…}message of at most 16 KiB; wrong signature:bad_ticketand close; expired ticket:expiredand close; - it registers the connection in the process's registry;
- only then, it announces its presence in Redis;
- it sends the list of friends who are present, and the list of teammates if the ticket has a team;
- it enters its main loop.
The order of steps 2 and 3 matters. Announcing first risks a friend arriving at the instant the connection isn't registered yet: that wake-up would find nobody, and the list would go out stale.
Each connection is a tokio task that waits for several things at once in a single select! loop:
// ws.rs: a connection's loop (simplified).
let recompute = tokio::select! {
incoming = socket.recv() => /* a message from the browser: focus, reaction */,
Some(message) = queue.recv() => { send_to_client(socket, connection, &message).await?; false }
() = connection.notified() => true, // a friend arrived or left
_ = ticker.tick() => { // every 20 s
send_message(socket, Message::Ping(Bytes::new())).await?;
state.presence.touch(grid, player, connection_id).await?;
true
}
() = &mut expiry => { /* {"type":"expired"} then close */ }
() = shutdown_requested(&mut shutdown) => return Ok(close_code::AWAY),
};
if recompute {
// rereads the presence from Redis and only sends the list if it changed
}
The notified() wake-up relies on a tokio::sync::Notify, which keeps at most one token when nobody is waiting. Ten friend arrivals while the task is sending a message melt into a single wake-up: the task doesn't replay the events, it rereads the full state from Redis, and one reread is enough for all of them.
On close, the connection removes its presence and, if it was showing a cell to its team, publishes a "no more cell" so that no highlight stays stuck. This cleanup is an asynchronous function called explicitly: in Rust, a destructor (Drop) can't wait for a network operation.
05 Redis as a bus
Each grid has six Redis channels. Each Rust process subscribes to all the grids at once with PSUBSCRIBE stationdoku:live:grid:*, recognizes the channel by its name and files its connections by grid. When a message arrives, it only hands it to the connections concerned.
| Channel | What it carries | Published by |
|---|---|---|
grid:{g} |
the presence of a given player may have changed | the Rust server |
grid:{g}:scores |
a right answer | Python |
grid:{g}:finished |
a finished game (won, lost, given up) | Python |
grid:{g}:notices |
a team move, an invitation, an availability | Python |
grid:{g}:reactions |
a reaction from one friend to another | the Rust server |
grid:{g}:focus |
the cell a teammate has opened | the Rust server |
Sorting is done with what each connection knows from its own ticket:
| Message | Delivered to connections that… | Never to |
|---|---|---|
| score, end of K's game | follow K | K themselves |
| reaction from A to B | belong to B and follow A | A |
| cell opened by A | have the same team key and list A among their teammates | A |
| notice for a team | have that team key | nobody is excluded |
The recipient names the sender by looking up their key in its own ticket: the server needs no global table of players, and a recipient never learns a code it didn't already know. A reaction additionally requires that the recipient follow the sender back: a stranger who follows you can't send you one.
Team notices have a content (data, 4,096 bytes at most) that the Rust server never reads: it checks the envelope (version, keys, recipients) and forwards the rest as is. Adding a new type of notice therefore requires no change to the Rust code. The messages Python publishes are still validated on receipt: a malformed message is delivered to nobody and leaves a warning in the logs, a sign that the two sides no longer agree on the format.
Several Rust processes can run side by side, each with its own connections: Redis is the only link between them too. If the subscription drops, the process reconnects with a wait that doubles on each failure, from 1 to 30 seconds, then wakes all its connections so they reread the state.
A slow client slows nobody down
Each connection has a queue of 16 messages, and delivery never waits:
// hub.rs: deliver without ever waiting for a slow client.
fn push(&self, message: ServerMessage) -> bool {
match self.queue.try_send(message) {
Ok(()) => true,
Err(TrySendError::Full(_)) => {
self.dropped_count.fetch_add(1, Ordering::Relaxed);
// counted, reported at most once a minute, then forgotten
false
}
Err(TrySendError::Closed(_)) => false,
}
}
If a phone on a bad connection doesn't read fast enough, the extra messages are dropped for it alone. Since they're signals, the next reload of the grid catches up on what's missing.
06 Publishing from Python
On the Python side, publishing happens after the transaction that makes the message true is committed, on the channel of the grid concerned, in compact JSON. The messages of a single request go out together, in a single Redis pipeline:
# game/live_bus.py
async def publish(*messages: Message | None) -> bool:
pending = [message for message in messages if message is not None]
if not pending:
return True
if not is_enabled():
return False
try:
async with _redis().pipeline(transaction=False) as pipeline:
for message in pending:
pipeline.publish(message.channel, message.payload)
await pipeline.execute()
except Exception:
logger.warning("Publishing live messages failed", exc_info=True)
return False
return True
The Redis client has a 200 ms timeout, and the function never raises an error: a move never depends on Redis. If Redis or the Rust server goes down, the game carries on exactly as before, only the real-time extras disappear.
In a team, each move locks the team's row in the database and rereads the state under that lock. Two teammates who play the same cell at the same instant go through one after the other: the second gets "cell already taken", with no penalty. The database is only written once per move, whatever the team's size.
A move's notice carries what it changed, a delta: the score, the error and hint counters, the number of open answers in each cell, the cell found. Bob's page applies it without asking the API anything, with the animation that goes with it. But it first checks that the counters follow on: exactly one more cell found, error or hint than what it displays. Otherwise, a message was lost or arrived out of order, and it reloads the grid. A delta too big for the notice (over 4,096 bytes) is simply left out: the page reloads then too.
07 Presence, a lease that expires
Knowing who has the grid open relies on one Redis sorted set per grid, stationdoku:live:grid:{g}:presence.
- One entry per connection, not per player (
key:connection): with two tabs open, you stay present as long as either one is. - The score is the expiry time, in milliseconds, one minute later: it's a lease, which the connection renews every 20 seconds. A process that crashes stops renewing, and its entries die on their own.
- The key expires 2 minutes after the last write: the presence of an abandoned grid disappears entirely.
Each write changes the set and publishes an event: so that no crash and no other process can slip in between the two, these are Lua scripts, which Redis runs as a single block. The one that announces an arrival only publishes if the entry is new, so a simple renewal wakes nobody:
-- touch.lua
local added = redis.call('ZADD', KEYS[1], ARGV[2], ARGV[3])
redis.call('EXPIRE', KEYS[1], ARGV[4])
if added == 1 then
redis.call('PUBLISH', ARGV[1], ARGV[5])
end
return added
The scripts are embedded in the binary and sent with EVALSHA, falling back to EVAL if Redis has forgotten them. A sweep, every 15 seconds and in each process, removes the expired leases and notifies once per player, even if several processes sweep at the same time.
Computing the list to display is a pure function, easy to test: keep the unexpired entries, translate each key into a friend code thanks to the ticket, remove your own, deduplicate, sort. And since Redis pub/sub doesn't keep messages published during a reconnection, each connection redoes this computation every 20 seconds: a lost event costs at most 20 seconds of delay. The list is only sent back to the browser if it changed, so this heartbeat costs no traffic.
08 What the browser can send
After the hello, the page only sends two things, and Python has nothing to do with them:
- the open cell,
{"type":"focus","cell":[3,12]}, ornullwhen it closes. Going through the API would cost a request, a transaction and a publication on every tap, for information worth nothing a second later; - a reaction,
{"type":"react","to":"ABCD2345EFGH","reaction":"clap"}, among five possible ones (clap,fire,wow,muscle,rocket), never free text.
Both go through a token bucket per connection: 10 cells of headroom and one more token every 100 ms for focus, 3 reactions of headroom and one more every 2 seconds for reactions. The bucket takes the time as a parameter instead of reading it itself, which makes it testable without waiting:
// reaction.rs
pub fn try_take(&mut self, now: Instant) -> bool {
let elapsed = now.saturating_duration_since(self.refilled_at);
let earned = elapsed.as_nanos() / self.refill.as_nanos();
// … adds the tokens earned, capped at `burst`, then takes one if any are left
}
Any malformed or refused message is ignored with no response: sending back an error would only help someone probing the server, an honest client never sends one.
09 Tickets change, connections stay
The API signs a new ticket every time the grid loads, and its exp moves forward each time: two tickets taken a few seconds apart are always different. In a team, the page reloads the grid after certain notices: reconnecting on every new ticket would make the whole team's presence flicker on every move.
The ticket therefore arrives with a connection key, a SHA-256 of its content without exp, truncated to 16 characters. It changes when the grid, the friends or the team change, never when only the expiry moves forward. The page only reconnects if it changes; otherwise, it keeps the new ticket for the next reconnection, which happens once every 12 hours.
When Bob joins Alice's team mid-game, his response contains a ticket with the team: his page reconnects. Alice's ticket, signed earlier, doesn't know him yet; but the joined notice is addressed to the team key, which Alice already has. Her page reloads the grid, receives a ticket that lists Bob and reconnects in turn. During that fraction of a second, Alice already receives Bob's moves, but not yet the cell he opened.
10 In production
- Caddy sends
/live/*to the real-time server's container, before the application's rule. There's only one real-time container, no blue-green switch like for the application: a deploy cuts all the WebSockets, and the pages reconnect with an exponential wait, from 1 to 30 seconds, give or take 20% so that not every tab comes back at the same instant. - The Docker image is built in two stages: compilation in a Rust image, then the single binary copied into a minimal Debian, run by an unprivileged user. It has no
curl: the binary can probe its own/live/healthroute (stationdoku-live --healthcheck), which answers after aPINGto Redis. - Shutdown is clean: on
SIGTERM, the server stops accepting connections, asks each one to close and gives them up to 5 seconds to remove their presence from Redis, so friends see the departure right away. - The secret is Django's, passed to the container: changing the
SECRET_KEYchanges everything at once. Tickets in flight become invalid, and pages receive new ones on their next load. - The logs never contain anything identifying: no player identifier, no key, no friend code, no ticket, no notice content. One line per minute sums up the activity: open connections, notices, scores, endings, reactions and cells delivered, messages dropped. A refused signature shows up there as a warning: the sign that the real-time server's secret is no longer Django's.
11 When something breaks
| What goes down | What you see | Why it's not a big deal |
|---|---|---|
| The real-time server | the friends row disappears, the page retries | the whole game goes through the API |
| Redis | same | nothing durable lives there; a move never depends on Redis |
| A pub/sub message | an announcement is missing; presence at most 20 s late | the recompute every 20 s and the next reload catch up |
| A slow client | it misses a few messages | the others don't suffer; it catches up on reload |
| A Rust process crashes | its players look present for 60 s too long | the leases expire, the sweep announces the departures |
| Two moves on the same cell | the second gets "cell already taken", with no penalty | the team's lock queues the moves |
12 Testing both sides
Python signs, Rust verifies: if one changes the format, the order of the fields or the key derivation without the other, everything stops silently. Shared test vectors (a ticket, a score, a game ending, a notice, in JSON) are therefore checked by the tests of both languages. The Rust server also has its unit tests in each module, and end-to-end tests that open real WebSockets against a real Redis (ignored by default, run by CI). Before each merge, CI checks the formatting, runs clippy with warnings as errors, and runs all these tests.
13 Why Rust
The real-time server spends its life waiting: for messages from the browser, messages from Redis, timers. With tokio, each connection is a lightweight task that waits for all of this at once in a single select! loop, without blocking a thread. The compiler guarantees that no shared data is modified by two tasks at the same time: each process's connection registry is a table behind an RwLock, none of whose locks survives an .await, and the counters are atomic. The whole thing fits in a single binary, with a handful of dependencies: axum for HTTP and WebSockets, tokio, the redis client, serde for JSON, hmac, sha2 and base64 for the tickets, tracing for the logs.
The code is also commented for people learning Rust: ownership and borrowing, Arc, select! and cancellation, bounded channels, Notify, errors with thiserror and anyhow, lifetimes… each notion of the language is explained where it's used.