Why we put chat messages in Redis streams (and plan to move old ones to MySQL)
The BoostServerTech Chat project stores every message in Redis. An in-memory data store that Rubén Pérez (
@anarthal) already knows will need to be replaced for older messages down the road.
He did it anyway. Here's why and what the code looks like.
Rubén is the author of Boost.MySQL and co-maintainer of Boost.Redis. He built this chat server as a case study in composing Boost libraries for a real application.
**The fit**
Chat messages have a specific access pattern: append only, read backward (newest first), scoped to a room. Redis streams match this almost exactly. Each room (chat group) is a stream. Writing a message is `XADD`. Reading history is `XREVRANGE`. Redis assigns each entry a unique, time ordered ID, so you get message ordering and cursor-based pagination for free. No schema migrations, indexing decisions, or ORM.
A SQL table could do this. But messages are generated at a fast pace and most SQL databases would struggle with this insertion heavy flow. It would require serious performance tuning for a workload that Redis handles natively.
**Storing a message**
When a user sends a message, the server appends it to the room's Redis stream. The "\*" tells Redis to auto assign a stream ID:
// Compose the request. XADD appends to the room's stream
// and auto-assigns an ID.
redis::request req;
for (const auto& msg : messages)
req.push("XADD", room_id, "*", "payload",
serialize_redis_message(msg));
// Execute it. All XADDs go out in one round trip.
redis::generic_response res;
error_code ec;
co_await conn_.async_exec(req, res, asio::redirect_error(ec));
Three things worth noting:
1. Multiple `XADD` commands get pushed into a single `redis::request`. Boost.Redis pipelines them over one connection, so even if a client sends several messages at once, it's one round trip.
2. This is a C++20 coroutine. The `co_await` suspends until Redis responds, but the thread is free to handle other work while it waits.
3. XADD accepts an arbitrary list of (key, value) string pairs. We are using a single key named “payload” that contains the message serialized as JSON. This allows arbitrary nesting.
**Serialization without boilerplate**
Each message is stored as a JSON payload inside the stream entry. The wire format is a simple struct:
struct redis_wire_message
{
std::string_view content;
std::int64_t timestamp;
std::int64_t user_id;
};
BOOST_DESCRIBE_STRUCT(redis_wire_message, (), (content, timestamp, user_id))
That `BOOST_DESCRIBE_STRUCT` macro registers the struct's members for compile time reflection. Boost.JSON picks it up automatically: `boost::json::value_from(msg)` serializes it, `boost::json::try_value_to<redis_wire_message>(jv)` deserializes it. No hand-written `to_json/from_json` functions. Add a field to the struct and the serialization updates itself.
This is one of those spots where Boost libraries click together in a way that's hard to replicate with unrelated dependencies. Describe provides the reflection, JSON consumes it. Three lines replace what would otherwise be two hand maintained serialization functions.
**The tradeoff**
Redis keeps everything in memory. That's what makes it fast, and it's also the obvious problem. Right now, the server runs with Redis persistence enabled, so data survives restarts. But as message volume grows, keeping the full history in RAM stops making sense.
The plan is to eventually offload old messages to MySQL for archival. The message layer is already isolated behind its own service interface, so swapping in a tiered storage strategy (recent messages from Redis, older ones from MySQL) touches one component. Nothing else needs to know.
But "eventually" involves a lot. The migration boundary is full of questions. Do you move messages after a time window? After a count threshold? Do you do it inline during reads, or as a background job? What happens to cursor based pagination when the data lives in
