Publish/Subscribe (or pub/sub) is a software engineering pattern that has been used for decades but often required dedicated messaging servers and specialized knowledge to be useful.
Redis pub/sub is a lean and simple implementation of the publish/subscribe pattern which has been a feature of all Redis servers since the 2.0 release. This means it’s easy to use wherever Redis is deployed, and a fast and simple pub/sub system can be built quickly by deploying Redis.
To understand whether Redis pub/sub is a good fit for an app it’s important first to understand the design and goal of pub/sub, then to consider the particulars of the Redis pub/sub implementation.
Reducing dependencies and compartmentalizing knowledge are key goals in designing scalable software. The publish/subscribe messaging pattern, AKA pub/sub, separates the parts of your software that publish messages from the parts of your software that act on messages. To better grok this, let’s look at some examples:
The pub/sub pattern helps you avoid thinking too much about the subscribers when working on the publishers, and vice versa. In the examples above we can imagine several of the subscribers being interested in the messages from any one publisher. The subscribers might even want to listen to all publishers.
In pub/sub a publisher doesn’t need to know about its subscribers – it pushes a message to a channel (often called a topic in other systems) and moves on. Any subscribers that happen to be listening on that channel when the message is published will receive it.
Subscribers are designed to listen to one or more channels and react to the messages as they come in. If a subscriber is unable to keep up with the pace of messages published, it will miss some messages. This is a useful design: it allows the system to scale beyond the capacity of any slow subscribers. Publishers move on quickly without being slowed down by subscriber behavior.
Pub/sub is a pattern used in scaling software – it’s important to understand what sort of scaling it helps with. An important distinction is between pub/sub and message queueing.
In the queueing pattern, a queue (or list in Redis) buffers the messages to be processed while a pool of workers pops items off the list and handles them. In this model, scaling the size of your pool of workers scales the speed with which you process your queue, because each message is passed to only one worker. All of the workers handle any given message in the exact same way.
In pub/sub on the other hand, the system attempts to deliver all of a channel’s messages to all of its subscribers. It’s a many-to-many pattern, where each of the different subscribers do something unique with the message – one writes it to a durable log, one sends it to a Slack channel, one rings a bell in a local sales office, etc.
In short, pub/sub scales message delivery, and queueing scales message workload processing. Redis is frequently used for both of these goals. See Sidekiq for a popular example of queueing with Redis.
Pub/sub is a pattern that has been around for a very long time, and while the core pattern is the same the specific features vary considerably from implementation to implementation.
Redis pub/sub is a lightweight, fast implementation. To best understand its design, it will be useful to look at some features that are not a part of Redis pub/sub:
Unlike most Redis operations, which may be written to disk, Redis pub/sub is non-persistent. Published messages are passed directly to subscribers and then dropped, with no record retained in Redis’s memory or on disk.
This can sometimes be confusing for new users who hear that Redis pub/sub is often used to implement chat rooms. Many of us think of tools like Slack as chat rooms – you log in and see the recent messages and then receive all of the new ones. In fact the “seeing recent messages” is not a part of pub/sub at all, and must be handled by a separate means. Pub/sub only facilitates the delivery of new messages. In this sense pub/sub is like a live stream – when you turn it on you start getting information, but you don’t learn anything about what happened before you turned it on. Aside: Internet Relay Chat (IRC) uses the pub/sub model with no built-in history or message retention – that’s why the chat room metaphor is so common.
In fact you can still implement a chat room with Redis and pub/sub. To do so you’ll ensure that messages are not only published but also pushed onto a list so users can see the message history.
Subscribers are not guaranteed to receive messages in the pub/sub model. If a subscriber has networking trouble, fails to read messages quickly enough, or otherwise appears to not be attached when a message is published, it just won’t receive the message. Publishers can send messages to channels even when there are no subscribers listening – those messages are dropped.
Certain other messaging systems use read receipts or “acks” or might store a buffer for a subscriber which guards against short periods of disconnection. Redis chooses the simple option here: if you miss a message, you’ve missed the message. Messages that must reach their recipient need to be delivered by other means.
This tradeoff sounds worse than it is. Dropping message acknowledgement and subscriber-specific buffering lets Redis pub/sub process messages very quickly, and there are many systems that benefit from fair-weather message delivery.
Pub/sub is a solution for scaling, so it’s important to consider scaling not only your messages but the service you’re using to deliver the messages as well. Many messaging systems, including RabbitMQ and Kafka, were designed for high availability and reasonable scaling properties as a messaging service setup grows.
Redis Cluster (part of Redis since 3.0) has built-in support for Redis pub/sub, with an important caveat. Every published message is broadcast to every member of a Redis cluster, which can quickly cause large clusters to be overwhelmed with traffic.
The short answer to this is to only use pub/sub on individual Redis servers or on small pub/sub-specific clusters – both of these options can handle a large number of messages. Longer term Redis Cluster will have smarter features for routing messages only as needed, but that feature is still in the design phase.
Overall Redis pub/sub is a useful tool for scaling software. The absence of each of the features we looked at above is a design tradeoff, resulting in simplicity and speed for Redis at the expense of being a panacea for all messaging purposes.
Like many other parts of Redis it’s not tailored for every specialized use-case but it can take you a long way with a bit of understanding. Paired with the other features of Redis it becomes a very powerful tool in the developer’s toolbelt.
With that high-level overview complete, you’re probably ready to look at some pub/sub code examples. Take a look at Redis Pub/Sub: How To next.