Basic open-set intermediated peer-to-peer messaging with RabbitMQ

Everyone's familiar with the use of queue systems for workload distribution: push new work items into a queue, and have multiple workers pull items off the queue as they become available. Great.

However, there are other situations in which a messaging queue system like RabbitMQ can be useful. Here's one.

Twice in the past few months I've encountered a situation in which an open set of entities need to communicate with every other entity in the set. For example, imagine a table piled high with slips of paper on which numbers are written. A group of people sits around the table, drawing slips. People sometimes come and go from the table. Their group goal is to keep track of the highest number yet found on any slip: a distributed task that requires shared knowledge. They all remember the current highest number; when one of them finds a slip with a higher number, they shout it out to the rest, and they all remember the new number.

This problem, which we might (with tongues firmly in cheeks) term the “counting philosophers”, summarizes a whole category of systems: distributed data updates; group chats; alert notification; even mailing lists.

Without a messaging system, the common approaches to this are:

• Configure each entity to communicate peer-to-peer with the others. This is fine for small, infrequently changing sets, but rapidly becomes unmanageable as either of those aspects change. What do you do when 3 out of 4 of your peers all received your transaction update, but the fourth returned an error? This inevitably results in each entity maintaining a queue for each of its peers. That's state spread around on unreliable individual hosts. Not good.
• Use a network-level solution such as broadcast or multicast. This is fine for communication within a single datacenter, but it's not particularly reasonable for larger volumes of data. The ability of each entity to respond to upstream network issues is negligible.
• Build a centralized clearing house (chat server, log aggregator…).

I think RabbitMQ makes a pretty good clearing house. The approach is simple: each entity in the peer group creates their own queue on the server, binding it to a shared exchange with a shared routing key. Voila! A message arriving at the exchange with that key is distributed to every entity's queue. New entities bind their queue to the exchange, and immediately begin receiving new messages. When an entity leaves, its queue can remain (and fill up) or be discarded. RabbitMQ takes care of the reliability of the queues; from the client's perspective, the notifications are fire-and-forget.

Here's an example (all.py) in Python. Start RabbitMQ, then open two shells. Start python in each.

# Shell 1
import all
all.queue_setup("agent1")

# Shell 2
import all
all.queue_setup("agent2")

Now you can send a message from either shell, and it's printed in both windows:

>>> all.notify_others("Hello, world! (Agent 1)")
>>> Received: Hello, world! (Agent 1)

(Yes, these counting philosophers hear their own shouts.)

It's not much; in fact, it's almost trivial. However, it's so much better than doing it yourself that it bears publishing… and so here it is.

Posted at 2009-09-17 15:15:00 by Richard Newman • Link to Basic open-set int…