Key-Shared Subscription Revisted

We’re finally ready to test the Key-Shared Subscription!

Failover Subscriptions allowed us to scale the number of consumers up to the number of partitions while maintaining ordering on a per partition basis. When using keyed messages, this also provided us with messaging ordering on a per key basis since messages with the same key are routed to the same partition. But what if you need to scale the number of consumers beyond the number of partitions and still need ordering on a per key basis? This is where the Key-Shared Subscription comes in.

Instead of routing messages based on partition like the Failover Subscription, the Key-Shared Subscription routes messages to consumers based on a hash map of the key. This hash map is based on the number of consumers and not the number of partitions. Generally, if you have two consumers, you can think of each consumer being assigned half the hash map of the keys even though we have three partitions. If you have four consumers (more than our three partitions!), the map of keys will generally be divided into four allowing us to scale the number of consumers being the number of partitions while keeping message ordering based on key! Let’s test out the Key-Shared Subscription.

Consult the documentation if you want more information on the definition of the key hashing. You may see groupings of keys being routed together differently than you see in the video based on the order you start and stop your consumers. This is discussed in the documentation. However, you should always see messages with the same key routed to the same consumer.

You also likely noticed that which keys are assigned to which consumers is changing as you change the number of consumers. To have control over this, you can use sticky key ranges, creating predefined key ranges for specific consumers. Just be sure you cover the entire key range when you create your consumers.