Partitioned Topics

Before discussing how increasing the number of partitions of our topic allows for increased throughput, it’s important to learn a little about Pulsar’s unique architecture.

Pulsar’s architecture is more sophisticated than other platforms. The brokers are responsible for managing all the communication and the processing of the topics. Broker nodes are stateless: they don’t store data except caches for efficiency.

In contrast, the bookies (Apache BookKeeper) nodes are responsible for storage. They have state and are where the messages are stored.

This allows for a highly scalable elastic architecture where the number of brokers or bookies can be quickly scaled independently.

When the throughput of a single topic partition becomes too large for a single broker, we can increase the number of partitions. Each individual partition can be serviced by different brokers, increasing the overall throughput of the topic. This can be done quickly because the topic ownership by a broker is independent of where the data is stored.

Let’s increase the number of partitions of our topic keyedtest from 1 to 2 using ClusterAdmin in the package AdminClient. You will need to make the following changes to ClusterAdmin before executing:

• Change the path in CREDENTIALS_URL (line 39)
• Change topic to use your student id (e.g. persistent://summitstudent1/developer/keyedtest) (line 78)

The code on lines 143-148 will change the number of partitions:

int partitions = 2;
try{
  pulsarAdmin.topics().updatePartitionedTopic(topic, partitions);
} catch (PulsarAdminException e) {
  e.printStackTrace();
}

Once you have increased the number of partitions for topic keyedtest, let’s retest a few of the subscription types. We will use KeyedProducer and SNConsumer in project PulsarKeyShared to test both Failover and Key-Shared.

Failover: Start 3 instances of SNConsumer in project PulsarKeyShared with a Failover subscription type. Execute KeyedProducer to send 100 messages.

You should notice that one instance of the consumer will receive all messages from partition-0 while another instance receives all messages from partition-1. The third instance will remain idle. Messages will be in-order on a per-key basis.

But what if you need to scale the number of consumers beyond the number of partitions while still maintaining order on a per key basis? Use a Key-Shared subscription!

Key-Shared: Repeat the above test with 3 instances of SNConsumer with a Key-Shared subscription.

Notice that all three consumers receive messages. Messages of a given key are guaranteed to be sent to the same consumer and messages for a given key are consumed in order. It’s possible a single consumer will receive messages from multiple partitions.