Geo-Replication of Kafka or Pulsar Messages

As a final exercise, we are going to enable geo-replication of your georep namespace and consume messages from the second Pulsar cluster. This exercise requires a second cluster and will not be possible to complete if you have spun up your own StreamNative Hosted cluster.

The following has already been done to help enable geo-replication:

• create a new entry in the cluster list with credentials to the second cluster
• provide your student id tenant permissions to use those credentials, restricting the use of cluster credentials to specific tenants is part of Pulsar’s built-in multi-tenancy

Now that your tenant has access to the second cluster, as a tenant admin you need to enable geo-replication for specific namespaces. This will enable geo-replication for all topics in each namespace. Geo-replication can also be managed at the topic level.

The topic configurations must match on the source and destination servers (e.g. partitioned versus non-partitioned and number of partitions for partitioned topics). We have already created a tenant on the destination server with your student id. We’ve also created a georep namespace and a topic called sharedtopic with one partition.

We have already configured credentials to the second cluster inside the first cluster you have been working with throughout the class. Your tenant has also been provided permissions to use these credentials. Now, you just need to turn on geo-replication for the georep namespace inside your tenant.

Before executing EnableGeoRep.java, make the following changes:

• update CREDENTIALS_URL to point to the OAuth2 json file in your resources folder
• edit the namespace to include the tenant for your student id

Execute EnableGeoRep.java. Only messages written after enabling geo-replication will be sent to the second cluster.

Make the following changes to PulsarConsumerClusterB.java:

• edit the credentialsUrl to point to the OAuth2 json file in the resources folder
• Use the following as the serviceUrl of the second server (train2):

pulsar+ssl://pc-5ca42d11.aws-use2-production-snci-pool-kid.streamnative.aws.snio.cloud:6651

• Edit the topic to use the tenant with your student id

Execute PulsarConsumerClusterB.java in terminal. Open a second terminal and produce messages to the first cluster using KProducer.java or PulsarProducer.java.

Check terminal with PulsarConsumerClusterB.java and you should see messages consumed that were moved from the first cluster to the second cluster using geo-replication. Do not hesitate to reach out to training if you do not see any messages consumed. We are happy to help troubleshoot.