Using Pulsar Multi-Tenancy with Kafka

We’ve now worked with two applications:

• Kafka Transactions Application: connect producer and consumer with a JWT Token
• Kafka Schema Application: connect producer and consumer with OAuth2, connect to schema registry with a JWT Token or OAuth2

The ONE StreamNative Platform supports multi-tenancy for producing and consuming messages. Topics are defined by tenant/namespace/topic. Up until this point in the class, you have been publishing to the public/default tenant/namespace. With Kafka, you should not include public/default in the topic definition. Schema and KStreams data still remains in the public tenant when using multi-tenancy. See note below on using tenant.namespace.topic convention with Kafka.

Consider an example where each student taking this course is an organization within a company. Before switching from Kafka to Pulsar, each organization had their own Kafka cluster. After switching to Pulsar, each organization produces and consumes messages to and from namespaces and topics within their own tenant. Policies like restricting publish rates can be applied to avoid issues with noisy neighbors across the cluster. Other features like georep can be applied at the namespace or topic level, with permissions to publish to other clusters defined at the cluster and tenant level.

In the following examples we convert both the transactions and schema applications to use multi-tenancy.

You currently have the follow topic configuration in client.properties.transactions:

#topics
topic1=kafkastudent3-transactions1
topic2=kafkastudent3-transactions2

This will publish messages to the public/default namespace. When using public/default, you should not specify these in the topic name when using Kafka. You are currently sharing public/default with all other students in the class, which is why we wanted you to use your student id in the topic name.

We have created a tenant corresponding to your student id. Only you have access to this tenant. We have also created three namespaces in your tenant that you will use for the remainder of the course. These are transactions, schema, and georep.

To convert your transactions application to use your new tenant/namespace, edit client.properties.transacations as follows with your student id. Note: the video uses / to separate tenant/namespace/topic. For compatibility it is recommended to use tenant.namespace.topic. More information can be found here.

#topics
topic1=kafkastudent3.transactions.kafkastudent3-transactions1
topic2=kafkastudent3.transactions.kafkastudent3-transactions2

You currently have the follow topic configuration in client.properties.schema:

#topics

topic1=kafkastudent3-schemainput

To convert your schema application to use your new tenant/namespace, edit client.properties.schema as follows with your student id.

#topics

topic1=kafkastudent3.schema.kafkastudent3-schemainput

Even when using multi-tenancy, the schema registry remains in the public tenant:

• If you are using a JWT Token to connect to the schema registry, continue to use public in the basic.auth.user.info
• If you are using OAuth2 to connect to the schema registry, keep tenant=”public” in the sasl.jass.config