Kafka on StreamNative Testing Guide

If this is your first time deploying a StreamNative Cluster, we suggest using Serverless. You’ll have the option of deploying to AWS, Google Cloud, or Microsoft Azure. The Cluster will deploy using the Rapid Release Channel of the Classic Engine, with the most up-to-date support for the Kafka protocol, while also supporting the Pulsar protocol. This will also be the most cost effective use of your $200 in free credits. Only select regions are available when using Serverless.

Dedicated Clusters provide users the ability to size their clusters with dedicated resources and stronger isolation, still fully managed using StreamNative’s cloud resources. Contact StreamNative if you are interested in testing a Dedicated Cluster.

BYOC (Bring Your Own Cloud) Clusters are fully managed by StreamNative, but use resources in your Cloud Provider. The $200 in free credits can be used for testing a BYOC Cluster but requires the additional steps of providing StreamNative access to your Cloud Account, deploying a Cloud Connection, and deploying a Cloud Environment. You can then deploy the BYOC Cluster to the Cloud Environment in your Cloud Provider. Additional notes on BYOC Clusters:

• Both the Classic Engine (AWS, Google Cloud, and Azure) and the Ursa Engine (AWS) are available with BYOC Clusters.
• If deploying the Classic Engine, be sure to use the Rapid Release Channel for the best Kafka protocol support. The Classic Engine provides full support for Pulsar and Kafka protocols, with extremely low latency due to its use of BookKeeper as the storage layer.
• The Ursa Engine, currently in Public Preview and without support for transactions and topic compaction, is built directly on Lakehouse Storage for relaxed latency workloads. This greatly reduces infrastructure cost by reducing inter-AZ traffic and removes the complexity and cost of deploying BookKeeper. The Ursa Engine also supports direct integration with Databricks Unity Catalog (Public Preview), Snowflake Open Catalog (Polaris) (coming soon) and S3 Tables (coming soon).

We hope these descriptions help in choosing which cluster type to use when testing Kafka on StreamNative. Additional details can be found here.

If your cluster is Serverless, you’re all set to start testing Kafka with your cluster! Serverless clusters always use the Rapid Release Channel of the Classic Engine for best Kafka protocol support, and always have the Kafka protocol enabled.

If your cluster is Dedicated, we need to verify the Kafka protocol is enabled and the cluster uses the Rapid Release Channel.

• The Release Channel can be viewed by navigating to the Cluster Configuration page and clicking Configuration in the left pane. For testing the Kafka protocol, the Release Channel should say “rapid”. It is not possible to change the Release Channel of an existing cluster. If you require testing the Kafka protocol, you should create a new cluster.

• To check if the Kafka protocol is enabled for your cluster, navigate to the Cluster Dashboard and select the Overview Tab. In Access Points, If the Kafka protocol is enabled, you will see Kafka Service URL (TCP) and Kafka Schema Registry URL (HTTPS).

• If the Kafka endpoints are missing, it is possible to enable Kafka on an existing cluster. From the Cluster Dashboard, select Configuration in the left pane. Select Edit Cluster in the top right corner. You should have the option to enable Kafka on StreamNative under Features. If the option is unavailable, contact StreamNative support.
• Kafka on StreamNative is enabled by default for all new Dedicated Clusters.

If your cluster is BYOC, support for Kafka will depend on the engine of your cluster (Classic Engine or Ursa Engine). The engine type can be viewed on the Organization or Cluster Dashboards.

• If using the Classic Engine, follow the same directions as a Dedicated Cluster to check if your cluster is ready for testing Kafka (e.g. confirm the Rapid Release Channel and the Kafka Access Points).
• If using the Ursa Engine, the cluster is ready for testing with Kafka. Be aware transactions and topic compaction currently are not supported with the Ursa Engine, which also affects support for KStreams and KSqlDB.

If using a non admin service account, you will need to apply produce and consume permissions on the target namespace for your topic (by default Kafka will use the public tenant and default namespace). If using Kafka schema registry, also apply produce permissions to public/__kafka_schemaregistry/__schema-registry for your service account after creation.

API Keys have an expiration date, can be revoked early (but not extended), and are assigned to a specific instance. Each service account can have multiple API Keys. If your existing Kafka application uses basic.auth.user.info, the API Key can be used in the following format: “token:<API Key>”, where token can be any short string.

OAuth2 is supported for Java Clients using the OauthLoginCallbackHandler described here. This option is available if required, but will likely require additional changes to your existing Kafka code like updating the client library versions to support the callback handler, and adding the callback handler library.

Update the Kafka Service URL (TCP) (bootstrap.servers) and Kafka Schema Registry URL (HTTPS) (schema.registry.url). These can both be found under Access Points by navigating to the Overview Tab of the Cluster Dashboard.

Minimal changes will need to be made to your code when using an API Key. You will like only need to update the password to have the format “token:<API Key>” where token is a short string.

An example client.properties file will look as follows:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=’user’ password=’token:<API Key>’

In Java Client code:

final String apiKey = "<API Key>";
final String token = "token:" + apiKey;
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
props.put("sasl.mechanism", "PLAIN");
props.put("sasl.jaas.config", String.format("org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user\" password=\"%s\";",token));