Kafka in a Project
Deploying Kafka in a project
Get Started
To get started with creating and managing a Kafka Cluster in a project, you first need to deploy the Kafka operator and the ZooKeeper operator in the workspace where the project exists.
After deploying the Kafka operator, create Kafka Clusters by applying a KafkaCluster
custom resource on each attached cluster in a project’s namespace. Refer to the Kafka operator repository for examples of the custom resources and their configurations.
Example Deployment
If you need to manage these custom resources across all clusters in a project, it is recommended you use Project Deployments which enables you to leverage GitOps to deploy the resources. Otherwise, you will need to create the custom resources manually in each cluster.
This example deployment walks you through first deploying a ZooKeeper cluster and then a Kafka cluster in a project namespace. The result of this procedure is a running ZooKeeper cluster and Kafka cluster ready for use in your project’s namespace.
Set the
PROJECT_NAMESPACE
environment variable to the name of your project’s namespace:CODEexport PROJECT_NAMESPACE=<project namespace>
Create a ZooKeeper Cluster custom resource in your project’s namespace:
CODEkubectl apply -f - <<EOF apiVersion: zookeeper.pravega.io/v1beta1 kind: ZookeeperCluster metadata: name: zookeeper namespace: ${PROJECT_NAMESPACE} spec: replicas: 1 EOF
Check the status of your ZooKeeper cluster using
kubectl
:CODEkubectl -n ${PROJECT_NAMESPACE} get zookeeperclusters
Download the sample Kafka Cluster file.
Update the following attribute
zkAddresses
, replacingzookeeper-client.zookeeper:2181
withzookeeper-client.<project namespace>:2181
. You can usesed
to update the file:MacOS sed
CODE# If you are using sed that comes installed on macOS sed -i '' "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml
GNU sed
CODE# If you are using GNU sed sed -i "s/zookeeper-client.zookeeper:2181/zookeeper-client.${PROJECT_NAMESPACE}:2181/g" simplekafkacluster.yaml
Verify the file contains the following line:
CODE... zkAddresses: - "zookeeper-client.<project namespace>:2181" ...
Apply the
KafkaCluster
to your project’s namespace:CODEkubectl apply -n ${PROJECT_NAMESPACE} -f simplekafkacluster.yaml
Check the status of your Kafka cluster using
kubectl
:CODEkubectl -n ${PROJECT_NAMESPACE} get kafkaclusters
With both the ZooKeeper cluster and Kafka cluster running in your project’s namespace, refer to the Kafka Operator documentation for information on how to test and verify they are working as expected in. When performing those steps, ensure you substitute: zookeeper-client.<project namespace>:2181
anywhere that the zookeeper client address is mentioned.
Delete Kafka Custom Resources
Follow these steps to delete the Kafka custom resources.
View all Kafka resources in the cluster:
CODEkubectl get kafkaclusters -A kubectl get kafkausers -A Kubectl get kafkatopics -A
Delete a
KafkaCluster
example:CODEkubectl -n ${PROJECT_NAMESPACE} delete kafkacluster <name of KafkaCluster>