Enterprise: Upgrade images used by Catalog Applications

To reduce the risk caused by vulnerabilities present in older Kafka and Zookeeper images, you should upgrade image references in theZookeeperCluster and KafkaCluster custom resources on attached clusters.

Please note that you must perform this step after completing your upgrade of DKP. You should perform this step after upgrading the zookeeper-operator and kafka-operator applications themselves.

Note that these images can be upgraded individually, on a cluster-by-cluster basis.

Upgrading a Zookeeper image reference in a ZookeeperCluster resource

The DKP catalog in 2.7 includes an updated Zookeeper 3.7.2 image, which does not contain CVEs present in earlier versions. This image is hosted at ghcr.io/mesosphere/zookeeper:0.2.15-d2iq. To update an existing Zookeeper cluster to use this image, perform the following steps:

1. Ensure that your workloads that use this Zookeeper cluster are compatible with Zookeeper 3.7.2

2. Set the spec.image.repository and spec.image.tag fields of each ZookeeperCluster custom resource to reference the new image.
   For example, you can usekubectl patch to upgrade the image in a ZookeeperCluster named zk-3 that has been manually created in a namespace ws-1 as follows:

   CODE

   kubectl patch zookeepercluster -n ws-1 zk-3 --type='json' -p='[{"op": "replace", "path": "/spec/image/repository", "value": "ghcr.io/mesosphere/zookeeper"}, {"op": "replace", "path": "/spec/image/tag", "value": "0.2.15-d2iq"}]'

For more information, refer to Zookeeper Operator documentation: https://github.com/pravega/zookeeper-operator/blob/master/README.md#trigger-the-upgrade-manually

Upgrading the Kafka image reference in a KafkaCluster resource

The DKP catalog in 2.7 includes an updated Kafka 3.4.1 image, which does not contain CVEs present in earlier versions. This image is hosted at ghcr.io/banzaicloud/kafka:2.13-3.4.1. To update an existingKafkaClusterto use this image, perform the following steps:

1. Ensure that all workloads that use this Kafka cluster are compatible with this version of Kafka.

2. Identify the current version of Kafka, which is specified in the .spec.clusterImage field of the KafkaCluster resource.

3. Ensure that the spec.readOnlyConfig field of the KafkaCluster contains a inter.broker.protocol.version=X.Y.Z line, where X.Y.Z stands for the current version of Kafka, and no other lines setting this value are present in the KafkaCluster resource. This ensures all the brokers in the cluster can communicate with each other as the cluster is upgraded using the specified protocol version.
   
   For example, if you use the kubectl edit command to perform this step on a KafkaCluster currently using the ghcr.io/banzaicloud/kafka:2.13-2.8.1 image, the resulting KafkaCluster manifest should resemble this:

   CODE

   apiVersion: kafka.banzaicloud.io/v1beta1
   ...
   spec:
     clusterImage: "ghcr.io/banzaicloud/kafka:2.13-2.8.1"
     ...
     readOnlyConfig: |
       auto.create.topics.enable=false
       cruise.control.metrics.topic.auto.create=true
       cruise.control.metrics.topic.num.partitions=1
       cruise.control.metrics.topic.replication.factor=2
       inter.broker.protocol.version=2.8.1
       ...

   For more information, refer to Kafka upgrade documentation: https://kafka.apache.org/documentation/#upgrade

4. Wait until kafka-operator reconciles the broker protocol change. kubectl get -A kafkaclusters should report ClusterRunning status for this KafkaCluster.

5. Set the spec.clusterImage to the chosen image.
   In the example above, the KafkaCluster manifest will look like this:

   CODE

   ...
   spec:
     clusterImage: "ghcr.io/banzaicloud/kafka:2.13-3.4.1" 
     ...
     readOnlyConfig: |
       ...
       inter.broker.protocol.version=2.8.1
       ...

6. Wait until kafka-operator fully applies the image change. After that, the cluster will run a new version of Kafka, but will still use the old protocol version.

7. Verify the behavior and performance of this Kafka cluster and workloads that use it.
   WARNING: This verification should not be postponed, because the next step can make a rollback to the old version of Kafka impossible.

8. Bump the protocol version to the current one by modifying the spec.readOnlyConfig field so that inter.broker.protocol.version equals the new Kafka version.
   In the example above, the `KafkaCluster` manifest will look like this:

   CODE

   ...
   spec:
     clusterImage: "http://ghcr.io/banzaicloud/kafka:2.13-3.4.1" 
     ...
     readOnlyConfig: |
       ...
       inter.broker.protocol.version=3.4.1
       ...

9. Wait until kafka-operator fully applies the protocol version change. The upgrade of the Kafka image for this cluster is now complete.