Certificate updates

Kaptain’s default Certificate Authority or CA, which is handled by the Kommander component of DKP, creates and renews certificates automatically and periodically. A CronJob enabled by default verifies if a new certificate has been issued since the last run, and if so, updates the Secret with the new CA bundle, and restarts the authservice pod to make the new certificate effective.

You can establish the regularity of these jobs or manually trigger the reconciliation of your system with a renewed certificate to avoid or minimize downtime.

Enable or disable automatic certificate verification

Enable or disable a CronJob that periodically scans for any newly issued certificate.

The ingress.enableSecretRotation field of the ConfigMap containing Kaptain’s configuration is set to true by default, allowing the CronJob to run automatically:

CODE

ingress:
    enableSecretRotation: true
    secretRotationSchedule: "0 0 * * *"

To disable the job, set the enableSecretRotation field to false.

Change the regularity of the CronJob

You can modify the regularity of the CronJob to suit your needs. A CronJob that runs more regularly consumes larger amounts of resources, but minimizes possible downtime. Less regular CronJobs save up resources but increase the possibility of experiencing downtime. The default CronJob is set to run every day at 12 am.

The reconciliation is triggered by the CronJob and not by the certificate creation. That is why you may experience downtime when a new certificate has been issued, and the CronJob has not fetched the new value yet.

To change the regularity of the jobs, modify the secretRotationSchedule of the Kaptain ingress, following the standards for Cron scheduling expressions. In the following example, the schedule has been set to every 6 hours:

CODE

ingress:
    enableSecretRotation: true
    secretRotationSchedule: "* */6 * * *"

Trigger certificate reconciliation manually

You can also run the CronJob on-demand. This is useful when, for example, you know the Secret has been updated with a new certificate, but you don’t want to wait for the next scheduled run to restart the pods. If your cluster is experiencing downtime due to expired certificates, this command will make your cluster available again:

CODE

kubectl -n kaptain-ingress create job --from=cronjob/kaptain-ingress-cert-secret-rotation rotation-1

Verification

You can verify that the CronJob or manually triggered job ran successfully by verifying the kubeflow-oidc-ca-bundle secret and the authservice pod.

Verify the secret

Verify that the kubeflow-oidc-ca-bundle secret has been updated:

CODE

kubectl get secret kubeflow-oidc-ca-bundle -n kaptain-ingress

The output should look similar to this:

CODE

NAME                      TYPE     DATA   AGE
kubeflow-oidc-ca-bundle   Opaque   1      109m

If the secret has been recreated successfully, the value under AGE will be recent.

Verify the pods

Verify if the authservice pod has been restarted:

CODE

kubectl get pod authservice-0 -n kaptain-ingress

The output should look like this:

CODE

NAME                              READY   STATUS    RESTARTS   AGE
authservice-0.                    1/1     Running   0          128m

If the pod has been restarted successfully, the value under AGE will be recent.