Pre-provisioned Air-gapped GPU: Create a Management Cluster

If your cluster is in an air-gapped environment or you have a local registry, you must provide additional arguments when creating the cluster. These tell the cluster where to locate the local registry to use by defining the URL.

CODE

export REGISTRY_URL=<https/http>://<registry-address>:<registry-port>
export REGISTRY_CA=<path to the CA on the bastion>
export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>

REGISTRY_URL: the address of an existing registry accessible in the VPC that the new cluster nodes will be configured to use a mirror registry when pulling images.
REGISTRY_CA: (optional) the path on the bastion machine to the registry CA. Konvoy will configure the cluster nodes to trust this CA. This value is only needed if the registry is using a self-signed certificate and the AMIs are not already configured to trust this CA.
REGISTRY_USERNAME: optional, set to a user that has pull access to this registry.
REGISTRY_PASSWORD: optional if username is not set.

Create a Cluster with GPU AMI

If a custom AMI was created using Konvoy Image Builder, the custom ami id is printed and written to ./manifest.json.

To use the built ami with Konvoy, specify it with the --ami flag when calling cluster create in Step 1.

For GPU Steps in Pre-provisioned section of the documentation to use the overrides/nvidia.yaml.

Additional helpful information can be found in the NVIDIA Device Plug-in for Kubernetes instructions and the Installation Guide of Supported Platforms.

Before you create a new DKP cluster below, choose an external load balancer or virtual IP and use the corresponding dkp create cluster command.

Name Your Cluster

The cluster name may only contain the following characters: a-z, 0-9, ., and -. Cluster creation will fail if the name has capital letters. See Kubernetes for more naming information.

When specifying the cluster-name, you must use the same cluster-name as used when defining your inventory objects.

By default, the control-plane Nodes will be created in 3 different zones. However, the default worker Nodes will reside in a single Availability Zone. You may create additional node pools in other Availability Zones with the dkp create nodepool command.

Follow these steps:

Give your cluster a unique name suitable for your environment.
Set the environment variable:

CODE

export CLUSTER_NAME=<preprovisioned-example>

Create an Air-gapped Kubernetes Cluster

Once you’ve defined the infrastructure and control plane endpoints, you can proceed to creating the cluster by following these steps to create a new pre-provisioned cluster.

DKP uses local static provisioner as the default storage provider for a pre-provisioned environment. However, localvolumeprovisioner is not suitable for production use. You should use a Kubernetes CSI compatible storage that is suitable for production.

After disabling localvolumeprovisioner, you can choose from any of the storage options available for Kubernetes. To make that storage the default storage, use the commands shown in this section of the Kubernetes documentation: https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

The following command relies on the pre-provisioned cluster API infrastructure provider to initialize the Kubernetes control plane and worker nodes on the hosts defined in the inventory.

The create cluster command below includes the --self-managed flag. A self-managed cluster refers to one in which the CAPI resources and controllers that describe and manage it are running on the same cluster they are managing.

Execute this command to create a cluster with a GPU AMI using the default external load balancer option:

CODE

dkp create cluster preprovisioned \
  --cluster-name=${CLUSTER_NAME} \
  --control-plane-endpoint-host <control plane endpoint host> \
  --control-plane-endpoint-port <control plane endpoint port, if different than 6443> \
  --pre-provisioned-inventory-file preprovisioned_inventory.yaml \
  --ssh-private-key-file <path-to-ssh-private-key> \
  --registry-mirror-url=${REGISTRY_URL} \
  --registry-mirror-cacert=${REGISTRY_CA} \
  --registry-mirror-username=${REGISTRY_USERNAME} \
  --registry-mirror-password=${REGISTRY_PASSWORD} \
  --ami <ami> \
  --self-managed

If your environment uses HTTP/HTTPS proxies, you must include the flags --http-proxy, --https-proxy, and --no-proxy and their related values in this command for it to be successful. More information is available in Configuring an HTTP/HTTPS Proxy.

Virtual IP ALTERNATIVE - if you don’t have an external LB, and wish to use a VIRTUAL IP provided by kube-vip, specify these flags example below::

CODE

dkp create cluster preprovisioned \
    --cluster-name ${CLUSTER_NAME} \
    --control-plane-endpoint-host 196.168.1.10 \
    --virtual-ip-interface eth1

The output from this command is shortened here for reading clarity, but should start like this:

CODE

Generating cluster resources
cluster.cluster.x-k8s.io/preprovisioned-example created
cont.........

Create the node pool after cluster creation:

CODE

dkp create nodepool aws -c ${CLUSTER_NAME} \
  --instance-type p2.xlarge \
  --ami-id=${AMI_ID_FROM_KIB} \
  --replicas=1 ${NODEPOOL_NAME} \
  --kubeconfig=${CLUSTER_NAME}.conf

Use the wait command to monitor the cluster control-plane readiness:Use the wait command to monitor the cluster control-plane readiness:
CODE
```
kubectl wait --for=condition=ControlPlaneReady "clusters/${CLUSTER_NAME}" --timeout=30m
```
Output:
CODE
```
cluster.cluster.x-k8s.io/preprovisioned-example condition met
```

NOTE: Depending on the cluster size, it will take a few minutes to create.

When the command completes, you will have a running Kubernetes cluster! Use this command to get the Kubernetes kubeconfig for the new cluster and proceed to installing the DKP Kommander UI:

CODE

dkp get kubeconfig -c ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf

If changing the Calico encapsulation, D2iQ recommends changing it after cluster creation, but before production.

Audit logs

To modify Control Plane Audit logs settings using the information contained in the page Configure the Control Plane.

Further Steps:

Virtual IP ALTERNATIVE - if you don’t have an external LB, and wish to use a VIRTUAL IP provided by kube-vip, specify these flags example below:

CODE

dkp create cluster preprovisioned \
    --cluster-name ${CLUSTER_NAME} \
    --control-plane-endpoint-host 196.168.1.10 \
    --virtual-ip-interface eth1

The output from this command is shortened here for reading clarity, but should start like this:

CODE

Generating cluster resources
cluster.cluster.x-k8s.io/preprovisioned-example created
cont.........

Use the wait command to monitor the cluster control-plane readiness:

CODE

kubectl wait --for=condition=ControlPlaneReady "clusters/${CLUSTER_NAME}" --timeout=30m

Output:

CODE

cluster.cluster.x-k8s.io/preprovisioned-example condition met

NOTE: Depending on the cluster size, it will take a few minutes to create.

When the command completes, you will have a running Kubernetes cluster! Use this command to get the Kubernetes kubeconfig for the new cluster and proceed to installing the DKP Kommander UI:

CODE

dkp get kubeconfig -c ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf

If changing the Calico encapsulation, D2iQ recommends changing it after cluster creation, but before production.

Audit Logs

To modify Control Plane Audit logs settings using the information contained in the page Configure the Control Plane.

Further Steps:

For more customized cluster creation, access the Pre-Provisioned Additional Configurations section.

NOTE: The section mentioned above is for overrides for your clusters, custom flags, and more that specify the secret as part of the create cluster command. If these are not specified, the overrides for your nodes will not be applied.

Cluster Verification

If you want to monitor or verify the installation of your clusters, refer to:

Verify your Cluster and DKP Installation.

Next Step:

Pre-provisioned Air-gapped GPU: Configure MetalLB