Pre-provisioned Air-gapped with GPU: Create Managed Clusters Using the DKP CLI

After initial cluster creation, you have the ability to create additional clusters from the CLI. In a previous step, the new cluster was created as Self-managed which allows it to be a Management cluster or a stand alone cluster. Subsequent new clusters are not self-managed as they will likely be Managed or Attached clusters to this Management Cluster.

When creating Managed clusters, you do not need to create and move CAPI objects, or install the Kommander component. Those tasks are only done on Management clusters!

Choose a Workspace for the New Cluster

1. If you have an existing Workspace name, run this command to find the name:
   ⚠️ NOTE: If you need to create a new Workspace, follow the instructions to Create a Workspace.

   CODE

   kubectl get workspace -A

2. When you have the Workspace name, set the WORKSPACE_NAMESPACE environment variable:

   CODE

   export WORKSPACE_NAMESPACE=<workspace_namespace>

Name Your Cluster

The cluster name may only contain the following characters: a-z, 0-9, ., and -. Cluster creation will fail if the name has capital letters. See Kubernetes for more naming information.

When specifying the cluster-name, you must use the same cluster-name as used when defining your inventory objects.

By default, the control-plane Nodes will be created in 3 different zones. However, the default worker Nodes will reside in a single Availability Zone. You may create additional node pools in other Availability Zones with the dkp create nodepool command.

Follow these steps:

1. Give your cluster a unique name suitable for your environment.

2. Set the environment variable:

export CLUSTER_NAME=<preprovisioned-additional>

Create a Cluster with GPU AMI

If a custom AMI was created using Konvoy Image Builder, the custom ami id is printed and written to ./manifest.json. To use the built ami with Konvoy, specify it with the --ami flag when calling cluster create in Step 1 in the next section where you create your Kubernetes cluster.

Create a Kubernetes Cluster

Once you’ve defined the infrastructure and control plane endpoints, you can proceed to creating the cluster by following these steps to create a new pre-provisioned cluster.

Before you create a new DKP cluster below, choose an external load balancer (LB) or virtual IP and use the corresponding dkp create cluster command.

In a pre-provisioned environment, use the Kubernetes CSI and third party drivers for local volumes and other storage devices in your data center.

DKP uses local static provisioner as the default storage provider for a pre-provisioned environment. However, localvolumeprovisioner is not suitable for production use. You should use a Kubernetes CSI compatible storage that is suitable for production.

After disabling localvolumeprovisioner, you can choose from any of the storage options available for Kubernetes. To make that storage the default storage, use the commands shown in this section of the Kubernetes documentation: https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

The following command relies on the pre-provisioned cluster API infrastructure provider to initialize the Kubernetes control plane and worker nodes on the hosts defined in the inventory YAML previously created.

1. This command uses the default external load balancer (LB) option:

   CODE

   dkp create cluster preprovisioned \
     --cluster-name ${CLUSTER_NAME} \
     --control-plane-endpoint-host <control plane endpoint host> \
     --control-plane-endpoint-port <control plane endpoint port, if different than 6443> \
     --pre-provisioned-inventory-file preprovisioned_inventory.yaml \
     --ssh-private-key-file <path-to-ssh-private-key> \
     --registry-mirror-url=${_REGISTRY_URL} \
     --registry-mirror-cacert=${_REGISTRY_CA} \
     --registry-mirror-username=${_REGISTRY_USERNAME} \
     --registry-mirror-password=${_REGISTRY_PASSWORD} \
     --namespace=${WORKSPACE_NAMESPACE}

   If your environment uses HTTP/HTTPS proxies, you must include the flags --http-proxy, --https-proxy, and --no-proxy and their related values in this command for it to be successful. More information is available in Configuring an HTTP/HTTPS Proxy.

2. Use the wait command to monitor the cluster control-plane readiness:

   CODE

   kubectl wait --for=condition=ControlPlaneReady "clusters/${CLUSTER_NAME}" --timeout=30m

   Output:

   CODE

   cluster.cluster.x-k8s.io/preprovisioned-additional condition met

NOTE: Depending on the cluster size, it will take a few minutes to create.

Manually Attach a DKP CLI Cluster to the Management Cluster

1. Find out the name of the created Cluster, so you can reference it later:

   CODE

   kubectl -n <workspace_namespace> get clusters

2. Attach the cluster by creating a KommanderCluster:

   CODE

   cat << EOF | kubectl apply -f -
   apiVersion: kommander.mesosphere.io/v1beta1
   kind: KommanderCluster
   metadata:
     name: <cluster_name>
     namespace: <workspace_namespace>
   spec:
     kubeconfigRef:
       name: <cluster_name>-kubeconfig
     clusterRef:
       capiCluster:
         name: <cluster_name>
   EOF

Next Step:

Day 2 - Cluster Operations Management