Skip to main content
Skip table of contents

BYOS (Bring Your Own Storage) to DKP Clusters

Ceph can be used as the CSI Provisioner in some environments. For environments where Ceph was installed before installing DKP, you can reuse your existing Ceph installation to satisfy the storage requirements of DKP Applications.

This guide assumes you have a Ceph cluster that is not managed by DKP.

Refer to Rook Ceph Configuration for information on how to configure the Ceph instance installed by DKP for use by DKP platform applications.

Disable DKP Managed Ceph

Disable rook-ceph in your installer config since the default config of DKP has already installed a Ceph Cluster.

Disable rook-ceph in the installer config to prevent DKP from installing a Ceph cluster:

CODE
apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  rook-ceph:
    enabled: false
  rook-ceph-cluster:
    enabled: false
  ...
...

The DKP instances of velero and grafana-loki rely on the storage provided by Ceph. Before installing the Kommander component of DKP, be sure to configure appropriate Ceph resources for their usage as detailed in the next section.

Creating DKP Compatible Ceph Resources

This section walks you through the creation of CephObjectStore and then a set of ObjectBucketClaims, which can be consumed by either velero and grafana-loki.

Typically, Ceph is installed in the rook-ceph namespace, which is the default namespace if you have followed the Quickstart - Rook Ceph Documentation guide.

This guide assumes your Ceph instance is installed in the rook-ceph namespace. Configure the variable CEPH_NAMESPACE in subsequent steps as it is applicable to your environment.

Creating CephObjectStore

There are two ways to install Ceph:

Helm Chart Values

This section is relevant if you have installed Ceph using helm install or some other managed Helm resource mechanism.

If you have applied any configuration overrides to your Rook Ceph operator, ensure it was deployed with currentNamespaceOnly set to false (It is the default value, so unless you have applied any overrides, it will be false by default). This ensures that the Ceph Operator in the rook-ceph namespace is able to monitor and manage resources in other namespaces such as kommander.

Ensure the following configuration for rook-ceph helm chart:

CODE
# This is the default value, so need to overwrite if you are just using the defaults as-is
currentNamespaceOnly: false

You must enable the following configuration overrides for the rook-ceph-cluster helm chart:

YAML
cephObjectStores:
  - name: dkp-object-store
    # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration
    spec:
      metadataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Must use replicated pool ONLY. Erasure coding is not supported.
        replicated:
          size: 3
      dataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
        # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
        # Configure this according to your CephCluster specification.
        erasureCoded:
          dataChunks: 3
          codingChunks: 1
      preservePoolsOnDelete: false
      gateway:
        port: 80
        instances: 2
        priorityClassName: system-cluster-critical
        resources:
          limits:
            cpu: "750m"
            memory: "1Gi"
          requests:
            cpu: "250m"
            memory: "500Mi"
      healthCheck:
        bucket:
          interval: 60s
    storageClass:
      enabled: true
      name: dkp-object-store
      reclaimPolicy: Delete

Managing resources directly

  1. Set a variable to refer to the namespace the AppDeployments are created in.
    NOTE: This is the kommander namespace on the management cluster or Workspace namespace on all other clusters.

    CODE
    export CEPH_NAMESPACE=rook-ceph
    export NAMESPACE=kommander
  2. Create CephObjectStore in the same namespace as the CephCluster:

    CODE
    cat <<EOF | kubectl apply -f -
    apiVersion: ceph.rook.io/v1
    kind: CephObjectStore
    metadata:
      name: dkp-object-store
      namespace: ${CEPH_NAMESPACE}
    spec:
      metadataPool:
        # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
        failureDomain: osd
        # Must use replicated pool ONLY. Erasure coding is not supported.
        replicated:
          size: 3
      dataPool:
        # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
        failureDomain: osd
        # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
        # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
        # Configure this according to your CephCluster specification.
        erasureCoded:
          dataChunks: 3
          codingChunks: 1
      preservePoolsOnDelete: false
      gateway:
        port: 80
        instances: 2
        priorityClassName: system-cluster-critical
        resources:
          limits:
            cpu: "750m"
            memory: "1Gi"
          requests:
            cpu: "250m"
            memory: "500Mi"
      healthCheck:
        bucket:
          interval: 60s
    EOF
  3. Wait for the CephObjectStore to be Connected:

    NONE
    $ kubectl get cephobjectstore -A       
    NAMESPACE   NAME               PHASE
    rook-ceph   dkp-object-store   Progressing
    ...
    ...
    rook-ceph   dkp-object-store   Connected
  4. Create a StorageClass to consume the object storage:

    YAML
    cat <<EOF | kubectl apply -f -
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: dkp-object-store
    parameters:
      objectStoreName: dkp-object-store
      objectStoreNamespace: ${CEPH_NAMESPACE}
    provisioner: ${CEPH_NAMESPACE}.ceph.rook.io/bucket
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    EOF

Creating ObjectBucketClaims

  1. Once the Object Store is Connected, create the ObjectBucketClaim in the same namespace as velero and grafana-loki .
    This results in the creation of ObjectBucket , which creates Secrets that are consumed by velero and grafana-loki.

    1. For grafana-loki:

      YAML
      cat <<EOF | kubectl apply -f -
      apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      metadata:
        name: dkp-loki
        namespace: ${NAMESPACE}
      spec:
        additionalConfig:
          maxSize: 80G
        bucketName: dkp-loki
        storageClassName: dkp-object-store
      EOF
    2. For velero:

      YAML
      cat <<EOF | kubectl apply -f -
      apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      metadata:
        name: dkp-velero
        namespace: ${NAMESPACE}
      spec:
        additionalConfig:
          maxSize: 10G
        bucketName: dkp-velero
        storageClassName: dkp-object-store
      EOF
  2. Wait for the ObjectBuckets to be Bound by executing the following command:

    CODE
    kubectl get objectbucketclaim -n${NAMESPACE} -ocustom-columns='NAME:.metadata.name,PHASE:.status.phase'

    which should display something similar to:

    CODE
    NAME         PHASE
    dkp-loki     Bound
    dkp-velero   Bound

Configure Loki to use S3 Compatible Storage

If you wish to use your own storage in DKP that is S3 compatible, create a secret that contains your AWS secret credentials.

CODE
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: base64EncodedValue
  AWS_SECRET_ACCESS_KEY: base64EncodedValue
kind: Secret
metadata:
  name: dkp-loki #If you want to configure a custom name here also use it in the step below
  namespace: kommander

Overriding velero and grafana-loki Configuration

Once all the buckets are in the Bound state, DKP applications are now ready to be installed with the following configuration overrides populated in the installer config:

YAML
cat <<EOF | kubectl apply -f -
apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  grafana-loki:
    enabled: true
    values: |
      loki:
        structuredConfig:
          storage_config:
            aws:
              s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"
      ingester:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      querier:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      queryFrontend:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      compactor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      ruler:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      distributor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
  velero:
    enabled: true
    values: |
      configuration:
        provider: "aws"
        backupStorageLocation:
          bucket: dkp-velero
          config:
            region: dkp-object-store
            s3Url: http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/
      credentials:
        # This secret is owned by the ObjectBucketClaim. A ConfigMap and a Secret with same name as bucket are created.
        extraSecretRef: dkp-velero
EOF

This installer config can be merged with your installer config with any other relevant configuration before installing DKP.

Overriding project-grafana-loki Configuration

When installing project level grafana loki, its configuration needs to be overridden in a similar manner to workspace level grafana loki, so that the project logs can be persisted in Ceph storage.

The following overrides need to be applied to project-grafana-loki :

CODE
loki:
  structuredConfig:
    storage_config:
      aws:
        s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"

These overrides can be applied from the UI directly while substituting the ${CEPH_NAMESPACE} appropriately.

If you are using using CLI, follow these steps:

  1. Set NAMESPACE to project namespace and CEPH_NAMESPACE to Ceph install namespace:

    CODE
    export CEPH_NAMESPACE=rook-ceph
    export NAMESPACE=my-project
  2. Create a ConfigMap to apply the configuration overrides:

    CODE
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    data:
      values.yaml: |
        loki:
          structuredConfig:
            storage_config:
              aws:
                s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/proj-loki-${NAMESPACE}"
    kind: ConfigMap
    metadata:
      name: project-grafana-loki-ceph
      namespace: ${NAMESPACE}
    EOF
  3. Create the AppDeployment with a reference to the above ConfigMap:
    NOTE: The clusterSelector can be adjusted according to your needs.

    CODE
    cat <<EOF | kubectl apply -f -
    apiVersion: apps.kommander.d2iq.io/v1alpha3
    kind: AppDeployment
    metadata:
      name: project-grafana-loki
      namespace: ${NAMESPACE}
    spec:
      appRef:
        kind: ClusterApp
        name: project-grafana-loki-0.48.6
      clusterSelector: {}
      configOverrides:
        name: project-grafana-loki-ceph
    EOF

The project level Grafana Loki creates an ObjectBucketClaim and assumes that the Ceph operator is montitoring the project namespace, so there is no need to create ObjectBucketClaim manually.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.