Skip to main content
Skip table of contents

BYOS (Bring Your Own Storage) to DKP Clusters

Ceph can be used as the CSI Provisioner in some environments. For environments where Ceph was installed before installing DKP, you can reuse your existing Ceph installation to satisfy the storage requirements of DKP Applications.

This guide assumes you have a Ceph cluster that is not managed by DKP.

Refer to Rook Ceph Configuration for information on how to configure the Ceph instance installed by DKP for use by DKP platform applications.

Disable DKP Managed Ceph

Disable rook-ceph in your installer config since the default config of DKP has already installed a Ceph Cluster.

Disable rook-ceph in the installer config to prevent DKP from installing a Ceph cluster:

CODE 
apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  rook-ceph:
    enabled: false
  rook-ceph-cluster:
    enabled: false
  ...
...

The DKP instances of velero and grafana-loki rely on the storage provided by Ceph. Before installing the Kommander component of DKP, be sure to configure appropriate Ceph resources for their usage as detailed in the next section.

Creating DKP Compatible Ceph Resources

This section walks you through the creation of CephObjectStore and then a set of ObjectBucketClaims, which can be consumed by either velero and grafana-loki.

Typically, Ceph is installed in the rook-ceph namespace, which is the default namespace if you have followed the Quickstart - Rook Ceph Documentation guide.

This guide assumes your Ceph instance is installed in the rook-ceph namespace. Configure the variable CEPH_NAMESPACE in subsequent steps as it is applicable to your environment.

Creating CephObjectStore

There are two ways to install Ceph:

Helm Chart Values

This section is relevant if you have installed Ceph using helm install or some other managed Helm resource mechanism.

If you have applied any configuration overrides to your Rook Ceph operator, ensure it was deployed with currentNamespaceOnly set to false (It is the default value, so unless you have applied any overrides, it will be false by default). This ensures that the Ceph Operator in the rook-ceph namespace is able to monitor and manage resources in other namespaces such as kommander.

Ensure the following configuration for rook-ceph helm chart:

CODE 
# This is the default value, so need to overwrite if you are just using the defaults as-is
currentNamespaceOnly: false

You must enable the following configuration overrides for the rook-ceph-cluster helm chart:

YAML 
cephObjectStores:
  - name: dkp-object-store
    # see https://github.com/rook/rook/blob/master/Documentation/CRDs/Object-Storage/ceph-object-store-crd.md#object-store-settings for available configuration
    spec:
      metadataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Must use replicated pool ONLY. Erasure coding is not supported.
        replicated:
          size: 3
      dataPool:
        # The failure domain: osd/host/(region or zone if available) - technically also any type in the crush map
        failureDomain: osd
        # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
        # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
        # Configure this according to your CephCluster specification.
        erasureCoded:
          dataChunks: 3
          codingChunks: 1
      preservePoolsOnDelete: false
      gateway:
        port: 80
        instances: 2
        priorityClassName: system-cluster-critical
        resources:
          limits:
            cpu: "750m"
            memory: "1Gi"
          requests:
            cpu: "250m"
            memory: "500Mi"
      healthCheck:
        bucket:
          interval: 60s
    storageClass:
      enabled: true
      name: dkp-object-store
      reclaimPolicy: Delete

Managing resources directly

  1. Set a variable to refer to the namespace the AppDeployments are created in.
    NOTE: This is the kommander namespace on the management cluster or Workspace namespace on all other clusters.

    CODE 
    export CEPH_NAMESPACE=rook-ceph
export NAMESPACE=kommander

  2. Create CephObjectStore in the same namespace as the CephCluster:

    CODE 
    cat <<EOF | kubectl apply -f -
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: dkp-object-store
  namespace: ${CEPH_NAMESPACE}
spec:
  metadataPool:
    # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
    failureDomain: osd
    # Must use replicated pool ONLY. Erasure coding is not supported.
    replicated:
      size: 3
  dataPool:
    # The failure domain: osd/host/(region or zone if available) - technically, any type in the crush map
    failureDomain: osd
    # Data pool can use either replication OR erasure coding. Consider the following example scenarios:
    # Erasure Coding is used here with 3 data chunks and 1 parity chunks which assumes 4 OSDs exist.
    # Configure this according to your CephCluster specification.
    erasureCoded:
      dataChunks: 3
      codingChunks: 1
  preservePoolsOnDelete: false
  gateway:
    port: 80
    instances: 2
    priorityClassName: system-cluster-critical
    resources:
      limits:
        cpu: "750m"
        memory: "1Gi"
      requests:
        cpu: "250m"
        memory: "500Mi"
  healthCheck:
    bucket:
      interval: 60s
EOF

  3. Wait for the CephObjectStore to be Connected:

    NONE 
    $ kubectl get cephobjectstore -A       
NAMESPACE   NAME               PHASE
rook-ceph   dkp-object-store   Progressing
...
...
rook-ceph   dkp-object-store   Connected

  4. Create a StorageClass to consume the object storage:

    YAML 
    cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dkp-object-store
parameters:
  objectStoreName: dkp-object-store
  objectStoreNamespace: ${CEPH_NAMESPACE}
provisioner: ${CEPH_NAMESPACE}.ceph.rook.io/bucket
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

Creating ObjectBucketClaims

  1. Once the Object Store is Connected, create the ObjectBucketClaim in the same namespace as velero and grafana-loki .
    This results in the creation of ObjectBucket , which creates Secrets that are consumed by velero and grafana-loki.

    1. For grafana-loki:

      YAML 
      cat <<EOF | kubectl apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: dkp-loki
  namespace: ${NAMESPACE}
spec:
  additionalConfig:
    maxSize: 80G
  bucketName: dkp-loki
  storageClassName: dkp-object-store
EOF

    2. For velero:

      YAML 
      cat <<EOF | kubectl apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: dkp-velero
  namespace: ${NAMESPACE}
spec:
  additionalConfig:
    maxSize: 10G
  bucketName: dkp-velero
  storageClassName: dkp-object-store
EOF

  2. Wait for the ObjectBuckets to be Bound by executing the following command:

    CODE 
    kubectl get objectbucketclaim -n${NAMESPACE} -ocustom-columns='NAME:.metadata.name,PHASE:.status.phase'

    which should display something similar to:

    CODE 
    NAME         PHASE
dkp-loki     Bound
dkp-velero   Bound

Configure Loki to use S3 Compatible Storage

If you wish to use your own storage in DKP that is S3 compatible, create a secret that contains your AWS secret credentials.

CODE 
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: base64EncodedValue
  AWS_SECRET_ACCESS_KEY: base64EncodedValue
kind: Secret
metadata:
  name: dkp-loki #If you want to configure a custom name here also use it in the step below
  namespace: kommander

Overriding velero and grafana-loki Configuration

Once all the buckets are in the Bound state, DKP applications are now ready to be installed with the following configuration overrides populated in the installer config:

YAML 
cat <<EOF | kubectl apply -f -
apiVersion: config.kommander.mesosphere.io/v1alpha1
kind: Installation
apps:
  grafana-loki:
    enabled: true
    values: |
      loki:
        structuredConfig:
          storage_config:
            aws:
              s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"
      ingester:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      querier:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      queryFrontend:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      compactor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      ruler:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
      distributor:
        extraEnvFrom:
          - secretRef:
              name: dkp-loki # Optional: This is the default value
  velero:
    enabled: true
    values: |
      configuration:
        provider: "aws"
        backupStorageLocation:
          bucket: dkp-velero
          config:
            region: dkp-object-store
            s3Url: http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/
      credentials:
        # This secret is owned by the ObjectBucketClaim. A ConfigMap and a Secret with same name as bucket are created.
        extraSecretRef: dkp-velero
EOF

This installer config can be merged with your installer config with any other relevant configuration before installing DKP.

Overriding project-grafana-loki Configuration

When installing project level grafana loki, its configuration needs to be overridden in a similar manner to workspace level grafana loki, so that the project logs can be persisted in Ceph storage.

The following overrides need to be applied to project-grafana-loki :

CODE 
loki:
  structuredConfig:
    storage_config:
      aws:
        s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/dkp-loki"

These overrides can be applied from the UI directly while substituting the ${CEPH_NAMESPACE} appropriately.

If you are using using CLI, follow these steps:

  1. Set NAMESPACE to project namespace and CEPH_NAMESPACE to Ceph install namespace:

    CODE 
    export CEPH_NAMESPACE=rook-ceph
export NAMESPACE=my-project

  2. Create a ConfigMap to apply the configuration overrides:

    CODE 
    cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  values.yaml: |
    loki:
      structuredConfig:
        storage_config:
          aws:
            s3: "http://rook-ceph-rgw-dkp-object-store.${CEPH_NAMESPACE}.svc:80/proj-loki-${NAMESPACE}"
kind: ConfigMap
metadata:
  name: project-grafana-loki-ceph
  namespace: ${NAMESPACE}
EOF

  3. Create the AppDeployment with a reference to the above ConfigMap:
    NOTE: The clusterSelector can be adjusted according to your needs.

    CODE 
    cat <<EOF | kubectl apply -f -
apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: project-grafana-loki
  namespace: ${NAMESPACE}
spec:
  appRef:
    kind: ClusterApp
    name: project-grafana-loki-0.48.6
  clusterSelector: {}
  configOverrides:
    name: project-grafana-loki-ceph
EOF

The project level Grafana Loki creates an ObjectBucketClaim and assumes that the Ceph operator is montitoring the project namespace, so there is no need to create ObjectBucketClaim manually.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close