Scalability and Operations for the MinIO Object Store

Learn how to configure and expand MinIO Object Store in Kaptain.

Kaptain includes the MinIO Object Store running in a distributed mode, which allows assembling multiple storage resources into a single object storage cluster. As MinIO server pods and persistent volumes are distributed across several nodes, the object store can withstand multiple node failures and still ensure full data protection.

Prerequisites

You provisioned a DKP cluster using v2.2 or later.

Configuring MinIO cluster

By default, the MinIO cluster consists of 4 servers, each with 2 volumes and a total capacity of 40Gi. To ensure data redundancy and availability, approximately half of the configured storage will be used for erasure coding sets.

The default configuration will leave approximately 20Gi for user data, which is generally sufficient for storing semi-structured and small binary data, such as pipeline artifacts and model files. If you intend to store datasets or other files in the object store, you should estimate how much additional data you need to store and then double that estimate to determine how much additional space to reserve.

To increase the storage capacity, the default settings can be altered during the installation of Kaptain by configuring the following properties:

minio.servers - the number of MinIO server pods to deploy in the pool. Kaptain deploys each of the MinIO servers to a separate node to increase the redundancy and to make the object store more fault-tolerant. Before changing this property, ensure that the cluster has a sufficient number of nodes to deploy all the servers.
minio.volumesPerServer - the number of Persistent Volume Claims to generate for each MinIO server pod in the pool. The minimum total number of disks required for MinIO in distributed mode is 4, which is the minimum required for erasure coding.
minio.storageCapacity - the total capacity of a single MinIO server pool.

WARNING: By default, MinIO shards the objects across N/2 data and N/2 parity drives. Therefore, it is recommended to specify the minio.storageCapacity property as twice the intended usage.

To change the default MinIO cluster settings, create a configuration file with the properties described below:

CODE

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ${WORKSPACE_NAMESPACE}
  name: kaptain-overrides
data:
  values.yaml: |
    core:
      minio:
        servers: 4
        volumesPerServer: 4
        storageCapacity: 10Gi
EOF

This configuration will deploy 4 MinIO servers; each server will use 4 volumes with 10Gi capacity per volume.
To learn more about MinIO distributed mode and replication, see the official documentation.

To install Kaptain with the provided parameters, refer to the Install Kaptain documentation.

Known limitations

It is not possible to expand the MinIO cluster after you have deployed it with Kaptain.
- Ensure you configure sufficient storage capacity to meet your requirements in the Kaptain configuration. Do this before deploying MinIO in your Kaptain cluster.
- Ensure the StorageClass used for MinIO storage provisioning supports Persistent Volumes Claims expansion. This must be explicitly enabled with allowVolumeExpansion: true.