Fluent Bit

Fluent Bit is the DKP choice of open-source log collection and forwarding tool.

On the Management cluster, Fluentbit is disabled by default. The amount of admin logs ingested to Loki requires additional disk space to be configured on the grafana-loki-minio Minio Tenant. Enabling admin logs may use around 2GB/day per node. See Release Notes 2.3.0 | Configuring-the-Grafana-Loki-Minio-Tenant for more details on how to configure the Minio Tenant.

Audit Log Collection

Auditing in Kubernetes provides a way to chronologically document the actions taken on a cluster. On Kommander, by default, audit logs are collected and stored for quick indexing. Viewing and accessing can be done via the Grafana logging UI.

To adjust the default Audit Policy log backend configuration, you must modify the log retention settings by Configuring the Control Plane before creating the cluster. This needs to be done prior to creating the cluster since it cannot be edited after creation.

Collecting systemd logs from a non-default path

By default, Fluent Bit pods are configured to collect systemd logs from the /var/log/journal/ path on cluster nodes.

If systemd-journald running as a part of the OS on the nodes uses a different path for writing logs, you will need to override configuration of the fluent-bit AppDeployment to make Fluent Bit collect systemd logs.

To configure the Fluent Bit AppDeployment to collect systemd logs from a non-default path, follow these steps (all kubectl and dkp invocations refer to the management cluster):

Execute the following command to get the namespace of the workspace in which you would like to configure Fluent Bit:
CODE
```
dkp get workspaces
```
And copy the value under the NAMESPACE column for your workspace.
Set the WORKSPACE_NAMESPACE variable to the namespace copied in the previous step:
CODE
```
export WORKSPACE_NAMESPACE=<WORKSPACE_NAMESPACE>
```
Identify the systemd-journald log data storage path on the nodes of the clusters in the workspace by using the OS documentation and examining the systemd configuration.
Usually it will be either /var/log/journal (typically used when systemd-journald is configured to store logs permanently; in this case the default Fluent Bit configuration should work) or /run/log/journal (typically used when systemd-journald is configured to use a volatile storage).

Extract the default Helm values used by the Fluent Bit App:

CODE

kubectl get -n ${WORKSPACE_NAMESPACE} configmaps fluent-bit-0.19.21-d2iq-defaults -o=jsonpath='{.data.values\.yaml}' > fluent-bit-values.yaml

Edit the resulting file fluent-bit-values.yaml by removing all sections except for extraVolumes, extraVolumeMounts and config.inputs. The result should look similarly to this:

CODE

extraVolumes:
# we create this to have a persistent tail-db directory an all nodes
# otherwise a restarted fluent-bit would rescrape all tails
- name: tail-db
  hostPath:
    path: /var/log/tail-db
    type: DirectoryOrCreate
# we create this to get rid of error messages that would appear on non control-plane nodes
- name: kubernetes-audit
  hostPath:
    path: /var/log/kubernetes/audit
    type: DirectoryOrCreate
# needed for kmsg input plugin
- name: uptime
  hostPath:
    path: /proc/uptime
    type: File
- name: kmsg
  hostPath:
    path: /dev/kmsg
    type: CharDevice

extraVolumeMounts:
- name: tail-db
  mountPath: /tail-db
- name: kubernetes-audit
  mountPath: /var/log/kubernetes/audit
- name: uptime
  mountPath: /proc/uptime
- name: kmsg
  mountPath: /dev/kmsg

config:
  inputs: |
    # Collect audit logs, systemd logs, and kernel logs.
    # Pod logs are collected by the fluent-bit deployment managed by logging-operator.
    [INPUT]
        Name tail
        Alias kubernetes_audit
        Path /var/log/kubernetes/audit/*.log
        Parser kubernetes-audit
        DB /tail-db/audit.db
        Tag audit.*
        Refresh_Interval 10
        Rotate_Wait 5
        Mem_Buf_Limit 135MB
        Buffer_Chunk_Size 5MB
        Buffer_Max_Size 20MB
        Skip_Long_Lines Off
    [INPUT]
        Name systemd
        Alias kubernetes_host
        DB /tail-db/journal.db
        Tag host.*
        Max_Entries 1000
        Read_From_Tail On
        Strip_Underscores On
    [INPUT]
        Name kmsg
        Alias kubernetes_host_kernel
        Tag kernel

Add the following item to the list under the extraVolumes key:

CODE

- name: kubernetes-host
  hostPath:
    path: <path to systemd logs on the node>
    type: Directory

Add the following item to the list under the extraVolumeMounts key:
CODE
```
- name: kubernetes-host
  mountPath: <path to systemd logs on the node>
```
These items will make Kubernetes mount systemd logs into Fluent Bit pods.
Add the following line into the [INPUT] entry identified by Name systemd and Alias kubernetes_host.
CODE
```
Path <path to systemd logs on the node>
```
This is needed to make Fluent Bit actually collect the mounted logs

Assuming that the path to systemd logs on the node is /run/log/journal, the result will look similarly to this:

CODE

extraVolumes:
# we create this to have a persistent tail-db directory an all nodes
# otherwise a restarted fluent-bit would rescrape all tails
- name: tail-db
  hostPath:
    path: /var/log/tail-db
    type: DirectoryOrCreate
# we create this to get rid of error messages that would appear on non control-plane nodes
- name: kubernetes-audit
  hostPath:
    path: /var/log/kubernetes/audit
    type: DirectoryOrCreate
# needed for kmsg input plugin
- name: uptime
  hostPath:
    path: /proc/uptime
    type: File
- name: kmsg
  hostPath:
    path: /dev/kmsg
    type: CharDevice
- name: kubernetes-host
  hostPath:
    path: /run/log/journal
    type: Directory

extraVolumeMounts:
- name: tail-db
  mountPath: /tail-db
- name: kubernetes-audit
  mountPath: /var/log/kubernetes/audit
- name: uptime
  mountPath: /proc/uptime
- name: kmsg
  mountPath: /dev/kmsg
- name: kubernetes-host
  mountPath: /run/log/journal

config:
  inputs: |
    # Collect audit logs, systemd logs, and kernel logs.
    # Pod logs are collected by the fluent-bit deployment managed by logging-operator.
    [INPUT]
        Name tail
        Alias kubernetes_audit
        Path /var/log/kubernetes/audit/*.log
        Parser kubernetes-audit
        DB /tail-db/audit.db
        Tag audit.*
        Refresh_Interval 10
        Rotate_Wait 5
        Mem_Buf_Limit 135MB
        Buffer_Chunk_Size 5MB
        Buffer_Max_Size 20MB
        Skip_Long_Lines Off
    [INPUT]
        Name systemd
        Alias kubernetes_host
        Path /run/log/journal
        DB /tail-db/journal.db
        Tag host.*
        Max_Entries 1000
        Read_From_Tail On
        Strip_Underscores On
    [INPUT]
        Name kmsg
        Alias kubernetes_host_kernel
        Tag kernel

Create a ConfigMap manifest with override values from fluent-bit-values.yaml:

CODE

cat <<EOF >fluent-bit-overrides.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ${WORKSPACE_NAMESPACE}
  name: fluent-bit-overrides
data:
  values.yaml: |
$(cat fluent-bit-values.yaml | sed 's/^/    /g')
EOF

Create a ConfigMap from the manifest above:
CODE
```
kubectl apply -f fluent-bit-overrides.yaml
```

Edit the fluent-bit AppDeployment to set the value of spec.configOverrides.name to the name of the created ConfigMap. (You can use the steps in the procedure, Deploy an Application with a Custom Configuration as a guide.)

CODE

dkp edit appdeployment -n ${WORKSPACE_NAMESPACE} fluent-bit

After your editing is complete, the AppDeployment resembles this example:

CODE

apiVersion: apps.kommander.d2iq.io/v1alpha3
kind: AppDeployment
metadata:
  name: fluent-bit
  namespace: ${WORKSPACE_NAMESPACE}
spec:
  appRef:
    name: fluent-bit-0.19.21
    kind: ClusterApp
  configOverrides:
    name: fluent-bit-overrides

Log in into the Grafana logging UI of your workspace and verify that logs with a label log_source=kubernetes_host are now present in Loki.

Related information

For information on related topics or procedures, refer to the following:

Logging