Fluentbit is shipped (along with kibana and elasticsearch) as one the Konvoy logging stack components [1]. By default, the “es” fluentbit’s output plugin is set to send logs to the local Elasticsearch service. However, the “es” output plugin can be configured to use a remote elasticsearch backend by modifying the default “es” plugin output configuration and removing the dependency on the elasticsearch addon.
The elasticsearch addon should be disabled in the cluster.yaml:
- name: elasticsearch
enabled: false
- name: elasticsearch-curator
enabled: false
- name: elasticsearchexporter
enabled: false
Then fluentbit’s output configuration should be specified as follows:
- name: fluentbit
enabled: true
values: |
config:
outputs: |
[OUTPUT]
Name es
Match *
Host <Elasticsearch endpoint (ip_addr or URL)>
Port <PORT>
HTTP_User <username>
HTTP_Passwd <password>
tls On
Time_Key @ts
Logstash_Format On
Logstash_Prefix kubernetes_cluster
Retry_Limit 10
Buffer_Size 512KB
In Konvoy 1.7.X and previous releases, elasticsearch is configured as a dependency, therefore fluentbit addon must be patched to eliminate that requirement. To remove the requirement, please execute:
> kubectl patch addon.kubeaddons.mesosphere.io/fluentbit -n kubeaddons --patch='{"spec":{"requires":null}}' --type merge
Without removing the requirement, konvoy won’t be able to deploy the fluentbit addon:
> kubectl get addons fluentbit -n kubeaddons
fluentbit false waitingOnDependency 1.5.7-2
> kubectl describe addons fluentbit -n kubeaddons
Name: fluentbit
Namespace: kubeaddons
Labels: kubeaddons.mesosphere.io/name=fluentbit
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning MissingDeps 3m35s (x79 over 18m) kubeaddons-controller no addons found matching required dependency label: {MatchLabels:map[kubeaddons.mesosphere.io/name:elasticsearch] MatchExpressions:[]}
In Konvoy 1.8.X this is not necessary as the dependency was removed.
References:
[1] https://docs.d2iq.com/dkp/konvoy/1.8/logging/
[2] https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#configuration-parameters