fluentbit-fluentd logging architecture¶
Blueprints: 1. osh-logging-framework
Releated Specs: 1. OSH logging monitoring and alerting: https://review.openstack.org/#/c/482687/
OpenStack-Helm defines a centralized logging mechanism to provide insight into the state of the OpenStack services and infrastructure components as well as underlying Kubernetes platform. Among the requirements for a logging platform, where log data can come from and where log data need to be delivered are very variable. To support various logging scenarios, OpenStack-Helm should provide a flexible mechanism to meet with certain operation needs. This spec proposes fast and lightweight log forwarder and full featured log aggregator complementing each other providing a flexible and reliable solution. Especially, Fluentbit is proposed as a log forwarder and Fluentd is proposed as a main log aggregator and processor.
The requirements for a logging collector/aggregator include:
Log collection daemon runs on each node to forward logs to aggregator
Log collection daemon should have a minimal server footprint
Log aggregator deployment runs on a selected node as deployment
Ability to apply custom metadata and uniform format to logs
Log aggregator should have HA capability
Log aggregator should have a flexible output capability to choose from
Log aggregator is able to send data to Elasticsearch and Kafka
Log aggregator should be scalable
Logging Use Cases¶
Example uses for centralized logging with Fluentbit and Fluentd include:
Cover the following logging use cases https://review.openstack.org/#/c/482687/
Collect logs from the node by Fluentbit
Every Fluentbit send logs to Fluentd with Kubernetes metadata attached
Fluentd then attaches Kubernetes and/or OpenStack metadata
Fluentd properly filters and categorizes logs
Fluentd send aggregated logs to Elasticsearch for the internal use cases
Aggregator also send aggregated logs to Kafka for external tools to consume
Fluentbit, Fluentd meet OpenStack-Helm's logging requirements for gathering, aggregating, and delivering of logged events. Fluntbit runs as a daemonset on each node and mounts the /var/lib/docker/containers directory. The Docker container runtime engine directs events posted to stdout and stderr to this directory on the host. Fluentbit then forward the contents of that directory to Fluentd. Fluentd runs as deployment at the designated nodes and expose service for Fluentbit to forward logs. Fluentd should then apply the Logstash format to the logs. Fluentd can also write Kubernetes and OpenStack metadata to the logs. Fluentd will then forward the results to Elasticsearch and to optionally Kafka. Elasticsearch indexes the logs in a logstash-* index by default. Kafka stores the logs in a 'logs' topic by default. Any external tool can then consume the 'logs' topic.
The proposal includes the following:
Helm chart for Fluentbit-Fluentd Combination
The above chart must include sensible configuration values to make the logging platform usable by default. These include: proper input configurations for both Fluentbit and Fluentd, proper output configurations for both Fluentbit and Fluentd, proper metadata and formats applied to the logs via Fluentd.
All services running within the platform should be subject to the security practices applied to the other OpenStack-Helm charts.
To minimize the performance impacts, the following should be considered:
Sane defaults for log retention and rotation policies
- Primary assignees:
sungil (Sungil Im) jayahn (Jaesuk Ahn)
Fluentbit-Fluentd logging chart
All charts should follow design approaches applied to all other OpenStack-Helm charts, including the use of helm-toolkit.
All charts require valid and sensible default values to provide operational value out of the box.
Testing should include Helm tests for each of the included charts as well as an integration test in the gate.
Documentation should be included for each of the included charts as well as documentation detailing the requirements for a usable monitoring platform, preferably with sane default values out of the box.