Kafka Driver Deployment Guide¶
The Kafka Driver is an experimental messaging transport backend in oslo.messaging. The driver maps the base oslo.messaging capabilities for notification message exchange onto v2.0 of the Apache Kafka distributed streaming platform. More detail regarding the Apache Kafka server is available from the Apache Kafka website.
The Kafka driver only supports use for sending and receiving
oslo.messaging notifications. Specifically, the Kafka driver
does not support oslo.messaging RPC transfers. Communications between
the driver and Kafka server backend uses a binary protocol over TCP
that defines all APIs as request response message pairs. The Kafka
driver integrates the
confluent-kafka Python client for full
protocol support and utilizes the Producer API to publish notification
messages and the Consumer API for notification listener subscriptions.
The driver is able to work with a single instance of a Kafka server or
a clustered Kafka server deployment.
Hybrid Messaging Deployment¶
Oslo.messaging provides a mechanism to configure separate messaging backends for RPC and notification communications. This is supported through the definition of separate RPC and notification transport urls in the service configuration. When the Kafka driver is deployed for oslo.messaging notifications, a separate driver and messaging backend must be deployed for RPC communications. For these hybrid messaging configurations, either the rabbit or amqp drivers can be deployed for oslo.messaging RPC.
Topics and vhost Support¶
The Kafka topic is the feed name to which records are published. Topics in Kafka are multi-subscriber such that a topic can have zero, one or many consumers that subscribe to the data written to it. In oslo.messaging, a notification listener subscribes to a topic in a supplied target that is directly mapped by the driver to the Kafka topic. The Kafka server architecture does not natively support vhosts. In order to support the presence of a vhost in the transport url provided to the driver, the topic created on the Kafka server will be appended with the virtual host name. This creates a unique topic per virtual host but note there is otherwise no access control or isolation provided by the Kafka server.
The Kafka driver provides support for listener pools. This capability is realized by mapping the listener pool name to a Kafka server consumer group name. Each record published to a topic will be delivered to one consumer instance within each subscribing pool (e.g. consumer group). If a listener pool name is not assigned to the notification listener, a single default consumer group will be used by the Kafka driver and all listeners will be assigned to that group and the messages will effectively be load balanced across the competing listener instances.
A primary functional difference between a Kafka server and a classic broker queue is that the offset or position of the message read from the commit log is controlled by the listener (e.g. consumer). The driver will advance the offset it maintains linearly as it reads message records from the server. To ensure that duplicate messages are not generated during downtime or communication interruption, the driver will synchronously commit the consumed messages prior to the notification listener dispatch. Due to this, the driver does not support the re-queue operation and the driver can not replay messages from a Kafka partition.
In order to run the driver the
confluent-kafka Python client must be
installed. The Kafka driver integrates a Python client based on librdkafka
for full protocol support and utilizes the Producer API to publish
notification messages and the Consumer API for notification listener
Source packages for the
confluent-kafka library are available via PyPI.
Since the Kafka driver is an optional extension to oslo.messaging
these packages are not installed by default. Use the
tag when installing oslo.messaging in order to pull in these extra
$ python -m pip install oslo.messaging[kafka]
Transport URL Enable¶
In oslo.messaging, the
transport_url parameters define the OpenStack
service backends for RPC and Notify. The URL is of the form:
Where the transport value specifies the RPC or notification backend as
kafka, etc. To specify and enable the
Kafka driver for notifications, in the section
[oslo_messaging_notifications] of the service configuration file,
transport_url = kafka://username:password@kafkahostname:9092
Note, that if a
transport_url parameter is not specified in the
[oslo_messaging_notifications] section, the value of
transport_url will be used for both RPC and notification backends.
It is recommended that the default configuration options provided by
the Kafka driver be used. The configuration options can be modified
oslo_messaging_kafka section of the service
Notification Listener Options¶
The compression codec for all data generated by the producer, valid values are: none, gzip, snappy, lz4, zstd. Note that the legal option of this depends on the kafka version, please refer to kafka documentation.
The plugin for the Kafka oslo.messaging driver is supported by
DevStack. As the Kafka driver can only be deployed for notifications,
the plugin supports the deployment of several message bus configurations.
[localrc] section of
local.conf, the devstack-plugin-kafka
plugin repository must be enabled. For example:
enable_plugin kafka https://opendev.org/openstack/devstack-plugin-kafka
Set the Kafka and Scala version and location variables if needed for the configuration
NOTIFY_ variables will define the message bus
configuration that will be used. The hybrid configurations will allow
for the rabbit and amqp drivers to be used for the RPC transports
while the kafka driver will be used for the notification transport. The
setting of the service variables will select which messaging
intermediary is enabled for the configuration: