Storage drivers options

Storage drivers options

Storage back ends

The Message service supports several different storage back ends (storage drivers) for storing management information, messages and their metadata. The recommended storage back end is MongoDB. For information on how to specify the storage back ends, see Drivers options.

When the storage back end is chosen, the corresponding back-end options become active. For example, if Redis is chosen as the management storage back end, the options in [drivers:management_store:redis] section become active.

Storage layer pipelines

A pipeline is a set of stages needed to process a request. When a new request comes to the Message service, first it goes through the transport layer pipeline and then through one of the storage layer pipelines depending on the type of operation of each particular request. For example, if the Message service receives a request to make a queue-related operation, the storage layer pipeline will be queue pipeline. The Message service always has the actual storage controller as the final storage layer pipeline stage.

By setting the options in the [storage] section of zaqar.conf, you can add additional stages to these storage layer pipelines:

  • Claim pipeline
  • Message pipeline with built-in stage available to use:
    • zaqar.notification.notifier - sends notifications to the queue subscribers on each incoming message to the queue, in other words, enables notifications functionality.
  • Queue pipeline
  • Subscription pipeline

The storage layer pipelines options are empty by default, because additional stages can affect the performance of the Message service. Depending on the stages, the sequence in which the option values are listed does matter or not.

You can add external stages to the storage layer pipelines. For information how to write and add your own external stages, see Writing stages for the storage pipelines tutorial.

Options

The following tables detail the available options:

Description of storage configuration options
Configuration option = Default value Description
[storage]  
claim_pipeline = (List) Pipeline to use for processing claim operations. This pipeline will be consumed before calling the storage driver’s controller methods.
message_pipeline = (List) Pipeline to use for processing message operations. This pipeline will be consumed before calling the storage driver’s controller methods.
queue_pipeline = (List) Pipeline to use for processing queue operations. This pipeline will be consumed before calling the storage driver’s controller methods.
subscription_pipeline = (List) Pipeline to use for processing subscription operations. This pipeline will be consumed before calling the storage driver’s controller methods.
Description of MongoDB configuration options
Configuration option = Default value Description
[drivers:management_store:mongodb]  
database = zaqar (String) Database name.
max_attempts = 1000 (Integer) Maximum number of times to retry a failed operation. Currently only used for retrying a message post.
max_reconnect_attempts = 10 (Integer) Maximum number of times to retry an operation that failed due to a primary node failover.
max_retry_jitter = 0.005 (Floating point) Maximum jitter interval, to be added to the sleep interval, in order to decrease probability that parallel requests will retry at the same instant.
max_retry_sleep = 0.1 (Floating point) Maximum sleep interval between retries (actual sleep time increases linearly according to number of attempts performed).
reconnect_sleep = 0.02 (Floating point) Base sleep interval between attempts to reconnect after a primary node failover. The actual sleep time increases exponentially (power of 2) each time the operation is retried.
ssl_ca_certs = None (String) The ca_certs file contains a set of concatenated “certification authority” certificates, which are used to validate certificates passed from the other end of the connection.
ssl_cert_reqs = CERT_REQUIRED (String) Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values CERT_NONE``(certificates ignored), ``CERT_OPTIONAL``(not required, but validated if provided), or ``CERT_REQUIRED``(required and validated). If the value of this parameter is not ``CERT_NONE, then the ssl_ca_cert parameter must point to a file of CA certificates.
ssl_certfile = None (String) The certificate file used to identify the local connection against mongod.
ssl_keyfile = None (String) The private keyfile used to identify the local connection against mongod. If included with the certifle then only the ssl_certfile is needed.
uri = None (String) Mongodb Connection URI. If ssl connection enabled, then ssl_keyfile, ssl_certfile, ssl_cert_reqs, ssl_ca_certs need to be set accordingly.
[drivers:message_store:mongodb]  
database = zaqar (String) Database name.
max_attempts = 1000 (Integer) Maximum number of times to retry a failed operation. Currently only used for retrying a message post.
max_reconnect_attempts = 10 (Integer) Maximum number of times to retry an operation that failed due to a primary node failover.
max_retry_jitter = 0.005 (Floating point) Maximum jitter interval, to be added to the sleep interval, in order to decrease probability that parallel requests will retry at the same instant.
max_retry_sleep = 0.1 (Floating point) Maximum sleep interval between retries (actual sleep time increases linearly according to number of attempts performed).
partitions = 2 (Integer) Number of databases across which to partition message data, in order to reduce writer lock %. DO NOT change this setting after initial deployment. It MUST remain static. Also, you should not need a large number of partitions to improve performance, esp. if deploying MongoDB on SSD storage.
reconnect_sleep = 0.02 (Floating point) Base sleep interval between attempts to reconnect after a primary node failover. The actual sleep time increases exponentially (power of 2) each time the operation is retried.
ssl_ca_certs = None (String) The ca_certs file contains a set of concatenated “certification authority” certificates, which are used to validate certificates passed from the other end of the connection.
ssl_cert_reqs = CERT_REQUIRED (String) Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided. It must be one of the three values CERT_NONE``(certificates ignored), ``CERT_OPTIONAL``(not required, but validated if provided), or ``CERT_REQUIRED``(required and validated). If the value of this parameter is not ``CERT_NONE, then the ssl_ca_cert parameter must point to a file of CA certificates.
ssl_certfile = None (String) The certificate file used to identify the local connection against mongod.
ssl_keyfile = None (String) The private keyfile used to identify the local connection against mongod. If included with the certifle then only the ssl_certfile is needed.
uri = None (String) Mongodb Connection URI. If ssl connection enabled, then ssl_keyfile, ssl_certfile, ssl_cert_reqs, ssl_ca_certs need to be set accordingly.
Description of Redis configuration options
Configuration option = Default value Description
[drivers:management_store:redis]  
max_reconnect_attempts = 10 (Integer) Maximum number of times to retry an operation that failed due to a redis node failover.
reconnect_sleep = 1.0 (Floating point) Base sleep interval between attempts to reconnect after a redis node failover.
uri = redis://127.0.0.1:6379 (String) Redis connection URI, taking one of three forms. For a direct connection to a Redis server, use the form “redis://host[:port][?options]”, where port defaults to 6379 if not specified. For an HA master-slave Redis cluster using Redis Sentinel, use the form “redis://host1[:port1][,host2[:port2],...,hostN[:portN]][?options]”, where each host specified corresponds to an instance of redis-sentinel. In this form, the name of the Redis master used in the Sentinel configuration must be included in the query string as “master=<name>”. Finally, to connect to a local instance of Redis over a unix socket, you may use the form “redis:/path/to/redis.sock[?options]”. In all forms, the “socket_timeout” option may be specified in the query string. Its value is given in seconds. If not provided, “socket_timeout” defaults to 0.1 seconds.
[drivers:message_store:redis]  
max_reconnect_attempts = 10 (Integer) Maximum number of times to retry an operation that failed due to a redis node failover.
reconnect_sleep = 1.0 (Floating point) Base sleep interval between attempts to reconnect after a redis node failover.
uri = redis://127.0.0.1:6379 (String) Redis connection URI, taking one of three forms. For a direct connection to a Redis server, use the form “redis://host[:port][?options]”, where port defaults to 6379 if not specified. For an HA master-slave Redis cluster using Redis Sentinel, use the form “redis://host1[:port1][,host2[:port2],...,hostN[:portN]][?options]”, where each host specified corresponds to an instance of redis-sentinel. In this form, the name of the Redis master used in the Sentinel configuration must be included in the query string as “master=<name>”. Finally, to connect to a local instance of Redis over a unix socket, you may use the form “redis:/path/to/redis.sock[?options]”. In all forms, the “socket_timeout” option may be specified in the query string. Its value is given in seconds. If not provided, “socket_timeout” defaults to 0.1 seconds.
Description of SQLAlchemy configuration options
Configuration option = Default value Description
[drivers:management_store:sqlalchemy]  
uri = sqlite:///:memory: (String) An sqlalchemy URL
Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.