.. (c) Copyright 2016 Hewlett-Packard Development Company, L.P. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ****************** Swift Plugin Guide ****************** WARNING: The Swift plugin is currently EXPERIMENTAL as incremental indexing is only provided via an experimental OSLO messaging middleware patch while other indexing methodologies are explored for Swift. See :ref:`proxy-server.conf` for additional information. Integration is provided via a plugin. There are multiple configuration settings required for proper indexing and incremental updates. Some of the settings are specified in Searchlight configuration files. Others are provided in Swift configuration files. Swift Configuration ==================== reseller_admin_role ------------------- Users with the Keystone role defined in `reseller_admin_role` (`ResellerAdmin` by default) can operate on any account. The auth system sets the request context variable `reseller_request` to True if a request is coming from a user with this role. Searchlight needs this role for its service user to access all of the swift accounts during initial indexing. The `searchlight` user and `service` project being referred to here are defined in the `service_credentials` section of `searchlight.conf` file. If any of the Swift plugins are enabled, this role must be added prior to running `searchlight-manage index sync`. :: openstack role add --user searchlight --project service ResellerAdmin .. _proxy-server.conf: proxy-server.conf ----------------- Incremental indexing for searchlight is typically provided via OSLO messaging. The Swift service currently doesn't send notifications, but work has been started to investigate more performant ways to achieve indexing. In the meantime, experimental support for providing notifications via middleware is provided in the following patch: * https://review.openstack.org/#/c/249471 #. Apply the patch to the Swift proxy server #. Make the below configuration changes to `proxy-server.conf` #. Restart the Swift proxy server :: # Add the following new section [filter:oslomiddleware] paste.filter_factory = swift.common.middleware.oslo_notifications:filter_factory publisher_id = swift.localhost #Replace ,, and for your environment values transport_url = rabbit://:@:/ notification_driver = messaging notification_topics = notifications # Add oslomiddleware to pipeline:main # see example below. [pipeline:main] pipeline = catch_errors gatekeeper ...... oslomiddleware proxy-logging proxy-server .. note:: Restart the swift proxy API service (s-proxy) after making changes. Starting in Newton, Searchlight can share the same notification topic as other services, because it uses a messaging pool. Searchlight Configuration ========================= Searchlight resource configuration options are shown below with their configuration file and default values. See :ref:`searchlight-plugins` for common options with their default values, general configuration information, and an example complete configuration. .. note:: Unless you are changing to a non-default value, you do not need to specify any of the following configuration options. After enabling or disabling a plugin you do need to restart the searchlight services (`searchlight-api` and `searchlight-listener`). After enabling a Swift plugin, you will also need to run the sync job: `searchlight-manage index sync --type OS::Swift::Account` searchlight.conf ---------------- Plugin: OS::Swift::Account ^^^^^^^^^^^^^^^^^^^^^^^^^^ :: [resource_plugin:os_swift_account] enabled = true resource_group_name = searchlight # Specify same value as in swift proxy-server.conf for reseller_prefix reseller_prefix = AUTH_ .. note:: `os_swift_account` is disabled by default. You need to explicitly set enabled = True as shown above. Plugin: OS::Swift::Container ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: [resource_plugin:os_swift_container] enabled = true resource_group_name = searchlight .. note:: `os_swift_container` is disabled by default. You need to explicitly set enabled = True as shown above. Plugin: OS::Swift::Object ^^^^^^^^^^^^^^^^^^^^^^^^^ :: [resource_plugin:os_swift_object] enabled = true resource_group_name = searchlight .. note:: `os_swift_object` is disabled by default. You need to explicitly set enabled = True as shown above. local.conf (devstack) --------------------- At this time we recommend that you manually enable the Searchlight plugins and middleware for Swift after devstack has completed stacking. Please follow the instructions above. Release Notes ============= 0.2.0.0 (Mitaka) ---------------- Notifications must be configured properly for searchlight to process incremental updates. Searchlight must use its own topic. Use the following:: notification_driver = messaging notification_topics = searchlight_indexer Large scale swift cluster support is targeted at a future release, but we encourage trial deployments to help us address issues as soon as possible. Swift did not generate notifications for account/container/object CRUD during the Mitaka release. This means that search results will not include incremental updates after the initial indexing. However, there is a patch available to enable notifications via oslo messaging for the Mitaka release. * https://review.openstack.org/#/c/249471 For devstack, the easiest way to test is:: cd /opt/stack/swift git review -x 249471 Searchlight developers/installers should apply the above patch in Swift when using Searchlight with the Swift Mitaka release. We are working with the Swift team to create a supported incremental indexing methodology for future releases. Alternatively, you may set up a cron job to re-index swift account/container/objects periodically to get updated information. The recommendation is to use the notifications, because a full re-indexing will not be performant in large installations. :: searchlight-manage index sync --type OS::Swift::Account The Searchlight Swift plugin resource types follow the hierarchy similar to Swift concepts :: OS::Swift:Acccount(Parent) -> OS:Swift::Container(Child) -> OS::Swift::Object(Grand Child) which means indexing is initiated by specifying only the top parent (OS::Swift::Account) and that will in-turn index all the child plugins(Container and Object) Searchlight is adding indexing isolation in the Newton release via a concept called resource group isolation. This will better support re-indexing scalability. Additional properties can be similarly protected with the `admin_only_fields` under each plugin's configuration section. Glob-like patterns are supported. For instance:: [resource_plugin:os_swift_object] admin_only_fields=x-meta-admin*