Current Series Release Notes

17.0.0-23

Prelude

The OpenStack 2026.2 (Watcher 17.0.0) release advances Watcher’s modernization efforts, delivering a major milestone in the eventlet removal initiative, broader adoption of openstacksdk, more accurate cluster modelling, and stronger security defaults. This release focuses on making Watcher deployments more maintainable, more correct in their optimization decisions, and safer by default.

This release marks a turning point in the eventlet removal initiative: all Watcher services (API, Decision Engine, and Applier) now run in native threading mode by default. Eventlet, previously the default concurrency library, is now deprecated and will be removed in a future release.

The migration from legacy per-service clients to openstacksdk continues across the codebase. Building on the novaclient migration from the previous release, the Cinder, Keystone, and Placement integrations now use openstacksdk, removing the python-cinderclient and python-keystoneclient dependencies and aligning all integrations on OpenStack’s standard client library.

Optimization decisions are now more accurate and better aligned with Nova’s placement constraints. The vm_workload_consolidation strategy enforces allocation-based capacity checks in addition to utilization-based checks, producing more conservative and correct migration plans. Local disk accounting has been improved for boot-from-volume instances and flavors with ephemeral or swap devices, preventing valid destination hosts from being wrongly rejected. Compute scope availability_zones filtering now correctly limits audits to the specified zones.

Reliability of the cluster data model has been significantly hardened. A reentrant deadlock affecting the compute, storage, and baremetal models was fixed and a race condition that could silently lose model updates during periodic synchronization has been resolved. A lazily-populated per-node allocation cache further improves performance by avoiding redundant iterations when computing node resource usage.

New feature capabilities include support for default_parameters on audit templates (API microversion 1.7), allowing operators to pre-configure strategy parameter values directly on a template so that audits inherit them automatically. The action catalog also grows with two new actions: delete, which permanently removes a server instance and its associated resources, and shelve, which frees a host’s vCPU and RAM while preserving the instance data.

Finally, this release strengthens security defaults. Policy enforcement was added to the webhook trigger endpoint, requiring appropriate privileges to trigger event-driven audits, and service debug logging was hardened so the messaging transport URL is no longer written to logs, avoiding accidental exposure of embedded credentials.

Upgrade Notes

  • The error_message field in API error responses has changed format. Previously, ParsableErrorMiddleware double-encoded WSME fault bodies, so error_message was a JSON-encoded string that clients had to parse a second time. It is now a JSON object with faultcode, faultstring, and debuginfo keys. Clients that called json.loads() on the error_message value should access it directly as a dict (e.g. response['error_message']['faultstring']). For more details, please see Bug #2169114 <https://bugs.launchpad.net/watcher/+bug/2169114>_.

  • Watcher now requires Python 3.11 or newer. The last release of Watcher to support Python 3.10 was 2026.1. Ensure Python 3.11 or newer is available before upgrading.

  • The deprecated watcher/api/app.wsgi script has been removed, along with the watcher-api-wsgi script generated by the pbr wsgi_scripts extension, which is itself deprecated and scheduled for removal from pbr. The Watcher API is now published exclusively as the watcher.wsgi.api Python module.

    Deployment tooling should reference the watcher.wsgi.api:application module path if the chosen WSGI server supports it, as gunicorn and uWSGI do:

    [uwsgi]
module = watcher.wsgi.api:application

    Servers that require a file on disk instead, such as Apache mod_wsgi, can be pointed at the installed watcher/wsgi/api.py module, which exposes the same application object:

    WSGIScriptAlias / /usr/local/lib/python3/dist-packages/watcher/wsgi/api.py

  • Eventlet has been removed from Watcher. All services now run exclusively in native threading mode. The OS_WATCHER_DISABLE_EVENTLET_PATCHING environment variable no longer has any effect and can be dropped from service environments. Deployments that were still opting back into the deprecated eventlet mode must validate their configuration under the threading model.

  • The MAAS (Metal as a Service) integration has been removed from Watcher. It was marked as experimental due to the lack of testing, maintenance and documentation, and deprecated in the 2026.1 release. The [maas_client] configuration section and its url, api_key and timeout options no longer exist and are ignored if still present in watcher.conf. The change_node_power_state action now manages bare metal nodes through Ironic only. Deployments that relied on MAAS to power compute nodes on and off must enroll those nodes in Ironic. The optional python-libmaas dependency is no longer used by Watcher.

  • The standalone watcher-api server has been removed, along with its console script and manual page. It was built on oslo_service.wsgi.Server, which is eventlet based and cannot run under the threading service backend. The Watcher API must now be deployed as a WSGI application behind a WSGI server such as uwsgi or Apache mod_wsgi, using the watcher.wsgi.api:application module path. Deployments that already serve the API over uwsgi or mod_wsgi are unaffected.

  • The [api] host, [api] port, [api] workers and [api] enable_ssl_api configuration options have been removed. They were only consumed by the standalone watcher-api server, which has been removed as well. The listen address, listen port, worker count and TLS termination are now configured in the WSGI server that serves the API. Any of these options left in watcher.conf are ignored and should be dropped.

Bug Fixes

  • Fixed building of the storage data model failing when the cinder API reports a pool capacity attribute (such as provisioned_capacity_gb) as None. Some backends report None for these values when the driver does not support it as they are optional in Cinder API.

    See: https://bugs.launchpad.net/watcher/+bug/2170714 for more details.

  • Fixes a bug in the Cinder cluster data model collector where building the storage data model would fail when a volume had no host assigned yet, for example while it was still in the creating state. Such volumes are now skipped and will be picked up in a later collection cycle or via notifications once their creation completes. For more details, please see Bug #2167852.

  • Fixed an AttributeError raised in the decision engine when processing Cinder notifications for pools whose storage driver does not report all optional pool fields. Some drivers (like the nfs driver) do not report fields such as total_volumes, free_capacity_gb, provisioned_capacity_gb or allocated_capacity_gb that the lvm driver does, so assuming those fields are always present in the notification payload was wrong. These fields are now only updated when present.

    See: https://bugs.launchpad.net/watcher/+bug/2125060 for more details.

  • The Grafana datasource helper is no longer instantiated when the grafana datasource is not listed in the datasources option of the [watcher_datasources] group. Previously it was built every time a strategy resolved its datasource backend, which logged GrafanaHelper authentication token not configured at CRITICAL level on every audit for deployments that do not use Grafana.

    See: https://bugs.launchpad.net/watcher/+bug/2152623 for more details.

17.0.0

New Features

  • Added two new actions: delete and shelve.

    The delete action permanently removes a server instance and all its associated resources via the Nova API. It is automatically skipped when the target instance does not exist. Deletion cannot be reverted.

    The shelve action shelves a server instance via the Nova API, freeing compute resources (vCPU and RAM) on the host while preserving the instance data. It is automatically skipped when the target instance does not exist or is already in SHELVED or SHELVED_OFFLOADED state. The action accepts both states as valid postconditions, since Nova may offload the instance immediately depending on the backend storage.