2026.2 Series Release Notes

17.0.0.0rc1-5

Prelude

The OpenStack 2026.2 (Watcher 17.0.0) release advances Watcher’s modernization efforts, delivering a major milestone in the eventlet removal initiative, broader adoption of openstacksdk, more accurate cluster modelling, and stronger security defaults. This release focuses on making Watcher deployments more maintainable, more correct in their optimization decisions, and safer by default.

This release marks a turning point in the eventlet removal initiative: all Watcher services (API, Decision Engine, and Applier) now run in native threading mode by default. Eventlet, previously the default concurrency library, is now deprecated and will be removed in a future release.

The migration from legacy per-service clients to openstacksdk continues across the codebase. Building on the novaclient migration from the previous release, the Cinder, Keystone, and Placement integrations now use openstacksdk, removing the python-cinderclient and python-keystoneclient dependencies and aligning all integrations on OpenStack’s standard client library.

Optimization decisions are now more accurate and better aligned with Nova’s placement constraints. The vm_workload_consolidation strategy enforces allocation-based capacity checks in addition to utilization-based checks, producing more conservative and correct migration plans. Local disk accounting has been improved for boot-from-volume instances and flavors with ephemeral or swap devices, preventing valid destination hosts from being wrongly rejected. Compute scope availability_zones filtering now correctly limits audits to the specified zones.

Reliability of the cluster data model has been significantly hardened. A reentrant deadlock affecting the compute, storage, and baremetal models was fixed and a race condition that could silently lose model updates during periodic synchronization has been resolved. A lazily-populated per-node allocation cache further improves performance by avoiding redundant iterations when computing node resource usage.

New feature capabilities include support for default_parameters on audit templates (API microversion 1.7), allowing operators to pre-configure strategy parameter values directly on a template so that audits inherit them automatically. The action catalog also grows with two new actions: delete, which permanently removes a server instance and its associated resources, and shelve, which frees a host’s vCPU and RAM while preserving the instance data.

Finally, this release strengthens security defaults. Policy enforcement was added to the webhook trigger endpoint, requiring appropriate privileges to trigger event-driven audits, and service debug logging was hardened so the messaging transport URL is no longer written to logs, avoiding accidental exposure of embedded credentials.

Bug Fixes

  • Fixes a bug in the Cinder cluster data model collector where building the storage data model would fail when a volume had no host assigned yet, for example while it was still in the creating state. Such volumes are now skipped and will be picked up in a later collection cycle or via notifications once their creation completes. For more details, please see Bug #2167852.

17.0.0.0rc1

New Features

  • Added two new actions: delete and shelve.

    The delete action permanently removes a server instance and all its associated resources via the Nova API. It is automatically skipped when the target instance does not exist. Deletion cannot be reverted.

    The shelve action shelves a server instance via the Nova API, freeing compute resources (vCPU and RAM) on the host while preserving the instance data. It is automatically skipped when the target instance does not exist or is already in SHELVED or SHELVED_OFFLOADED state. The action accepts both states as valid postconditions, since Nova may offload the instance immediately depending on the backend storage.

  • The cluster data model (ModelRoot) now maintains a lazily-populated cache of per-node allocated resources (vcpu, memory, disk), kept in sync by topology-changing methods (map_instance, unmap_instance, migrate_instance, remove_instance, remove_node). This avoids redundant O(N) iterations over a node’s instances on repeated calls to get_node_used_resources / get_node_free_resources.

  • Audit Templates now support a default_parameters field that allows operators to pre-configure strategy parameter values directly on the template. Audits created from a template with default_parameters set will inherit those values automatically, without requiring parameters to be specified at audit creation time. Explicitly provided audit parameters take precedence over template defaults.

    The default_parameters field requires a strategy to be associated with the audit template, and the provided values are validated against the strategy’s parameter schema at creation and update time.

    This feature is available via Watcher API microversion 1.7 and is exposed through the POST /v1/audit_templates and PATCH /v1/audit_templates/{audit_template_ident} endpoints. The field is returned in GET responses when the client requests API microversion 1.7 or higher.

  • Enhances local disk accounting for boot-from-volume instances and when flavors has ephemeral and/or swap devices. Watcher was unconditionally using the flavor’s root disk value as local compute node disk consumption for all instances, including those booting disk on Cinder storage rather than the local compute node. This caused the model to wrongly estimate local disk usage and migration strategies to reject valid destination hosts due to perceived disk constraints.

    For more information: https://blueprints.launchpad.net/watcher/+spec/improve-instances-disk-usage-model

Upgrade Notes

  • Connection settings for Cinder should be added directly to the [cinder] section of the configuration now, instead of [cinder_client].

  • All Watcher services (API, Decision Engine, and Applier) now run in native threading mode by default. Previously, Eventlet was the default concurrency library. For more information, please check eventlet removal documentation.

  • Connection settings for Keystone should be added directly to the [keystone] section of the configuration now, instead of [keystone_client].

  • Connection settings for the Placement service should be added directly to the [placement] section of the configuration now, instead of [placement_client].

Deprecation Notes

  • The cinder_client configuration options (api_version, endpoint_type, region_name) are deprecated and will be removed in a future release. Operators should migrate to the keystoneauth adapter configuration options in the [cinder] configuration group.

  • The Eventlet concurrency mode is now deprecated and will be removed in a future release. To temporarily re-enable Eventlet mode, set the environment variable OS_WATCHER_DISABLE_EVENTLET_PATCHING=false in the service configuration. Users are encouraged to migrate to native threading mode.

  • The keystone_client configuration options are deprecated and will be removed in a future release. Operators should migrate to the keystoneauth adapter configuration options in the [keystone] configuration group.

  • The [placement_client] configuration options are deprecated and will be removed in a future release. Operators should migrate to the keystoneauth adapter configuration options in the [placement] configuration group.

Security Issues

  • Hardened service debug logging so service start and stop messages no longer include the configured messaging transport URL. This avoids writing credentials embedded in the transport URL to service logs when debug logging is enabled. This is a security hardening enhancement and not a security vulnerability fix.

  • Adds policy enforcement to the webhook trigger endpoint (POST /v1/webhooks/{audit_uuid}). When [api] enable_webhooks_auth is True (the default), the new webhook:trigger policy rule is enforced, requiring the caller to hold the admin, administrator, or service role. Requests that do not satisfy the rule are rejected with HTTP 403. When enable_webhooks_auth is False the endpoint remains unauthenticated and no policy check is applied. For more details, please see Bug #2161771.

Bug Fixes

  • Compute scope availability_zones filtering now correctly limits audits to the specified zones. Previously, nodes were not properly matched, causing all nodes to be included or excluded unexpectedly. See: https://bugs.launchpad.net/watcher/+bug/1988981

  • Fixed a reentrant deadlock in ModelRoot, StorageModelRoot, and BaremetalModelRoot where methods such as map_instance(), unmap_instance(), and migrate_instance() could permanently deadlock the calling thread. The fix replaces the global semaphore with a per-instance threading.RLock. As a reentrant lock, it allows the same thread to re-acquire it safely, eliminating the deadlock.

    See: https://bugs.launchpad.net/watcher/+bug/2152254 for more details.

  • Fixed a race condition where cluster data model (CDM) updates delivered by oslo.messaging notification handlers could be silently lost during a periodic CDM synchronization.

    Notifications that arrive during a synchronization now wait until the new model is current before applying their updates, ensuring no state changes are lost.

    See: https://bugs.launchpad.net/watcher/+bug/2152645 for more details.

  • The vm_workload_consolidation strategy now enforces allocation-based capacity checks in addition to utilization-based checks. Previously, only actual CPU, RAM, and disk utilization was verified, which allowed the algorithm to pack more instances onto a node than Nova’s placement service would permit. The strategy now also checks that flavor-level allocations (vcpus, memory, disk) do not exceed node capacity, producing more conservative but correct migration plans that align with Nova’s placement constraints.

    See: Bug #2156473 for more details.

  • The /v1/audits/detail endpoint now accepts the strategy query parameter to filter audits by strategy UUID or name. Previously, using the --strategy filter together with --detail on openstack optimize audit list would fail because the detail endpoint did not declare the strategy parameter.

    See: Bug #2162057 for more details.

Other Notes

  • Replaced python-cinderclient with the openstacksdk block-storage proxy in CinderHelper. Usage of cinderclient has been removed and python-cinderclient has been removed as a dependency.

  • The data_model list API (GET /v1/data_model) now returns only a fixed set of fields for compute node and server elements instead of exposing every field present on the internal ComputeNode and Instance objects. Any internal field not included in the frozen allowlist is silently excluded from the response. All fields that were documented and available in the previous release are still returned.

  • Added support for openstacksdk as an alternative to keystoneclient. Usage of keystoneclient has been removed and python-keystoneclient has been removed as a dependency.

  • The Placement helper now uses OpenStackSDK instead of raw keystoneauth1 HTTP calls. This aligns the Placement integration with the SDK-based approach already used for Nova and Keystone.