Current Series Release Notes

22.0.0-62

New Features

  • Added the [expressions] allowed_languages option, which lists the expression languages workflows may use (yaql and jinja by default). An operator can restrict it - for example to yaql only - to disable a language entirely: a workflow that uses a disabled language is rejected when it is created and its expressions are refused at evaluation time.

  • Cron trigger processing can now run as a dedicated Mistral component: mistral-server --server periodic. This avoids running one processing loop per API worker. When deploying the dedicated periodic server, set [cron_trigger] run_in_api = False on the API nodes so they stop processing cron triggers themselves.

Upgrade Notes

  • A new option [cron_trigger] run_in_api (default True) controls whether the API service processes cron triggers. The default preserves the previous behavior; nothing changes unless you deploy the new dedicated periodic server and disable the processing in the API. Note that if you disable run_in_api without running a periodic server, cron triggers will stop firing.

  • A migration adds indexes to the scheduled_jobs_v2 table, used by the default scheduler, on (execute_at, captured_at) and on key. Run mistral-db-manage upgrade head when upgrading.

  • A new [scheduler] in_memory_workers option (default 10) sets the size of the thread pool the default scheduler uses to run jobs whose execution time has come.

  • The mistral-wsgi-api WSGI script, previously generated by pbr, has been removed. Deployment tooling should instead reference the Python module path mistral.wsgi:application if the chosen WSGI server supports it (gunicorn, uWSGI), or implement a .wsgi script wrapping it otherwise (mod_wsgi).

Deprecation Notes

  • Processing cron triggers in the API service is deprecated and will be removed in the next cycle, along with the [cron_trigger] run_in_api option. At that point, deployments using cron triggers will need to run the dedicated periodic server (mistral-server --server periodic). The API service logs a deprecation warning at startup while it still processes cron triggers.

  • PostgreSQL support is deprecated and not usable anymore. MySQL / MariaDB are the supported production databases. The PostgreSQL references have been removed from the documentation.

Security Issues

  • Added a new cron_triggers:publicize policy (admin_only) and enforcement on create when scope is public. Previously, the cron trigger POST endpoint hardcoded the scope to private and ignored the scope field from the request body, so cron triggers could never be created as public. The scope field is now properly passed through to the database.

  • Added a new dynamic_actions:publicize policy (admin_only) and enforcement on both create and update operations when scope is public. Previously, the dynamic actions POST endpoint did not persist the scope field and neither POST nor PUT enforced a publicize policy, allowing any user with create or update access to make dynamic actions public.

  • Added a new environments:publicize policy (admin_only) and enforcement on both create and update operations when scope is public. Previously, the environment POST endpoint did not accept the scope field in the request body and neither POST nor PUT enforced a publicize policy, allowing any user with update access to make environments public.

  • Added a new workbooks:publicize policy (admin_only) and enforcement on both create and update operations when scope is public. Previously, any project owner could make workbooks public as there was no publicize policy check on the workbook endpoints.

  • List endpoints now cap the number of returned resources at the new [api] max_limit option (default 1000). Previously a list request with no limit (or a very large one) loaded the entire table into memory and serialized it in a single response, which a caller could use to exhaust the API and database memory (amplified by all_projects=true for admins). Requests are now clamped to max_limit and callers page further using the returned next marker.

  • Operators can now disable the Jinja expression language by removing it from [expressions] allowed_languages. This reduces the attack surface exposed to untrusted workflows (expression evaluation runs in the engine and can, for instance, be abused to build very large objects and exhaust resources). The default keeps both YAQL and Jinja enabled so that existing workflows are unaffected on upgrade.

  • The yaml_parse workflow expression function now uses Mistral’s hardened YAML loader (mistral.utils.safe_yaml) instead of the plain yaml.safe_load. The plain loader resolves YAML anchors/aliases and is vulnerable to the “billion laughs” entity-expansion denial of service; yaml_parse is callable from any workflow expression with attacker-controlled input, so a workflow author could exhaust the engine memory. Anchors/aliases are now treated as plain text and are not expanded. yaml_dump likewise no longer emits aliases.

  • Added a soft address-space (memory) limit that can be applied to the engine process, configured with the new [engine] memory_limit_mb option (in MiB, 0 disables it, which is the default). When set, a runaway allocation - for instance a workflow expression building a huge data structure such as {{ [0] * 2000000000 }} - raises a MemoryError that is turned into a regular expression evaluation error, instead of exhausting the host memory and taking the engine down. Operators running untrusted workflows are encouraged to set it to the memory budget of the engine process. Note that the limit applies to the whole engine process, not only to expression evaluation.

    This bounds memory, not CPU time. It does not protect against expressions that exhaust CPU without allocating much memory, such as a Jinja template with nested loops (for example {% for i in range(100000) %}{% for j in range(100000) %}...). Operators exposing the engine to untrusted workflows should also consider disabling the Jinja expression language entirely (see the [expressions] allowed_languages option).

  • REST API resources no longer expose secret-bearing fields when they are logged. The controllers commonly log a resource with LOG.*("... %s", resource), and Resource.__str__ used to render every attribute verbatim, including workflow input, output, params (which carries env), result, published / published_global and environment variables. Those fields routinely hold client-supplied secrets (action passwords, API keys, tokens), so they were leaking into the logs on execution/task/action-execution create and update. These attributes are now masked (***) in __str__.

  • The serialized security context is no longer leaked when scheduler objects are logged. ScheduledJob.auth_ctx and DelayedCall.auth_context store a serialized MistralContext that includes the Keystone auth token and service catalog. The ORM __repr__ used to dump every column, so a warning such as “Unable to capture a scheduled job” (logged on the routine multi-node capture race) leaked the token. These columns are now masked in __repr__, and the scheduler warning logs only the job id.

  • The evaluation context is no longer included in expression evaluation errors. When a YAQL or Jinja expression failed to evaluate, the raised error (and the related log messages) embedded the whole evaluation context, which can contain sensitive data such as the Keystone auth token and the service catalog. That error is persisted in the task state_info and written to the logs, leaking the token. Evaluation errors now report only the expression and the underlying error, never the context.

  • The osprofiler DB trace no longer logs the SQL bind parameters. Mistral binds persisted context rows (e.g. scheduled_jobs_v2.auth_ctx, delayed_calls_v2.auth_context) and workflow inputs into those parameters, so they could contain the serialized security context (Keystone auth token, service catalog) and other secrets. Only the SQL statement and timing are logged now. This only affected deployments running with [profiler] enabled = True and trace_sqlalchemy = True.

  • The execution report endpoint (GET /v2/executions/<id>/report) and the sub-executions endpoints (GET /v2/executions/<id>/executions and GET /v2/tasks/<id>/executions) now enforce the executions:get RBAC policy, like the other execution-read endpoints. Previously they skipped acl.enforce, so an operator policy.yaml restricting execution reads was not applied to these two routes. Tenant isolation was unaffected (results are still project-scoped); this only bypassed custom policy restrictions.

  • Several logs and error messages that could leak secrets have been fixed:

    • Failing to instantiate an action no longer embeds the action input (which can hold ssh passwords/keys, HTTP Authorization headers, SMTP passwords, tokens) in the error - that error is also persisted in the API-visible task state_info.

    • The std.http action no longer logs its auth (HTTP Basic password / bearer token), cookies or sensitive params in clear text.

    • The engine RPC client no longer logs the raw workflow input and params (which carry the env and action credentials) when starting a workflow.

    • The notification server now redacts publisher configuration (e.g. webhook headers such as X-Auth-Token) before logging.

    • The Jinja evaluator no longer logs the rendered expression result at debug level, which could contain secrets from the workflow context.

    A new mistral.utils.redact.redact_sensitive helper masks the values of sensitive-looking keys (password, token, secret, auth, api_key, credential, cookie, private_key) in arbitrary data before logging.

  • The API now rejects request bodies sent with an XML content type (text/xml, application/xml or any +xml media type) with a 415 Unsupported Media Type response. Mistral does not support XML, but wsme used to deserialize XML bodies with an unhardened xml.etree.ElementTree parser, before any authorization or schema validation. On deployments running an old libexpat (< 2.6.0, which predates the built-in entity-expansion protection), this exposed the API to XML entity-expansion (“billion laughs”) denial-of-service attacks. XML bodies are now refused up front, on every endpoint.

  • All code_sources and dynamic_actions API policies are now restricted to admin users only (admin_only). Previously, these policies defaulted to admin_or_owner, allowing any project owner to create, read, update, and delete code sources and dynamic actions. Operators who need to restore the previous behavior can override the relevant policies in their policy.yaml.

  • The publicize policy for workflows, actions, and event triggers is now restricted to admin users only (admin_only). Previously, any project owner could make these resources public. A new code_sources:publicize policy has also been added with the same admin_only default, and publicize checks are now enforced on both create and update operations for code sources and event triggers where they were previously missing.

  • The std.http action and the webhook notifier now enforce an outbound egress policy to mitigate server-side request forgery (SSRF). Both used to connect to any user-supplied URL, so a workflow author could make the executor/engine reach the cloud metadata service (169.254.169.254) or internal endpoints. Requests to non-http(s) schemes and to hosts that resolve to a denied CIDR are now rejected.

    The deny-list is fully operator-controlled through the new [action_std_http] denied_cidrs option, which defaults to loopback and link-local (127.0.0.0/8, ::1/128, 169.254.0.0/16, fe80::/10 - so the metadata service and localhost are blocked out of the box). Operators can widen it (e.g. add RFC1918) or narrow it - even to an empty list - to re-enable those targets. An optional [action_std_http] allowed_hosts restricts connections to an explicit host allow-list. Operators who do not want std.http at all can instead disable the action via the action provider denylist.

    In addition, std.http now applies a default request timeout ([action_std_http] default_timeout, default 60s) when the action does not set its own, and can reject an over-large response via [action_std_http] max_response_size_bytes (default 5 MiB, above the 1 MiB persistable result size; set 0 to disable). The webhook notifier no longer follows redirects and uses a bounded timeout.

  • The Keystone auth_token and service_catalog are now masked in the workflow execution context (replaced with ***). Previously, their real values were persisted in the database and accessible to workflow authors via $.openstack.auth_token and $.openstack.service_catalog YAQL expressions, which could allow token exfiltration. Actions are not affected as they receive their authentication token through the RPC context.

Bug Fixes

  • The default scheduler no longer keeps one thread per pending job. Pending jobs are held in a single in-memory queue and run by a bounded thread pool, so a large number of scheduled jobs (for instance long wait-before delays) no longer results in a matching number of threads.

  • The default scheduler’s in-memory job collection is now guarded by a lock and always cleaned up, and pending jobs are cancelled on shutdown so they can no longer keep the engine process alive.

  • Fixed the action execution heartbeat checker losing its administrative security context: after processing the first expired action of a batch, the context was replaced by a non-admin project-less context (a regression of the root_execution_id logging feature). Subsequent project-scoped lookups then failed with DBEntityNotFoundError, rolling back the whole batch, and the checker retried the same batch forever, leaving expired actions (e.g. of cancelled workflow executions) in the RUNNING state indefinitely. The checker now keeps its administrative context and only enriches it with the root execution id used for logging. Additionally, an action execution whose task or workflow execution no longer exists is now skipped with a warning instead of aborting the whole heartbeat checker iteration.

  • Fixed a regression that caused the READ COMMITTED transaction isolation level to no longer be applied to database connections. The isolation level was set as a side effect of setup_db(), whose startup calls were removed. On MySQL/MariaDB servers using the default REPEATABLE READ isolation level, this broke the named lock synchronization between engines: with several engines running, concurrent completion of parallel branches could raise DBDuplicateEntry errors when creating a “join” task, leaving the workflow execution stuck in the RUNNING state. The isolation level is now set on the engine as soon as it is created, regardless of the code path that triggers the creation.

  • Fixed a keystone authentication failure when a workflow running under a trust-scoped token triggers another keystone call (for example, creating a cron trigger from within a workflow). The previous implementation passed the user’s token to keystoneclient.Client directly, which internally issues POST /v3/auth/tokens to re-authenticate. Keystone rejects this for trust-scoped tokens. The client now uses the admin_token keystoneauth1 plugin, which consumes the existing token without re-authenticating. See LP#2048851.

  • A fields query parameter that names a non-existent field on a single-resource GET (e.g. GET /v2/executions/<id>?fields=bogus) now returns a 400 Bad Request instead of an unhandled 500 with a traceback. The value is validated centrally in fields_list_to_cls_fields_tuple.

  • The API now configures logging when run as a WSGI application (mistral.wsgi:application under uWSGI/gunicorn). Previously only the mistral-server launcher called oslo.log setup, so a WSGI-hosted API fell back to Python’s default logging, which drops everything below WARNING - the INFO-level API logs never appeared. init_wsgi() now sets up oslo.log right after parsing the configuration, so the log level and formatting from mistral.conf are honored.

Other Notes

  • A scheduled job is run once by the default scheduler; a failure is logged but not retried by the scheduler itself. If the owning engine crashes, another engine re-runs the job from the job store after [scheduler] pickup_job_after seconds.

  • The REST API documentation is now published as a curated OpenStack API reference at https://docs.openstack.org/api-ref/workflow/, built from the api-ref/ source tree and covering every Mistral v2 resource. The previous auto-generated “REST API V2” page in the user documentation has been removed; requests to its old URL are redirected to the new reference.