Current Series Release Notes

23.0.0-97

New Features

  • New [engine] options let operators cap the load a single workflow execution can put on engines, executors and the message queue. All of them default to 0, which means no limit.

    • max_with_items_length: maximum number of items a with-items task can iterate over. If the list is longer, the task fails and on-error can handle it.

    • max_with_items_concurrency: maximum number of actions a with-items task runs in parallel. A higher or missing concurrency is lowered to this value.

    • max_task_executions: maximum number of times the same task can run within one workflow execution, for example in a loop. Above it, the task and the workflow fail without running on-error, so that a loop cannot go on through it.

  • A new [engine] use_trusts option, disabled by default, runs each workflow execution started through the API under a Keystone trust. Mistral renews the token from that trust before it expires, so a workflow can run longer than the token lifetime. The trust is deleted when the execution completes, including when it fails, so a failed execution that is rerun uses the token of the user who asked for the rerun, without renewal. If the trust cannot be created, for example with a trust-scoped token, the execution uses the caller’s token as before.

Upgrade Notes

  • An abandoned action thread keeps running until the action returns or the executor process exits, and any side effects it still has happen after the timeout was reported. Actions that never return leave threads behind for the life of the executor process.

  • [yaql] limit_iterators now defaults to 100000 and [yaql] memory_quota to 104857600 bytes (100 MiB) instead of -1. A YAQL expression returning or iterating over a collection of more than 100000 items (e.g. $.items, $.items.first(), $.items.select(...)), or handling any single value larger than 100 MiB (including a large action result read with task().result), now fails. Raise these options, or set them to -1, for workflows that legitimately process larger data.

  • std.sleep or std.echo waits above [action_std_sleep] max_seconds (3600 by default) now fail. 0 disables the limit.

  • Deployments that let non-admin users list code sources or dynamic actions across projects must also grant the new list:all_projects policies.

  • The default value of the [DEFAULT] scheduler_type option changed from legacy to default. The default scheduler stores its jobs in the scheduled_jobs_v2 table, while the legacy one uses delayed_calls_v2: jobs still pending in delayed_calls_v2 are not picked up by the default scheduler. Make sure no such jobs remain before upgrading, or set scheduler_type = legacy explicitly to keep the previous behaviour.

  • executions() now returns plain dictionaries, with created_at and updated_at as strings, like execution() and tasks().

  • Executions started by a one-shot cron trigger and still running during the upgrade will not delete the trigger trust on completion; remove it from Keystone by hand if needed.

  • triggered_by is now a reserved workflow param and is rejected when creating an execution, a cron trigger or an event trigger.

  • std.email times out after 30 seconds by default and rejects SMTP servers on loopback or link-local addresses; empty [action_std_email] denied_cidrs to use a relay on the executor host. Failure messages no longer include the SMTP error.

  • Setting [action_logging] sensitive_headers replaces the new default list (Authorization, X-Auth-Token, Cookie, …). The std.http response body moved from INFO to DEBUG.

  • Jinja expressions calling a method that modifies a dict, a list or a set in place (update, append, pop, …) now fail, including on values built in the expression itself, e.g. {% set l = [] %}{{ l.append(1) }}. Build new values instead, e.g. {{ dict(_.my_dict, key='value') }} or {{ _.my_list + ['item'] }}. Setting [jinja] immutable_sandbox = False restores the previous behavior; it is unsafe with untrusted workflows and will be removed in a future release.

  • Mistral admin rights for Keycloak users now come from the realm role set in the new [keycloak_oidc] admin_role option (default admin, empty to disable). The new [keycloak_oidc] admin_realm option limits this to one realm; set it when the Keycloak server hosts realms managed by other parties. The user ID now comes from the token sub claim.

  • subworkflow_depth is now a reserved workflow param and is rejected when creating an execution, a cron trigger or an event trigger.

  • Unknown keys at the top level of a workflow definition, such as a typo like ttags, are now rejected when a workflow is created, updated or validated. They were silently ignored before. Existing workflows and executions are not affected.

  • The std.ssh_proxied action now refuses any proxy_command that is not listed in the new [action_std_ssh] allowed_proxy_commands config option (empty by default). If you use proxy_command, add its exact value to this option.

  • private_key_filename must now point inside [action_std_ssh] private_key_dir. SSH to loopback and link-local addresses is rejected by default; adjust [action_std_ssh] denied_cidrs if needed. The executor’s SSH agent and default keys are no longer tried implicitly; set private_key_filename (e.g. id_rsa) explicitly.

  • Workflows combining timeout and retry on a task no longer retry it when the timeout fires: timeout now bounds the whole task, including all its retry iterations. Raise the task timeout if retries after a slow attempt are expected.

Deprecation Notes

  • The legacy value of the [DEFAULT] scheduler_type option is deprecated and will be removed in a future release. Use the default scheduler instead.

Security Issues

  • After an action timeout the default executor still waited for the action thread to finish, so a never-ending action pinned an executor worker forever. The timeout is now reported right away and the action thread is abandoned; its late result is discarded.

  • Timed-out action threads abandoned by the default executor were not bounded, so a workflow could repeatedly run never-ending actions with a short timeout and leak threads in the executor process. The new [executor]/max_abandoned_action_threads option (default 64, 0 means unlimited) caps them: once reached, the executor fails new actions until some abandoned threads finish.

  • Workflow expressions now have default resource limits that block common bombs such as <% range(0, 10000000000).sum() %> or <% 'a' * 1000000000 %>. Jinja refuses string or list repetition above 10 million items, YAQL refuses list repetition above [yaql] limit_iterators items, and both refuse integer products, powers and left shifts above 100000 bits. This is not a complete protection: CPU time and total memory are still not bounded, and other string operations (padding, replace, join, …) can still build large values. [engine] memory_limit_mb or a container memory limit remains the backstop.

  • std.sleep and the delay of std.echo held an executor thread for an unbounded time. The wait is now limited by the new [action_std_sleep] max_seconds option.

  • Listing code sources or dynamic actions with all_projects=true now requires the new code_sources:list:all_projects or dynamic_actions:list:all_projects policy (admin only by default). Previously any user allowed to list them could read the code of every project.

  • The executions() expression function returned live database objects, and execution(), task(), tasks(), global() and env() returned data shared with them, so an expression such as {{ executions()[0].update({...}) }} could modify executions of the project. These functions now return detached copies.

  • The event of an event trigger is now restricted to letters, digits and _.:-, and is escaped when building the notification filter. It was compiled as a regular expression, so a user could prevent the event engine from starting its listeners or burn its CPU with a backtracking pattern.

  • Fixed a cross-project isolation flaw when deleting workbooks and environments. A regular (non-admin) project member could delete a public resource belonging to another project, because the deletion looked the object up with a query that includes other projects’ public resources and performed no ownership check. Deletions now resolve the object within the caller’s own project and enforce check_db_obj_access(), consistently with workflow and action definitions.

  • Deleting a code source or a dynamic action now checks that it belongs to the caller’s project. Previously, if the delete policy was relaxed from its admin-only default, a user could delete another project’s public code source or dynamic action.

  • Updating or deleting an event trigger now requires the caller to own it (or be an admin). Any user could update another project’s public event trigger and delete any event trigger whose id was known. Updates also validate the workflow and its input, as on creation.

  • Fixed a command injection on the gateway host in std.ssh_proxied: host was formatted unescaped into the nc <host> 22 command run on the gateway. host must now be a hostname or an IP address and is shell-quoted.

  • The trigger that started an execution is now stored by the engine in the execution runtime context instead of being read from the user-editable description. A user could forge triggered_by in a description to make Mistral delete the trust of a live cron or event trigger.

  • Fixed a cross-project isolation flaw when updating workbooks and environments. A regular (non-admin) project member could modify — and unpublish — a public workbook or environment belonging to another project, because the update looked the object up with a query that includes other projects’ public resources and performed no ownership check before writing. Updates now resolve the object within the caller’s own project and enforce check_db_obj_access(), consistently with workflow and action definitions.

  • std.email had no SMTP socket timeout, so a silent peer held an executor thread forever, and it returned connection errors to the caller, making it a port scanner. It now uses [action_std_email] timeout, checks the SMTP server against [action_std_email] allowed_hosts and denied_cidrs, and only logs failure details.

  • std.http no longer logs credentials and full response bodies by default. [action_logging] sensitive_headers now defaults to the usual authentication headers and is matched case-insensitively, sensitive keys of the json payload are masked, the raw request body is no longer logged, and the response body is logged at DEBUG only, truncated to the new [action_logging] response_body_log_limit.

  • Jinja expressions are now evaluated in an immutable sandbox. Previously an expression such as {{ _.my_var.update({...}) }} could modify the workflow data it was reading, and the change could be persisted with the next update of the workflow execution.

  • The Keycloak auth handler now takes the project, user and roles from the token only once it is verified, and drops identity headers sent by the client such as X-User-Id or X-Service-Roles. Token verification also works again with PyJWT 2.

  • With py_mini_racer, std.javascript ran scripts without any time or memory limit. Scripts are now bounded by the new [action_std_javascript] timeout (30s) and max_memory_mb (256) options; 0 disables a limit. pyv8 and v8eval do not enforce them. max_memory_mb is best-effort: V8 runs inside the executor process, and a single large allocation can still abort it with a fatal out-of-memory error.

  • A new [engine]/max_subworkflow_depth option limits how deep sub-workflows can be nested below a root execution. A task that would start a deeper sub-workflow fails. Without it, a self-recursive workflow could exhaust engine, database and executor capacity. It defaults to 0 (unlimited).

  • The cluster maintenance endpoint is now guarded by policy. Reading the status (GET /maintenance) requires the new maintenance:get rule and changing it (PUT /maintenance) requires the new maintenance:update rule; both default to admin only (rule:admin_only). Previously the endpoint enforced no policy, so any authenticated token holder could read the status or switch the cluster to PAUSED - a cluster-wide operation that pauses running executions across all projects. Operators who need to grant these to non-admins can override the rules in policy.yaml.

  • Fixed a resource-sharing privilege escalation. Only the owner of a workflow may now share it (create a membership for it). Previously the share endpoint (POST /v2/workflows/<id>/members) only checked that the caller could see the workflow and that it was private. A project that had a workflow shared with it and accepted the share can read the definition through the shared-resource query, and could therefore re-share the owner’s private workflow with a third project without the owner’s consent. The derived membership was owned by the re-sharing project, so the real owner could neither see nor revoke it, while the third project could accept it and read and execute the owner’s private workflow. The share operation is now rejected with 403 unless the caller is the resource owner.

  • Looking up a workflow, workbook, ad-hoc action, environment, cron trigger, code source or dynamic action by name now prefers the caller’s own object over a public or shared one of another project with the same name. Previously the match was arbitrary, so a task or execution could use another project’s workflow, action or environment.

  • The engine-internal workflow params task_execution_id, root_execution_id and index are now rejected with HTTP 400 when creating an execution, a cron trigger or an event trigger, and can no longer be set through a sub-workflow input. A user knowing a task execution UUID of another project could attach an execution to it and have the engine update that task on completion.

  • Fixed a remote code execution vulnerability in the std.ssh_proxied action. The action passed a caller-supplied proxy_command straight to paramiko.ProxyCommand, which runs it as a local subprocess on the executor host - before any SSH authentication and regardless of the SSH credentials or gateway supplied. Any authenticated, project-scoped user could therefore run arbitrary commands on every Mistral executor by submitting the action through the standard action-execution API.

    proxy_command is now rejected unless its exact value appears in the new [action_std_ssh] allowed_proxy_commands allow-list, which is empty by default. Existing deployments that rely on a proxy_command must add the exact command string to this option; otherwise the action fails with a clear error instead of executing it.

  • std.ssh and std.ssh_proxied accepted any absolute private_key_filename, letting a workflow use any key readable by the executor. Keys are now resolved inside the new [action_std_ssh] private_key_dir (~/.ssh by default), and the hosts the executor connects to are checked against the new [action_std_ssh] allowed_hosts and denied_cidrs options.

  • Single-object GET endpoints now reject unknown names in the fields query parameter with 400 Bad Request, as list endpoints already did. Unknown names shifted the returned values, so a request such as ?fields=bogus,name,trust_id exposed the hidden trust_id of a cron or event trigger under the name key.

Bug Fixes

  • The event engine now listens for notifications on the transport set by [oslo_messaging_notifications] transport_url. It used to listen on the RPC transport from [DEFAULT] transport_url, so event triggers never fired when notifications were sent to a separate bus. When [oslo_messaging_notifications] transport_url is not set, the [DEFAULT] one is still used.

  • The token of a workflow started by a cron or event trigger is now renewed from the trigger’s trust before it expires.

  • The host option of the [engine], [executor] and [event_engine] sections, and [event_engine]/listener_pool_name, now accept any string, as documented, instead of only valid hostnames or IP addresses. Previously a value such as executor_pool prevented Mistral from starting. See bug 1912850.

  • create_or_update_cron_trigger now resolves the existing trigger within the caller’s own project, so that when several projects have a cron trigger with the same name the caller’s own one is updated instead of another project’s (possibly public) trigger. The periodic scheduler’s update_cron_trigger bookkeeping is intentionally left unchanged, as it legitimately updates triggers across projects; the cron trigger REST API exposes no update operation.

  • Creating or updating a workflow or ad-hoc action definition (including implicitly through a workbook) no longer touches a definition of another project that happens to have the same name. Previously, when several projects had a definition with the same name, an admin request could pick an arbitrary one of them, move it into the admin’s project and fail with an unhandled duplicate entry error (HTTP 500). The existing definition to update is now looked up by its full unique key (name, namespace and project id).

  • Deleting an ad-hoc action definition is now checked like updating one: a non-admin user can no longer delete a public action definition of another project, such as the preinstalled std.wait_ssh.

    When several projects have an action definition with the same name, a delete now removes the one of the caller’s project, instead of any of them.

  • Deleting a dynamic action in a non-default namespace no longer fails with an internal error.

  • The event engine no longer stops processing notifications when one of them fails, for example when an expression from the event definitions file cannot be evaluated against it. The error is now logged and the notification is skipped. A failure to start the workflow of one event trigger no longer prevents the other triggers of the same event from running. See bug 1857615.

  • The [engine]/execution_field_size_limit_kb check now compares the exact field size against the limit. Previously the size was truncated to whole kilobytes, so values up to almost 1 KB over the limit were accepted. Error outputs that are truncated to fit the limit are now cut so that the whole stored value fits. See bug 1797136.

  • Requesting resource fields that are not database columns with the fields query parameter (for example input or interface of a workflow, or actions of a code source) no longer shifts the returned values onto the wrong field names. Workflow input and interface are now computed when requested.

  • Policies applied before a task starts (pause-before, wait-before, timeout and concurrency) are now honored by join tasks. They were previously ignored, so for example a join task with with-items ran all its iterations at once. See bug 1833273 and bug 1718754.

  • A join task in a loop now applies its pause-before, wait-before, wait-after and timeout policies on each pass, not only on the first one. A timeout left over from a previous pass no longer fails the next one. Its retry count is also reset on each pass, and a pass that completes while the workflow is paused now runs its transitions on resume.

  • Filtering executions on a JSON field such as input, output, params or published now matches resources whose field contains all the given keys with the given values. Previously, the filter only matched when the whole field was equal to the given dictionary, so a filter like input={"method": "GET"} returned nothing for a std.http execution with other input keys.

  • Fixed an IndexError raised while wrapping an exception that has no arguments (for example some cinderclient exceptions) when the executor sends an action result to the engine. The action execution is now correctly moved to the ERROR state instead of staying RUNNING. See bug 2122024.

  • Updating an ad-hoc action definition now verifies that the caller is allowed to modify it, consistently with workflow definitions: a non-admin request can no longer modify a public action definition of another project, and system action definitions are protected from modification by non-admin users at the database layer. In addition, when several projects have an action definition with the same name, an update now always targets the definition of the caller’s project instead of an arbitrary one.

  • Updating a code source or a dynamic action definition no longer risks modifying an object of another project that happens to have the same name. The existing object is now resolved within the caller’s own project (by name, namespace and project id) before writing, and an ownership check is applied, consistently with the other definition types. These resources are admin-only by default policy, so this is a hardening / data-integrity fix rather than a cross-tenant escalation.

  • update_workbook now verifies ownership before writing. Previously it was only protected indirectly (a workbook update cascades into the workflow-definition update, which was guarded); making the workflow lookup project-scoped removed that incidental protection, so the guard is now applied to the workbook itself.

  • Fixed a retried “with-items” task with “concurrency” running some items several times and skipping others, so that the task result contained duplicated values. See bug 1892043.

  • A workflow whose output exceeds [engine]/execution_field_size_limit_kb now ends in ERROR state with the size error in its output. Previously it was marked as SUCCESS with an empty output. When output-on-error does not fit into the limit, it is dropped so that the error message can still be stored. See bug 1796592.

  • Trust lookups and deletions now use the keystone configured in [keystone_authtoken] auth_url, the same one trusts are created in, instead of the first identity endpoint of the service catalog. This fixes “Could not find trust” errors and leaked trusts in deployments with one keystone per region. See LP#1862940.

  • A named lock is now released when an engine error is raised while it is held. Previously, the lock row could stay in the named_locks table when the error was handled by the caller, for example when a task with a wait-before policy failed to evaluate its action input.

  • Rerunning a task no longer creates a new execution for a downstream task that it had already triggered and that failed or was cancelled. That execution is rerun instead, so the workflow can succeed. See https://launchpad.net/bugs/1815190.

  • Fixed resuming a workflow that was paused again while it was starting an existing task, for example a task paused by “pause-before”. The task is now run instead of being duplicated by a new task execution.

  • Fixed a workflow staying RUNNING forever after resume when all the inbound tasks of a “join” task completed while the workflow was paused. See https://launchpad.net/bugs/1893506.

  • A task iteration delayed by the retry or wait-before policy no longer runs while its workflow is paused. The task is put back in IDLE state and runs again when the workflow is resumed. A join task delayed by retry also waits for the resume. If the workflow is cancelled or failed, the delayed task is cancelled or failed too instead of running, and the retry policy no longer schedules new iterations. Cancelling a paused workflow now also cancels the tasks it parked in IDLE, including tasks stopped by pause-before, so they no longer block a later rerun of the workflow. See bug 1769012.

  • A task with a retry policy no longer fails when its publish clause can’t be evaluated on an intermediate result that the retry policy then discards (e.g. with continue-on). The error is only raised if the task actually completes with that result. See bug 1878167.

  • DSL schema validation errors are now short and say where the problem is, for example Invalid DSL in task 't1': Additional properties are not allowed ('on-sucess' was unexpected). They no longer include the whole JSON schema and the input, which could be thousands of lines.

  • [action_std_http] denied_cidrs could be bypassed with addresses that reach a denied host under another form, and these are now refused:

    • the unspecified addresses 0.0.0.0/8 and ::/128, which Linux connects to the local host, are added to the default deny-list. Deployments that set denied_cidrs explicitly do not get the new defaults and may add both ranges to their list;

    • an IPv4-mapped IPv6 address such as ::ffff:127.0.0.1 is checked through the IPv4 address it embeds;

    • a NAT64 address of the well-known prefix 64:ff9b::/96 is checked through the IPv4 address it embeds, which a NAT64 gateway connects to: 64:ff9b::a9fe:a9fe reaches 169.254.169.254.

  • A task that hits its timeout policy is no longer retried by its retry policy, and a retry iteration that was already scheduled when the timeout fired no longer runs the task again. The timed out task now stays in ERROR. See bug 1767352.

  • Creating a cron trigger or an event trigger now returns HTTP 403 with a clear message instead of HTTP 500 when Keystone refuses to create the trust. This happens for example when the request uses a trust-scoped token whose trust does not have allow_redelegation enabled. See bug 1721508.

  • Listing action executions with the params or task_name filter, or tasks with the result, reset or env filter, now returns 400 with a clear message instead of 500. These fields are not stored on the resource, so the filters never worked. They are removed from the API reference.

  • Fixed a rerun of a “with-items” task getting stuck in RUNNING when its list of items became shorter than in the previous run, for example after a rerun with a different environment.