WSGI Server Process Management¶
Graceful Shutdowns with
Swift has always supported graceful WSGI server shutdown via
This causes the manager process to fall out of its
ensure-all-workers-are-running loop, close all workers’ listen sockets,
and exit. Closing the listen sockets causes all new
accept calls to
fail, but does not impact any established connections.
The workers are re-parented, likely to PID 1, and are discoverable with
swift-orphans. When the
accept call fails, it waits for the
GreenPool to complete, then exits. Each worker
continues processing the current request, then closes the connection.
Note that clients will get connection errors if they try to re-use a
connection for further requests.
Prior to the introduction of seamless reloads (see below), a common reload strategy was to perform a graceful shutdown followed by a fresh service start.
Seamless Reloads with
Beginning with Swift 2.24.0, WSGI servers support seamless reloads via
SIGUSR1. This allows servers to restart to pick up configuration or
code changes while being minimally-disruptive to clients. The process
is as follows:
Manager process receives
USR1signal. This causes the process to fall out of its loop ensuring that all workers are running and instead begin reloading. The workers continue servicing client requests as long as their listen sockets remain open.
Manager process forks. The new child knows about all the existing workers and their listen sockets; it will be responsible for closing the old worker listen sockets so they stop accepting new connections.
Manager process re-exec’s itself. It picks up new configuration and code while maintaining the same PID as the old manager process. At this point only the socket-closer is tracking the old workers, but everything (including old workers) remains a child of the new manager process. As a result, old workers are not discoverable with
swift-oldiesmay be useful, but will also find the manager process.
New manager process forks off new workers, each with its own listen socket. Once all workers have started and can accept new connections, the manager notifies the socket-closer via a pipe. The socket-closer closes the old worker listen sockets so they stop accepting new connections, then exits.
Old workers continue servicing any in-progress connections, while new connections are picked up by new workers. Once an old worker completes all of its oustanding requests, it exits.
All old workers have now exited. Only new code and configs are in use.
Beginning with Swift 2.33.0, a new
swift-reload helper is included
to help validate the reload process. Given a PID, it will
Validate that the PID seems to belong to a Swift WSGI server manager process,
Check that the config file used by that PID is currently valid,
USR1signal to initiate a reload, and
Wait for the new workers to come up (indicating the reload is complete) before exiting.