freezer.scheduler.coordination module

Coordination layer for running freezer-scheduler as a cluster.

Several schedulers configured with the same client_id form a cluster. To freezer-api they look like a single client and therefore all poll the same set of jobs; coordination decides which member actually runs each job so backups are not duplicated.

The mechanism is entirely scheduler-side and uses tooz:

• The tooz group is named after the shared client_id (the cluster identity). Members are told apart by a unique per-process member id (<hostname>-<pid>), the hostname being kept only for readability in get_members() output and logs. tooz heartbeats the membership at the cadence the backend session timeout requires (tunable via the backend URL, e.g. redis://host:6379?timeout=30).

• Job ownership is decided with a consistent hash ring built once per poll tick from the live group members (see refresh()). Because every member feeds the same member set and the same job id into the same ring, they all agree on the owner with no messaging.

• Before running a job the owner takes a per-job distributed lock, which guards the brief window when membership is changing.

All coordination calls fail safe: if the backend is unreachable ownership cannot be determined and the lock cannot be taken, so jobs are skipped rather than run without coordination (which would risk duplicate backups). refresh() also self-heals: it (re)connects and (re)joins the group, so a backend that was down at startup or a member evicted during an outage recovers on a later poll tick without a restart.

class freezer.scheduler.coordination.SchedulerCoordinator(client_id: str, backend_url: str)

Bases: object

Wraps tooz group membership, hash-ring ownership and job locks.

is_owner(job_id: str) → bool

Return True if this member owns job_id on the current ring.

Uses the ring built by the last refresh(); without one (backend unreachable, not a group member) everything is skipped.

job_lock(job_id: str) → Iterator[bool]

Yield True if the per-job distributed lock was acquired.

Fail-safe: if the lock is held elsewhere or the backend is unreachable, yields False and the caller must not run the job. The lock is released automatically when the block exits (and, if this member dies, when its tooz heartbeat expires).

property member_id: str

refresh() → None

Rebuild the hash ring from live members; call once per tick.

Self-healing: (re)connects and (re)joins as needed, covering a backend unreachable at startup and eviction after an outage. Fail-safe: on any error no ring is kept, so is_owner() returns False and jobs are skipped rather than run uncoordinated.

start() → None

Connect to the backend and join the group.

Idempotent: a repeated call (e.g. from the daemon’s restart-on-error loop) is a no-op once connected, so connections and heartbeat threads cannot accumulate. Heartbeating is handled by tooz itself (start_heart).

stop() → None