Notifications in Masakari

Similar to other OpenStack services Masakari emits notifications to the message bus with the Notifier class provided by oslo.messaging-doc. From the notification consumer point of view a notification consists of two parts: an envelope with a fixed structure defined by oslo.messaging and a payload defined by the service emitting the notification. The envelope format is the following:

{
    "priority": <string, selected from a predefined list by the sender>,
    "event_type": <string, defined by the sender>,
    "timestamp": <string, the isotime of when the notification emitted>,
    "publisher_id": <string, defined by the sender>,
    "message_id": <uuid, generated by oslo>,
    "payload": <json serialized dict, defined by the sender>
}
oslo.messaging provides below choices of notification drivers:

Driver

Description

messaging

Send notifications using the 1.0 message format

messagingv2

Send notifications using the 2.0 message format (with a message envelope)

routing

Configurable routing notifier (by priority or event_type)

log

Publish notifications via Python logging infrastructure

test

Store notifications in memory for test verification

noop

Disable sending notifications entirely

So notifications can be completely disabled by setting the following in Masakari configuration file:

[oslo_messaging_notifications]
driver = noop

Masakari supports only Versioned notifications.

Versioned notifications

Masakari code uses the masakari.rpc.get_notifier call to get a configured oslo.messaging Notifier object and it uses the oslo provided functions on the Notifier object to emit notifications. The configuration of the returned Notifier object depends on the parameters of the get_notifier call and the value of the oslo.messaging configuration options driver and topics. The versioned notification the the payload is not a free form dictionary but a serialized oslo.versionedobjects-doc.

For example the wire format of the segment.update notification looks like the following:

{
    "event_type": "api.update.segments.start",
    "timestamp": "2018-11-27 14:32:20.396940",
    "payload": {
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "recovery_method": "auto",
            "name": "test",
            "service_type": "compute",
            "id": 877,
            "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
        }, "
        "masakari_object.version": "1.0",
        "masakari_object.namespace": "masakari"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:test-virtualbox",
    "message_id": "e6322900-025d-4dd6-a3a1-3e0e1e9badeb"
}

The serialized oslo versionedobject as a payload provides a version number to the consumer so the consumer can detect if the structure of the payload is changed. Masakari provides the following contract regarding the versioned notification payload:

  • the payload version defined by the masakari_object.version field of the payload will be increased only if the syntax or the semantics of the masakari_object.data field of the payload is changed.

  • a minor version bump indicates a backward compatible change which means that only new fields are added to the payload so a well written consumer can still consume the new payload without any change.

  • a major version bump indicates a backward incompatible change of the payload which can mean removed fields, type change, etc in the payload.

  • there is an additional field ‘masakari_object.name’ for every payload besides ‘masakari_object.data’ and ‘masakari_object.version’. This field contains the name of the Masakari internal representation of the payload type. Client code should not depend on this name.

Existing versioned notifications

  • This provides the list of existing versioned notifications with sample payloads.

Event type Notification class Payload class Sample
error.exception ExceptionNotification ExceptionPayload 
{
    "event_type": "segment.create.error",
    "message_id": "e5405591-1d19-4a8c-aa92-4d551165d863",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": {
                "masakari_object.data": {
                    "exception": "DBError",
                    "exception_message": "(pymysql.err.Internal Error) (1054, u\"Unknown column 'name' in 'field list'\" ...",
                    "function_name": "raise_mysql_exception",
                    "module_name": "pymysql.err",
                    "traceback": "Traceback (most recent call last):\n  File \"/opt/stack/masakari/masakari/ha/api.py\", line ..."
                },
                "masakari_object.name": "ExceptionPayload",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "name": "testT6",
            "recovery_method": "auto",
            "service_type": "compute"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "ERROR",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-28 14:24:27.902437"
}
create.host.end HostApiNotification HostApiPayload 
{
    "event_type": "host.create.end",
    "message_id": "e437834a-73e1-4c47-939a-83f6aca2e7ac",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "id": 70,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE",
            "uuid": "d6a2d900-1977-48fd-aa52-ad7a41fc068b"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:09:30.737034"
}
create.host.start HostApiNotification HostApiPayload 
{
    "event_type": "host.create.start",
    "message_id": "0ed836cc-353a-40bc-b86b-d89e6632d838",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:09:30.716747"
}
delete.host.end HostApiNotification HostApiPayload 
{
    "event_type": "host.delete.end",
    "message_id": "64d61bcf-c875-41c3-b795-19a076f6de96",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE",
            "uuid": "3d8d1751-9cab-4a48-8801-96f102200077"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:35:09.882636"
}
delete.host.start HostApiNotification HostApiPayload 
{
    "event_type": "host.delete.start",
    "message_id": "b5914f94-99dd-42fa-aaf3-3cedacda6b67",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "id": 71,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE",
            "uuid": "3d8d1751-9cab-4a48-8801-96f102200077"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:31:47.451466"
}
update.host.end HostApiNotification HostApiPayload 
{
    "event_type": "host.update.end",
    "message_id": "e7f85d49-7d02-4713-b90b-433f8e447558",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "id": 70,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE",
            "uuid": "d6a2d900-1977-48fd-aa52-ad7a41fc068b"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:13:25.361394"
}
update.host.start HostApiNotification HostApiPayload 
{
    "event_type": "host.update.start",
    "message_id": "d1a3ae84-7f41-4884-bc3f-fa34c7cd1424",
    "payload": {
        "masakari_object.data": {
            "control_attributes": "TEST",
            "failover_segment": {
                "masakari_object.data": {
                    "created_at": "2018-11-27T09:26:30Z",
                    "deleted": false,
                    "deleted_at": null,
                    "description": null,
                    "id": 877,
                    "name": "test",
                    "recovery_method": "auto",
                    "service_type": "compute",
                    "updated_at": "2018-11-27T09:54:50Z",
                    "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
                },
                "masakari_object.name": "FailoverSegment",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "failover_segment_id": "89597691-bebd-4860-a93e-1b6e9de34b9e",
            "fault": null,
            "id": 70,
            "name": "fake-mini",
            "on_maintenance": false,
            "reserved": false,
            "type": "COMPUTE",
            "uuid": "d6a2d900-1977-48fd-aa52-ad7a41fc068b"
        },
        "masakari_object.name": "HostApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:13:25.298007"
}
create.notification.end NotificationApiNotification NotificationApiPayload 
{
    "event_type": "notification.create.end",
    "message_id": "500447b9-4797-4090-9189-b56bc3521b75",
    "payload": {
        "masakari_object.data": {
            "fault": null,
            "generated_time": "2017-06-13T15:34:55Z",
            "id": 1,
            "notification_uuid": "e6b1996f-7792-4a65-83c3-23f2d4721eb0",
            "payload": {
                "process_name": "nova-compute"
            },
            "source_host_uuid": "d4ffe3a4-b2a8-41f3-a2b0-bae3b06fc1a3",
            "status": "new",
            "type": "VM"
        },
        "masakari_object.name": "NotificationApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:46:25.496514"
}
create.notification.start NotificationApiNotification NotificationApiPayload 
{
    "event_type": "notification.create.start",
    "message_id": "5e2e4699-0bbd-4583-b1e2-a87c458f84eb",
    "payload": {
        "masakari_object.data": {
            "fault": null,
            "generated_time": "2017-06-13T15:34:55Z",
            "payload": {
                "process_name": "nova-compute"
            },
            "source_host_uuid": "d4ffe3a4-b2a8-41f3-a2b0-bae3b06fc1a3",
            "status": "new",
            "type": "VM"
        },
        "masakari_object.name": "NotificationApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 13:46:23.060352"
}
process.notification.end NotificationApiNotification NotificationApiPayload 
{
    "event_type": "notification.process.end",
    "message_id": "c081eb25-7450-4fa2-bb19-ae6d4466e14e",
    "payload": {
        "masakari_object.data": {
            "fault": null,
            "generated_time": "2017-06-13T15:34:55Z",
            "id": 47,
            "notification_uuid": "15505a8c-8856-4f3d-9747-55b6e899c0f5",
            "payload": {
                "process_name": "nova-compute"
            },
            "source_host_uuid": "6bfaf80d-7592-4ea8-ad12-60d45476d056",
            "status": "ignored",
            "type": "VM"
        },
        "masakari_object.name": "NotificationApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-engine:fake-mini",
    "timestamp": "2018-12-20 05:26:05.075917"
}
process.notification.error NotificationApiNotification NotificationApiPayload 
{
    "event_type": "notification.process.error",
    "message_id": "5f3c9705-b3fb-41f9-a4e0-4868db93178c",
    "payload": {
        "masakari_object.data": {
            "fault": {
                "masakari_object.data": {
                    "exception": "str",
                    "exception_message": "Failed to execute process recovery workflow.",
                    "function_name": "_handle_notification_type_process",
                    "module_name": "masakari.engine.manager",
                    "traceback": "Traceback (most recent call last):\n  File \"/opt/stack/masakari/masakari/engine/manager.py\", line ..."
                },
                "masakari_object.name": "ExceptionPayload",
                "masakari_object.namespace": "masakari",
                "masakari_object.version": "1.0"
            },
            "generated_time": "2017-06-13T15:34:55Z",
            "id": 51,
            "notification_uuid": "0adb94e0-8283-4702-9793-186d4ed914e8",
            "payload": {
                "process_name": "nova-compute"
            },
            "source_host_uuid": "6bfaf80d-7592-4ea8-ad12-60d45476d056",
            "status": "running",
            "type": "PROCESS"
        },
        "masakari_object.name": "NotificationApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "ERROR",
    "publisher_id": "masakari-engine:fake-mini",
    "timestamp": "2018-12-20 06:21:19.315761"
}
process.notification.start NotificationApiNotification NotificationApiPayload 
{
    "event_type": "notification.process.start",
    "message_id": "285be756-ac29-4b78-9e2b-9756f5077012",
    "payload": {
        "masakari_object.data": {
            "fault": null,
            "generated_time": "2017-06-13T15:34:55Z",
            "id": 47,
            "notification_uuid": "15505a8c-8856-4f3d-9747-55b6e899c0f5",
            "payload": {
                "process_name": "nova-compute"
            },
            "source_host_uuid": "6bfaf80d-7592-4ea8-ad12-60d45476d056",
            "status": "new",
            "type": "VM"
        },
        "masakari_object.name": "NotificationApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-engine:fake-mini",
    "timestamp": "2018-12-20 05:26:05.002421"
}
create.segment.end SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.create.end",
    "message_id": "b8478b31-5943-4495-8867-e8291655f660",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "id": 850,
            "name": "test",
            "recovery_method": "auto",
            "service_type": "compute",
            "uuid": "5cce639c-da08-4e78-b615-66c88aa49d50"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-22 09:25:12.813483"
}
create.segment.start SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.create.start",
    "message_id": "e44cb15b-dcba-409e-b0e1-9ee103b9a168",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "name": "test",
            "recovery_method": "auto",
            "service_type": "compute"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-22 09:25:12.393979"
}
delete.segment.end SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.delete.end",
    "message_id": "00184d05-7a96-4021-b44e-03912a6c0b0d",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "name": "test2",
            "recovery_method": "auto",
            "service_type": "compute",
            "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 14:36:07.457369"
}
delete.segment.start SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.delete.start",
    "message_id": "e6c32ecb-eacc-433d-ba8c-6390ea3da6d2",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "id": 877,
            "name": "test2",
            "recovery_method": "auto",
            "service_type": "compute",
            "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 14:36:07.442538"
}
update.segment.end SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.update.end",
    "message_id": "3fbe50a5-9175-4161-85f0-e502f9024657",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "id": 877,
            "name": "test2",
            "recovery_method": "auto",
            "service_type": "compute",
            "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 14:32:20.417745"
}
update.segment.start SegmentApiNotification SegmentApiPayload 
{
    "event_type": "segment.update.start",
    "message_id": "e6322900-025d-4dd6-a3a1-3e0e1e9badeb",
    "payload": {
        "masakari_object.data": {
            "description": null,
            "fault": null,
            "id": 877,
            "name": "test",
            "recovery_method": "auto",
            "service_type": "compute",
            "uuid": "89597691-bebd-4860-a93e-1b6e9de34b9e"
        },
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.namespace": "masakari",
        "masakari_object.version": "1.0"
    },
    "priority": "INFO",
    "publisher_id": "masakari-api:fake-mini",
    "timestamp": "2018-11-27 14:32:20.396940"
}