Logging API for firewall-rules

This document describes MidoNet’s implementation of logging for firewall rules.

FWaaS v2.0 2 will be implemented in newton, However, first of all we implement MidoNet for mitaka with FWaaS v1.0 because development schedule between networking-midonet and neutron does not match.

Note that eventually we aim to unify this spec and spec for upstream 1. Thus, this spec is written following spec for upstream. In addition, we should keep on watch spec for upstream because the spec have not completed yet.

Problem Description

Operator wants to

• monitor network traffic in system to detect illegal traffic, or to solve unexpected communication error across L3.

• pass audit for system.

Tenant wants to

• monitor network traffic in tenant to detect illegal traffic, or to solve unexpected communication error across L3.

• pass audit for tenant

In Neutron, traffic accepted/denied on routers are managed by FWaaS. However, logging is currently a missing feature in FWaaS. Thus, requirements above cannot be satisfied.

Proposed Change

The scope of this spec:

• logging API for operator and tenant.

• logging format for operator and tenant.

How tenants can consume outputted logs are out of scope. Logging format that will be sent to tenants are out of scope. Where the logs are generated is also out of scope.

Plugin

Add ‘logging-resource’ extension alias in the supported extension aliases list.

MidoNet plugin implement the CRUD methods for logging resource and firewall log objects.

Expected API behavior

The events related to firewall rules will collect:

1. ACCEPT event

2. DROP event

Operators and tenants can specify what kind of events they want to log. Note that only logging of firewall rules that are created explicitly are gathered.

1. ACCEPT/DROP or ALL (collect all ACCEPT/DROP events of firewalls).

2. firewall uuid.

REST API

In this spec, two resources are newly defined.

LoggingResource is root resource for logging. This model defines policy of logging. e.g. how can we see logs, where logs are going to be collected.

FirewallLog is logging resource for specific resource. Though this spec proposes only FWaaS, many logging resources for specific resource will be supported in the future. (e.g. security group)

In addition, firewall can be associated with multiple logging resource only when the tenant_ids are different to allow operators and tenants to specify same firewall as logging resource. This allows operators to gather log in all of system.

Note that there is no difference in outputted format between tenant’s logging resource and operator’s logging resource. If operator wants to operate description above, some validations are needed to solve why the log is outputted.

The rough sketch that contains future consideration is following;

TenantA---------LoggingResourceA------FirewallLogA-----------
                       |                                    |
                       |--------------SecurityGroupLogA     |------firewallA
                                                            |
AdminTenant-----LoggingResourceB------FirewallLogB-----------
                       |
                       |--------------FirewallLogC
                       |
                       |--------------SecurityGroupLogB

LoggingResource

Attribute Name	Type	CRUD	Required	Description
id	string (UUID)	CR	generated	ID of the LoggingResource
name	string	CRU	No	Defined LoggingResource name
description	string	CRU	No	Description for the LoggingResource
tenant_id	string	CR	No	Tenant ID of LoggingResource object owner
enabled	string	CRU	No	Enable/disable for the LoggingResource. log is gathered only when this flag is enable.

FirewallLog

Attribute Name	Type	CRUD	Required	Description
id	string (UUID)	CR	generated	ID of the FirewallLog
description	string	CRU	No	Description for FirewallLog
tenant_id	string	CR	No	Tenant ID of FirewallLog object owner
fw_event	string	CRU	No	Event of firewall. ACCEPT/DROP/ALL can be specified. ALL is set as default.
firewall_id	string (UUID)	CR	Yes	ID of firewall instance

API list is as follows. Note that api path prefix ‘logging’ may conflict with upstream. However, we keep this prefix to keep consitency with upstream. To avoid confusion, we should do following things.

• Separate our plugin configuration from upstream.

• Note to user that not to use both plugins together.

Object	URI	Type
logging-resource	/logging/logging_resources	POST
logging-resource	/logging/logging_resources	GET
logging-resource	/logging/logging_resources/{id}	GET
logging-resource	/logging/logging_resources/{id}	DELETE
logging-resource	/logging/logging_resources/{id}	PUT
firewall-log	/logging/logging_resources/{id}/firewall_logs	POST
firewall-log	/logging/logging_resources/{id}/firewall_logs	GET
firewall-log	/logging/logging_resources/{id}/firewall_logs/{id}	GET
firewall-log	/logging/logging_resources/{id}/firewall_logs/{id}	DELETE
firewall-log	/logging/logging_resources/{id}/firewall_logs/{id}	PUT

REST API Examples

To Create a LoggingResource to manage security event log, following API can be used:

JSON Request

POST /v2.0/logging/logging_resources
{
    "logging_resource": {
        "name": "firewall_log",
        "description": "Get traffic flow of firewall",
        "enabled": True
     }
}

Response:

Response:
{
   "logging_resource": {
       "id": "46ebaec0-0570-43ac-82f6-60d2b03168c4",
       "tenant_id": "8d4c70a21fed4aeba121a1a429ba0d04",
       "name": "firewall_log",
       "description": "Get traffic flow of firewall",
       "enabled": True
   }
}

To Create a FirewallLog to collect security event of the firewall, following API can be used:

JSON Request

POST /v2.0/logging/logging_resources/46ebaec0-0570-43ac-82f6-60d2b03168c4/firewall_logs
{
    "firewall_log": {
        "description": "Collecting all traffic passing the firewall",
        "fw_event": "ALL",
        "firewall_id: "21aeda2a-a52f-4e81-9e64-7edeb59fa25b"
    }
}

Response:

{
"firewall_log": {
    "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
    "tenant_id": "8d4c70a21fed4aeba121a1a429ba0d04",
    "logging_resource_id": "46ebaec0-0570-43ac-82f6-60d2b03168c4",
    "description": "Collecting all traffic passing the firewall",
    "fw_event": "ALL",
    "firewall_id": "21aeda2a-a52f-4e81-9e64-7edeb59fa25b"
    }
}

REST API Impact

The new resources:

LOGGING_PREFIX = '/logging'
FW_EVENT_ACCEPT = 'ACCEPT'
FW_EVENT_DROP = 'DROP'
FW_EVENT_ALL = 'ALL'
FW_EVENTS = [FW_EVENT_ACCEPT, FW_EVENT_DROP, FW_EVENT_ALL]
LOG_COMMON_FIELDS = {
    'id': {'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None},
           'is_visible': True, 'primary_key': True},
    'tenant_id': {'allow_post': True, 'allow_put': False,
                  'required_by_policy': True, 'is_visible': True},
    'logging_resource_id': {'allow_post': False, 'allow_put': False,
                            'is_visible': True}
}

RESOURCE_ATTRIBUTE_MAP = {
    'logging_resources': {
        'id': {'allow_post': False, 'allow_put': False,
               'validate': {'type:uuid': None}, 'is_visible': True,
               'primary_key': True},
        'tenant_id': {'allow_post': True, 'allow_put': False,
                      'required_by_policy': True, 'is_visible': True},
        'name': {'allow_post': True, 'allow_put': True,
                 'validate': {'type:string': attr.NAME_MAX_LEN},
                 'default': '', 'is_visible': True},
        'description': {'allow_post': True, 'allow_put': True,
                        'validate': {'type:string': attr.LONG_DESCRIPTION_MAX_LEN},
                        'default': '', 'is_visible': True},
        'enabled': {'allow_post': True, 'allow_put': True,
                    'is_visible': True, 'default': False,
                    'convert_to': attr.convert_to_boolean},
        'firewall_logs': {'allow_post': False, 'allow_put': False,
                          'is_visible': True}
    }
}

SUB_RESOURCE_ATTRIBUTE_MAP = {
    'firewall_logs': {
        'parent': {'collection_name': 'logging_resources',
                   'member_name': 'logging_resource'},
        'parameters': dict((LOG_COMMON_FIELDS),
                      **{
                        'description': {
                            'allow_post': True, 'allow_put': True,
                            'validate': {'type:string': None},
                            'default': None, 'is_visible': True},
                        'firewall_id': {
                            'allow_post': True, 'allow_put': False,
                            'is_visible': True,
                            'validate': {'type:uuid': None}},
                        'fw_event': {
                            'allow_post': True, 'allow_put': True,
                            'is_visible': True,
                            'validate': {'type:values': FW_EVENT},
                            'default': 'ALL'}
                      })
    },
}

Logging format

Following items can be shown as follows. Eventually, we catch up neutron behavior that agent collects logs and sends logs to specified location from user. Therefore, outputted items should be unified with neutron after supporting function in neutron.

Item	Description
tenant_id	Tenant ID of targeted firewall
timestamp	Time of the event is happened The time is based on ISO8601, time zone is UTC
firewall UUID	UUID of neutron firewall
firewall rule UUID	UUID of neutron firewall rule
router UUID	UUID of neutron router
source IP address	Source IP address of the communication
destination IP address	Destination IP address of the communication
source L4 port	Source L4 port of the communication
destination L4 port	Destination L4 port of the communication
protocol	IANA protocol number
action	ACCEPT/DROP

Logging out location

Currently, operators can only access directly to file on host that midolman is running to consume log-data. File location has format: /var/log/midolman/logging/fw-<firewall-log-uuid>.log How generated log files are sent to tenants is up to the operator. Backend implementation and/or log collector are expected to handle log rotation. In the case with MidoNet, log rotation policy can be configured using its configuration tool.

DB Model impact

To avoid competition of table name with upstream, we add specific initial to head of table names. Note that upstream DB will be reused in newton or later and DB in networking-midonet will be deleted.

The LoggingResource model has the following attributes:

midonet_logging_resources

Attribute Name	Type	Access	Default Value	Validation/ Conversion	Description
id	uuid	RO	generated	uuid	Identity
tenant_id	uuid	RO	N/A	uuid	Id of tenant that created this LoggingResource
name	string	RW	N/A	none	LoggingResource name
description	string	RW	N/A	none	LoggingResource description
enabled	bool	RW	False	Boolean	Enable/disable log

The FirewallLog model would look like:

midonet_firewall_logs

Attribute Name	Type	Access	Default Value	Validation/ Conversion	Description
id	uuid	RO	generated	uuid	Identity
logging_resource_id	uuid	RO	N/A	uuid	LoggingResource UUID
tenant_id	uuid	RO	generated	uuid	Tenant creates logging
description	string	RW	N/A	none	FirewallLogging description
fw_event	enum	RW	N/A	enum	ACCEPT/DROP & ALL (collect all ACCEPT/DROP events)
firewall_id	uuid	RW(No update)	N/A	uuid	Firewalls UUID is enabled logging

Quota

Firewall log is managed by Quota. Default value of firewall log is 10 that is same number as firewall. Basically, both Quota value for firewall and firewall log should be aligned.

CLI Impact

Additional methods will be added to python-neutronclient to create, update, delete, list, get logging resource and firewall logging.

Checking support resource logging

For logging resource:

neutron logging-create --name <logging-resource-name>
                       [--enable <True/False>]
                       [--description <logging-resource-description>]
neutron logging-list
neutron logging-update <logging-resource-name-or-id>
                       [--name ...]
                       [--description ...]
                       [--enable <True/False>]
neutron logging-show <logging-resource-name-or-id>
neutron logging-delete <logging-resource-name-or-id>

For firewalls logging:

neutron logging-firewall-create <logging-resource-name-or-id> <firewall-id>
                                [--description <firewall-log description>]
                                [--fw-event <ACCEPT/DROP/ALL>]
neutron logging-firewall-list <logging-resource-name-or-id>
neutron logging-firewall-update <logging-resource-name-or-id> <firewall-log-id>
                                [--description ...]
                                [--fw-event ...]
neutron logging-firewall-show <logging-resource-name-or-id> <firewall-log-id>
neutron logging-firewall-delete <logging-resource-name-or-id> <firewall-log-id>

Other Deployer Impact

Set quota for firewall log in quotas section of neutron.conf.

quota_firewall_log = 10

References

1

http://docs-draft.openstack.org/09/203509/41/check/gate-neutron-specs-docs/34a11fa//doc/build/html/specs/newton/logging-API-for-security-group-rules.html

2

https://github.com/openstack/neutron-specs/blob/master/specs/newton/fwaas-api-2.0.rst