Logging API for firewall-rules

This document describes MidoNet’s implementation of logging for firewall rules.

FWaaS v2.0 will be implemented in newton, However, first of all we implement MidoNet for mitaka with FWaaS v1.0 because development schedule between networking-midonet and neutron does not match.

Note that eventually we aim to unify this spec and spec for upstream [1]. Thus, this spec is written following spec for upstream. In addition, we should keep on watch spec for upstream because the spec have not completed yet.

Problem Description

Operator wants to
  • monitor network traffic in system to detect illegal traffic, or to solve unexpected communication error across L3.
  • pass audit for system.
Tenant wants to
  • monitor network traffic in tenant to detect illegal traffic, or to solve unexpected communication error across L3.
  • pass audit for tenant

In Neutron, traffic accepted/denied on routers are managed by FWaaS. However, logging is currently a missing feature in FWaaS. Thus, requirements above cannot be satisfied.

Proposed Change

The scope of this spec:

  • logging API for operator and tenant.
  • logging format for operator and tenant.

How tenants can consume outputted logs are out of scope. Logging format that will be sent to tenants are out of scope. Where the logs are generated is also out of scope.

Plugin

Add ‘logging-resource’ extension alias in the supported extension aliases list.

MidoNet plugin implement the CRUD methods for logging resource and firewall log objects.

Expected API behavior

The events related to firewall rules will collect:

  1. ACCEPT event
  2. DROP event

Operators and tenants can specify what kind of events they want to log. Note that only logging of firewall rules that are created explicitly are gathered.

  1. ACCEPT/DROP or ALL (collect all ACCEPT/DROP events of firewalls).
  2. firewall uuid.

REST API

In this spec, two resources are newly defined.

LoggingResource is root resource for logging. This model defines policy of logging. e.g. how can we see logs, where logs are going to be collected.

FirewallLog is logging resource for specific resource. Though this spec proposes only FWaaS, many logging resources for specific resource will be supported in the future. (e.g. security group)

In addition, firewall can be associated with multiple logging resource only when the tenant_ids are different to allow operators and tenants to specify same firewall as logging resource. This allows operators to gather log in all of system.

Note that there is no difference in outputted format between tenant’s logging resource and operator’s logging resource. If operator wants to operate description above, some validations are needed to solve why the log is outputted.

The rough sketch that contains future consideration is following;

TenantA---------LoggingResourceA------FirewallLogA-----------
                       |                                    |
                       |--------------SecurityGroupLogA     |------firewallA
                                                            |
AdminTenant-----LoggingResourceB------FirewallLogB-----------
                       |
                       |--------------FirewallLogC
                       |
                       |--------------SecurityGroupLogB

LoggingResource

Attribute Name Type CRUD Required Description
id string (UUID) CR generated ID of the LoggingResource
name string CRU No Defined LoggingResource name
description string CRU No Description for the LoggingResource
tenant_id string CR No Tenant ID of LoggingResource object owner
enabled string CRU No Enable/disable for the LoggingResource. log is gathered only when this flag is enable.

FirewallLog

Attribute Name Type CRUD Required Description
id string (UUID) CR generated ID of the FirewallLog
description string CRU No Description for FirewallLog
tenant_id string CR No Tenant ID of FirewallLog object owner
fw_event string CRU No Event of firewall. ACCEPT/DROP/ALL can be specified. ALL is set as default.
firewall_id string (UUID) CR Yes ID of firewall instance

API list is as follows. Note that api path prefix ‘logging’ may conflict with upstream. However, we keep this prefix to keep consitency with upstream. To avoid confusion, we should do following things.

  • Separate our plugin configuration from upstream.
  • Note to user that not to use both plugins together.
Object URI Type
logging-resource /logging/logging_resources POST
logging-resource /logging/logging_resources GET
logging-resource /logging/logging_resources/{id} GET
logging-resource /logging/logging_resources/{id} DELETE
logging-resource /logging/logging_resources/{id} PUT
firewall-log /logging/logging_resources/{id}/firewall_logs POST
firewall-log /logging/logging_resources/{id}/firewall_logs GET
firewall-log /logging/logging_resources/{id}/firewall_logs/{id} GET
firewall-log /logging/logging_resources/{id}/firewall_logs/{id} DELETE
firewall-log /logging/logging_resources/{id}/firewall_logs/{id} PUT

REST API Examples

To Create a LoggingResource to manage security event log, following API can be used:

JSON Request

POST /v2.0/logging/logging_resources
{
    "logging_resource": {
        "name": "firewall_log",
        "description": "Get traffic flow of firewall",
        "enabled": True
     }
}

Response:

Response:
{
   "logging_resource": {
       "id": "46ebaec0-0570-43ac-82f6-60d2b03168c4",
       "tenant_id": "8d4c70a21fed4aeba121a1a429ba0d04",
       "name": "firewall_log",
       "description": "Get traffic flow of firewall",
       "enabled": True
   }
}

To Create a FirewallLog to collect security event of the firewall, following API can be used:

JSON Request

POST /v2.0/logging/logging_resources/46ebaec0-0570-43ac-82f6-60d2b03168c4/firewall_logs
{
    "firewall_log": {
        "description": "Collecting all traffic passing the firewall",
        "fw_event": "ALL",
        "firewall_id: "21aeda2a-a52f-4e81-9e64-7edeb59fa25b"
    }
}

Response:

{
"firewall_log": {
    "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
    "tenant_id": "8d4c70a21fed4aeba121a1a429ba0d04",
    "logging_resource_id": "46ebaec0-0570-43ac-82f6-60d2b03168c4",
    "description": "Collecting all traffic passing the firewall",
    "fw_event": "ALL",
    "firewall_id": "21aeda2a-a52f-4e81-9e64-7edeb59fa25b"
    }
}

REST API Impact

The new resources:

LOGGING_PREFIX = '/logging'
FW_EVENT_ACCEPT = 'ACCEPT'
FW_EVENT_DROP = 'DROP'
FW_EVENT_ALL = 'ALL'
FW_EVENTS = [FW_EVENT_ACCEPT, FW_EVENT_DROP, FW_EVENT_ALL]
LOG_COMMON_FIELDS = {
    'id': {'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None},
           'is_visible': True, 'primary_key': True},
    'tenant_id': {'allow_post': True, 'allow_put': False,
                  'required_by_policy': True, 'is_visible': True},
    'logging_resource_id': {'allow_post': False, 'allow_put': False,
                            'is_visible': True}
}

RESOURCE_ATTRIBUTE_MAP = {
    'logging_resources': {
        'id': {'allow_post': False, 'allow_put': False,
               'validate': {'type:uuid': None}, 'is_visible': True,
               'primary_key': True},
        'tenant_id': {'allow_post': True, 'allow_put': False,
                      'required_by_policy': True, 'is_visible': True},
        'name': {'allow_post': True, 'allow_put': True,
                 'validate': {'type:string': attr.NAME_MAX_LEN},
                 'default': '', 'is_visible': True},
        'description': {'allow_post': True, 'allow_put': True,
                        'validate': {'type:string': attr.LONG_DESCRIPTION_MAX_LEN},
                        'default': '', 'is_visible': True},
        'enabled': {'allow_post': True, 'allow_put': True,
                    'is_visible': True, 'default': False,
                    'convert_to': attr.convert_to_boolean},
        'firewall_logs': {'allow_post': False, 'allow_put': False,
                          'is_visible': True}
    }
}

SUB_RESOURCE_ATTRIBUTE_MAP = {
    'firewall_logs': {
        'parent': {'collection_name': 'logging_resources',
                   'member_name': 'logging_resource'},
        'parameters': dict((LOG_COMMON_FIELDS),
                      **{
                        'description': {
                            'allow_post': True, 'allow_put': True,
                            'validate': {'type:string': None},
                            'default': None, 'is_visible': True},
                        'firewall_id': {
                            'allow_post': True, 'allow_put': False,
                            'is_visible': True,
                            'validate': {'type:uuid': None}},
                        'fw_event': {
                            'allow_post': True, 'allow_put': True,
                            'is_visible': True,
                            'validate': {'type:values': FW_EVENT},
                            'default': 'ALL'}
                      })
    },
}

Logging format

Following items can be shown as follows. Eventually, we catch up neutron behavior that agent collects logs and sends logs to specified location from user. Therefore, outputted items should be unified with neutron after supporting function in neutron.

Item Description
tenant_id Tenant ID of targeted firewall
timestamp Time of the event is happened The time is based on ISO8601, time zone is UTC
firewall UUID UUID of neutron firewall
firewall rule UUID UUID of neutron firewall rule
router UUID UUID of neutron router
source IP address Source IP address of the communication
destination IP address Destination IP address of the communication
source L4 port Source L4 port of the communication
destination L4 port Destination L4 port of the communication
protocol IANA protocol number
action ACCEPT/DROP

Logging out location

Currently, operators can only access directly to file on host that midolman is running to consume log-data. File location has format: /var/log/midolman/logging/fw-<firewall-log-uuid>.log How generated log files are sent to tenants is up to the operator. Backend implementation and/or log collector are expected to handle log rotation. In the case with MidoNet, log rotation policy can be configured using its configuration tool.

DB Model impact

To avoid competition of table name with upstream, we add specific initial to head of table names. Note that upstream DB will be reused in newton or later and DB in networking-midonet will be deleted.

The LoggingResource model has the following attributes:

midonet_logging_resources

Attribute Name Type Access Default Value Validation/ Conversion Description
id uuid RO generated uuid Identity
tenant_id uuid RO N/A uuid Id of tenant that created this LoggingResource
name string RW N/A none LoggingResource name
description string RW N/A none LoggingResource description
enabled bool RW False Boolean Enable/disable log

The FirewallLog model would look like:

midonet_firewall_logs

Attribute Name Type Access Default Value Validation/ Conversion Description
id uuid RO generated uuid Identity
logging_resource_id uuid RO N/A uuid LoggingResource UUID
tenant_id uuid RO generated uuid Tenant creates logging
description string RW N/A none FirewallLogging description
fw_event enum RW N/A enum ACCEPT/DROP & ALL (collect all ACCEPT/DROP events)
firewall_id uuid RW(No update) N/A uuid Firewalls UUID is enabled logging

Quota

Firewall log is managed by Quota. Default value of firewall log is 10 that is same number as firewall. Basically, both Quota value for firewall and firewall log should be aligned.

CLI Impact

Additional methods will be added to python-neutronclient to create, update, delete, list, get logging resource and firewall logging.

Checking support resource logging

For logging resource:

neutron logging-create --name <logging-resource-name>
                       [--enable <True/False>]
                       [--description <logging-resource-description>]
neutron logging-list
neutron logging-update <logging-resource-name-or-id>
                       [--name ...]
                       [--description ...]
                       [--enable <True/False>]
neutron logging-show <logging-resource-name-or-id>
neutron logging-delete <logging-resource-name-or-id>

For firewalls logging:

neutron logging-firewall-create <logging-resource-name-or-id> <firewall-id>
                                [--description <firewall-log description>]
                                [--fw-event <ACCEPT/DROP/ALL>]
neutron logging-firewall-list <logging-resource-name-or-id>
neutron logging-firewall-update <logging-resource-name-or-id> <firewall-log-id>
                                [--description ...]
                                [--fw-event ...]
neutron logging-firewall-show <logging-resource-name-or-id> <firewall-log-id>
neutron logging-firewall-delete <logging-resource-name-or-id> <firewall-log-id>

Other Deployer Impact

Set quota for firewall log in quotas section of neutron.conf.

quota_firewall_log = 10