QoS is defined as the ability to guarantee certain network requirements like bandwidth, latency, jitter, and reliability in order to satisfy a Service Level Agreement (SLA) between an application provider and end users.
Network devices such as switches and routers can mark traffic so that it is handled with a higher priority to fulfill the QoS conditions agreed under the SLA. In other cases, certain network traffic such as Voice over IP (VoIP) and video streaming needs to be transmitted with minimal bandwidth constraints. On a system without network QoS management, all traffic will be transmitted in a “best-effort” manner making it impossible to guarantee service delivery to customers.
QoS is an advanced service plug-in. QoS is decoupled from the rest of the OpenStack Networking code on multiple levels and it is available through the ml2 extension driver.
Details about the DB models, API extension, and use cases are out of the scope of this guide but can be found in the Neutron QoS specification.
QoS supported rule types are now available as VALID_RULE_TYPES
in QoS rule types:
Any QoS driver can claim support for some QoS rule types
by providing a driver property called
supported_rules
, the QoS driver manager will recalculate rule types
dynamically that the QoS driver supports.
The following table shows the Networking back ends, QoS supported rules, and traffic directions (from the VM point of view).
Rule \ back end | Open vSwitch | SR-IOV | Linux bridge |
---|---|---|---|
Bandwidth limit | Egress \ Ingress | Egress (1) | Egress \ Ingress |
Minimum bandwidth | Egress | ||
DSCP marking | Egress | Egress |
Note
In the most simple case, the property can be represented by a simple Python list defined on the class.
For an ml2 plug-in, the list of supported QoS rule types and parameters is defined as a common subset of rules supported by all active mechanism drivers. A QoS rule is always attached to a QoS policy. When a rule is created or updated:
Valid DSCP mark values are even numbers between 0 and 56, except 2-6, 42, 44, and 50-54. The full list of valid DSCP marks is:
0, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 46, 48, 56
To enable the service on a cloud with the architecture described in Networking architecture, follow the steps below:
On the controller nodes:
Add the QoS service to the service_plugins
setting in
/etc/neutron/neutron.conf
. For example:
service_plugins = \
neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,
neutron.services.metering.metering_plugin.MeteringPlugin,
neutron.services.qos.qos_plugin.QoSPlugin
Optionally, set the needed notification_drivers
in the [qos]
section in /etc/neutron/neutron.conf
(message_queue
is the
default).
Optionally, in order to enable the floating IP QoS extension qos-fip
,
set the service_plugins
option in /etc/neutron/neutron.conf
to
include both router
and qos
. For example:
service_plugins = router, qos
In /etc/neutron/plugins/ml2/ml2_conf.ini
, add qos
to
extension_drivers
in the [ml2]
section. For example:
[ml2]
extension_drivers = port_security, qos
Edit the configuration file for the agent you are using and set the
extensions
to include qos
in the [agent]
section of the
configuration file. The agent configuration file will reside in
/etc/neutron/plugins/ml2/<agent_name>_agent.ini
where agent_name
is the name of the agent being used (for example openvswitch
).
For example:
[agent]
extensions = qos
On the network and compute nodes:
Edit the configuration file for the agent you are using and set the
extensions
to include qos
in the [agent]
section of the
configuration file. The agent configuration file will reside in
/etc/neutron/plugins/ml2/<agent_name>_agent.ini
where agent_name
is the name of the agent being used (for example openvswitch
).
For example:
[agent]
extensions = qos
Optionally, in order to enable QoS for floating IPs, set the extensions
option in the [agent]
section of /etc/neutron/l3_agent.ini
to
include fip_qos
. If dvr
is enabled, this has to be done for all the
L3 agents. For example:
[agent]
extensions = fip_qos
As rate limit doesn’t work on Open vSwitch’s internal
ports,
optionally, as a workaround, to make QoS bandwidth limit work on
router’s gateway ports, set ovs_use_veth
to True
in DEFAULT
section in /etc/neutron/l3_agent.ini
[DEFAULT] ovs_use_veth = True
Note
QoS currently works with ml2 only (SR-IOV, Open vSwitch, and linuxbridge are drivers enabled for QoS).
When using overlay networks (e.g., VxLAN), the DSCP marking rule only applies to the inner header, and during encapsulation, the DSCP mark is not automatically copied to the outer header.
In order to set the DSCP value of the outer header, modify the dscp
configuration option in /etc/neutron/plugins/ml2/<agent_name>_agent.ini
where <agent_name>
is the name of the agent being used
(e.g., openvswitch
):
[agent]
dscp = 8
In order to copy the DSCP field of the inner header to the outer header,
change the dscp_inherit
configuration option to true in
/etc/neutron/plugins/ml2/<agent_name>_agent.ini
where <agent_name>
is the name of the agent being used (e.g., openvswitch
):
[agent]
dscp_inherit = true
If the dscp_inherit
option is set to true, the previous dscp
option
is overwritten.
If projects are trusted to administrate their own QoS policies in
your cloud, neutron’s file policy.json
can be modified to allow this.
Modify /etc/neutron/policy.json
policy entries as follows:
"get_policy": "rule:regular_user",
"create_policy": "rule:regular_user",
"update_policy": "rule:regular_user",
"delete_policy": "rule:regular_user",
"get_rule_type": "rule:regular_user",
To enable bandwidth limit rule:
"get_policy_bandwidth_limit_rule": "rule:regular_user",
"create_policy_bandwidth_limit_rule": "rule:regular_user",
"delete_policy_bandwidth_limit_rule": "rule:regular_user",
"update_policy_bandwidth_limit_rule": "rule:regular_user",
To enable DSCP marking rule:
"get_policy_dscp_marking_rule": "rule:regular_user",
"create_dscp_marking_rule": "rule:regular_user",
"delete_dscp_marking_rule": "rule:regular_user",
"update_dscp_marking_rule": "rule:regular_user",
To enable minimum bandwidth rule:
"get_policy_minimum_bandwidth_rule": "rule:regular_user",
"create_policy_minimum_bandwidth_rule": "rule:regular_user",
"delete_policy_minimum_bandwidth_rule": "rule:regular_user",
"update_policy_minimum_bandwidth_rule": "rule:regular_user",
QoS policies are only created by admins with the default policy.json
.
Therefore, you should have the cloud operator set them up on
behalf of the cloud projects.
If projects are trusted to create their own policies, check the trusted
projects policy.json
configuration section.
First, create a QoS policy and its bandwidth limit rule:
$ openstack network qos policy create bw-limiter
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| description | |
| id | 5df855e9-a833-49a3-9c82-c0839a5f103f |
| is_default | False |
| name | bw-limiter |
| project_id | 4db7c1ed114a4a7fb0f077148155c500 |
| rules | [] |
| shared | False |
+-------------------+--------------------------------------+
$ openstack network qos rule create --type bandwidth-limit --max-kbps 3000 \
--max-burst-kbits 2400 --egress bw-limiter
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | egress |
| id | 92ceb52f-170f-49d0-9528-976e2fee2d6f |
| max_burst_kbps | 2400 |
| max_kbps | 3000 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
Note
The QoS implementation requires a burst value to ensure proper behavior of bandwidth limit rules in the Open vSwitch and Linux bridge agents. If you do not provide a value, it defaults to 80% of the bandwidth limit which works for typical TCP traffic.
Second, associate the created policy with an existing neutron port.
In order to do this, user extracts the port id to be associated to
the already created policy. In the next example, we will assign the
bw-limiter
policy to the VM with IP address 192.0.2.1
.
$ openstack port list
+--------------------------------------+-----------------------------------+
| ID | Fixed IP Addresses |
+--------------------------------------+-----------------------------------+
| 0271d1d9-1b16-4410-bd74-82cdf6dcb5b3 | { ... , "ip_address": "192.0.2.1"}|
| 88101e57-76fa-4d12-b0e0-4fc7634b874a | { ... , "ip_address": "192.0.2.3"}|
| e04aab6a-5c6c-4bd9-a600-33333551a668 | { ... , "ip_address": "192.0.2.2"}|
+--------------------------------------+-----------------------------------+
$ openstack port set --qos-policy bw-limiter \
88101e57-76fa-4d12-b0e0-4fc7634b874a
In order to detach a port from the QoS policy, simply update again the port configuration.
$ openstack port unset --qos-policy 88101e57-76fa-4d12-b0e0-4fc7634b874a
Ports can be created with a policy attached to them too.
$ openstack port create --qos-policy bw-limiter --network private port1
+-----------------------+--------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | unbound |
| binding_vnic_type | normal |
| created_at | 2017-05-15T08:43:00Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | |
| dns_assignment | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.10.4', subnet_id='292f8c1e-...' |
| id | f51562ee-da8d-42de-9578-f6f5cb248226 |
| ip_address | None |
| mac_address | fa:16:3e:d9:f2:ba |
| name | port1 |
| network_id | 55dc2f70-0f92-4002-b343-ca34277b0234 |
| option_name | None |
| option_value | None |
| port_security_enabled | False |
| project_id | 4db7c1ed114a4a7fb0f077148155c500 |
| qos_policy_id | 5df855e9-a833-49a3-9c82-c0839a5f103f |
| revision_number | 6 |
| security_group_ids | 0531cc1a-19d1-4cc7-ada5-49f8b08245be |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| trunk_details | None |
| updated_at | 2017-05-15T08:43:00Z |
+-----------------------+--------------------------------------------------+
You can attach networks to a QoS policy. The meaning of this is that any compute port connected to the network will use the network policy by default unless the port has a specific policy attached to it. Internal network owned ports like DHCP and internal router ports are excluded from network policy application.
In order to attach a QoS policy to a network, update an existing network, or initially create the network attached to the policy.
$ openstack network set --qos-policy bw-limiter private
Note
Configuring the proper burst value is very important. If the burst value is set too low, bandwidth usage will be throttled even with a proper bandwidth limit setting. This issue is discussed in various documentation sources, for example in Juniper’s documentation. Burst value for TCP traffic can be set as 80% of desired bandwidth limit value. For example, if the bandwidth limit is set to 1000kbps then enough burst value will be 800kbit. If the configured burst value is too low, achieved bandwidth limit will be lower than expected. If the configured burst value is too high, too few packets could be limited and achieved bandwidth limit would be higher than expected.
The created policy can be associated with an existing floating IP.
In order to do this, user extracts the floating IP id to be associated to
the already created policy. In the next example, we will assign the
bw-limiter
policy to the floating IP address 172.16.100.18
.
$ openstack floating ip list
+--------------------------------------+---------------------+------------------+------+-----+
| ID | Floating IP Address | Fixed IP Address | Port | ... |
+--------------------------------------+---------------------+------------------+------+-----+
| 1163d127-6df3-44bb-b69c-c0e916303eb3 | 172.16.100.9 | None | None | ... |
| d0ed7491-3eb7-4c4f-a0f0-df04f10a067c | 172.16.100.18 | None | None | ... |
| f5a9ed48-2e9f-411c-8787-2b6ecd640090 | 172.16.100.2 | None | None | ... |
+--------------------------------------+---------------------+------------------+------+-----+
$ openstack floating ip set --qos-policy bw-limiter d0ed7491-3eb7-4c4f-a0f0-df04f10a067c
In order to detach a floating IP from the QoS policy, simply update the floating IP configuration.
$ openstack floating ip set --no-qos-policy d0ed7491-3eb7-4c4f-a0f0-df04f10a067c
Or use the unset
action.
$ openstack floating ip unset --qos-policy d0ed7491-3eb7-4c4f-a0f0-df04f10a067c
Floating IPs can be created with a policy attached to them too.
$ openstack floating ip create --qos-policy bw-limiter public
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2017-12-06T02:12:09Z |
| description | |
| fixed_ip_address | None |
| floating_ip_address | 172.16.100.12 |
| floating_network_id | 4065eb05-cccb-4048-988c-e8c5480a746f |
| id | 6a0efeef-462b-4312-b4ad-627cde8a20e6 |
| name | 172.16.100.12 |
| port_id | None |
| project_id | 916e39e8be52433ba040da3a3a6d0847 |
| qos_policy_id | 5df855e9-a833-49a3-9c82-c0839a5f103f |
| revision_number | 1 |
| router_id | None |
| status | DOWN |
| updated_at | 2017-12-06T02:12:09Z |
+---------------------+--------------------------------------+
The QoS bandwidth limit rules attached to a floating IP will become
active when you associate the latter with a port. For example, to associate
the previously created floating IP 172.16.100.12
to the instance port with
fixed IP 192.168.222.5
:
$ openstack port show a7f25e73-4288-4a16-93b9-b71e6fd00862
+-----------------------+--------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------+
| admin_state_up | UP |
| ... | ... |
| device_id | 69c03d70-53e8-4030-9c02-675c47f0b06b |
| device_owner | compute:nova |
| dns_assignment | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='192.168.222.5', subnet_id='...' |
| id | a7f25e73-4288-4a16-93b9-b71e6fd00862 |
| ip_address | None |
| mac_address | fa:16:3e:b5:1a:cc |
| name | |
| network_id | ea602456-3ea8-4989-8981-add6182b4ceb |
| option_name | None |
| option_value | None |
| port_security_enabled | False |
| project_id | 916e39e8be52433ba040da3a3a6d0847 |
| qos_policy_id | None |
| revision_number | 6 |
| security_group_ids | 77436c73-3a29-42a7-b544-d47f4ea96d54 |
| status | ACTIVE |
| subnet_id | None |
| tags | |
| trunk_details | None |
| updated_at | 2017-12-05T15:48:54Z |
+-----------------------+--------------------------------------------------+
$ openstack floating ip set --port a7f25e73-4288-4a16-93b9-b71e6fd00862 \
0eeb1f8a-de96-4cd9-a0f6-3f535c409558
Note
For now, the L3 agent floating IP QoS extension only uses
bandwidth_limit
rules. Other rule types (like DSCP marking) will be
silently ignored for floating IPs. A QoS policy that does not contain any
bandwidth_limit
rules will have no effect when attached to a
floating IP.
If floating IP is bound to a port, and both have binding QoS bandwidth rules, the L3 agent floating IP QoS extension ignores the behavior of the port QoS, and installs the rules on the appropriate device in the router namespace.
Each project can have at most one default QoS policy, although it is not mandatory. If a default QoS policy is defined, all new networks created within this project will have this policy assigned, as long as no other QoS policy is explicitly attached during the creation process. If the default QoS policy is unset, no change to existing networks will be made.
In order to set a QoS policy as default, the parameter --default
must be
used. To unset this QoS policy as default, the parameter --no-default
must
be used.
$ openstack network qos policy create --default bw-limiter
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| description | |
| id | 5df855e9-a833-49a3-9c82-c0839a5f103f |
| is_default | True |
| name | bw-limiter |
| project_id | 4db7c1ed114a4a7fb0f077148155c500 |
| rules | [] |
| shared | False |
+-------------------+--------------------------------------+
$ openstack network qos policy set --no-default bw-limiter
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| description | |
| id | 5df855e9-a833-49a3-9c82-c0839a5f103f |
| is_default | False |
| name | bw-limiter |
| project_id | 4db7c1ed114a4a7fb0f077148155c500 |
| rules | [] |
| shared | False |
+-------------------+--------------------------------------+
Administrators are able to enforce policies on project ports or networks. As long as the policy is not shared, the project is not be able to detach any policy attached to a network or port.
If the policy is shared, the project is able to attach or detach such policy from its own ports and networks.
You can modify rules at runtime. Rule modifications will be propagated to any attached port.
$ openstack network qos rule set --max-kbps 2000 --max-burst-kbits 1600 \
--ingress bw-limiter 92ceb52f-170f-49d0-9528-976e2fee2d6f
$ openstack network qos rule show \
bw-limiter 92ceb52f-170f-49d0-9528-976e2fee2d6f
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| direction | ingress |
| id | 92ceb52f-170f-49d0-9528-976e2fee2d6f |
| max_burst_kbps | 1600 |
| max_kbps | 2000 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
Just like with bandwidth limiting, create a policy for DSCP marking rule:
$ openstack network qos policy create dscp-marking
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| description | |
| id | d1f90c76-fbe8-4d6f-bb87-a9aea997ed1e |
| is_default | False |
| name | dscp-marking |
| project_id | 4db7c1ed114a4a7fb0f077148155c500 |
| rules | [] |
| shared | False |
+-------------------+--------------------------------------+
You can create, update, list, delete, and show DSCP markings with the neutron client:
$ openstack network qos rule create --type dscp-marking --dscp-mark 26 \
dscp-marking
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| dscp_mark | 26 |
| id | 115e4f70-8034-4176-8fe9-2c47f8878a7d |
| name | None |
| project_id | |
+----------------+--------------------------------------+
$ openstack network qos rule set --dscp-mark 22 \
dscp-marking 115e4f70-8034-4176-8fe9-2c47f8878a7d
$ openstack network qos rule list dscp-marking
+--------------------------------------+----------------------------------+
| ID | DSCP Mark |
+--------------------------------------+----------------------------------+
| 115e4f70-8034-4176-8fe9-2c47f8878a7d | 22 |
+--------------------------------------+----------------------------------+
$ openstack network qos rule show \
dscp-marking 115e4f70-8034-4176-8fe9-2c47f8878a7d
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| dscp_mark | 22 |
| id | 115e4f70-8034-4176-8fe9-2c47f8878a7d |
| name | None |
| project_id | |
+----------------+--------------------------------------+
$ openstack network qos rule delete \
dscp-marking 115e4f70-8034-4176-8fe9-2c47f8878a7d
You can also include minimum bandwidth rules in your policy:
$ openstack network qos policy create bandwidth-control
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| description | |
| id | 8491547e-add1-4c6c-a50e-42121237256c |
| is_default | False |
| name | bandwidth-control |
| project_id | 7cc5a84e415d48e69d2b06aa67b317d8 |
| revision_number | 1 |
| rules | [] |
| shared | False |
+-------------------+--------------------------------------+
$ openstack network qos rule create \
--type minimum-bandwidth --min-kbps 1000 --egress bandwidth-control
+------------+--------------------------------------+
| Field | Value |
+------------+--------------------------------------+
| direction | egress |
| id | da858b32-44bc-43c9-b92b-cf6e2fa836ab |
| min_kbps | 1000 |
| name | None |
| project_id | |
+------------+--------------------------------------+
A policy with a minimum bandwidth ensures best efforts are made to provide no less than the specified bandwidth to each port on which the rule is applied. However, as this feature is not yet integrated with the Compute scheduler, minimum bandwidth cannot be guaranteed.
It is also possible to combine several rules in one policy:
$ openstack network qos rule create --type bandwidth-limit \
--max-kbps 50000 --max-burst-kbits 50000 bandwidth-control
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| id | 0db48906-a762-4d32-8694-3f65214c34a6 |
| max_burst_kbps | 50000 |
| max_kbps | 50000 |
| name | None |
| project_id | |
+----------------+--------------------------------------+
$ openstack network qos policy show bandwidth-control
+-------------------+-------------------------------------------------------------------+
| Field | Value |
+-------------------+-------------------------------------------------------------------+
| description | |
| id | 8491547e-add1-4c6c-a50e-42121237256c |
| is_default | False |
| name | bandwidth-control |
| project_id | 7cc5a84e415d48e69d2b06aa67b317d8 |
| revision_number | 4 |
| rules | [{u'max_kbps': 50000, u'type': u'bandwidth_limit', |
| | u'id': u'0db48906-a762-4d32-8694-3f65214c34a6', |
| | u'max_burst_kbps': 50000, |
| | u'qos_policy_id': u'8491547e-add1-4c6c-a50e-42121237256c'}, |
| | {u'direction': |
| | u'egress', u'min_kbps': 1000, u'type': u'minimum_bandwidth', |
| | u'id': u'da858b32-44bc-43c9-b92b-cf6e2fa836ab', |
| | u'qos_policy_id': u'8491547e-add1-4c6c-a50e-42121237256c'}] |
| shared | False |
+-------------------+-------------------------------------------------------------------+
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.