Quota Usage for Cyborg Resources

Launchpad blueprint: https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-resource-quota

There are multiple ways to slice an OpenStack cloud. Imposing quota on these various slices puts a limitation on the amount of resources that can be consumed which helps to guarantee “fairness” or fair distribution of resources at the creation time. If a particular project needs more resources, the concept of quota gives the ability to increase the resource count on-demand, given that the system constraints are not exceeded.

Problem description

At present in Cyborg we don’t have the concept of Quota on acceleration resources, so users can consume as many resources as they want. Quotas are tied closely to physical resources and billable entities, hence from Cyborg’s perspective, it helps to limit the allocation and consumption of a particular kind of resources at a certain value.

In place of implementing quota like other services, we want to enable the unified limit which is provided by Keystone to manage our quota limit[1]. With unified limits, all limits will be set in Keystone and enforced by oslo.limit. So we decided to implement quota usage part first. Once the oslo.limit is ready for other services, Cyborg will invoke oslo.limit to get the limit information and do limit check etc.

This specs aims at the implementation of quota usage in Cyborg. As the oslo.limit is not finished yet, we can directly set the value of limit manually, and reserved the function calling oslo.limit with a “pass” inside.

Use cases

Alice is an admin. She would like to have a feature which will give her details of Cyborg acceleration resource consumptions so that she can manage her resources appropriately.

She might run into following scenarios:

  • Ability to know current resource consumption.

  • Ability to prohibit overuse by a project.

  • Prevent situation where users in a project get starved because users in other project consume all the resource. “Quota Management” would help to gurantee “fairness”.

  • Prevent DOS kind of attacks, abuse or error by users, which leads to an excessive amount of resources allocation.

Proposed change

Proposed changes are introducing a Quota_Usage Table which primarily stores the quota usage assigned for each resource in a project, and a Reservation Table to store every modification of resource usage.

When a new resource allocation request comes, the ‘reserved’ field in the Quota usages table will be updated. This acceleration resource is being used to set up VM. For example, the fpga quota hardlimit is 5 and 3 fgpas have already been used, then two new fpga requests come in. Since we have 3 fpgas already used, the ‘used’ field will be set to 3. Now the ‘reserved’ field will be set to 2 untill the fpga attachment is successful. Once the attachment is done this field will be reset to 0, and the ‘used’ count will be updated from 3 to 5. So at this moment, hardlimit is 5, used is 5 and in-progress is 0. So there is one more request comes in, this request will be rejected since there is not enough quota available.

In general,

Resource quota available = Resource hard_limit - [ (Resource reserved + Resources already allocated for project)]

In this specs, we just focus on the update of quota usage and we will not check if one user has already exceed his quota limit. The limit management will be set in Keystone in the future and we just need to invoke the oslo.limit.

Alternatives

At present there is no quota infrastructure in Cyborg.

Adding Quota Management layer at the Orchestration layer could be an alternative.However, our approach will give a finer view of resource consumptions at the IaaS layer which can be used while provisioning Cyborg resources.

Data model impact

New Quota usages and reservation table will be introduced to Cyborg database to store quota consumption for each resource in a project.

Quota usages table:

Field

Type

Null

Key

Default

Extra

created_at updated_at id project_id resource reserved used

datetime datetime int(11) varchar(255) varchar(255) int(11) int(11)

YES YES NO YES NO NO NO

PRI MUL

NULL NULL NULL NULL NULL NULL NULL

auto_increment

Quota reservation table:

Field

Type

Null

Key

Default

Extra

created_at updated_at deleted_at deleted id uuid usage_id project_id resource delta expire

datetime datetime datetime tinyint(1) int(11) varchar(36) int(11) varchar(255) varchar(255) int(11) datetime

YES YES YES YES NO NO NO YES YES NO YES

PRI

MUL MUL

NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL

auto_increment

We will also introduce QuotaEngine class which represents the set of recognized quotas and DbQuotaDriver class which performs check to enforcement of quotas and also allows to obtain quota information.

REST API impact

Not sure if we need to expose GET quota usage before oslo.limit settle down.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Xinran WANG

Other contributors: None

Work Items

  • Introduce Quota usages and Reservation table in Cyborg databases.

  • Update these two tables during allocation and deallocation of resources.

  • Reserve the place of function which will invoke oslo.limit with a “pass” inside.

  • Add rollback mechanism when allocation fails.

Dependencies

None

Testing

  • Each commit will be accompanied with unit tests.

  • Gate functional tests will also be covered.

Documentation Impact

None