Vitrage manual tests

General

The purpose of these tests is to manually check functionality that is not tested using tempest tests, and to double-check the correctness and validity of a devstack in general.

Vitrage Health

Services

Services Status

Run

sudo systemctl status devstack@vitrage-graph.service
sudo systemctl status devstack@vitrage-persistor.service
sudo systemctl status devstack@vitrage-notifier.service
sudo systemctl status devstack@vitrage-api.service

Expected result

Make sure that the status is active.

Check the logs for WARNING/ERROR/Exception/traceback

Services Restart

Run

sudo service devstack@vitrage-graph restart
sudo service devstack@vitrage-notifier restart
sudo service devstack@vitrage-persistor restart

Expected result

Make sure the restart is quick and does not reach a timeout

Services Information

Run

vitrage service list

Expected result

+----------------------------------+------------+-------------+---------------------------+
| Name                             | Process Id | Hostname    | Created At                |
+----------------------------------+------------+-------------+---------------------------+
| ApiWorker_worker_0               |       1084 | my-devstack | 2019-03-13T14:31:46+00:00 |
| EvaluatorWorker_worker_0         |       1082 | my-devstack | 2019-03-13T14:31:46+00:00 |
| MachineLearningService_worker_0  |       5956 | my-devstack | 2019-03-13T10:30:54+00:00 |
| PersistorService_worker_0        |      22536 | my-devstack | 2019-03-13T14:14:15+00:00 |
| SnmpParsingService_worker_0      |       6170 | my-devstack | 2019-03-13T10:30:56+00:00 |
| VitrageNotifierService_worker_0  |      22746 | my-devstack | 2019-03-13T14:14:27+00:00 |
| vitrageuWSGI_worker_1            |       2847 | my-devstack | 2019-03-13T10:30:47+00:00 |
| vitrageuWSGI_worker_2            |       2848 | my-devstack | 2019-03-13T10:30:47+00:00 |
+----------------------------------+------------+-------------+---------------------------+

Processes

Run

ps -aux | grep vitrage-graph

Expected result

vitrage-graph: master process

vitrage-graph: EvaluatorWorker

vitrage-graph: ApiWorker

There may be more than one EvaluatorWorker processes, according to the definition in vitrage.conf (the default is one worker).

Healthcheck API

Run

vitrage healthcheck

Expected result

+----------+---------+
| Field    | Value   |
+----------+---------+
| detailed | False   |
| reasons  | ['OK'] |
+----------+---------+

Vitrage Help

Run

vitrage -h

Expected result

Should display all Vitrage commands.

CLI commands

Most of the functionality is covered in tempest tests, but we have no automatic tests for the CLI itself.

Topology

Run

vitrage topology show

Expected result

Should display a graph with Vitrage entities and relationships.

Templates

Template Validate

What to execute

Expected results

vitrage template validate

Error, –path is required

vitrage template validate -path ./v1_template.yaml

Validation failed - Unknown template type

vitrage template validate –path ./v1_template.yaml –type standard

Template validation is OK

vitrage template validate –path ./v1_template.yaml –type definition

Validation failed, definition template can not contain scenarios block

vitrage template validate –path ./v2_high_cpu_load.yaml

Template validation is OK

vitrage template validate –path .

Validates all templates in the path. Some succeed and some fail.

vitrage template validate –path ./v2_definition_template.yaml

Template validation is OK

vitrage template validate –path v2_equivalence.yaml

No validation

vitrage template validate –path v3_high_mem_consumption.yaml

Template validation is OK

Template Add

What to execute

Expected results

vitrage template list

An empty list

vitrage template add

Error, –path is required

vitrage template add –path ./v1_template.yaml

Template added with status ERROR: Unknown template type

vitrage template add –path ./v1_template.yaml –type kuku

–type: invalid choice: kuku

vitrage template add –path ./v1_template.yaml –type standard

Template added with status LOADING

vitrage template add –path ./v1_template.yaml –type standard

ERROR: Duplicate template name

vitrage template list

One template with status ACTIVE

vitrage template delete

No output

vitrage template list

An empty list

vitrage template add –path ./v2_high_cpu_load.yaml

Template added with status LOADING

vitrage template add –path ./v2_definition_template.yaml

Template added with status LOADING

vitrage template add –path ./v2_with_include.yaml

Template added with status LOADING

vitrage template add –path ./v2_with_invalid_include.yaml

ERROR: Trying to include a template that does not exist

vitrage template add –path v2_equivalence.yaml

Template added with status LOADING

vitrage template add –path v3_high_mem_consumption.yaml

Template validation is OK

vitrage template list

Five templates with status ACTIVE and one in ERROR

vitrage template show <uuid>

Shows the template json representation

Templates with parameters

What to execute

Expected results

vitrage template validate –path v3_with_params.yaml

Failed to resolve parameter - template_name

vitrage template validate –path v3_with_params.yaml –params template_name=template1

Template validation is OK

vitrage template validate –path v3_with_params.yaml –params alarm_name=alarm1

Failed to resolve parameter - template_name

vitrage template validate –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1

Template validation is OK

vitrage template add –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1

Template added with status LOADING

vitrage template add –path v2_with_params.yaml –params template_name=t1 alarm_name=a1 alarm_type=zabbix

Template added with status LOADING

vitrage template show <uuid>

Shows the template json representation. All parameters are resolved and the parameters section is removed.

Deduced Alarms and RCA

What to execute

Expected results

create an instance in Nova

An instance is created

vitrage alarm list

An empty list

vitrage event post –type=”High CPU load” –details=’ {“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

Make sure to use hostname that contains an instance No output

vitrage alarm list

Shows ‘High CPU load’ on the host and also an alarm on the instance.

vitrage alarm show <uuid>

Shows the alarm details. Verify for both alarms.

vitrage rca show <uuid>

Shows the alarm RCA graph. Verify for both alarms.

vitrage alarm count

Shows one WARNING and one CRITICAL alarm

Resources

What to execute

Expected results

vitrage resource list

Shows a list of resources from different datasources. Does not show alarms

vitrage resource list –filter ….. TBD

Shows a list of resources that match the given filter.

vitrage resource show <uuid>

Shows a the defails of the selected resource.

vitrage resource count

Shows how many resources there are of every type.

Multi Tenancy

TBD

Datasources

Note: The resources and alarms are visible only to the tenant that created them.

  • For a resource that was created in the UI, check the Vitrage entity graph of the same tenant.

  • For a resource that was created in the CLI, check the Vitrage entity graph of admin tenant.

Basic OpenStack datasources

What to execute

Expected results

Create an instance/volume/network/stack

A new entity immediately appears in Vitrage entity graph and is connected to the right neighbors.

Delete these resources

The entities are immediately removed from the graph

Static Topology

What to execute

Expected results

Copy switch_and_nic.yaml under /etc/vitrage/static_datasources

The switche and nic should appear in the graph within 30 seconds

Aodh

What to execute

Expected results

aodh alarm create –name ‘test_1’ –state alarm –event-type ‘my.event’ –type event –query ‘traits.resource_id=string::my-devstack’

An alarm is created in Aodh with state alarm. Make sure to use as traits.resource_id the name of your devstack.

vitrage alarm list

The Aodh alarm appears and is connected to the devstack

aodh alarm create –name ‘Gnocchi alarm 1’ –type gnocchi_resources_threshold –resource-type instance –resource-id f9335fc1-f3bf-4915-bed2-2c7350628ac9 –metric cpu_util –threshold 0.001 –granularity 300 –state alarm –aggregation-method mean –comparison-operator gt

An alarm is created in Aodh with state alarm. Make sure to assign –resource-id with a valid instance uuid.

vitrage alarm list

Two Aodh alarms, connected to different resources

aodh alarm delete <alarm UUID>

vitrage alarm list

One Aodh alarm remains

Notifiers

Nova Notifier

What to execute

Expected results

vitrage template add –path ./host_down.yaml

Template added with status LOADING

nova service-list

The state of nova-compute is up

vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

Make sure to use hostname of your devstack. No output.

vitrage alarm list

A compute.host.down alarm, connected to the right host

nova service-list

The state of nova-compute is down

vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

Make sure to use hostname of your devstack. No output.

nova service-list

The state of nova-compute is up

Webhook Notifier

Configure a web client

Copy test_web_server.py under /opt/stack Run: sudo pip install web.py

Test the Webhook Notifier

What to execute

Expected results

vitrage webhook list

Empty list

python /opt/stack/test_web_server.py 8001

server started (in a different console window)

python /opt/stack/test_web_server.py 8002

server started (in a different console window)

vitrage webhook add –url http://localhost:8001/alarm

Outputs the webhook details

vitrage webhook add –url http://localhost:8002/alarm –regex ‘{“vitrage_type”: “doctor”}’

Outputs the webhook details

vitrage webhook list

A list with the details of both webhooks

vitrage webhook show <webhook uuid>

Shows the details of the webhook

vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

Both webhooks print the details of compute.host.down alarm to the console. The webhook on port 8001 prints also the details of the deduced alarms to the console.

vitrage webhook delete <webhook uuid>

Deletes a webhook

vitrage webhook list

Shows only one webhook

vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

The deleted webhook does not print anything. The other webhook prints the cleared alarm(s).

Mistral Notifier

What to execute

Expected results

mistral workflow-create ./workflow1.yaml

The workflow is created

vitrage template add –path ./v3_execute_mistral.yaml

Template added with status LOADING

vitrage event post –type=”alarm_for_mistral” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’

Make sure to use hostname of your devstack. No output.

mistral execution-list

workflow1 is in the list.

mistral execution-get <uuid of the latest execution>

Shows details about workflow1 execution.

mistral execution-get-input <uuid of the latest execution>

“farewell”: “my-devstack”

Zaqar Notifier

TBD

SNMP Notifier

TBD

UI Tests

The UI tests are included in this document, so we’ll have a complete set of manual sanity tests in one place.

Alarm Banner

What to execute

Expected results

Go to compute->instances menu

The alarm banner should be on the top right corner with the correct number of alarms

Click on the alarm banner

You should be redirected to Vitrage alarms view

Alarm View

What to execute

Expected results

Raise an alarm of type doctor (examples above)

The alarm appears in the Active Alarms list

Filter By alarm type doctor

Only doctor alarms remain

Clear the doctor filter

All alarms appear

Sort by name

Alarms are sorted

Click the RCA button for the High CPU load alarm

An RCA graph displays the alarms on the host and on the instance(s). Make sure all properties are ok.

Switch to Alarm History tab

Several alarms should appear with different statuses. The alarms that are currently active should not have an Ended value.

Click the RCA button for one of the alarms

An RCA graph displays the alarm(s) as inactive.

Change the Ended filter and click Fetch Alarms

The list of alarms is different

Filter By resource type nova.host

Only alarms on the host are displayed

Go back to Active Alarms

The list of active alarms is displayed

Topology View

What to execute

Expected results

Go to Topology View

The sunburst shows the compute hierarchy in different colors.

Switch tenants

The number of instances changes accordingly

Drill down to the host and instances

The sunburst changes. On the left there may be alarms

Click the RCA button on one of the alarms

The RCA view is opened

Entity Graph

What to execute

Expected results

Open the Entity Graph, zoom in and out

The graph changes accordingly

Click the Pin button, drag 1-2 entities, and refresh

The graph is not changed

Click the Unpin button

The graph is changed

Double-click two entites to pin them and drag them to different places. Then refresh the graph.

The pinned entities stay in the same location. The rest of the graph is changed.

Click several entities, one at the time

The properties of the selected entity appear on the left panel

Write a text in the search box

All entities with the selected text are highlighted

Switch to a different tenant and see how the graph changes

There are less entities (all instances are gone)

Filter by a specific Heat stack, modify the details level

The graph changes accordingly

Template View

Template View for tenant

What to execute

Expected results

Open the Template view under Project->Vitrage, check the list of templates

Few templates appear with Template validation is OK Details. There are no Add and Delete buttons.

Write a name in the filter

Only templates with this name appear

Clear the filter

All templates appear

Click the Show icon

The template content is displayed

Expand some of the entities, relationships and scenarios

The details are displayed

Switch to Yaml View

A json representation is displayed

Switch back to Simple View

The simple view is displayed

Template View for admin

What to execute

Expected results

Open the Template view under Admin->Vitrage and do the same checks as for tenant

Delete all existing templates

Templates are deleted, the list is empty

Click Add template and add v1_template.yaml. In the Type drop-down, select definition

ERROR: A template definition can not contain includes or scenarios blocks

Click Add template and add v1_template.yaml. In the Type drop-down, select standard

The template is added with status ACTIVE

Click Add template and add v2_definition_template.yaml

The template is added with status ACTIVE

Click Add template and add v2_equivalence.yaml

The template is added with status ACTIVE and and details No Validation

Click Show icon for templates of types standard (v1, v2 and v3), equivalence and definition

All templates are displayed correctly

Click Add template and add v2_wrong.yaml

ERROR: Action definition must contain action_target field. The template is not added.

Templates with parameters

What to execute

Expected results

Click Add template and add v2_with_params.yaml

Error: Failed to resolve parameter - alarm_type The template is not added.

Click Add template and add v2_with_params.yaml. Add parameter alarm_type.

Error: Failed to resolve parameter - alarm_name The template is not added.

Click Add template and add v2_with_params.yaml. Add parameters alarm_type, alarm_name, template_name and new_state. Add and remove another parameter before clicking Done.

The template is added with status ACTIVE

Click the Show icon

There is no parameters section. All parameters are resolved with the given values.

Click Add template and add v2_with_params.yaml. Add parameters alarm_type and alarm_name only

The template is added with status ACTIVE

Click the Show icon

There is no parameters section. alarm_type and alarm_name parameters are resolved with the given values. new_state has default value of ERROR.

Perform similar tests for v3_with_params.yaml

Click Add template and add v3_with_default_params.yaml

The template is added with status ACTIVE