Database synchronization error-handling¶

This document describes how to error-handling database synchronization errors.

Database Synchronization Description¶

If there are differences in the Pod information (Number of Pods, Pod name) between TackerDB and Kubernetes, Tacker will synchronize the TackerDB information with the Kubernetes information for all VNF instance, and synchronize according to the scale policy defined in the VNFD.

Note

The interval of database synchronization tasks between TackerDB and Kubernetes is 300 seconds. You can change it in /etc/tacker/tacker.conf

$ vim /etc/tacker/tacker.conf
db_synchronization_interval = 300

Database Synchronization Error¶

There are some errors in database synchronization and some error-handling operations.

The maximum or minimum number of pods is out of range
Error compute scale_level
LCM operation
- Conflict with LCM operation
- Abnormal LCM operation status

The maximum or minimum number of pods is out of range¶

During synchronization, Tacker will check the range of the maximum or minimum number of pods. If it finds out of range, Tacker will output an error log and not update the database.

When tacker-conductor.log contains the following error log, it means the maximum or minimum number of pods is out of range.

Note

If you don’t have tacker-conductor.log, you can execute the following CLI command to find log.

journalctl -u devstack@tacker-conductor

Error log:

Failed to update database vnf 81c4be9d-25ad-4726-8640-f2c4c326de2e vdu: VDU1. Pod num is out of range. pod_num: 5

Error-handling operations:

To solve this error, you can get with the following ways.

To check with the range of the maximum or minimum number of pods in VNFD.
To change the definition of number of pods in Kubernetes.

Check with the range of the maximum or minimum number of pods in VNFD¶

Note

VNFD files included in VNF Package defines the range of the maximum or minimum number of pods in vdu_profile. The following description uses helloworld3_df_simple.yaml as an example.

$ cat helloworld3_df_simple.yaml
tosca_definitions_version: tosca_simple_yaml_1_2

description: Simple deployment flavour for Sample VNF

imports:
  - etsi_nfv_sol001_common_types.yaml
  - etsi_nfv_sol001_vnfd_types.yaml
  - helloworld3_types.yaml

...

  node_templates:
    VNF:
      type: company.provider.VNF
      properties:
        flavour_description: A simple flavour

    VDU1:
      type: tosca.nodes.nfv.Vdu.Compute
      properties:
        name: curry-probe-test001
        description: kubernetes controller resource as VDU
        vdu_profile:
          min_number_of_instances: 1
          max_number_of_instances: 3

...

Note

vdu_profile.min_number_of_instances defines the minimum number of pods. vdu_profile.max_number_of_instances defines the maximum number of pods.

You can follow these steps to find VNFD files:

a.Find VNF instance ID in tacker-conductor.log¶

Execute the following CLI command to find tacker-conductor.log.

$ cat /opt/stack/logs/tacker-conductor.log | grep "Failed to update database vnf"

Note

If you don’t have tacker-conductor.log, you can execute the following CLI commands to view the log.

journalctl -u devstack@tacker-conductor

Result:

Failed to update database vnf 81c4be9d-25ad-4726-8640-f2c4c326de2e Pod num is out of range. pod_num: 5

Note

VNF instance ID in this case is 81c4be9d-25ad-4726-8640-f2c4c326de2e.

b. Find VNFD ID using VNF instance ID¶

Execute the following CLI command to find VNFD ID.

$ openstack vnf package list --filter '(eq,vnfdId,68014f5a-1f6b-47b2-914b-11bf0d1bd820)' -c 'Id'

Result:

+---------+--------------------------------------+
| Field   | Value                                |
+---------+--------------------------------------+
| VNFD ID | 68014f5a-1f6b-47b2-914b-11bf0d1bd820 |
+---------+--------------------------------------+

c. Find VNF package ID using VNFD ID¶

Execute the following CLI command to find VNF package ID.

$ openstack vnf package list --filter '(eq,vnfdId,68014f5a-1f6b-47b2-914b-11bf0d1bd820)' -c 'Id'

Result:

+--------------------------------------+
| Id                                   |
+--------------------------------------+
| eceb8b32-efb5-4940-a065-cef47fcdd973 |
+--------------------------------------+

d.Find VNFD files with VNF package ID¶

Execute the following CLI command to find VNF package ID.

$ openstack vnf package download eceb8b32-efb5-4940-a065-cef47fcdd973 --vnfd --file sample_vnfd.zip
$ unzip sample_vnfd.zip
$ vim sample_vnfd/Definitions/helloworld3_df_simple.yaml

Change the definition of number of pods in Kubernetes¶

Note

You can see Check with the range of the maximum or minimum number of pods in VNFD to find Kubernetes deployment name with helloworld3_df_simple.yaml in VDU1.properties.name.

You need to log in to the master node of the Kubernetes first, and execute the following CLI command to change the definition of number of pods in Kubernetes.

$ kubectl edit deploy curry-probe-test001 -n kube-system
apiVersion: apps/v1
kind: Deployment
...
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: curry-probe-test001
...

Note

Change the number of pods in spec.replicas.

Error compute scale_level¶

During synchronization (add or delete vnfc resource), Tacker will compute scale_level. When the pod numbers in Kubernetes are different from the initial number plus the scale number in the VNFD. Tacker will output an error log and do not update database.

When tacker-conductor.log contains the following error log, it means compute scale_level error.

Note

If you don’t have tacker-conductor.log, you can execute the following CLI command to show tacker-conductor.log.

journalctl -u devstack@tacker-conductor

Error log:

Error computing 'scale_level'. current Pod num: 4 delta: 2. vnf: 81c4be9d-25ad-4726-8640-f2c4c326de2e vdu: VDU1

Error-handling operations:

To solve this error, you can get with the following ways.

To check with the scale_level in VNFD.
To change the definition of number of pods in Kubernetes

Check with scale_level in VNFD (take vdu1 as an example)¶

Note

helloworld3_df_simple.yaml defines the scale_level of vdu1 in vdu1_scaling_aspect_deltas.properties.deltas.delta_1.number_of_instances, and initial delta in vdu1_initial_delta.properties.initial_delta.number_of_instances.

$ cat helloworld3_df_simple.yaml
tosca_definitions_version: tosca_simple_yaml_1_2

description: Simple deployment flavour for Sample VNF

imports:
  - etsi_nfv_sol001_common_types.yaml
  - etsi_nfv_sol001_vnfd_types.yaml
  - helloworld3_types.yaml

...

  policies:
    - scaling_aspects:
        type: tosca.policies.nfv.ScalingAspects
        properties:
          aspects:
            vdu1_aspect:
              name: vdu1_aspect
              description: vdu1 scaling aspect
              max_scale_level: 2
              step_deltas:
                - delta_1

    - vdu1_initial_delta:
        type: tosca.policies.nfv.VduInitialDelta
        properties:
          initial_delta:
            number_of_instances: 1
        targets: [ VDU1 ]

    - vdu1_scaling_aspect_deltas:
        type: tosca.policies.nfv.VduScalingAspectDeltas
        properties:
          aspect: vdu1_aspect
          deltas:
            delta_1:
              number_of_instances: 1
        targets: [ VDU1 ]

    - instantiation_levels:
        type: tosca.policies.nfv.InstantiationLevels
        properties:
          levels:
            instantiation_level_1:
              description: Smallest size
              scale_info:
                vdu1_aspect:
                  scale_level: 0
            instantiation_level_2:
              description: Largest size
              scale_info:
                vdu1_aspect:
                  scale_level: 2
          default_level: instantiation_level_1

    - vdu1_instantiation_levels:
        type: tosca.policies.nfv.VduInstantiationLevels
        properties:
          levels:
            instantiation_level_1:
              number_of_instances: 1
            instantiation_level_2:
              number_of_instances: 3
        targets: [ VDU1 ]

Note

You can follow these steps to find VNFD files in Check with the range of the maximum or minimum number of pods in VNFD from a.Find VNF instance ID in tacker-conductor.log to d.Find VNFD files with VNF package ID

Change the definition of number of pods in Kubernetes¶

The pod numbers should satisfy that the pod numbers in Kubernetes minus the initial increment is a multiple of the scale level.

Note

You can See Check with scale_level in VNFD (take vdu1 as an example) to find initial delta and scale level, and refer to Change the definition of number of pods in Kubernetes for details.

LCM operation¶

Conflict with LCM operation¶

There are two kinds of conflicts:

Database synchronization occurs while LCM operation is in progress.
LCM operation occurs during DB synchronization.

Database synchronization occurs while a LCM operation is in progress¶

When tacker-conductor.log contains the following info log, it means database synchronization conflict with LCM operation, and database synchronization will skip.

Note

If you don’t have tacker-conductor.log, you can execute the following CLI command to show tacker-conductor.log.

journalctl -u devstack@tacker-conductor

Info log:

There is an LCM operation in progress, so skip this DB synchronization. vnf:81c4be9d-25ad-4726-8640-f2c4c326de2e

Conflict resolution operations: Waiting for LCM operation completes and database synchronization will be repeated at a default time.

LCM operation occurs during DB synchronization¶

When LCM operation responds 409, it conflicts with Database synchronization.

Conflict resolution operations:

Execute LCM operation again after database synchronization completes.

Note

When completed, you can get the following information from log.

Debug log:

Ended sync_db

Abnormal LCM operation status¶

During synchronization, Tacker checks the operationState of VnfLcmOpOcc. For the same vnf instance, if FAILED_TEMP exists in the operationState, or the latest operationState is FAILED, Tacker will output an error log and do not update database.

Note

If you don’t have tacker-conductor.log, you can execute the following CLI command to show tacker-conductor.log.

journalctl -u devstack@tacker-conductor

Error log:

The LCM operation status of the vnf: 81c4be9d-25ad-4726-8640-f2c4c326de2e is abnormal, so skip this DB synchronization.

Error-handling operations:

To solve this error, you can get with the following ways.

For the operation state of FAILED_TEMP, please refer to VNF LCM error-handling.
For the operation state of FAILED, please perform other LCM operations on this vnf instance until the result is COMPLETED.

Database synchronization error-handling

Database synchronization error-handling¶

Database Synchronization Description¶

Database Synchronization Error¶

The maximum or minimum number of pods is out of range¶

Check with the range of the maximum or minimum number of pods in VNFD¶

a.Find VNF instance ID in tacker-conductor.log¶

b. Find VNFD ID using VNF instance ID¶

c. Find VNF package ID using VNFD ID¶

d.Find VNFD files with VNF package ID¶

Change the definition of number of pods in Kubernetes¶

Error compute scale_level¶

Check with scale_level in VNFD (take vdu1 as an example)¶

Change the definition of number of pods in Kubernetes¶

LCM operation¶

Conflict with LCM operation¶

Database synchronization occurs while a LCM operation is in progress¶

LCM operation occurs during DB synchronization¶

Abnormal LCM operation status¶

tacker 9.0.0.0rc2.dev7

Page Contents