REM Node Install Guide

The current version of the VMs being upgraded is known as the downlevel version, and the version that the VMs are being upgraded to is known as the uplevel version.

A rolling upgrade is a procedure where each VM is replaced, one at a time, with a new VM running the uplevel version of software. The Rhino VM Automation nodes are designed to allow rolling upgrades with little or no service outage time.

As with installation, upgrades and rollbacks use the SIMPL VM. The user starts the upgrade process by running csar update on the SIMPL VM. SIMPL VM destroys, in turn, each downlevel node and replaces it with an uplevel node. This is repeated until all nodes have been upgraded.

Configuration for the uplevel nodes is uploaded in advance. As nodes are recreated, they immediately pick up the uplevel configuration and resume service.

If an upgrade goes wrong, rollback to the previous version is also supported.

See the Rolling upgrades and patches page for detailed instructions on how to perform an upgrade.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have:

If you have configured your custom VMs to use authentication or SSL encryption when connecting to Cassandra via CQL as per Cassandra security configuration, you must also configure your Cassandra servers to support your security configuration.

Use the cqlsh tool to create the CDS keyspace and tables as follows.

First create the keyspace. If your Cassandra cluster has 1 or 2 nodes, use the following statement.

If your Cassandra cluster has 3 or 4 nodes, use the following statement instead.

Finally, if your Cassandra cluster has 5 or more nodes, use the following statement instead.

Regardless of the number of nodes, create five tables as follows.

Your CDS is now ready for use.

The SIMPL Virtual Appliance provides orchestration software to create, verify, configure, destroy and upgrade instances of your custom VM and the optional supporting REM and SGC VMs. Following the initial deployment, you will only need the SIMPL VM to perform configuration changes, patching or upgrades - it is not required for normal operation of the deployment.

The custom application solution uses Metaswitch Deployment Manager (MDM) to co-ordinate installation, upgrades, scale and healing (replacement of failed instances). MDM is a virtual appliance that provides state monitoring, DNS and NTP services to the deployment. It is deployed as a pool of at least three virtual machines.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

Anyone can perform these MOP steps.

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

This can either be done on your local Linux machine or on a SIMPL VM.

The Solution Definition File (SDF) contains all the information required to set up your cluster. It is therefore crucial to ensure all information in the SDF is correct before beginning the deployment. One SDF should be written per deployment.

It is recommended that the SDF is written before starting the deployment. The SDF must be named sdf-rvt.yaml.

In addition, you will need to write a secrets file and upload its contents to QSG. For security, the SDF no longer contains plaintext values of secrets (such as the password to access the VM host). Instead, the SDF contains secret IDs which refer to secrets stored in QSG.

See the various pages in the Writing an SDF section for more detailed information.

Example SDFs are included in every CSAR and can also be found at Example SDFs. We recommend that you start from a template SDF and edit it as desired instead of writing an SDF from scratch.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have access to a local computer (referred to in this procedure as the local computer) with a network connection and access to the vSphere client.

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

Follow the SIMPL VM Documentation on how to deploy the SIMPL VM and set up the configuration.

Create configuration YAML files relevant for your node type on the SIMPL VM. Store these files in the same directory as your prepared SDF.

See Example configuration YAML files for example configuration files.

Generate a template secrets.yaml file by running csar secrets create-input-file --sdf <path to SDF>.

Replace the value of any secrets in your SDF with a secret ID. The secret ID and corresponding secret value should be written in secrets.yaml.

Run the command csar secrets add <path to secrets.yaml template> to add the secrets to the secret store.

Refer to the Refer to the SIMPL VM documentation for more information.

MDM requires the following certificates and keys. Refer to the MDM documentation for more details.

If the CA used is an in-house CA, keep the CA private key safe so that you can generate a new static certificate and private key from the same CA in the future. Add the other credentials to QSG as described in MDM service group.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have access to the SIMPL VM, and the SIMPL VM must have the right permissions on the VMware vSphere deployment.

In the below steps, replace parameters marked with angle brackets (such as <deployment ID>) with values as follows. (Replace the angle brackets as well, so that they are not included in the final command to be run.)

Validate the configuration for the REM nodes to ensure that each REM node can properly self-configure.

To validate the configuration after creating the YAML files, run

rvtconfig validate -t rem -i <yaml-config-file-directory>

on the SIMPL VM from the resources subdirectory of the REM CSAR.

Upload the configuration for the REM nodes to the CDS. This will enable each REM node to self-configure when they are deployed in the next step.

To upload configuration after creating the YAML files and validating them as described above, run

rvtconfig upload-config -c <CDS address> <CDS auth args> -t rem -i <yaml-config-file-directory> (--vm-version-source this-rvtconfig | --vm-version <vm version>)

on the SIMPL VM from the resources subdirectory of the REM CSAR.

See Example configuration YAML files for example configuration files.

Run csar deploy --vnf rem --sdf <path to SDF>.

This will validate the SDF, and generate the terraform template. After successful validation, this will upload the image, and deploy the number of REM nodes specified in the SDF.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

Anyone can perform these MOP steps.

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

This can either be done on your local Linux machine or on a SIMPL VM.

The Solution Definition File (SDF) contains all the information required to set up your cluster. It is therefore crucial to ensure all information in the SDF is correct before beginning the deployment. One SDF should be written per deployment.

It is recommended that the SDF is written before starting the deployment. The SDF must be named sdf-rvt.yaml.

In addition, you will need to write a secrets file and upload its contents to QSG. For security, the SDF no longer contains plaintext values of secrets (such as the password to access the VM host). Instead, the SDF contains secret IDs which refer to secrets stored in QSG.

See the various pages in the Writing an SDF section for more detailed information.

Example SDFs are included in every CSAR and can also be found at Example SDFs. We recommend that you start from a template SDF and edit it as desired instead of writing an SDF from scratch.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have:

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

Follow the SIMPL VM Documentation on how to deploy the SIMPL VM and set up the configuration.

Create configuration YAML files relevant for your node type on the SIMPL VM. Store these files in the same directory as your prepared SDF.

See Example configuration YAML files for example configuration files.

Generate a template secrets.yaml file by running csar secrets create-input-file --sdf <path to SDF>.

Replace the value of any secrets in your SDF with a secret ID. The secret ID and corresponding secret value should be written in secrets.yaml.

Run the command csar secrets add <path to secrets.yaml template> to add the secrets to the secret store.

Refer to the Refer to the SIMPL VM documentation for more information.

You can find the possible flavors in the Flavors section, and it is recommended to use the same flavor name as described there.

Some node types share flavors. If the same flavor is to be used for multiple node types, only create it once.

You have now created the OpenStack flavor you will need when following the procedure to install the nodes on OpenStack virtual machines.

MDM requires the following certificates and keys. Refer to the MDM documentation for more details.

If the CA used is an in-house CA, keep the CA private key safe so that you can generate a new static certificate and private key from the same CA in the future. Add the other credentials to QSG as described in MDM service group.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have access to the SIMPL VM, and the SIMPL VM must have the right permissions on the OpenStack deployment.

In the below steps, replace parameters marked with angle brackets (such as <deployment ID>) with values as follows. (Replace the angle brackets as well, so that they are not included in the final command to be run.)

The SIMPL VM creates one server group per VM, and one security group per interface on each VM. OpenStack sets limits on the number of server groups and security groups through quotas.

View the quota by running openstack quota show <project id> on OpenStack Controller node. This shows the maximum number of various resources.

You can view the existing server groups by running openstack server group list. Similarly, you can find the security groups by running openstack security group list

If the quota is too small to accommodate the new VMs that will be deployed, increase it by running
openstack quota set --<quota field to increase> <new quota value> <project ID>. For example:
openstack quota set --server-groups 100 125610b8bf424e61ad2aa5be27ad73bb

Validate the configuration for the REM nodes to ensure that each REM node can properly self-configure.

To validate the configuration after creating the YAML files, run

rvtconfig validate -t rem -i <yaml-config-file-directory>

on the SIMPL VM from the resources subdirectory of the REM CSAR.

Upload the configuration for the REM nodes to the CDS. This will enable each REM node to self-configure when they are deployed in the next step.

To upload configuration after creating the YAML files and validating them as described above, run

rvtconfig upload-config -c <CDS address> <CDS auth args> -t rem -i <yaml-config-file-directory> (--vm-version-source this-rvtconfig | --vm-version <vm version>)

on the SIMPL VM from the resources subdirectory of the REM CSAR.

See Example configuration YAML files for example configuration files.

Run csar deploy --vnf rem --sdf <path to SDF>.

This will validate the SDF, and generate the heat template. After successful validation, this will upload the image, and deploy the number of REM nodes specified in the SDF.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

Anyone can perform these MOP steps.

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

This can either be done on your local Linux machine or on a SIMPL VM.

The Solution Definition File (SDF) contains all the information required to set up your cluster. It is therefore crucial to ensure all information in the SDF is correct before beginning the deployment. One SDF should be written per deployment.

It is recommended that the SDF is written before starting the deployment. The SDF must be named sdf-rvt.yaml.

In addition, you will need to write a secrets file and upload its contents to QSG. For security, the SDF no longer contains plaintext values of secrets (such as the password to access the VM host). Instead, the SDF contains secret IDs which refer to secrets stored in QSG.

See the various pages in the Writing an SDF section for more detailed information.

Example SDFs are included in every CSAR and can also be found at Example SDFs. We recommend that you start from a template SDF and edit it as desired instead of writing an SDF from scratch.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have access to a local computer (referred to in this procedure as the local computer) with a network connection and access to the vCloud client.

This page references an external document: the SIMPL VM Documentation. Ensure you have a copy available before proceeding.

Follow the SIMPL VM Documentation on how to deploy the SIMPL VM and set up the configuration.

Create configuration YAML files relevant for your node type on the SIMPL VM. Store these files in the same directory as your prepared SDF.

See Example configuration YAML files for example configuration files.

Generate a template secrets.yaml file by running csar secrets create-input-file --sdf <path to SDF>.

Replace the value of any secrets in your SDF with a secret ID. The secret ID and corresponding secret value should be written in secrets.yaml.

Run the command csar secrets add <path to secrets.yaml template> to add the secrets to the secret store.

Refer to the Refer to the SIMPL VM documentation for more information.

MDM requires the following certificates and keys. Refer to the MDM documentation for more details.

If the CA used is an in-house CA, keep the CA private key safe so that you can generate a new static certificate and private key from the same CA in the future. Add the other credentials to QSG as described in MDM service group.

This procedure assumes that:

This procedure does not require a maintenance period. However, if you are integrating into a live network, we recommend that you implement measures to mitigate any unforeseen events.

This procedure does not impact service.

You must be a system operator to perform the MOP steps.

You must have access to the SIMPL VM, and the SIMPL VM must have the right permissions on the VMware vCloud deployment.

In the below steps, replace parameters marked with angle brackets (such as <deployment ID>) with values as follows. (Replace the angle brackets as well, so that they are not included in the final command to be run.)

Validate the configuration for the REM nodes to ensure that each REM node can properly self-configure.

To validate the configuration after creating the YAML files, run

rvtconfig validate -t rem -i <yaml-config-file-directory>

on the SIMPL VM from the resources subdirectory of the REM CSAR.

Upload the configuration for the REM nodes to the CDS. This will enable each REM node to self-configure when they are deployed in the next step.

To upload configuration after creating the YAML files and validating them as described above, run

rvtconfig upload-config -c <CDS address> <CDS auth args> -t rem -i <yaml-config-file-directory> (--vm-version-source this-rvtconfig | --vm-version <vm version>)

on the SIMPL VM from the resources subdirectory of the REM CSAR.

See Example configuration YAML files for example configuration files.

Run csar deploy --vnf rem --sdf <path to SDF>.

This will validate the SDF, and generate the terraform template. After successful validation, this will upload the image, and deploy the number of REM nodes specified in the SDF.

In the below steps, replace parameters marked with angle brackets (such as <deployment ID>) with values as follows. (Replace the angle brackets as well, so that they are not included in the final command to be run.)

You must have the SSH keys required to access the SIMPL VM and the REM VMs that are to be upgraded.

The SIMPL VM must have the right permissions on the VNFI. Refer to the SIMPL VM documentation for more information:

rvtconfig is a command-line tool for configuring and managing Rhino VM Automation VMs. All REM CSARs include this tool; once the CSAR is unpacked, you can find rvtconfig in the resources directory, for example:

The rest of this page assumes that you are running rvtconfig from the directory in which it resides, so that it can be invoked as ./rvtconfig. It assumes you use the uplevel version of rvtconfig, unless instructed otherwise. If it is explicitly specified you must use the downlevel version, you can find it here:

Log into the SIMPL VM and run the command simpl-version. The SIMPL VM version is displayed at the top of the output:

Ensure this is at least 6.13.3. If not, contact your Customer Care Representative to organise upgrading the SIMPL VM before proceeding with the upgrade of the REM VMs.

Output shown on this page is correct for version 6.13.3 of the SIMPL VM; it may differ slightly on later versions.

Your Customer Care Representative will have provided you with the uplevel REM CSAR. Use scp to copy this to /csar-volume/csar/ on the SIMPL VM.

Once the copy is complete, run csar unpack /csar-volume/csar/<filename> on the SIMPL VM (replacing <filename> with the filename of the CSAR, which will end with .zip).

The csar unpack command may fail if there is insufficient disk space available. If this occurs, SIMPL VM will report this with instructions to remove some CSARs to free up disk space. You can list all unpacked CSARs with csar list and remove a CSAR with csar remove <node type>/<version>.

On the SIMPL VM, run csar list.

Ensure that there is a REM CSAR listed there with the current downlevel version.

If you are upgrading to an image that doesn’t require patching, or have already applied the patch, skip this step.

To patch a set of VMs, rather than modify the code directly on the VMs, the procedure is instead to patch the CSAR on SIMPL VM and then upgrade to the patched CSAR.

If you have a patch to apply, it will be provided to you in the form of a .tar.gz file. Use scp to transfer this file to /csar-volume/csar/ on the SIMPL VM. Apply it to the uplevel CSAR by running csar efix rem/<uplevel version> <patch file>, for example, csar efix rem/3.2-3-1.0.0/csar-volume/csar/mypatch.tar.gz. This takes about five minutes to complete.

Check the output of the patching process states that SIMPL VM successfully created a patch. Example output for a patch named mypatch on version 3.2-3-1.0.0 and a vSphere deployment is:

You can verify that a patched CSAR now exists by running csar list again - you should see a CSAR named rem/<uplevel version>-<patch name> (for the above example that would be rem/3.2-3-1.0.0-mypatch).

For all future steps on this page, wherever you type the <uplevel version>, be sure to include the suffix with the patch name, for example 3.2-3-1.0.0-mypatch.

If the csar efix command fails, be sure to delete any partially-created patched CSAR before retrying the patch process. Run csar list as above, and if you see the patched CSAR, delete it with csar remove <CSAR>.

If you keep the configuration hosted on the SIMPL VM, find it and rename it to /home/admin/current-config. Verify the contents by running ls /home/admin/current-config and checking that at least the SDF (sdf-rvt.yaml) is present there. If it isn’t, or you prefer to keep your configuration outside of the SIMPL VM, then create this directory on the SIMPL VM:

Use scp to upload the SDF (sdf-rvt.yaml) to this directory.

On the SIMPL VM, run mkdir /home/admin/uplevel-config. This directory is for holding the uplevel configuration files.

Use scp (or cp if the files are already on the SIMPL VM, for example in /home/admin/current-config as detailed in the previous section) to copy the following files to this directory. Include configuration for the entire deployment, not just the REM nodes.

Open the /home/admin/uplevel-config/sdf-rvt.yaml file using vi. Find the vnfcs section, and within that the REM VNFC. Within the VNFC, locate the version field and change its value to the uplevel version, for example 3.2-3-1.0.0. Save and close the file.

You can verify the change you made by using diff -u2 /home/admin/current-config/sdf-rvt.yaml /home/admin/uplevel-config/sdf-rvt.yaml. The diff should look like this (context lines and line numbers may vary), with only a change to the version for the relevant node type:

The upgrade procedure requires a maintenance period. For upgrading nodes in a live network, implement measures to mitigate any unforeseen events.

Ensure you reserve enough time for the maintenance period, which must include the time for a potential rollback.

To calculate the time required for the actual upgrade or roll back of the VMs, The output will be similar to the following, stating how long it will take to do an upgrade or rollback of the REM VMs.

You must also reserve time for:

The csar update dry run command carries out more extensive validation of the SDF and VM states than rvtconfig validate does.

Carrying out this step now, before the upgrade is due to take place, ensures problems with the SDF files are identified early and can be rectified beforehand.

Please run the following command to execute the dry run.

Confirm the output does not flag any problems or errors. The end of the command output should look similar to this.

Please confirm the set of nodes you are upgrading looks correct, and that the software version against the service group correctly indicates the software version you are planning to upgrade to.

If you see any errors, please address them, then re-run the dry run command until it indicates success.

Only perform this step if this is the first, or only, node type being upgraded.

Run ./rvtconfig enter-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> --hours <MW duration in hours>. The output will look similar to:

This will prevent scheduled tasks running on the VMs until the time given in the output.

If at any point in the upgrade process you wish to confirm the end time of the maintenance window, you can run ./rvtconfig maintenance-window-status -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>.

Run rm -rf /home/admin/config-output on the SIMPL VM to remove that directory if it already exists. Then use the command ./rvtconfig compare-config -c <CDS address> <CDS auth args> -d <deployment ID> --input /home/admin/uplevel-config
--vm-version <downlevel version> --output-dir /home/admin/config-output -t rem to compare the live configuration to the configuration in the /home/admin/uplevel-config directory.

Example output is listed below:

You can then view the differences using commands such as cat /home/admin/config-output/sdf-rvt.yaml.diff (there will be one .diff file for every file that has differences). Aside from the version parameter in the SDF, there should normally be no other changes. If there are other unexpected changes, pause the procedure here and correct the configuration by editing the files in /home/admin/uplevel-config.

When performing a rolling upgrade, some elements of the uplevel configuration must remain identical to those in the downlevel configuration. The affected elements of the REM configuration are described in the following list:

The rvtconfig compare-config command reports any unsupported changes as errors, and may also emit warnings about other changes. For example:

Ensure you address the reported errors, if any, before proceeding. rvtconfig will not upload a set of configuration files that contains unsupported changes.

Run the command ./rvtconfig validate -t rem -i /home/admin/uplevel-config to check that the configuration files are correctly formatted, contain valid values, and are self-consistent. A successful validation with no errors or warnings produces the following output.

If the output contains validation errors, fix the configuration in the /home/admin/uplevel-config directory

If the output contains validation warnings, consider whether you wish to address them before performing the upgrade. The VMs will accept configuration that has validation warnings, but certain functions may not work.

Upload the configuration to CDS:

Check that the output confirms that configuration exists in CDS for both the current (downlevel) version and the uplevel version:

We recommend gathering diagnostic archives for all REM VMs in the deployment.

On the SIMPL VM, run the command

If <diags-bundle> does not exist, the command will create the directory for you.

Each diagnostic archive can be up to 200 MB per VM. Ensure you have enough disk space on the SIMPL VM to collect all diagnostics. The command will be aborted if the SIMPL VM does not have enough disk space to collect all diagnostic archives from all the VMs in your deployment specified in the provided SDF.

For each VM:

The output of the csar update command will look something like the following, repeated for each VM.

Once all VMs have been upgraded, you should see this success message, detailing all the VMs that were upgraded and the version they are now running, which should be the uplevel version.

If the upgrade fails, you will see Failed VNF instead of Successful VNF in the above output. There will also be more details of what went wrong printed before that. Refer to the Backout procedure below.

Run csar validate --vnf rem --sdf /home/admin/uplevel-config/sdf-rvt.yaml to perform some basic validation tests against the uplevel nodes.

This command first performs a check that the nodes are connected to MDM and reporting that they have successfully applied the uplevel configuration:

After that, it performs various checks on the health of the VMs' networking and services:

If all is well, then you should see the message All tests passed for CSAR 'rem/<uplevel version>'!.

If the VM validation fails, you can find details in the log file. The log file can be found in /var/log/csar/ansible_output-<timestamp>.log.

The msg field under each ansible task explains why the script failed.

If there are failures, investigate them with the help of your Customer Care Representative and the Troubleshooting pages.

Run ./rvtconfig leave-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>. This will allow scheduled tasks to run on the VMs again. The output should look like this:

If you have prepared verification tests for the deployment, run these now.

We recommend gathering diagnostic archives for all REM VMs in the deployment.

On the SIMPL VM, run the command

If <diags-bundle> does not exist, the command will create the directory for you.

Each diagnostic archive can be up to 200 MB per VM. Ensure you have enough disk space on the SIMPL VM to collect all diagnostics. The command will be aborted if the SIMPL VM does not have enough disk space to collect all diagnostic archives from all the VMs in your deployment specified in the provided SDF.

Only perform this step if this is the first, or only, node type being rolled back. You can also skip this step if the rollback is occurring immediately after a failed upgrade, such that the existing maintenance window is sufficient. You can check the remaining maintenance window time with ./rvtconfig maintenance-window-status -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>.

To start a new maintenance window (or extend an existing one), run ./rvtconfig enter-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> --hours <MW duration in hours>. The output will look similar to:

This will prevent scheduled tasks running on the VMs until the time given in the output.

If at any point in the rollback process you wish to confirm the end time of the maintenance window, you can run the above rvtconfig maintenance-window-status command.

To roll back the VMs, the procedure is essentially to perform an "upgrade" back to the downlevel version, that is, with <downlevel version> and <uplevel version> swapped. You can refer to the Begin the upgrade section above for details on the prompts and output of csar update.

Once the csar update command completes successfully, proceed with the next steps below.

If csar update fails, check the output for which VMs failed. For each VM that failed, run csar redeploy --vm <failed VM name> --sdf /home/admin/current-config/sdf-rvt.yaml.

If csar redeploy fails, contact your Customer Care Representative to start the recovery procedures.

If all the csar redeploy commands were successful, then run the previously used csar update command on the VMs that were neither rolled back nor redeployed yet.

Run ./rvtconfig delete-node-type-version -c <CDS address> <CDS auth args> -t rem --vm-version <uplevel version>
-d <deployment ID> --site-id <site ID> --ssh-key-secret-id <SSH key secret ID> to remove data for the uplevel version from CDS.

Example output from the command:

Check the versions are the correct way around, and then confirm this prompt to delete the uplevel data from CDS.

Run ./rvtconfig leave-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>. This will allow scheduled tasks to run on the VMs again. The output should look like this:

Perform verification tests to ensure the deployment is functioning as expected.

If applicable, contact your Customer Care Representative to investigate the cause of the upgrade failure.

The VNF validation tests can be used to run some basic checks on deployed VMs to ensure they have been deployed correctly. Tests include:

After deploying the VMs for a given VM type, and performing the configuration for those VMs, you can run the VNF validation tests for those VMs from the SIMPL VM.

Run the validation tests: csar validate --vnf <node-type> --sdf <path to SDF>

Here, <node-type> is one of rem.

If any of the tests fail, refer to the troubleshooting section.

Log in to the VM with the default credentials.

Run systemctl status rhino-element-manager to view the status of the REM service. It should be listed as active (running).

You can also check the jps command to ensure that the Tomcat process has started. It is listed in the output as Bootstrap.

From a PC which is on or can reach the same subnet as the REM node’s management interface, connect to https://<management IP address>:8443/rem/ with a web browser. You should be presented with a login page. From here you can use the credentials set up in the rem-vmpool-config.yaml file to log in.

Commands that read or modify CDS state take a --cds-address parameter (which is also aliased as --cds-addresses, --cassandra-contact-point, --cassandra-contact-points, or simply -c). For this parameter, specify the management address(es) of at least one machine hosting the CDS database. Separate multiple addresses with a space, for example --cds-address 1.2.3.4 1.2.3.5.

The upload-config and export-audit-history commands read secrets from QSG. If you have not yet uploaded secrets to QSG, you can specify a --secrets-file <file> argument, passing in the path to your secrets file (the YAML file which you pass to csar secrets add). QSG is only available on the SIMPL VM; if running rvtconfig on a platform other than the SIMPL VM, for example on the VM itself, then you must pass the --secrets-file argument.

Commands that read or modify CDS state may also require additional parameters if the CDS endpoints are configured to use authentication and/or SSL encryption as per Cassandra security configuration. If the CDS endpoints are configured to use authentication, you must pass the --cds-username argument with your configured password and either the --cds-password or --cds-password-secret-name argument with the configured password or its ID in the secrets file. If the CDS endpoints are configured to use SSL encryption, you must pass the --ssl flag and also pass either the --ssl-ca-certificate or --ssl-ca-certificate-secret-name argument containing a file with the SSL signing certificate, or its ID in the secrets file.

The various delete-node-type commands, and the report-group-status command, require an SSH private key to access the VMs. You can specify this key as either a path to the private key file with the --ssh-key argument, or as a secret ID with the --ssh-key-secret-id argument. If you are running rvtconfig on the SIMPL VM, the recommended approach is to use the secret ID of the SIMPL VM-specific private key that you specified in the SDF (see SIMPL VM SSH private key ). Otherwise, use the SSH private key file itself (copying it to the machine on which you are running rvtconfig, and deleting it once you have finished, if necessary).

For more information, run rvtconfig --help. You can also view help about a particular command using, for example, rvtconfig upload-config --help.

The following limitations apply when running rvtconfig on the SIMPL VM:

Compare the configuration in an input directory with the currently uploaded configuration in the CDS using the command:

rvtconfig compare-config -c <cds-mgmt-addresses> -t <node type> -i ~/yamls --output-dir <output-directory>
[--deployment-id <deployment ID>] [--site-id <site ID>] [(--vm-version-source [this-vm | this-rvtconfig | sdf-version] | --vm-version <vm_version>)]

This will compare the currently uploaded configuration in the CDS with the configuration in the local input directory.

The deployment ID, site ID, and version of configuration to look up in CDS will be automatically taken from the SDF. These can be overridden by using the --deployment-id, --site-id, and one of the --vm-version-source or --vm-version parameters respectively. For example, you can specify --vm-version <downlevel version> to check what has changed just before running an upgrade, where the version in the input SDF will be the uplevel version.

The files that have differences will be displayed, along with the number of different lines, and any errors or warnings about the changes themselves. Any errors will need to be corrected before you can run rvtconfig upload-config.

The command puts the full contents of each version of these files into the output directory, along with separate files showing the differences found. The command ignores non-YAML files and any secrets in YAML files. The files in this output directory use the suffix .local for a redacted version of the input file, .live for a redacted version of the live file, and .diff for a file showing the differences between the two.

Delete all deployment configuration from the CDS by running the command:

rvtconfig delete-deployment -c <cds-mgmt-addresses> -d <deployment-id> [--delete-audit-history]

Delete all state and configuration for a given node type and version from the CDS by running the command:

rvtconfig delete-node-type-version -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Delete all state and configuration for a given node type from the CDS by running the command:

rvtconfig delete-node-type-all-versions -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id>
--node-type <node type> (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [--delete-certificates] [-y]

Remove all state and configuration relating to a versions of the node type other than the specified version from CDS by running the command:

rvtconfig delete-node-type-retain-version -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node-type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Following an upgrade or rollback, you may wish to clean up keyspaces in the Cassandra ramdisk database from version(s) that are no longer in use. This conserves memory and disk space.

To clean up unused keyspaces, use the following command:

rvtconfig remove-unused-keyspaces -c <cds-mgmt-addresses> -d <deployment-id> -g <group-id> [-y]

Confirm that the active VM versions that the command identifies are correct. rvtconfig removes keyspaces relating to all other versions from Cassandra.

List all currently available configurations in the CDS by running the command:

rvtconfig list-config -c <cds-mgmt-addresses> -d <deployment-id>

This command will print a short summary of the configurations uploaded, the VM version they are uploaded for, and which VMs are commissioned in that version.

Retrieve the VM group configuration from the CDS by running the command:

rvtconfig dump-config -c <cds-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)
[--output-dir <output-dir>]

If the optional --output-dir <directory> argument is specified, then the configuration will be dumped as individual files in the given directory. The directory can be expressed as either an absolute or relative path. It will be created if it doesn’t exist.

If the --output-dir argument is omitted, then the configuration is printed to the terminal.

Display the current leader seed by running the command:

rvtconfig print-leader-seed -c <cds-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)

The command will display the current leader seed for the specified deployment, group, and VM version. A leader seed may not always exist, in which case the output will include No leader seed found. Conditions where a leader seed may not exist include:

Some configuration, for example Rhino or REM users' passwords, are configured in plaintext, but stored encrypted in CDS for security. rvtconfig automatically performs this encryption using a secrets private key which you configure in the SDF. This key must be a Fernet key, in Base64 format. Use the following rvtconfig command to generate a suitable secrets private key:

rvtconfig generate-private-key

Add the generated secrets private key to your secrets input file when adding secrets to QSG.

The rvtconfig enter-maintenance-window and rvtconfig leave-maintenance-window commands allow you to pause and resume scheduled tasks (Rhino restarts, SBB/activity cleanup, and Cassandra repair) on the VMs for a period of time. This is useful to avoid the scheduled tasks interfering with maintenance window activities, such as patching a VM or making substantial configuration changes.

To start a maintenance window, use

rvtconfig enter-maintenance-window -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id> [--hours <hours>]

Once started, the maintenance window can be extended by running the same command again (but not shortened). rvtconfig will display the end time of the maintenance window in the command output. Until this time, all scheduled tasks on all VMs in the specified site will not be run.

When the maintenance window is complete, use the following command:

rvtconfig leave-maintenance-window -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id>

Scheduled tasks will now resume as per their configured schedules.

To check whether or not a maintenance window is currently active, use the following command:

rvtconfig maintenance-window-status -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id>

The rvtconfig calculate-maintenance-window commands allows you to estimate how long an upgrade or rollback is expected to take, so that an adequate maintenance window can be scheduled.

To calculate the recommended maintenance window duration, use

rvtconfig calculate-maintenance-window -i ~/yamls -t <node type> -s <site-id> [--index-range <index range>]

During upgrade, when a downlevel VM is removed, it uploads Initconf and Rhino logs to the CDS. The log files are stored as encrypted data in the CDS.

Retrieve the VM logs for a deployment from the CDS by running the command:

rvtconfig export-log-history -c <cds-mgmt-addresses> -d <deployment-id> --zip-destination-dir <directory>
--secrets-private-key-id <secrets-private-key-id>

Some commands, upload-config for example, can be used with the special version values sdf-version, this-vm, and this-rvtconfig.

To view the real version strings associated with each of these special values:

rvtconfig describe-versions [-i ~/yamls]

Optional argument -i ~/yamls is required for the sdf-version value to be given. If it is called, the sdf-version will be found for each node type in the SDF. If a node type is expected but not printed this may be because the config yaml files for that node are invalid or not present in the ~/yamls directory.

If a special version value cannot be found, for example if this-vm is run on a SIMPL VM or the optional argument is not called, the describe-versions command will print N/A for that special version.

This command reports the status of each node in the given group, providing information to help inform which approach to take when recovering VMs.

It connects to each of the VMs in the group via SSH, as well as querying the CDS service. It then prints a detailed summary of status information for each VM, as well as a high level summary of the status of the group.

It does not log its output to a file. When using this command to aid in recovery operations, it’s good practice to redirect its output to a file locally on disk, which can then be used as part of any root cause analysis efforts afterwards.

On the SIMPL VM, run the command as follows, under the resources dir of the unpacked CSAR:

It is possible to obtain diagnostic files from RVT nodes with the command rvtconfig gather-diags. These diagnostic files include system files and solution configuration files, are packaged as a tar.gz file and deposited in the given output directory. Depending on the node type there will be different kinds of solution configuration files. These files can be crucial to troubleshoot problems on the VMs.

./rvtconfig gather-diags --sdf <SDF File> -t <node type> --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --output-dir <output-directory>

If you need to quickly check the initconf.log file from a certain VM or VMs, it is possible to do it with the command rvtconfig initconf-log. This command executes a tail on the initconf.log file of the specified VM or VMs and dumps it to the standard output.

rvtconfig initconf-log --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --ip-addresses <Space separated VM IP address list> --tail <num lines>

From RVT 4.1-3-1.0.0, the TSN nodes can be deployed with Cassandra version {cassandra-version-three} or {cassandra-version-four}. Both Cassandra versions are installed in the VM Image, but only one is active. The commands rvtconfig cassandra-upgrade and rvtconfig cassandra-upgrade-sstables allow you to perform a Cassandra upgrade from {cassandra-version-three} to {cassandra-version-four} on a running TSN VM 4.1-3-1.0.0 or higher. These two commands must only be run if the target TSNs are running Cassandra {cassandra-version-three}.

To upgrade a single TSN node from {cassandra-version-three} to {cassandra-version-four} you can run ./rvtconfig cassandra-upgrade --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Address> for every TSN VM, one by one, and not in parallel.

Once all TSN nodes have been upgraded to {cassandra-version-four}, we must perform a sstables upgrade operation with the following command ./rvtconfig cassandra-upgrade-sstables --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Addresses>

Additionally the command rvtconfig cassandra-status prints the cassandra database status for the specified CDS IP addresses. Here is a couple of examples:

From RVT 4.1-3-1.0.0, the TSNs' CDS database can be backed up and restored. This provides a faster recovery procedure in case TSN upgrades go wrong.

To backup the CDS of a running TSN cluster, run ./rvtconfig backup-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --output-dir <backup-cds-bundle-dir> --ssh-key-secret-id <SSH key secret ID> -c <CDS address> <CDS auth args>

To restore the CDS of a running TSN cluster, run ./rvtconfig restore-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --snapshot-file <backup-cds-bundle-dir>/tsn_cassandra_backup.tar --ssh-key-secret-id <SSH key secret ID> -c <CDS Address> <CDS auth args>

During maintenance windows which involve upgrading TSN nodes, the command rvtconfig set-desired-running-state allows you stop/start the configuration tasks performed by the initconf that read from the CDS database in all non-TSN VMs. This operation does not stop the non-TSN VMs or the initconf process within it. But it instructs the initconf to pause or resume, the configuration tasks, while operating normally under traffic.

To pause initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Stopped.

To resume initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Started.

You cannot schedule two Rhino restarts on any one VM within 30 minutes of each other. (Such configuration would be excessive anyway; outside of exceptional circumstances, you only need to run these tasks at most once per day per VM.)

Additionally, two nodes in a group cannot restart Rhino within 30 minutes of each other. This is to prevent having a period where there are too few Rhino nodes to handle incoming traffic. While Rhino will normally restart in much less than 30 minutes, all traffic does need to drain from the node first, which can take some time.

All the above restrictions are checked by rvtconfig: configuration that doesn’t satisfy these requirements will not be accepted.

Scheduled Rhino restarts are applied per Rhino VM node, so they are defined under each virtual-machine element. For clarity, the examples below omit various fields that would normally be required.

Scheduled Cassandra repairs are executed on the whole TSN cluster, so they are set globally for all the virtual-machines element. For clarity, the examples below omit various fields that would normally be required.

Under site-parameters, all of the following are required for the Rhino VM Automation product:

Under the vnfcs section in each site, you list that site’s service groups. For REM VMs, each service group consists of the following fields:

For each service group, include a vim-configuration section with the flavor information, which varies according to the target VNFI type:

When deploying to VMware vSphere, include a vnfci-vim-options section for each instance with the following fields set:

For example:

For OpenStack, no vnfci-vim-options section is required.

To add secrets to QSG, first create a YAML file describing the secrets and their plaintext values. Next, pass the input file to the csar secrets add command. See the SIMPL VM documentation for instructions on how to create a template file, fill it in, and use csar secrets add.

When deploying a VM, SIMPL VM reads the values from QSG and passes them as bootstrap parameters. Likewise, when you run rvtconfig upload-config, rvtconfig will read secrets from QSG before encrypting them and storing them in CDS.

If you need to update the value of a secret (for example, if the password to the VM host is changed), edit your input file and run csar secrets add again. Any secrets already existing in QSG will be overwritten with their new values from the file.

MDM requires two traffic types: management and signaling, which must be on separate subnets.

Each MDM instance needs one IP address on each subnet. The management subnet does not necessarily have to be the same as the management subnet that the REM VMs are assigned to, but the network firewalling and topology does need to allow for communication between the REM VMs' management addresses and the MDM instances' management addresses, and as such it is simplest to use the same subnet as a matter of practicality.

For MDM product options, you must include the consul token and custom topology data.

Use YAML’s |- block-scalar style for the JSON blob, which will keep all newlines except the final one. Overall, the product options should look like this:

See Subnets and traffic schemes.

The following is a list of REM-specific product options in the SDF. All listed product options must be included in a product-options: → <node type> section, for example:

Each physical network attached to the VNFI must be at least 100Mb/s Ethernet (1Gb/s or better is preferred).

As a security measure, we recommend that you set up network firewalls to prevent traffic flowing between subnets. Note however that the VMs' software will send traffic over a particular subnet only when the subnet includes the traffic’s destination IP address; if the destination IP address is not on any of the VM’s subnets, it will use the management subnet as a default route.

If configuring routing rules for every destination is not possible, then an acceptable, but less secure, workaround is to firewall all interfaces except the management interface.

For all REM service groups in the SDF, where two or more service groups use a particular traffic type, this traffic type must be assigned to the same subnet throughout. For example, it is not permitted to use one subnet for management traffic on the REM VMs and a different subnet for management traffic on another VM type.

traffic types must each be assigned to a different subnet.

Once the VM has booted into multi-user mode, bootstrap normally takes about one minute.

SSH access to the VM is not possible until bootstrap has completed. If you want to monitor bootstrap from the console, log in as the sentinel user with the password you set in the SDF and examine the log file bootstrap/bootstrap.log. Successful completion is indicated by the line Bootstrap complete.

If bootstrap fails, an exception will be written to the log file. If the network-related portion of bootstrap succeeded but a failure occurred afterwards, the VM will be accessible over SSH and logging in will display a warning Automatic bootstrap failed.

Examine the log file bootstrap/bootstrap.log to see why bootstrap failed. In the majority of cases it will be down to an incorrect SDF or a missing or invalid bootstrap parameter. Destroy the VM and recreate it with the correct SDF or bootstrap parameters (it is not possible to run bootstrap more than once).

If you are sure you have the SDF or bootstrap parameters correct, or it is not obvious what is wrong, contact your Customer Care Representative.

After spinning up the VMs, configuration YAML files can be validated and uploaded to CDS using the rvtconfig tool. The rvtconfig tool can be run either on the SIMPL VM or any Rhino VM Automation VM.

The configuration process reads settings from YAML files. Each YAML file refers to a particular set of configuration options, for example, SNMP settings. The YAML files are validated against a YANG schema. The YANG schema is human-readable and lists all the possible options, together with a description. It is therefore recommended to reference the Configuration YANG schema while preparing the YAML files.

Some YAML files are shared between different node types. If a file with the same file name is required for two different node types, the same file must be used in both cases.

You will already have written a Solution Definition File (SDF) as part of the creation of the VMs. As the configuration process discovers other RVT nodes using the SDF, this SDF needs to be uploaded as part of the configuration.

The configuration process on the VMs starts after bootstrap completes. It is constantly listening for configuration to be written to CDS (via rvtconfig upload-config). Once it detects configuration has been uploaded, it will automatically download and validate it. Assuming everything passes validation, the configuration will then be applied automatically. This can take up to 20 minutes depending on node type.

The configuration process can be monitored using the report-initconf status tool. The tool can be run via an VM SSH session. Success is indicated by status=vm_converged.

Like bootstrap, errors are reported to the log file, located at initconf/initconf.log in the default user’s home directory.

initconf initialization failed due to an error: This indicates that initconf initialization has irrecoverably failed. Contact a Customer Care Representative for next steps.

Task <name> marked as permanently failed: This indicates that configuration has irrecoverably failed. Contact a Customer Care Representative for next steps.

<file> failed to validate against YANG schemas: This indicates something in one of the YAML files was invalid. Refer to the output to check which field was invalid, and fix the problem. For configuration validation issues, the VM doesn’t need to be destroyed and recreated. The fixed configuration can be uploaded using rvtconfig upload-config. The configuration process will automatically try again once it detects the uploaded configuration has been updated.

Other errors: If these relate to invalid field values or a missing license, it is normally safe to fix the configuration and try again. Otherwise, contact a Customer Care Representative.

The configuration process can raise the following SNMP alarms, which are sent to the configured notification targets (all with OID prefix 1.3.6.1.4.1.19808.2):

There are also limits on key size of the certificate signing request. The min key size is 2048-bit and the max is 8192-bit. The desired key size can be specified in the openssl command using the -newkey rsa:<keysize> option.

To generate the certificate signing request, run the following command, please change the keysize as desired:

The output files are the certificate signing request (rem.csr) and your private key (rem.key). The certificate signing request should be sent to the CA, who will then issue a certificate bundle for you. DO NOT DIRECTLY UPLOAD ANY OF THE CERTIFICATES GIVEN TO YOU BY THE CA. THEY MUST BE FORMATTED USING THE STEPS OUTLINED BELOW

Ensure that the CA issues your certificates in PEM (Privacy-Enhanced Mail) format. In addition, the private key must not have a passphrase (even an empty one).

A certificate bundle issued by a CA generally contains your certificate, their root certificate, and possibly one or more intermediate certificates. All certificates in the chain need to be merged into a single file in order to be uploaded for use with Tomcat. To format this correctly, follow these steps:

On the SIMPL VM place the certificate chain and private key files in the directory containing the YAML files, ensuring the files follow this naming scheme:

Validate the REM certificate chain has been formatted correctly by running this command from the directory containing the RVT YAML configuration files:

This command will output the subject and the issuer of each certificate in the chain in order.

There must be at least two certificates in the chain (the leaf and the root). If only one subject and one issuer is shown in the output, this means the certificate file is not valid. Run through the steps in section "Formatting the certificate bundle" again, then re-run the command above to validate the certificate file is in the correct format.

If this command outputs an error, then the chain has not been formatted correctly. Ensure each certificate in the chain is in PEM format and starts on a new line.

Check that each issuer matches the subject directly below it. This indicates they are chained in the correct order. If this is not the case then follow the steps in section "Formatting the certificate bundle" to put them in the right order, then retry the command above.

An example of a successful output with three certificates (one intermediate cert) looks like this:

The following example shows a failed output, where the issuer does not match the subject directly below it. In this case the root cert has been put before the leaf cert:

The config can then be uploaded, as it would be done for any change to the YAML configuration, using rvtconfig upload-config to the REM nodes. An example command is shown below:

where <cds-address> is the IP address of the CDS, <config-dir> is the path to the directory containing the YAML files.

This command will locate the files through the known filenames given above. It will then run a few basic checks on the files, such as checking whether the private key matches the certificate, and that the certificate is not due to expire in less than 30 days. If all checks pass, then the certificates will be uploaded to the CDS and installed by initconf. Otherwise, rvtconfig upload-config will inform you of any errors. Correct these and try again.

Note that you must provide both the certificate chain and private key. If you provide only one, rvtconfig upload-config will fail.

Below is a table indicating which user has permission to run commonly used commands.