This page explains how to do a major upgrade from version 4.1 of the SMO nodes.

The page is self-sufficient, that is, if you save or print this page, you have all the required information and instructions for upgrading SMO nodes. However, before starting the procedure, make sure you are familiar with the operation of Rhino VoLTE TAS nodes, this procedure, and the use of the SIMPL VM.

  • There are links in various places below to other parts of this book, which provide more detail about certain aspects of solution setup and configuration.

  • You can find more information about SIMPL VM commands in the SIMPL VM Documentation.

  • You can find more information on rvtconfig commands on the rvtconfig page.

Planning for the procedure

This procedure assumes that:

  • You are familiar with UNIX operating system basics, such as the use of vi and command-line tools like scp.

  • You are upgrading SMO VMs from version 4.1 to 4.2.

  • You have completed the steps in Prepare for the upgrade.

  • You have deployed a SIMPL VM, version 6.15.3 or later. Output shown on this page is correct for version 6.15.3 of the SIMPL VM; it may differ slightly on later versions.

Check you are using a supported VNFI version:

Platform Supported versions

OpenStack

Newton to Wallaby

VMware vSphere

6.7 and 7.0

Important notes

Important

Do not use these instructions for target versions whose major version component differs from 4.2.

Determine parameter values

In the below steps, replace parameters marked with angle brackets (such as <deployment ID>) with values as follows. (Replace the angle brackets as well, so that they are not included in the final command to be run.)

  • <deployment ID>: The deployment ID. You can find this at the top of the SDF. On this page, the example deployment ID mydeployment is used.

  • <site ID>: A number for the site in the form DC1 through DC32. You can find this at the top of the SDF.

  • <site name>: The name of the site. You can find this at the top of the SDF.

  • <MW duration in hours>: The duration of the reserved maintenance period in hours.

  • <CDS address>: The management IP address of the first TSN node.

  • <SIMPL VM IP address>: The management IP address of the SIMPL VM.

  • <CDS auth args> (authentication arguments): If your CDS has Cassandra authentication enabled, replace this with the parameters -u <username> -k <secret ID> to specify the configured Cassandra username and the secret ID of a secret containing the password for that Cassandra user. For example, ./rvtconfig -c 1.2.3.4 -u cassandra-user -k cassandra-password-secret-id …​.

    If your CDS is not using Cassandra authentication, omit these arguments.

  • <service group name>: The name of the service group (also known as a VNFC - a collection of VMs of the same type), which for Rhino VoLTE TAS nodes will consist of all SMO VMs in the site. This can be found in the SDF by identifying the SMO VNFC and looking for its name field.

  • <downlevel version>: The current version of the VMs. On this page, the example version 4.1-7-1.0.0 is used.

  • <uplevel version>: The version of the VMs you are upgrading to. On this page, the example version 4.2-8-1.0.0 is used.

  • <SSH key secret ID>: The secret store ID of the SSH key used to access the node. You can find this in the SDF, or by running csar secret status on the SIMPL VM.

  • <diags-bundle>`: The name of the diagnostics bundle directory. If this directory doesn’t already exist, it will be created.

Tools and access

You must have the SSH keys required to access the SIMPL VM and the SMO VMs that are to be upgraded.

The SIMPL VM must have the right permissions on the VNFI. Refer to the SIMPL VM documentation for more information:

Note

When starting an SSH session to the SIMPL VM, use a keepalive of 30 seconds. This prevents the session from timing out - SIMPL VM automatically closes idle connections after a few minutes.

When using OpenSSH (the SSH client on most Linux distributions), this can be controlled with the option ServerAliveInterval - for example, ssh -i <SSH private key file for SIMPL VM> -o ServerAliveInterval=30 admin@<SIMPL VM IP address>.

rvtconfig is a command-line tool for configuring and managing Rhino VoLTE TAS VMs. All SMO CSARs include this tool; once the CSAR is unpacked, you can find rvtconfig in the resources directory, for example:

$ cdcsars
$ cd smo/<uplevel version>
$ cd resources
$ ls rvtconfig
rvtconfig

The rest of this page assumes that you are running rvtconfig from the directory in which it resides, so that it can be invoked as ./rvtconfig. It assumes you use the uplevel version of rvtconfig, unless instructed otherwise. If it is explicitly specified you must use the downlevel version, you can find it here:

$ cdcsars
$ cd smo/<downlevel version>
$ cd resources
$ ls rvtconfig
rvtconfig

1. Preparation for upgrade procedure

These steps can be carried out in advance of the upgrade maintenance window. They should take less than 30 minutes to complete.

1.1 Ensure the SIMPL version is at least 6.15.3

Log into the SIMPL VM and run the command simpl-version. The SIMPL VM version is displayed at the top of the output:

SIMPL VM, version 6.15.3

Ensure this is at least 6.15.3. If not, contact your Customer Care Representative to organise upgrading the SIMPL VM before proceeding with the upgrade of the SMO VMs.

Output shown on this page is correct for version 6.15.3 of the SIMPL VM; it may differ slightly on later versions.

1.2 Verify the downlevel CSAR is present

On the SIMPL VM, run csar list.

Each listed CSAR will be of the form <node type>/<version>, for example, smo/4.1-7-1.0.0.

Ensure that there is a SMO CSAR listed there with the current downlevel version.

If the downlevel CSAR is not present, return to the pre-upgrade steps.

1.3 Reserve maintenance period

The upgrade procedure requires a maintenance period. For upgrading nodes in a live network, implement measures to mitigate any unforeseen events.

Ensure you reserve enough time for the maintenance period, which must include the time for a potential rollback.

To calculate the time required for the actual upgrade or roll back of the VMs, run rvtconfig calculate-maintenance-window -i /home/admin/uplevel-config -t smo --site-id <site ID>. The output will be similar to the following, stating how long it will take to do an upgrade or rollback of the SMO VMs.

Nodes will be upgraded sequentially

-----

Estimated time for a full upgrade of 3 VMs: 24 minutes
Estimated time for a full rollback of 3 VMs: 24 minutes

-----

Your maintenance window must include time for:

  • The preparation steps. Allow 30 minutes.

  • The upgrade of the VMs, as calculated above.

  • The rollback of the VMs, as calculated above.

  • Post-upgrade or rollback steps. Allow 15 minutes, plus time for any prepared verification tests.

In the example above, this would be 93 minutes.

Important

These numbers are a conservative best-effort estimate. Various factors, including IMS load levels, VNFI hardware configuration, VNFI load levels, and network congestion can all contribute to longer upgrade times.

These numbers only cover the time spent actually running the upgrade on SIMPL VM. You must add sufficient overhead for setting up the maintenance window, checking alarms, running validation tests, and so on.
Note

The time required for an upgrade or rollback can also be manually calculated.

For node types that are upgraded sequentially, like this node type, calculate the upgrade time by using the number of nodes. The first node takes 12 minutes, while later nodes take 12 minutes each.

You must also reserve time for:

  • The SIMPL VM to upload the image to the VNFI. Allow 2 minutes, unless the connectivity between SIMPL and the VNFI is particularly slow.

  • Any validation testing needed to determine whether the upgrade succeeded.

1.4 Upload SAS bundles

Upload the SMO SAS bundle for the uplevel version to the master SAS server in any site(s) containing the VMs to be upgraded. Your Customer Care Representative can provide you with the SAS bundle file.

2. Upgrade procedure

2.1 Run basic validation tests on downlevel nodes

Before starting the upgrade procedure, run VNF validation tests from the SIMPL VM against the downlevel nodes: csar validate --vnf smo --sdf /home/admin/current-config/sdf-rvt.yaml

This command performs various checks on the health of the VMs' networking and services:

================================
Running validation test scripts
================================
Running validation tests in CSAR 'smo/4.1-7-1.0.0'
Test running for: mydeployment-smo-1
Running script: check_ping_management_ip…​
Running script: check_can_sudo…​
Running script: check_converged…​
Running script: check_liveness…​
Running script: check_rhino_alarms…​
Detailed output can be found in /var/log/csar/ansible_output-2023-01-06-03-21-51.log

If all is well, then you should see the message All tests passed for CSAR 'smo/4.1-7-1.0.0'!.

If the VM validation fails, you can find details in the log file. The log file can be found in /var/log/csar/ansible_output-<timestamp>.log. The msg field under each ansible task explains why the script failed.

If there are failures, the upgrade cannot take place. Investigate them with the help of your Customer Care Representative and the Troubleshooting pages.

Once the VNF validation tests pass, you can proceed with the next step.

2.2 Verify downlevel config has no changes

Run rm -rf /home/admin/config-output on the SIMPL VM to remove that directory if it already exists. Using rvtconfig from the downlevel CSAR, run ./rvtconfig compare-config -c <CDS address> -d <deployment ID> --input /home/admin/current-config
--vm-version <downlevel version> --output-dir /home/admin/config-output -t smo to compare the live configuration to the configuration in the /home/admin/current-config directory.

Example output is listed below:

Validating node type against the schema: smo
Redacting secrets…​
Comparing live config for (version=4.1-7-1.0.0, deployment=mydeployment, group=RVT-smo.DC1) with local directory (version=4.2-8-1.0.0, deployment=mydeployment, group=RVT-smo.DC1)
Getting per-level configuration for version '4.1-7-1.0.0', deployment 'mydeployment', and group 'RVT-smo.DC1'
  - Found config with hash 7f6cc1f3df35b43d6286f19c252311e09216e6115f314d0cb9cc3f3a24814395

Wrote currently uploaded configuration to /tmp/tmprh2uavbh
Redacting secrets…​
Redacting SDF…​
No differences found in yaml files
Uploading this will have no effect unless secrets, certificates or licenses have changed, or --reload-resource-adaptors is specified

There should be no differences found, as the configuration in current-config should match the live configuration. If any differences are found, abort the upgrade process.

2.3 Disable scheduled tasks

Only perform this step if this is the first, or only, node type being upgraded.

Run ./rvtconfig enter-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> --hours <MW duration in hours>. The output will look similar to:

Maintenance window is now active until 04 Nov 2022 21:38:06 NZDT.
Use the leave-maintenance-window command once maintenance is complete.

This will prevent scheduled tasks running on the VMs until the time given in the output.

If at any point in the upgrade process you wish to confirm the end time of the maintenance window, you can run ./rvtconfig maintenance-window-status -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>.

2.4 Verify uplevel config has no unexpected or prohibited changes

Run rm -rf /home/admin/config-output on the SIMPL VM to remove that directory if it already exists. Then use the command ./rvtconfig compare-config -c <CDS address> <CDS auth args> -d <deployment ID> --input /home/admin/uplevel-config
--vm-version <downlevel version> --output-dir /home/admin/config-output -t smo to compare the live configuration to the configuration in the /home/admin/uplevel-config directory.

Example output is listed below:

Validating node type against the schema: smo
Redacting secrets…​
Comparing live config for (version=4.2-7-1.0.0, deployment=mydeployment, group=RVT-smo.DC1) with local directory (version=4.2-8-1.0.0, deployment=mydeployment, group=RVT-smo.DC1)
Getting per-level configuration for version '4.2-7-1.0.0', deployment 'mydeployment', and group 'RVT-smo.DC1'
  - Found config with hash 7f6cc1f3df35b43d6286f19c252311e09216e6115f314d0cb9cc3f3a24814395

Wrote currently uploaded configuration to /tmp/tmprh2uavbh
Redacting secrets…​
Found
  - 1 difference in file sdf-rvt.yaml

Differences have been written to /home/admin/config-output
Error: Line 110 exited with status 3

You can then view the differences using commands such as cat /home/admin/config-output/sdf-rvt.yaml.diff (there will be one .diff file for every file that has differences). Aside from the version parameter in the SDF, there should normally be no other changes. If there are other unexpected changes, pause the procedure here and correct the configuration by editing the files in /home/admin/uplevel-config.

When performing a rolling upgrade, some elements of the uplevel configuration must remain identical to those in the downlevel configuration. The affected elements of the SMO configuration are described in the following list:

  • The secrets-private-key-id in the SDF must not be altered.

  • The ordering of the VM instances in the SDF must not be altered.

  • The IP addresses and other networking information in the SDF must not be altered.

  • The Rhino node ID of any VM must not be altered.

  • The Diameter origin hosts or SIP local URI (configured in the various *-vmpool-config.yaml files) must not be altered.

  • All SGC-related configuration must not be altered. (Follow the instructions in Reconfiguring the SGC if you need to modify the SGC configuration.)

  • SNMP notification targets cannot be altered if SNMP was previously enabled for the SGC. (Follow the instructions in Reconfiguring the SGC’s SNMP subsystem if you need to reconfigure SNMP on the SGC.)

  • The SGC SNMP configuration cannot be disabled if it was previously enabled. (Follow the instructions in Reconfiguring the SGC’s SNMP subsystem if you need to reconfigure SNMP on the SGC.)

The rvtconfig compare-config command reports any unsupported changes as errors, and may also emit warnings about other changes. For example:

Found
  - 1 difference in file sdf-rvt.yaml

The configuration changes have the following ERRORS.
File sdf-rvt.yaml:
  - Changing the IP addresses, subnets or traffic type assignments of live VMs is not supported. Restore the networks section of the smo VNFC in the SDF to its original value before uploading configuration.

Ensure you address the reported errors, if any, before proceeding. rvtconfig will not upload a set of configuration files that contains unsupported changes.

2.5 Verify the SGC is healthy

First, establish an SSH connection to the management IP of the first SMO node.

Then, generate an sgc report using /home/sentinel/ocss7/<deployment ID>/<node-name>/current/bin/generate-report.sh. Copy the output to a local machine using scp. Untar the report. Open the file sgc-cli.txt from the extracted report. The first lines will look like this:

Preparing to start SGC CLI …​
Checking environment variables
[CLI_HOME]=[/home/sentinel/ocss7/<deployment ID>/<node-name>/ocss7-<version>/cli]
Environment is OK!
Determining SGC home, JAVA and JMX configuration
[SGC_HOME]=/home/sentinel/ocss7/<deployment ID>/<node-name>/ocss7-<version>
[JAVA]=/home/sentinel/java/current/bin/java (derived from SGC_HOME/config/sgcenv)
[JMX_HOST]=user override
[JMX_PORT]=user override
Done
---------------------------Environment--------------------------------
CLI_HOME: /home/sentinel/ocss7/<deployment ID>/<node-name>/ocss7-<version>/cli
JAVA: /home/sentinel/java/current/bin/java
JAVA_OPTS:  -Dlog4j2.configurationFile=file:/home/sentinel/ocss7/<deployment ID>/<node-name>/ocss7-<version>/cli/conf/log4j2.xml -Dsgc.home=/home/sentinel/ocss7/<deployment ID>/<node-name>/ocss7-<version>/cli
----------------------------------------------------------------------
127.0.0.1:10111 <node-name>> display-active-alarm;
Found <number of alarms> object(s):

The lines following this will describe the active alarms, if any. Depending on your deployment, some alarms (such as connection alarms to other systems that may be temporarily offline) may be expected and therefore can be ignored.

2.6 Collect diagnostics

We recommend gathering diagnostic archives for all SMO VMs in the deployment.

On the SIMPL VM, run the command ./rvtconfig gather-diags --sdf /home/admin/uplevel-config/sdf-rvt.yaml -t smo --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --output-dir <diags-bundle>.

If <diags-bundle> does not exist, the command will create the directory for you.

Each diagnostic archive can be up to 200 MB per VM. Ensure you have enough disk space on the SIMPL VM to collect all diagnostics. The command will be aborted if the SIMPL VM does not have enough disk space to collect all diagnostic archives from all the VMs in your deployment specified in the provided SDF.

Note

You are using the uplevel rvtconfig and uplevel SDF to gather diagnostics for downlevel VMs. This is intentional, as the uplevel rvtconfig features the gather-diags command that only works with the uplevel SDF due to schema changes.

2.7 Validate configuration

Run the command ./rvtconfig validate -t smo -i /home/admin/uplevel-config to check that the configuration files are correctly formatted, contain valid values, and are self-consistent. Ensure you use the uplevel version of rvtconfig. A successful validation with no errors or warnings produces the following output.

Validating node type against the schema: smo
YAML for node type(s) ['smo'] validates against the schema

If the output contains validation errors, fix the configuration in the /home/admin/uplevel-config directory and go back to the Update the configuration files for RVT 4.2 step.

If the output contains validation warnings, consider whether you wish to address them before performing the upgrade. The VMs will accept configuration that has validation warnings, but certain functions may not work.

2.8 Destroy downlevel SMO VMs

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major upgrade of the SMO nodes from 4.1 to 4.2 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Run csar delete --sdf /home/admin/downlevel-config/sdf-rvt.yaml --vnf smo --sites <site name>

Run ./rvtconfig delete-node-type-all-versions -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> -t smo --ssh-key-secret-id <SSH key secret ID>

2.9 Upload configuration

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major upgrade of the SMO nodes from 4.1 to 4.2 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Upload the uplevel configuration to the CDS. Ensure you use the uplevel version of rvtconfig.

./rvtconfig upload-config -c <CDS address> <CDS auth args> -t smo -i /home/admin/uplevel-config --vm-version <uplevel version>

Check that the output confirms that configuration exists in CDS for the uplevel version:

Validating node type against the schema: smo
Preparing configuration for node type smo…​
Checking differences between uploaded configuration and provided files
Getting per-level configuration for version '4.2-8-1.0.0', deployment 'mydeployment-smo', and group 'RVT-smo.DC1'
  - No configuration found
No uploaded configuration was found: this appears to be a new install or upgrade
Encrypting secrets…​
Wrote config for version '4.2-8-1.0.0', deployment ID 'mydeployment', and group ID 'RVT-smo.DC1'
Versions in group RVT-smo.DC1
=============================
  - Version: 4.2-8-1.0.0
    Config hash: f790cc96688452fdf871d4f743b927ce8c30a70e3ccb9e63773fc05c97c1d6ea
    Active: mydeployment-smo-1, mydeployment-smo-2, mydeployment-smo-3
    Leader seed:

2.10 Deploy uplevel SMO VMs

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so the SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major upgrade of the SMO nodes from 4.1 to 4.2 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Run csar deploy --sdf /home/admin/uplevel-config/sdf-rvt.yaml --vnf smo --sites <site name>

2.11 Run basic validation tests

Run csar validate --vnf smo --sdf /home/admin/uplevel-config/sdf-rvt.yaml to perform some basic validation tests against the uplevel nodes.

This command first performs a check that the nodes are connected to MDM and reporting that they have successfully applied the uplevel configuration:

========================
Performing healthchecks
========================
Commencing healthcheck of VNF 'smo'
Performing health checks for service group mydeployment-smo with a 0 second timeout
Running MDM status health-check for dc1-mydeployment-smo-1
dc1-mydeployment-smo-1: Current status 'complete', current state 'commissioned' - desired status 'complete', desired state 'commissioned'
Running MDM status health-check for dc1-mydeployment-smo-2
dc1-mydeployment-smo-2: Current status 'complete', current state 'commissioned' - desired status 'complete', desired state 'commissioned'
Running MDM status health-check for dc1-mydeployment-smo-3
dc1-mydeployment-smo-3: Current status 'complete', current state 'commissioned' - desired status 'complete', desired state 'commissioned'

After that, it performs various checks on the health of the VMs' networking and services:

================================
Running validation test scripts
================================
Running validation tests in CSAR 'smo/4.2-8-1.0.0'
Test running for: mydeployment-smo-1
Running script: check_ping_management_ip…​
Running script: check_can_sudo…​
Running script: check_converged…​
Running script: check_liveness…​
Running script: check_rhino_alarms…​
Detailed output can be found in /var/log/csar/ansible_output-2023-01-06-03-21-51.log

If all is well, then you should see the message All tests passed for CSAR 'smo/<uplevel version>'!.

If the VM validation fails, you can find details in the log file. The log file can be found in /var/log/csar/ansible_output-<timestamp>.log.

Running validation test scripts
================================
Running validation tests in CSAR 'smo/4.2-8-1.0.0'
Test running for: mydeployment-smo-1
Running script: check_ping_management_ip...
Running script: check_can_sudo...
Running script: check_converged...
Running script: check_liveness...
ERROR: Script failed. Specific error lines from the ansible output will be logged to screen. For more details see the ansible_output file (/var/log/csar/ansible_output-2023-01-06-03-40-37.log). This file has only ansible output, unlike the main command log file.

fatal: [mydeployment-smo-1]: FAILED! => {"ansible_facts": {"liveness_report": {"cassandra": true, "cassandra_ramdisk": true, "cassandra_repair_timer": true, "cdsreport": true, "cleanup_sbbs_activities": false, "config_hash_report": true, "docker": true, "initconf": true, "linkerd": true, "mdm_state_and_status_ok": true, "mdmreport": true, "nginx": true, "no_ocss7_alarms": true, "ocss7": true, "postgres": true, "rem": true, "restart_rhino": true, "rhino": true}}, "attempts": 1, "changed": false, "msg": "The following liveness checks failed: ['cleanup_sbbs_activities']", "supports_liveness_checks": true}
Running script: check_rhino_alarms...
Detailed output can be found in /var/log/csar/ansible_output-2023-01-06-03-40-37.log
***Some tests failed for CSAR 'smo/4.2-8-1.0.0' - see output above***

----------------------------------------------------------


WARNING: Validation script tests failed for the following CSARs:
  - 'smo/4.2-8-1.0.0'
See output above for full details

The msg field under each ansible task explains why the script failed.

If the validation tests fail because of unexpected Rhino alarms, a good place to start investigating is by logging into each node and running rhino-console listactivealarms. This will show you the alarm(s) in more detail.

Depending on your deployment, some Rhino alarms (such as connection alarms to other systems that may be temporarily offline, time warps and blocklist alarms) may be expected and therefore can be ignored.

If there are other failures, investigate them with the help of your Customer Care Representative and the Troubleshooting pages.

3. Post-upgrade procedure

3.1 Enable scheduled tasks

Run ./rvtconfig leave-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>. This will allow scheduled tasks to run on the VMs again. The output should look like this:

Maintenance window has been terminated.
The VMs will resume running scheduled tasks as per their configured schedules.

3.2 Run verification tests

If you have prepared verification tests for the deployment, run these now.

4. Post-acceptance

The upgrade of the SMO nodes is now complete.

After you have been running with the SMO nodes at the uplevel version for a while, you may want to perform post-acceptance tasks.

5. Backout Method of Procedure

First, gather the log history of the downlevel VMs. Run mkdir -p /home/admin/rvt-log-history and ./rvtconfig export-log-history -c <CDS address> <CDS auth args> -d <deployment ID> --zip-destination-dir /home/admin/rvt-log-history --secrets-private-key-id <secret ID>. The secret ID you specify for --secrets-private-key-id should be the secret ID for the secrets private key (the one used to encrypt sensitive fields in CDS). You can find this in the product-options section of each VNFC in the SDF.

Warning Make sure the <CDS address> used is one of the remaining available TSN nodes.

Next, how much of the backout procedure to run depends on how much progress was made with the upgrade. If you did not get to the point of running csar update, start from the Cleanup after backout section below.

If you ran csar update and it failed, the output will tell you which VMs failed to upgrade.

  Successfully updated smo VMs with indexes: 0,1
  Not started updating smo VMs with indexes: 3,4
  Failed whilst updating smo VM with index: 2

Perform a rollback of all the VMs listed under "Successfully updated" and "Failed whilst updating".

If you encounter further failures during recovery or rollback, contact your Customer Care Representative to investigate and recover the deployment.

5.1 Collect diagnostics

We recommend gathering diagnostic archives for all SMO VMs in the deployment.

On the SIMPL VM, run the command ./rvtconfig gather-diags --sdf /home/admin/uplevel-config/sdf-rvt.yaml -t smo --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --output-dir <diags-bundle>.

If <diags-bundle> does not exist, the command will create the directory for you.

Each diagnostic archive can be up to 200 MB per VM. Ensure you have enough disk space on the SIMPL VM to collect all diagnostics. The command will be aborted if the SIMPL VM does not have enough disk space to collect all diagnostic archives from all the VMs in your deployment specified in the provided SDF.

Note

You are using the uplevel rvtconfig and uplevel SDF to gather diagnostics for downlevel VMs. This is intentional, as the uplevel rvtconfig features the gather-diags command that only works with the uplevel SDF due to schema changes.

5.2 Disable scheduled tasks

Only perform this step if this is the first, or only, node type being rolled back. You can also skip this step if the rollback is occurring immediately after a failed upgrade, such that the existing maintenance window is sufficient. You can check the remaining maintenance window time with ./rvtconfig maintenance-window-status -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>.

To start a new maintenance window (or extend an existing one), run ./rvtconfig enter-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> --hours <MW duration in hours>. The output will look similar to:

Maintenance window is now active until 04 Nov 2022 21:38:06 NZDT.
Use the leave-maintenance-window command once maintenance is complete.

This will prevent scheduled tasks running on the VMs until the time given in the output.

If at any point in the rollback process you wish to confirm the end time of the maintenance window, you can run the above rvtconfig maintenance-window-status command.

5.3 Destroy uplevel SMO VMs

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major rollback of the SMO nodes from 4.2 to 4.1 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Run csar delete --sdf /home/admin/uplevel-config/sdf-rvt.yaml --vnf smo --sites <site name>

Run ./rvtconfig delete-node-type-all-versions -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID> -t smo --ssh-key-secret-id <SSH key secret ID>

5.4 Upload configuration

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major rollback of the SMO nodes from 4.2 to 4.1 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Upload again the downlevel configuration to the CDS. Ensure you use the downlevel version of rvtconfig.

./rvtconfig upload-config -c <CDS address> <CDS auth args> -t smo -i /home/admin/current-config --vm-version <downlevel version>

Check that the output confirms that configuration exists in CDS for the current (downlevel) version:

Validating node type against the schema: smo
Preparing configuration for node type smo…​
Checking differences between uploaded configuration and provided files
Getting per-level configuration for version '4.1-7-1.0.0', deployment 'mydeployment-smo', and group 'RVT-smo.DC1'
  - No configuration found
No uploaded configuration was found: this appears to be a new install or upgrade
Encrypting secrets…​
Wrote config for version '4.1-7-1.0.0', deployment ID 'mydeployment', and group ID 'RVT-smo.DC1'
Versions in group RVT-smo.DC1
=============================
  - Version: 4.1-7-1.0.0
    Config hash: 7f6cc1f3df35b43d6286f19c252311e09216e6115f314d0cb9cc3f3a24814395
    Active: mydeployment-smo-1, mydeployment-smo-2, mydeployment-smo-3
    Leader seed:

5.5 Deploy downlevel SMO VMs

Note

Hazelcast has been updated to the latest available release on 4.2 version. Some changes to the initial SGC installation are required compared with previous releases, so SMO online upgrades from 4.1 to 4.2 are not supported. An specific upgrade process that destroys and deploys new VMs is required to perform major rollback of the SMO nodes from 4.2 to 4.1 version.

See the OCSS7 Installation and Administration Guide - Only Upgrade Support Matrix for detailed information.

Run csar deploy --sdf /home/admin/current-config/sdf-rvt.yaml --vnf smo --sites <site name>

5.6 Cleanup after backout

  • If desired, remove the uplevel CSAR. On the SIMPL VM, run csar remove smo/<uplevel version>.

  • If desired, remove the uplevel config directories on the SIMPL VM with rm -rf /home/admin/uplevel-config. We recommend these files are kept in case the upgrade is attempted again at a later time.

5.7 Enable scheduled tasks

Run ./rvtconfig leave-maintenance-window -c <CDS address> <CDS auth args> -d <deployment ID> --site-id <site ID>. This will allow scheduled tasks to run on the VMs again. The output should look like this:

Maintenance window has been terminated.
The VMs will resume running scheduled tasks as per their configured schedules.

5.8 Verify service is restored

Perform verification tests to ensure the deployment is functioning as expected.

If applicable, contact your Customer Care Representative to investigate the cause of the upgrade failure.

Important

Before re-attempting the upgrade, ensure you have run the rvtconfig delete-node-type-version command, Attempting an upgrade while there is stale uplevel data in CDS can result in needing to completely redeploy one or more VMs.

You will also need to re-upload the uplevel configuration.
Previous page Next page
Rhino VoLTE TAS VMs Version 4.2