- 1. Check prerequisites
- 2. Prepare for breaking interface changes
- 3. Upload uplevel CSARs
- 4. Update the configuration files for RVT 4.1
- 4.1. Prepare the downlevel config directory
- 4.2. Create directories for RVT 4.1 configuration and for rollbacks
- 4.3. Make product-independent changes to the SDF for SIMPL 6.13.3
- 4.4. Generate SSH keys for the RVT nodes
- 4.5. Create a copy of the SDF for rollback purposes
- 4.6. Make product-specific changes to the SDF for RVT 4.1
- 4.7. Provision SIMPL SSH key on the RVT 4.0.0 nodes
- 4.8. Make configuration changes for RVT 4.1
- 4.9. Identify if any non-RVT nodes need access to ShCM
- 4.10. Identify if any RVT nodes are misordered
- 4.11. Backout procedure
- 5. Update the DNS entry for the vertical service codes feature
- 6. Validate the new configuration
This page describes steps required to prepare for a major upgrade from 4.0.0. They can be performed before the upgrade, outside of a maintenance window. However, the prerequisites might reveal the need for additional maintenance windows, so confirm the prerequisites prior to making a detailed upgrade plan.
We recommend that you upgrade to the latest available minor release of RVT 4.1.
Before starting the upgrade, check the following:
The SMO nodes need to be on version at least 4.0.0-34-1.0.0. If not, perform an upgrade of the SMO nodes to 4.0.0-34-1.0.0 first, following the RVT 4.0.0 VM Install Guide. This requires you to plan a separate maintenance window before starting the upgrade to RVT 4.1.
DNS changes are required during this upgrade. Please see section "Update the DNS entry for the vertical service codes feature" below for further details. These changes must be made before the upgrade commences, so please ensure they are in place and tested with sufficient time to spare.
A new RVT license must be installed before you commence upgrade to V4.1. Please contact your Customer Care Representative to obtain the updated license.
The TSN nodes need to be on one of the following versions:
If they are on a different version, contact your Customer Care Representative.
All other nodes need to be on version at least 4.0.0-9-1.0.0.
You have deployed a SIMPL VM, version 6.13.3 or later. Output shown in this document is correct for version 6.13.3 of the SIMPL VM; it may differ slightly on later versions.
If it is still on a lower version, upgrade it as per the SIMPL VM Documentation. SIMPL VM upgrades are out of scope for this document.
If you want to use RVT SIMon dashboards, you will need SIMon on version at least 13.5.0, and need to ensure the community string is set correctly. Contact your Customer Care Representative for more information.
You have access to the SSH keys used to access the SIMPL VM.
You have access to the SIMPL and MDM documentation.
From RVT 4.1 onwards, all deployments will have the same static set of SNMP OIDs. In RVT 4.0.0, the OIDs differed per deployment (but were preserved across upgrades). This means that during an upgrade to 4.1, you will be changed over to the new, static set. Ensure all monitoring systems are updated to accommodate this change. Contact your Customer Care Representative for Management Information Bases (MIBs) detailing all the new SNMP OIDs.
rhinoInstanceIdfor the HSS Data and Data Configuration REST API has changed. In RVT 4.0.0, the request URI was of the form
/rem/sentinel/api/hssdata/subscriberdata?rhinoInstanceId=Local&selectionKey=Metaswitch::::, but in RVT 4.1 the request URI is now of the form
/rem/sentinel/api/hssdata/subscriberdata?rhinoInstanceId=RVT-mag.<site ID>-<hostname>&selectionKey=Metaswitch::::. If you use this API, all calls will need to be made to the new URL once the MAG nodes have been upgraded. Prepare for this prior to starting the upgrade.
Your Customer Care Representative will have provided you with the uplevel TSN, MAG, ShCM, MMT GSM, and SMO CSARs.
scp to copy these to
/csar-volume/csar/ on the SIMPL VM.
Once the copy is complete, for each CSAR, run
csar unpack /csar-volume/csar/<filename> on the SIMPL VM
<filename> with the filename of the CSAR, which will end with
csar unpack command may fail if there is insufficient disk space available.
If this occurs, SIMPL VM will report this with instructions
to remove some CSARs to free up disk space.
You can list all unpacked CSARs with
csar list and remove a CSAR with
csar remove <node type>/<version>.
If you keep the configuration hosted on the SIMPL VM,
then the existing config should already be located in
/home/admin/current-config (your configuration folder may have a different name, as the folder name is not policed e.g. yours may be named
rvt-config, if this is the case then rename it to
Verify this is the case by running
and checking that the directory contains:
The Rhino license.
The current SDF for the deployment (in the format used by SIMPL 6.6 and SIMPL 6.7). This is the SDF titled 'sdf-rvt.yaml' which you will previously have used to manage the RVT 4.0 VMs.
Any certificates and private key files for REM, BSF, and/or XCAP:
<type>is one of
If it isn’t, or you prefer to keep your configuration outside of the SIMPL VM, then create this directory on the SIMPL VM:
scp to upload the files described above to this directory.
To create the directory for holding the uplevel configuration files, on the SIMPL VM, run:
cp /home/admin/current-config/* /home/admin/uplevel-config
to copy the configuration, which you will edit in place in the steps below.
In addition, create a directory to contain a specially tailored copy of the SDF, which you will use if a rollback is required:
At this point you should have the following directories on the SIMPL VM:
SIMPL 6.13.3 (used by RVT 4.1) has major changes in the SDF format compared to SIMPL 6.6/6.7 (used by RVT 4.0.0). Most notably, secrets are now stored in QSG.
Updating the SDF is independent of the RVT upgrade and as such is not described in this document.
Refer to the SIMPL VM Documentation for more details.
You can also refer to the list of "Deprecated SDF fields" described in
for all versions between your current SIMPL version up to and included SIMPL 6.13.3.
Make sure to make the changes to
As per the SIMPL VM documentation, product-specific changes need to be described in product documentation.
These will be described below in Make product-specific changes to the SDF for RVT 4.1.
If you are upgrading a deployment on OpenStack, ensure you specify your OpenStack release under the
openstack section in
Do NOT yet update the version in
In RVT 4.1, SSH access to VMs is only available using SSH keys, while on RVT 4.0, SSH access was possible both using passwords and SSH keys. A key will need to be provisioned to allow you access to RVT 4.1 VMs.
For 4.1, the SSH key must be in PEM format;
it must not be an OpenSSH formatted key (the default format of keys created by
If your existing key is OpenSSH format, or if you did not use SSH keys for access to RVT 4.0.0 VMs,
generate a new one.
You can create a PEM formatted SSH key pair
using the command
ssh-keygen -b 4096 -m PEM -f /home/admin/rvt-ssh-key.
This will prompt for a passphrase; we recommend setting one for security reasons.
Keep the file
rvt-ssh-key safe, as it will be used to connect to the RVT 4.1 VMs.
This key is meant to be used by people who need to access the VMs directly. It is advised to keep this key safe and share with only those who needs to access the VMs directly.
As any rollback to RVT 4.0.0 will need to be done using the upgraded SIMPL VM, you need an updated copy of the SDF to perform rollbacks. Before you make further updates to the SDF for RVT 4.1, create a copy:
cp /home/admin/uplevel-config/sdf-rvt.yaml /home/admin/rvt-rollback-sdf
In Update the SDF for SIMPL 6.13.3, you updated the SDF for SIMPL 6.13.3. We now make further changes to the SDF, to support RVT 4.1.
Some of these changes are due to secrets now being stored securely. We will first set the secret identifiers in the SDF, and then provide instructions on how to store their values in the secrets store.
/home/admin/uplevel-config/sdf-rvt.yaml file using
vnfcs section, and within that every RVT VNFC (
For each of them, make changes as follows :
secrets-private-keyhas been replaced by
secrets-private-key-id, with the value being stored in the secrets store. Please store off the current value of the
secrets-private-keyline as you will need this at a later stage, then replace this line with
primary-user-passwordline exists, store off the value of this parameter as you will need it at a later stage, then remove the line. Regardless of whether
primary-user-passwordwas previously present, you must now insert this line
primary-user-password-id: rvt-primary-user-password. This field is mandatory.
tsnVNFC, add the appropriate cassandra version option (
custom-optionssection as shown below:
product-options: tsn: cds-addresses: - 172.18.1.10 - 172.18.1.11 - 172.18.1.12 custom-options: - log-passwords - cassandra_version_3_11
Failure to add the
cassandra_version_3_11custom option to the SDF when performing major TSN upgrades from 4.0.0 to 4.1 4.0 will result in TSN 4.1 being deployed with Cassandra Version 4.1.1, thus, not being able to join the cassandra cluster.
Skip this step if using OpenStack - it is only applicable for vSphere based deployments. For each VNFC type, except the SMO nodes, under the
networkssection find the entry which has a
cluster. Remove the entry if present. If it is not present, move onto the next VNFC type.
For example, if the current
networks section look like this:
networks: - ip-addresses: ip: - 172.16.0.11 name: Management subnet: management traffic-types: - management - ip-addresses: ip: - 172.17.0.11 name: Cluster subnet: cluster traffic-types: - cluster - ip-addresses: ip: - 172.18.0.11 name: Signaling subnet: signaling traffic-types: - internal - diameter - sip - ss7
you would remove the second list entry, and end up with this:
networks: - ip-addresses: ip: - 172.16.0.11 name: Management subnet: management traffic-types: - management - ip-addresses: ip: - 172.18.0.11 name: Signaling subnet: signaling traffic-types: - internal - diameter - sip - ss7
cluster-configuration, find the
instancessection. For every instance, ensure there is a section
sshas follows, where
your public keyis either the contents of your pre-existing public key, or the contents of
/home/admin/rvt-ssh-key.pubif you generated one above:
ssh: authorized-keys: - <your public key> private-key-id: rvt-simpl-private-key-id
Update the VM versions for all the VM types (
smo). Find the
vnfcssection, and within each VNFC, locate the
versionfield and change its value to the uplevel version, for example
type: mag - version: 4.0.0-9-1.0.0 + version: 4.1-3-1.0.0 vim-configuration:
Save and close the file.
csar secrets auto-create-keys --sdf /home/admin/uplevel-config/sdf-rvt.yaml.
This will generate the SSH key with ID
rvt-simpl-private-key-id, this key will be used by SIMPL VM to connect to the RVT VMs so should not be shared or kept elsewhere.
Then, generate a template
secrets_input_file.yaml file by running:
csar secrets create-input-file --sdf /home/admin/uplevel-config/sdf-rvt.yaml
Open the file
vi and using the secrets you stored in the previous steps, fill the values in as follows:
rvt-secrets-private-key: The value of
/home/admin/current-config/sdf-rvt.yaml. Note that there are multiple occurrences of
sdf-rvt.yaml, but they should all be equal. If this is not the case, contact your Customer Care Representative.
rvt-primary-user-password: What you want the password of the
sentineluser to be. This password is used when logging into the VM through the VNFI console, when SSH connectivity can’t be established.
Run the command
csar secrets add secrets_input_file.yaml to add the secrets to the secret store.
In the previous step, you generated an SSH key for the SIMPL VM to use to connect to the RVT VMs. During RVT upgrade to V4.1 (or later), this SSH key will automatically be installed onto the VMs. However, SIMPL VM needs to connect to the RVT V4.0 VMs as part of the upgrade process. To allow this, this newly generated key must manually be copied to the RVT V4.0 VMs. Ensure you copy the key generated in the previous step, not the key you generated in step 4.4.
csar secrets get-value rvt-simpl-private-key-id.
From the output, copy-paste from the line
-----BEGIN RSA PRIVATE KEY----- up to (and including) the line
-----END RSA PRIVATE KEY-----.
Create the file
vi, and paste the private key.
Save and close the file.
chmod 600 /home/admin/rvt-simpl-private-key to change the permissions.
ssh-keygen -y -f /home/admin/rvt-simpl-private-key > /home/admin/rvt-simpl-private-key.pub to generate the public key.
Finally, provision this public key on all the RVT 4.0.0 VMs, by running, for the management IP of every RVT VM:
ssh-copy-id -i /home/admin/rvt-simpl-private-key sentinel@<management IP>,
entering the current VM password when prompted.
The output will then look as below:
Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'sentinel@<management IP>'" and check to make sure that only the key(s) you wanted were added.
Some fields in the configuration files have been removed, deprecated or added.
Open the following files inside the directory
edit them as instructed, and then save them.
common-config.yaml: If present, remove the field
mag-vmpool-config.yaml: Ensure every entry in the list
xcap.(including a period).
mmt-gsm-vmpool-config.yaml: If present, remove the field
naf-filter-config.yaml: If present, remove the section
cassandra-connectivityand the fields
sentinel-ipsmgw-config.yaml: If present, remove the field
smo-vmpool-config.yaml: If not using Sentinel IPSMGW (i.e.
sentinel-ipsmgw-enabledis set to
Remove the field
diameter-ro-origin-hostfrom every entry in the
Remove the file
sentinel-volte-cdma-config.yaml: If present, under
scc.service-continuityremove the field
atu-sti, and under
sisremove the field
originating-address(do NOT remove it under
xcap-data-update, if present, remove the fields
shcm-vmpool-config.yaml: Underneath every
vm-id, add a field
where the first entry gets node ID
101, the second entry node ID
102, and so on.
smo-vmpool-config.yaml: If present, remove the field
notifications, check that
sgc-notifications-enabledare all present. If any of them are missing, add them with a value of
To improve security of ShCM, from RVT 4.1 onwards only nodes on an allowlist are allowed to connect to ShCM.
This allowlist automatically includes all RVT nodes.
However, if for any reason a non-RVT node needs to connect to ShCM directly to integrate with the ShCM API,
edit the file
and add an
additional-client-addresses section under
deployment-config:shcm-service: additional-client-addresses: - <IP 1> - <IP 2> - <IP 3>
(adding or removing lines to match the number of IPs required as necessary).
/home/admin/uplevel-config, check the files
Within each of these files, confirm that the first occurrence of
is set to the smallest value of all occurrences of
rhino-node-id: yyy in that particular file.
If not, contact your Customer Care Representative to adjust the upgrade steps in this MOP.
The vertical service codes (VSC) feature on the MMT nodes uses the XCAP server to assist in the handling of vertical service codes. If you do not use this feature, this step can be skipped.
Previously, the DNS generation tool generated an entry of the form
This is not a valid XCAP domain, and no longer accepted by RVT.
Therefore it needs to be updated to
On the SIMPL VM, open the file
and find the value for
Replace the prefix
Then, change to the home directory by running
cd /home/admin, followed by
csar create-dns-entries --sdf /home/admin/uplevel-config/sdf-rvt.yaml --dns-ip <IP address of your primary DNS server> --domain <ims-domain-name>
<ims-domain-name> can be found as the value of
This will write a BIND file
Either provision it to the customer’s DNS server, or open this file in a text editor
and manually verify all DNS entries in this file are present in the customer’s DNS server.
In particular, ensure the presence of the new
We now check that the uplevel configuration files are correctly formatted, contain valid values, and are self-consistent.
For each node type
run the command
/home/admin/.local/share/csar/<node type>/<uplevel version>/resources/rvtconfig validate -t <node type> -i /home/admin/uplevel-config
For example …
/home/admin/.local/share/csar/mag/4.1-2-1.0.0/resources/rvtconfig validate -t mag -i /home/admin/uplevel-config
A successful validation with no errors or warnings produces the following output.
Validating node type against the schema: <node type> YAML for node type(s) ['<node type>'] validates against the schema
If the output contains validation errors,
fix the configuration in the
and refer to the previous steps to fix the issues.
If the output contains validation warnings, consider whether you wish to address them before performing the upgrade. The VMs will accept configuration that has validation warnings, but certain functions may not work.