rvtconfig tool

Configuration YAML files can be validated and uploaded to the CDS using the rvtconfig tool. The rvtconfig tool can be run either on the SIMPL VM or any Rhino VM Automation VM.

On the SIMPL VM, you can find the command in the resources subdirectory of any Rhino VM Automation (rem) CSAR, after it has been extracted using csar unpack.

/home/admin/.local/share/csar/<csar name>/<version>/resources/rvtconfig

On any Rhino VM Automation VM, the rvtconfig tool is in the PATH for the sentinel user and can be run directly by running:

rvtconfig <command>

The available rvtconfig commands are:

  • rvtconfig validate validates the configuration, even before booting any VMs by using the SIMPL VM.

  • rvtconfig upload-config validates, encrypts, and uploads the configuration to the CDS.

  • rvtconfig delete-deployment deletes a deployment from the CDS.

    Note Only use this when advised to do so by a Customer Care Representative.
  • rvtconfig delete-node-type-version deletes state and configuration for a specified version of a given node type from the CDS.

    Note This should only be used when there are no VMs of that version deployed.
  • rvtconfig delete-node-type-all-versions deletes state and configuration for all versions of a given node type from the CDS.

    Note Only use this after deleting all VMs for a given node type.
  • rvtconfig delete-node-type-retain-version deletes state and configuration for a given node type from the CDS, except for the specified version.

  • rvtconfig list-config displays a summary of the configurations stored in the CDS.

  • rvtconfig dump-config dumps the current configuration from the CDS.

  • rvtconfig print-leader-seed prints the current leader seed as stored in the CDS.

  • rvtconfig generate-private-key generates a new private key for use in the SDF.

  • rvtconfig enter-maintenance-window disables VMs' scheduled tasks for a period of time.

  • rvtconfig leave-maintenance-window re-enables VMs' scheduled tasks.

  • rvtconfig maintenance-window-status displays a message indicating whether there is an maintenance window period reserved or not.

  • rvtconfig export-audit-history exports the Rhino license audit history from the CDS.

  • rvtconfig export-log-history exports the quiesce log history from the CDS.

  • rvtconfig initconf-log retrieves initconf.log file from the specified remote RVT node.

  • rvtconfig describe-versions prints the current values of the versions of the VM found in the config and in the SDF.

  • rvtconfig compare-config compares currently uploaded config with a given set of configuration.

  • rvtconfig backup-cds creates a backup of the CDS database in tar format and retrieves it.

  • rvtconfig restore-cds uses CDS database backup taken with backup-cds to restore the CDS database to a previous state.

  • rvtconfig set-desired-running-state sets DesiredRunningState to stopped/started in MDM.

    • If --state Started or no --state is specified, all initconf processes of non-TSN VMs will pause their configuration loops.

    • If --state Stopped is specified, all initconf processes of non-TSN VMs will resume their configuration loops.

  • rvtconfig cassandra-upgrade performs a cassandra upgrade operation from {cassandra-version-three} to {cassandra-version-four}. This command can only be used after a Major TSN upgrade has been successfully executed to TSN 4.1 and the cassandra version running is {cassandra-version-three}. This operation must be done one TSN node at a time and no parallelization is allowed.

  • rvtconfig cassandra-status prints the cassandra database status of all the specified CDS IP addresses.

  • rvtconfig cassandra-upgrade-sstables upgrades sstables status once all TSN 4.1 nodes have been upgraded to Cassandra version {cassandra-version-four} with rvtconfig cassandra-upgrade

Common arguments

Commands that read or modify CDS state take a --cds-address parameter (which is also aliased as --cds-addresses, --cassandra-contact-point, --cassandra-contact-points, or simply -c). For this parameter, specify the management address(es) of at least one machine hosting the CDS database. Separate multiple addresses with a space, for example --cds-address 1.2.3.4 1.2.3.5.

The upload-config and export-audit-history commands read secrets from QSG. If you have not yet uploaded secrets to QSG, you can specify a --secrets-file <file> argument, passing in the path to your secrets file (the YAML file which you pass to csar secrets add). QSG is only available on the SIMPL VM; if running rvtconfig on a platform other than the SIMPL VM, for example on the VM itself, then you must pass the --secrets-file argument.

Commands that read or modify CDS state may also require additional parameters if the CDS endpoints are configured to use authentication and/or SSL encryption as per Cassandra security configuration. If the CDS endpoints are configured to use authentication, you must pass the --cds-username argument with your configured password and either the --cds-password or --cds-password-secret-name argument with the configured password or its ID in the secrets file. If the CDS endpoints are configured to use SSL encryption, you must pass the --ssl flag and also pass either the --ssl-ca-certificate or --ssl-ca-certificate-secret-name argument containing a file with the SSL signing certificate, or its ID in the secrets file.

The various delete-node-type commands, and the report-group-status command, require an SSH private key to access the VMs. You can specify this key as either a path to the private key file with the --ssh-key argument, or as a secret ID with the --ssh-key-secret-id argument. If you are running rvtconfig on the SIMPL VM, the recommended approach is to use the secret ID of the SIMPL VM-specific private key that you specified in the SDF (see SIMPL VM SSH private key ). Otherwise, use the SSH private key file itself (copying it to the machine on which you are running rvtconfig, and deleting it once you have finished, if necessary).

For more information, run rvtconfig --help. You can also view help about a particular command using, for example, rvtconfig upload-config --help.

rvtconfig limitations

The following limitations apply when running rvtconfig on the SIMPL VM:

  1. All files and directories mentioned in parameter values and the secrets file must reside within the root (/) filesystem of the SIMPL VM. A good way to ensure this is the case is to store files only in directories under /home/admin.

  2. rvtconfig assumes files specified without paths are located in the current directory. If multiple directories are involved, it is recommended to use absolute paths everywhere. (Relative paths can be used, but may not use .. to navigate out of the current directory.)


Note

The node type name and the rvtconfig command <node type> value are not always equal. For rvtconfig commands, the REM node type(s) use the rem value(s) respectively.

Verifying and uploading configuration

  1. Create a directory to hold the configuration YAML files.

    mkdir yamls
  2. Ensure the directory contains the following:

    • configuration YAML files

    • the Solution Definition File (SDF)

    • Rhino license for nodes running Rhino.

Note Do not create any subdirectories. Ensure the file names match the example YAML files.

Verifying configuration with validate

To validate configuration, run the command:

rvtconfig validate -t <node type> -i ~/yamls

where <node type> is the node type you want to verify, which can be rem. If there are any errors, fix them, move the fixed files to the yamls directory, and then re-run the above rvtconfig validate command on the yamls directory.

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Warning Once the files pass validation, store the YAML files in the CDS using the rvtconfig upload-config command.
Tip

If using the SIMPL VM, the rvtconfig validate command can be run before any of the other VMs are booted. We recommend that you validate all configuration before any of the VMs are booted.

Uploading configuration to the CDS with upload-config

To upload the YAML files to the CDS, run the command:

rvtconfig upload-config [--secrets-file <file>] -c <cds-mgmt-addresses> -t <node type> -i ~/yamls
[(--vm-version-source [this-vm | this-rvtconfig | sdf-version] | --vm-version <vm_version>)] [--reload-resource-adaptors]

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Note

The <cds-mgmt-addresses> value can either be any single CDS management IP address or a space-separated list of CDS management IP addresses.

If you would like to specify a version, you can use:

  • --vm-version to specify the exact version of the VM to target (as configuration can differ across a VM upgrade).

  • --vm-version-source to automatically derive the VM version from the given source. Failure to determine the version will result in an error.

    • Use this-rvtconfig when running the rvtconfig tool included in the CSAR for the target VM, to extract the version information packaged into rvtconfig.

    • Use this-vm if running the rvtconfig tool directly on the VM being configured, to extract the version information from the VM.

    • Option sdf-version extracts the version value written in the SDF for the given node.

If the version is not specified, then the version in the SDF will be compared to the this-rvtconfig or this-vm version (whichever is appropriate given how the rvtconfig command is run). If they match, this value will be used. Otherwise, the command will fail.

Note

Whatever way you enter the version, the value obtained must match the version in the SDF. Otherwise, the upload will fail.

Any YAML configuration values which are specified as secrets are marked as such in the YAML files' comments. These values will be encrypted using the generated private-key created by rvtconfig generate-private-key and prior to uploading the SDF. In other words, the secrets should be entered in plain text in the SDF, and the upload-config command takes care of encrypting them. Currently this applies to the following:

  • Rhino users' passwords

  • REM users' passwords

  • SSH keys for accessing the VM

  • the SNMPv3 authentication key and privacy key

  • the HTTPS key and certificate for REM.

Tip

Use the rvtconfig describe-versions command to view the exact version values provided by this-vm, this-rvtconfig, and sdf-version.

If the CDS is not yet available, this will retry every 30 seconds for up to 15 minutes. As a large Cassandra cluster can take up to one hour to form, this means the command could time out if run before the cluster is fully formed. If the command still fails after several attempts over an hour, troubleshoot Cassandra on the machines hosting the CDS database.

This command first compares the configuration files currently uploaded for the target version with those in the input directory. It summarizes which files are different, how many lines differ, and if there are any configuration changes that are unsupported (for example, changing the VMs' IP addresses). If there are any unsupported configuration changes, the config will not be uploaded. Follow the instructions in the error message(s) to revert unsupported changes in the configuration, then try again.

If the changes are valid, but any files are different, rvtconfig will prompt the user to confirm the differences are as expected before continuing with the upload. If the upload is canceled, and --output-dir is specified, then full details of any files with differences will be put into the given output directory, which rvtconfig creates if it doesn’t already exist.

Changes to secrets and non-YAML files cannot be detected due to encryption; they will not appear in the summary or detailed output. Any such changes will still be uploaded.

You can disable this pre-upload check on config differences using the --skip-diff flag (also aliased as -f).

Caution
Restarting resource adaptors

Specify the --reload-resource-adaptors option whenever you upload configuration where you have changed the values of any YAML configuration fields that require a restart of one or more Rhino resource adaptors (RAs). Which fields these are will be defined by your application.

The --reload-resource-adaptors option instructs initconf to restart RAs where required. USE THIS OPTION WITH CAUTION, as it will cause a short service outage across all nodes in the deployment. It is strongly advised that you only make changes requiring RA restarts during a maintenance window.

If you apply configuration changes that don’t include changes to any fields marked as needing an RA restart, then you do not need to specify the --reload-resource-adaptors option to rvtconfig upload-config.

If you apply configuration changes that include changes to such fields, and do not specify the --reload-resource-adaptors option, you may see Rhino alarms stating that restarting a certain resource adaptor(s) is required for configuration to take effect. You can clear these by manually restarting the affected RA(s), or Rhino itself, on the affected nodes at a convenient time.

Comparing existing configuration in the CDS with compare-config

Compare the configuration in an input directory with the currently uploaded configuration in the CDS using the command:

rvtconfig compare-config -c <cds-mgmt-addresses> -t <node type> -i ~/yamls --output-dir <output-directory>
[--deployment-id <deployment ID>] [--site-id <site ID>] [(--vm-version-source [this-vm | this-rvtconfig | sdf-version] | --vm-version <vm_version>)]

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Warning This will compare the currently uploaded configuration in the CDS with the configuration in the local input directory.

The deployment ID, site ID, and version of configuration to look up in CDS will be automatically taken from the SDF. These can be overridden by using the --deployment-id, --site-id, and one of the --vm-version-source or --vm-version parameters respectively. For example, you can specify --vm-version <downlevel version> to check what has changed just before running an upgrade, where the version in the input SDF will be the uplevel version.

The files that have differences will be displayed, along with the number of different lines, and any errors or warnings about the changes themselves. Any errors will need to be corrected before you can run rvtconfig upload-config.

The command puts the full contents of each version of these files into the output directory, along with separate files showing the differences found. The command ignores non-YAML files and any secrets in YAML files. The files in this output directory use the suffix .local for a redacted version of the input file, .live for a redacted version of the live file, and .diff for a file showing the differences between the two.

Note The contents of the files in the output directory are reordered and no longer have comments; these won’t match the formatting of the original input files, but contain the same information.

Deleting configuration from the CDS with delete-deployment

Delete all deployment configuration from the CDS by running the command:

rvtconfig delete-deployment -c <cds-mgmt-addresses> -d <deployment-id> [--delete-audit-history]

Note --delete-audit-history will delete stored audit logs with the deployment data.
Warning Only use this when advised to do so by a Customer Care Representative.
Warning Only use this after deleting all VMs of the deployment within the specified site. Functionality of all nodes of this type and version within the given site will be lost. These nodes will have to be deployed again to restore functionality.

Deleting state and configuration for a specific node type and version from the CDS with delete-node-type-version

Delete all state and configuration for a given node type and version from the CDS by running the command:

rvtconfig delete-node-type-version -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Note The argument -i ~/yamls is only needed if sdf-version is used.
Warning Only use this after deleting all VMs of this node type and version within the specified site. Functionality of all nodes of this type and version within the given site will be lost. These nodes will have to be deployed again to restore functionality.

Deleting all state and configuration for a specific node type from the CDS with delete-node-type-all-versions

Delete all state and configuration for a given node type from the CDS by running the command:

rvtconfig delete-node-type-all-versions -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id>
--node-type <node type> (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [--delete-certificates] [-y]

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Warning Only use this after deleting all VMs of this node type within the specified site. Functionality of all nodes of this type within the given site will be lost. These nodes will have to be deployed again to restore functionality.
Warning The --delete-certificates option should only be used when advised by a Customer Care Representative.

Deleting historical state and configuration for a given node type from the CDS with delete-node-type-retain-version

Remove all state and configuration relating to a versions of the node type other than the specified version from CDS by running the command:

rvtconfig delete-node-type-retain-version -c <cds-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node-type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Note The argument -i ~/yamls is only needed if sdf-version is used.
Warning The version specified in this command must be the only running VM version for this node type. i.e. do not use during an upgrade or rollback when multiple versions of the same node type may be running. All state and configuration relating to other versions will be deleted from CDS.

Removing unused Rhino-generated keyspaces

Following an upgrade or rollback, you may wish to clean up keyspaces in the Cassandra database from version(s) that are no longer in use. This conserves memory and disk space.

To clean up unused keyspaces, use the following command:

rvtconfig remove-unused-keyspaces -c <cds-mgmt-addresses> -d <deployment-id> -g <group-id> [-y]

Note Group ID syntax: RVT-<node type>.<site ID>
Example: RVT-tsn.DC1
Here, <node type> can be rem.

Confirm that the active VM versions that the command identifies are correct. rvtconfig removes keyspaces relating to all other versions from Cassandra.

Listing configurations available in the CDS with list-config

List all currently available configurations in the CDS by running the command:

rvtconfig list-config -c <cds-mgmt-addresses> -d <deployment-id>

This command will print a short summary of the configurations uploaded, the VM version they are uploaded for, and which VMs are commissioned in that version.

Retrieving configuration from the CDS with dump-config

Retrieve the VM group configuration from the CDS by running the command:

rvtconfig dump-config -c <cds-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)
[--output-dir <output-dir>]

Note Group ID syntax: RVT-<node type>.<site ID>
Example: RVT-tsn.DC1
Here, <node type> can be rem.

If the optional --output-dir <directory> argument is specified, then the configuration will be dumped as individual files in the given directory. The directory can be expressed as either an absolute or relative path. It will be created if it doesn’t exist.

If the --output-dir argument is omitted, then the configuration is printed to the terminal.

Note The arguments -i ~/yamls and -t <node type> are only needed if sdf-version is used.
Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).

Displaying the current leader seed with print-leader-seed

Display the current leader seed by running the command:

rvtconfig print-leader-seed -c <cds-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)

Note Group ID syntax: RVT-<node type>.<site ID>
Example: RVT-tsn.DC1
Here, <node type> can be rem.

The command will display the current leader seed for the specified deployment, group, and VM version. A leader seed may not always exist, in which case the output will include No leader seed found. Conditions where a leader seed may not exist include:

  • No deployment exists with the specified deployment, group, and VM version.

  • A deployment exists, but initconf has not yet initialized.

  • A deployment exists, but the previous leader seed has quiesced and a new leader seed has not yet been selected.

Note The arguments -i ~/yamls and -t <node type> are only needed if sdf-version is used.
Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).

Generating a secrets-private-key for Encrypting Secrets with generate-private-key

Some configuration, for example Rhino or REM users' passwords, are configured in plaintext, but stored encrypted in CDS for security. rvtconfig automatically performs this encryption using a secrets private key which you configure in the SDF. This key must be a Fernet key, in Base64 format. Use the following rvtconfig command to generate a suitable secrets private key:

rvtconfig generate-private-key

Add the generated secrets private key to your secrets input file when adding secrets to QSG.

Maintenance window support

The rvtconfig enter-maintenance-window and rvtconfig leave-maintenance-window commands allow you to pause and resume scheduled tasks (Rhino restarts, SBB/activity cleanup, and Cassandra repair) on the VMs for a period of time. This is useful to avoid the scheduled tasks interfering with maintenance window activities, such as patching a VM or making substantial configuration changes.

To start a maintenance window, use

rvtconfig enter-maintenance-window -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id> [--hours <hours>]

  • The <site-id> is in the form DC1 to DC32. It can be found in the SDF.

  • The number of hours defaults to 6 if not specified, and must be between 1 and 24 hours.

Once started, the maintenance window can be extended by running the same command again (but not shortened). rvtconfig will display the end time of the maintenance window in the command output. Until this time, all scheduled tasks on all VMs in the specified site will not be run.

Warning

Any scheduled tasks which are in progress at the time the maintenance window is started will continue until they are finished. If the maintenance window is starting around the time of a scheduled task as configured in the YAML files, it is advisable to manually check that the task is complete before starting maintenance (or run the rvtconfig enter-maintenance-window command in advance of the scheduled task time).

When the maintenance window is complete, use the following command:

rvtconfig leave-maintenance-window -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id>

Scheduled tasks will now resume as per their configured schedules.

To check whether or not a maintenance window is currently active, use the following command:

rvtconfig maintenance-window-status -c <cds-mgmt-addresses> -d <deployment-id> -S <site-id>

Retrieving deployment TAS audit logs with export-audit-history

During upgrade, when a downlevel VM is removed, it uploads the Rhino TAS audit logs to CDS. Retrieve the TAS audit logs for a deployment from the CDS by running the command:

rvtconfig export-audit-history [--secrets-file <file>] -c <cds-mgmt-addresses> -d <deployment-id> --zip-destination-dir <directory>

Note The audit logs retrieved are compressed in a zip file per VM and version.

Retrieving VM logs with export-log-history

During upgrade, when a downlevel VM is removed, it uploads Initconf and Rhino logs to the CDS. The log files are stored as encrypted data in the CDS.

Note Only the portions of the logs written during quiesce are stored.

Retrieve the VM logs for a deployment from the CDS by running the command:

rvtconfig export-log-history -c <cds-mgmt-addresses> -d <deployment-id> --zip-destination-dir <directory>
--secrets-private-key-id <secrets-private-key-id>

Note The --secrets-private-key-id must match the ID used in the SDF (secrets-private-key-id).
Note The Initconf and Rhino logs are exported in unencrypted zip files. The zip file names will consist of VM hostname, version, and type of log.

Viewing the values associated with the special sdf-version, this-vm, and this-rvtconfig versions with describe-versions

Some commands, upload-config for example, can be used with the special version values sdf-version, this-vm, and this-rvtconfig.

  • Calling sdf-version extracts the version from the value given in the SDF for a given node.

  • The this-vm option takes the version of the VM the command is being run from. This can only be used when the commands are run on a node VM.

  • Using this-rvtconfig extracts the version from the rvtconfig found in the directory the command is being run from. This can only be used on a SIMPL VM.

To view the real version strings associated with each of these special values:

rvtconfig describe-versions [-i ~/yamls]

Optional argument -i ~/yamls is required for the sdf-version value to be given. If it is called, the sdf-version will be found for each node type in the SDF. If a node type is expected but not printed this may be because the config yaml files for that node are invalid or not present in the ~/yamls directory.

If a special version value cannot be found, for example if this-vm is run on a SIMPL VM or the optional argument is not called, the describe-versions command will print N/A for that special version.

Reporting group status, to help guide VM recovery

This command reports the status of each node in the given group, providing information to help inform which approach to take when recovering VMs.

It connects to each of the VMs in the group via SSH, as well as querying the CDS service. It then prints a detailed summary of status information for each VM, as well as a high level summary of the status of the group.

It does not log its output to a file. When using this command to aid in recovery operations, it’s good practice to redirect its output to a file locally on disk, which can then be used as part of any root cause analysis efforts afterwards.

On the SIMPL VM, run the command as follows, under the resources dir of the unpacked CSAR:

./rvtconfig report-group-status -c <cds-mgmt-addresses> -d <deployment-id> \
  --g <group-id> --ssh-key-secret-id <simpl-private-key-id>
Note Group ID syntax: RVT-<node type>.<site ID>
Example: RVT-tsn.DC1
Here, <node type> can be rem.

Gathering diagnostics and initconf log files

It is possible to obtain diagnostic files from RVT nodes with the command rvtconfig gather-diags. These diagnostic files include system files and solution configuration files, are packaged as a tar.gz file and deposited in the given output directory. Depending on the node type there will be different kinds of solution configuration files. These files can be crucial to troubleshoot problems on the VMs.

./rvtconfig gather-diags --sdf <SDF File> -t <node type> --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --output-dir <output-directory>

Warning If <node type> is set to custom value, the user needs to include --custom-node-image-name <image name>, where <image name> must be set to the type parameter specified at that vnfc section for the corresponding service group in the SDF file (SDF Overview).
Warning If you need to quickly check the initconf.log file from a certain VM or VMs, it is possible to do it with the command rvtconfig initconf-log. This command executes a tail on the initconf.log file of the specified VM or VMs and dumps it to the standard output.

rvtconfig initconf-log --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --ip-addresses <Space separated VM IP address list> --tail <num lines>

Operate the TSN Cassandra Database

From RVT 4.1-3-1.0.0, the TSN nodes can be deployed with Cassandra version {cassandra-version-three} or {cassandra-version-four}. Both Cassandra versions are installed in the VM Image, but only one is active. The commands rvtconfig cassandra-upgrade and rvtconfig cassandra-upgrade-sstables allow you to perform a Cassandra upgrade from {cassandra-version-three} to {cassandra-version-four} on a running TSN VM 4.1-3-1.0.0 or higher. These two commands must only be run if the target TSNs are running Cassandra {cassandra-version-three}.

To upgrade a single TSN node from {cassandra-version-three} to {cassandra-version-four} you can run ./rvtconfig cassandra-upgrade --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Address> for every TSN VM, one by one, and not in parallel.

Once all TSN nodes have been upgraded to {cassandra-version-four}, we must perform a sstables upgrade operation with the following command ./rvtconfig cassandra-upgrade-sstables --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Addresses>

Additionally the command rvtconfig cassandra-status prints the cassandra database status for the specified CDS IP addresses. Here is a couple of examples:

  • ./rvtconfig cassandra-status --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Address 1> <TSN Address 2> …​

CDS Backup and Restore operations.

From RVT 4.1-3-1.0.0, the TSNs' CDS database can be backed up and restored. This provides a faster recovery procedure in case TSN upgrades go wrong.

To backup the CDS of a running TSN cluster, run ./rvtconfig backup-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --output-dir <backup-cds-bundle-dir> --ssh-key-secret-id <SSH key secret ID> -c <CDS address> <CDS auth args>

To restore the CDS of a running TSN cluster, run ./rvtconfig restore-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --snapshot-file <backup-cds-bundle-dir>/tsn_cassandra_backup.tar --ssh-key-secret-id <SSH key secret ID> -c <CDS Address> <CDS auth args>

Warning Only use restore-cds when advised to do so by a Customer Care Representative.

Control initconf configuration loop in non-TSN nodes.

During maintenance windows which involve upgrading TSN nodes, the command rvtconfig set-desired-running-state allows you stop/start the configuration tasks performed by the initconf that read from the CDS database in all non-TSN VMs. This operation does not stop the non-TSN VMs or the initconf process within it. But it instructs the initconf to pause or resume, the configuration tasks, while operating normally under traffic.

To pause initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Stopped.

To resume initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Started.

Previous page Next page
VM Build Container Version 3.2