rvtconfig

Commands that read or modify CDS state take a --cds-address parameter (which is also aliased as --cds-addresses, --cassandra-contact-point, --cassandra-contact-points, or simply -c). For this parameter, specify the management address(es) of at least one TSN node. Separate multiple addresses with a space, for example --cds-address 1.2.3.4 1.2.3.5.

The upload-config and export-audit-history commands read secrets from QSG. If you have not yet uploaded secrets to QSG, you can specify a --secrets-file <file> argument, passing in the path to your secrets file (the YAML file which you pass to csar secrets add). QSG is only available on the SIMPL VM; if running rvtconfig on a platform other than the SIMPL VM, for example on the VM itself, then you must pass the --secrets-file argument.

Commands that read or modify CDS state may also require additional parameters if the CDS endpoints are configured to use authentication and/or SSL encryption as per Cassandra security configuration. If the CDS endpoints are configured to use authentication, you must pass the --cds-username argument with your configured password and either the --cds-password or --cds-password-secret-name argument with the configured password or its ID in the secrets file. If the CDS endpoints are configured to use SSL encryption, you must pass the --ssl flag and also pass either the --ssl-ca-certificate or --ssl-ca-certificate-secret-name argument containing a file with the SSL signing certificate, or its ID in the secrets file.

The various delete-node-type commands, and the report-group-status command, require an SSH private key to access the VMs. You can specify this key as either a path to the private key file with the --ssh-key argument, or as a secret ID with the --ssh-key-secret-id argument. If you are running rvtconfig on the SIMPL VM, the recommended approach is to use the secret ID of the SIMPL VM-specific private key that you specified in the SDF (see SIMPL VM SSH private key ). Otherwise, use the SSH private key file itself (copying it to the machine on which you are running rvtconfig, and deleting it once you have finished, if necessary).

For more information, run rvtconfig --help. You can also view help about a particular command using, for example, rvtconfig upload-config --help.

The following limitations apply when running rvtconfig on the SIMPL VM:

Compare the configuration in an input directory with the currently uploaded configuration in the CDS using the command:

rvtconfig compare-config -c <cds-mgmt-addresses> -t <node type> -i ~/yamls --output-dir <output-directory>
[--deployment-id <deployment ID>] [--site-id <site ID>] [(--vm-version-source [this-vm | this-rvtconfig | sdf-version] | --vm-version <vm_version>)]

This will compare the currently uploaded configuration in the CDS with the configuration in the local input directory.

The deployment ID, site ID, and version of configuration to look up in CDS will be automatically taken from the SDF. These can be overridden by using the --deployment-id, --site-id, and one of the --vm-version-source or --vm-version parameters respectively. For example, you can specify --vm-version <downlevel version> to check what has changed just before running an upgrade, where the version in the input SDF will be the uplevel version.

The files that have differences will be displayed, along with the number of different lines, and any errors or warnings about the changes themselves. Any errors will need to be corrected before you can run rvtconfig upload-config.

The command puts the full contents of each version of these files into the output directory, along with separate files showing the differences found. The command ignores non-YAML files and any secrets in YAML files. The files in this output directory use the suffix .local for a redacted version of the input file, .live for a redacted version of the live file, and .diff for a file showing the differences between the two.

Delete all deployment configuration from the CDS by running the command:

rvtconfig delete-deployment -c <tsn-mgmt-addresses> -d <deployment-id> [--delete-audit-history]

Delete all state and configuration for a given node type and version from the CDS by running the command:

rvtconfig delete-node-type-version -c <tsn-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Delete all state and configuration for a given node type from the CDS by running the command:

rvtconfig delete-node-type-all-versions -c <tsn-mgmt-addresses> -d <deployment-id> --site-id <site-id>
--node-type <node type> (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [--delete-certificates] [-y]

Remove all state and configuration relating to a versions of the node type other than the specified version from CDS by running the command:

rvtconfig delete-node-type-retain-version -c <tsn-mgmt-addresses> -d <deployment-id> --site-id <site-id> --node-type <node-type>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls] | --vm-version <vm_version>) (--ssh-key SSH_KEY | --ssh-key-secret-id SSH_KEY_SECRET_ID) [-y]

Following an upgrade or rollback, you may wish to clean up keyspaces in the Cassandra ramdisk database from version(s) that are no longer in use. This conserves memory and disk space.

To clean up unused keyspaces, use the following command:

rvtconfig remove-unused-keyspaces -c <tsn-mgmt-addresses> -d <deployment-id> -g <group-id> [-y]

Confirm that the active VM versions that the command identifies are correct. rvtconfig removes keyspaces relating to all other versions from Cassandra.

List all currently available configurations in the CDS by running the command:

rvtconfig list-config -c <tsn-mgmt-addresses> -d <deployment-id>

This command will print a short summary of the configurations uploaded, the VM version they are uploaded for, and which VMs are commissioned in that version.

Retrieve the VM group configuration from the CDS by running the command:

rvtconfig dump-config -c <tsn-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)
[--output-dir <output-dir>]

If the optional --output-dir <directory> argument is specified, then the configuration will be dumped as individual files in the given directory. The directory can be expressed as either an absolute or relative path. It will be created if it doesn’t exist.

If the --output-dir argument is omitted, then the configuration is printed to the terminal.

Display the current leader seed by running the command:

rvtconfig print-leader-seed -c <tsn-mgmt-addresses> -d <deployment-id> --group-id <group-id>
(--vm-version-source [this-vm | this-rvtconfig | sdf-version -i ~/yamls -t <node type>] | --vm-version <vm_version>)

The command will display the current leader seed for the specified deployment, group, and VM version. A leader seed may not always exist, in which case the output will include No leader seed found. Conditions where a leader seed may not exist include:

Some configuration, for example Rhino or REM users' passwords, are configured in plaintext, but stored encrypted in CDS for security. rvtconfig automatically performs this encryption using a secrets private key which you configure in the SDF. This key must be a Fernet key, in Base64 format. Use the following rvtconfig command to generate a suitable secrets private key:

rvtconfig generate-private-key

Add the generated secrets private key to your secrets input file when adding secrets to QSG.

The rvtconfig enter-maintenance-window and rvtconfig leave-maintenance-window commands allow you to pause and resume scheduled tasks (Rhino restarts, SBB/activity cleanup, and Cassandra repair) on the VMs for a period of time. This is useful to avoid the scheduled tasks interfering with maintenance window activities, such as patching a VM or making substantial configuration changes.

To start a maintenance window, use

rvtconfig enter-maintenance-window -c <tsn-mgmt-addresses> -d <deployment-id> -S <site-id> [--hours <hours>]

Once started, the maintenance window can be extended by running the same command again (but not shortened). rvtconfig will display the end time of the maintenance window in the command output. Until this time, all scheduled tasks on all VMs in the specified site will not be run.

When the maintenance window is complete, use the following command:

rvtconfig leave-maintenance-window -c <tsn-mgmt-addresses> -d <deployment-id> -S <site-id>

Scheduled tasks will now resume as per their configured schedules.

To check whether or not a maintenance window is currently active, use the following command:

rvtconfig maintenance-window-status -c <tsn-mgmt-addresses> -d <deployment-id> -S <site-id>

The rvtconfig calculate-maintenance-window commands allows you to estimate how long an upgrade or rollback is expected to take, so that an adequate maintenance window can be scheduled.

To calculate the recommended maintenance window duration, use

rvtconfig calculate-maintenance-window -i ~/yamls -t <node type> -s <site-id> [--index-range <index range>]

During upgrade, when a downlevel VM is removed, it uploads Initconf, Rhino and SGC logs to the CDS. The log files are stored as encrypted data in the CDS.

Retrieve the VM logs for a deployment from the CDS by running the command:

rvtconfig export-log-history -c <tsn-mgmt-addresses> -d <deployment-id> --zip-destination-dir <directory>
--secrets-private-key-id <secrets-private-key-id>

Some commands, upload-config for example, can be used with the special version values sdf-version, this-vm, and this-rvtconfig.

To view the real version strings associated with each of these special values:

rvtconfig describe-versions [-i ~/yamls]

Optional argument -i ~/yamls is required for the sdf-version value to be given. If it is called, the sdf-version will be found for each node type in the SDF. If a node type is expected but not printed this may be because the config yaml files for that node are invalid or not present in the ~/yamls directory.

If a special version value cannot be found, for example if this-vm is run on a SIMPL VM or the optional argument is not called, the describe-versions command will print N/A for that special version.

This command reports the status of each node in the given group, providing information to help inform which approach to take when recovering VMs.

It connects to each of the VMs in the group via SSH, as well as querying the CDS service. It then prints a detailed summary of status information for each VM, as well as a high level summary of the status of the group.

It does not log its output to a file. When using this command to aid in recovery operations, it’s good practice to redirect its output to a file locally on disk, which can then be used as part of any root cause analysis efforts afterwards.

On the SIMPL VM, run the command as follows, under the resources dir of the unpacked CSAR:

It is possible to obtain diagnostic files from RVT nodes with the command rvtconfig gather-diags. These diagnostic files include system files and solution configuration files, are packaged as a tar.gz file and deposited in the given output directory. Depending on the node type there will be different kinds of solution configuration files. These files can be crucial to troubleshoot problems on the VMs.

./rvtconfig gather-diags --sdf <SDF File> -t <node type> --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --output-dir <output-directory>

If you need to quickly check the initconf.log file from a certain VM or VMs, it is possible to do it with the command rvtconfig initconf-log. This command executes a tail on the initconf.log file of the specified VM or VMs and dumps it to the standard output.

rvtconfig initconf-log --ssh-key-secret-id <SSH key secret ID> --ssh-username sentinel --ip-addresses <Space separated VM IP address list> --tail <num lines>

From RVT 4.1-3-1.0.0, the TSN nodes can be deployed with Cassandra version 3.11.13 or 4.1.3. Both Cassandra versions are installed in the VM Image, but only one is active. The commands rvtconfig cassandra-upgrade and rvtconfig cassandra-upgrade-sstables allow you to perform a Cassandra upgrade from 3.11.13 to 4.1.3 on a running TSN VM 4.1-3-1.0.0 or higher. These two commands must only be run if the target TSNs are running Cassandra 3.11.13.

To upgrade a single TSN node from 3.11.13 to 4.1.3 you can run ./rvtconfig cassandra-upgrade --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Address> for every TSN VM, one by one, and not in parallel.

Once all TSN nodes have been upgraded to 4.1.3, we must perform a sstables upgrade operation with the following command ./rvtconfig cassandra-upgrade-sstables --ssh-key-secret-id <SSH key secret ID> --ip-addresses <TSN Addresses>

Additionally the command rvtconfig cassandra-status prints the cassandra database status for the specified CDS IP addresses. Here is a couple of examples:

From RVT 4.1-3-1.0.0, the TSNs' CDS database can be backed up and restored. This provides a faster recovery procedure in case TSN upgrades go wrong.

To backup the CDS of a running TSN cluster, run ./rvtconfig backup-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --output-dir <backup-cds-bundle-dir> --ssh-key-secret-id <SSH key secret ID> -c <CDS address> <CDS auth args>

To restore the CDS of a running TSN cluster, run ./rvtconfig restore-cds --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --snapshot-file <backup-cds-bundle-dir>/tsn_cassandra_backup.tar --ssh-key-secret-id <SSH key secret ID> -c <CDS Address> <CDS auth args>

During maintenance windows which involve upgrading TSN nodes, the command rvtconfig set-desired-running-state allows you stop/start the configuration tasks performed by the initconf that read from the CDS database in all non-TSN VMs. This operation does not stop the non-TSN VMs or the initconf process within it. But it instructs the initconf to pause or resume, the configuration tasks, while operating normally under traffic.

To pause initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Stopped.

To resume initconf configuration tasks of all non-TSN VMs, run ./rvtconfig set-desired-running-state --sdf /home/admin/uplevel-config/sdf-rvt.yaml --site-id <site ID> --state Started.