Recovery of TSN VMs

When recovering multiple nodes, check whether any of the nodes to be recovered are reported as being the leader based on the output of the rvtconfig report-group-status command. If any of the nodes to be recovered are the current leader, recover the leader node first. This helps to speed up the handover of group leadership, so that the recovery will complete faster.

In general, use the csar heal operation where possible instead of csar redeploy. The csar heal operation requires that the initconf process is active on the VM, and that the VM can reach both the CDS and MDM services, as reported by rvtconfig report-group-status. If any of those pre-requisites are not met for csar heal, use csar redeploy instead.

When report-group-status reports that a single node cannot connect to CDS or MDM, it should be considered a VM specific fault. In that case, use csar redeploy instead of csar heal. But a widespread failure of all the VMs in the group to connect to CDS or MDM suggest a need to investigate the health of the CDS and MDM services themselves, or the connectivity to them.

When recovering multiple VMs, you don’t have to consistently use either csar redeploy or csar heal commands for all nodes. Choose the appropriate command for each VM according to the guidance on this page instead.

Care must be taken recovering TSN nodes, to avoid having too many nodes being down, according to the TSN limitations section above, where for example only 2 nodes can be down for TSN groups of 5 or more nodes.

In this context, a TSN node being down includes:

If multiple TSNs are malfunctioning, follow this order of precedence for recovery:

The Cassandra processes on the non-recovering nodes must be UN (up and normal). The Cassandra processes on nodes other than the recovering node cannot be in the DN state when the replacement TSN node is started. If this situation is detected, the recovering VMs will wait for the offending node until it is either restored to UP status, or the the node is removed from the cluster. In the edge case where 2 TSNs needed to be redeployed at the same time, this situation will automatically resolve once both replacement TSN nodes come up.

In some situations it may be possible to recover a whole TSN cluster without having to delete the full deployment and deploy it again. This is a delicate operation that must only be tried as a last resort and only when the following conditions are met.

The command sequence required should be as follows.

VMs should be healed one at a time, reassessing the group status using the rvtconfig report-group-status command after each heal operation, as detailed below.

See the 'Healing a VM' section of the SIMPL VM Documentation for details on the csar heal command.

The command should be run as follows:

VMs should be redeployed one at a time, reassessing the group status using the rvtconfig report-group-status command after each heal operation, as detailed below. Exceptions to this rules are noted on this page.

See the 'Healing a VM' section of the SIMPL VM Documentation for details on the csar redeploy command.

The command should be run as follows:

If the output of report-group-status indicates an unintended recovery to the wrong version, follow the procedure in Troubleshooting accidental VM recovery to recover.

In some cases after recovering 2 TSNs nodes simultaneously, manual intervention may be required, if they weren’t able to successfully heal. In this case, see the initconf.log on the affected VMs for guidance.

This case is detectable based on the following: