Recovery of MCP VMs

When recovering multiple nodes, check whether any of the nodes to be recovered are reported as being the leader based on the output of the rvtconfig report-group-status command. If any of the nodes to be recovered are the current leader, recover the leader node first. This helps to speed up the handover of group leadership, so that the recovery will complete faster.

In general, use the csar heal operation where possible instead of csar redeploy. The csar heal operation requires that the initconf process is active on the VM, and that the VM can reach both the CDS and MDM services, as reported by rvtconfig report-group-status. If any of those pre-requisites are not met for csar heal, use csar redeploy instead.

When report-group-status reports that a single node cannot connect to CDS or MDM, it should be considered a VM specific fault. In that case, use csar redeploy instead of csar heal. But a widespread failure of all the VMs in the group to connect to CDS or MDM suggest a need to investigate the health of the CDS and MDM services themselves, or the connectivity to them.

When recovering multiple VMs, you don’t have to consistently use either csar redeploy or csar heal commands for all nodes. Choose the appropriate command for each VM according to the guidance on this page instead.

In the case where the VMs are being recovered proactively to mitigate against an anticipated fault (such as running low on disk space), but are otherwise healthy and providing service, recover each VM one at at time, as a series of single VM recovery operations.

For cases where multiple VMs are malfunctioning and need to be recovered as a group, consider the cases below.

Recovery requires at least one Rhino node to be functioning. If the report-group-status output indicates that at least one Rhino node is healthy, the remaining nodes can be recovered one at at time. In the case where none of the Rhino nodes are functioning, and the group needs to be recovered as a whole, use the existing procedure for deleting and deploying the group as a whole. This includes a csar delete, a rvtconfig delete-node-type-version, followed by a csar deploy and a rvtconfig re-upload. See Backout procedure for the current platform within this guide for detailed steps related to backing out the group.

When recovering multiple nodes, VM recovery requires at least one Rhino node to be functioning in the Rhino cluster. If the report-group-status output indicates that at least one Rhino node is healthy, the remaining nodes can be recovered one at at time. In the mid-upgrade or mid-rollback case where none of the Rhino nodes on one side of the upgrade are functioning, and all nodes on that side of the upgrade need to be recovered as a whole, use the following procedure, in which the intent is to recover all VMs on version A, and leave the VMs on version B unchanged:

VMs should be healed one at a time, reassessing the group status using the rvtconfig report-group-status command after each heal operation, as detailed below.

See the 'Healing a VM' section of the SIMPL VM Documentation for details on the csar heal command.

The command should be run as follows:

VMs should be redeployed one at a time, reassessing the group status using the rvtconfig report-group-status command after each heal operation, as detailed below. Exceptions to this rules are noted on this page.

See the 'Healing a VM' section of the SIMPL VM Documentation for details on the csar redeploy command.

The command should be run as follows:

If the output of report-group-status indicates an unintended recovery to the wrong version, follow the procedure in Troubleshooting accidental VM recovery to recover.

A Rhino cluster expects at least half of the nodes to stay up during normal operation. An unplanned sudden loss of half or more of the nodes can cause the remaining half to go into a state called "waiting for primary component", in which the Rhino process restarts, and will not resume service until half or more of the nodes are present and available. This could be triggered for example by a hardware fault, or by redeploying too many nodes at the same time.

In any of the above cases, when the Rhino cluster enters this state, all available Rhino nodes will exhibit this state, observable by seeing this line in the Rhino section of the output of rvtconfig report-group-status:

To get past this state and restore service, heal or redeploy the faulty Rhino nodes, based on the Rhino section of the report-group-status.

Note that csar heal and initconf can be decoupled in this situation. The heal operation may timeout if not enough Rhino nodes are available. For example, if 4 of 5 Rhino nodes were lost, and the faulty nodes are being healed, the first csar heal operation will timeout while Rhino is waiting for primary component, because 2 of 5 Rhinos is not sufficient to reform the primary component.

In this situation, progress to heal the next node until sufficient nodes are restored. Once sufficient nodes are restored, rhino primary state will be restored and initconf will continue to completion on the affected nodes even after the csar heal has returned the timeout failure.

This rare case only applies when needing to recover 3 or more nodes, when all the persistence instances (see below) needing recovering simultaneously, and no Rhino nodes stayed up during the whole recovery process. It does not apply when you are already using one of the two procedures listed above in this page for intentionally recovering all nodes in the cluster:

Rhino uses a management database that is stored both in memory, and on-disk on the persistence instances, each of which run a PostgreSQL process. While Rhino is resilient to multiple failures, and can even withstand losing all persistence instances in some cases, there still exist cases where the sudden loss of all persistence instances can result in data loss.

The situation to mitigate against is very rare, but possible, in which a series of simultaneous node failures of all the persistence instances is followed by another event such as a power loss, causing the majority of the remaining Rhino processes to fail or be restarted, before the persistence instances are restored. This can only happen when all 3 persistence instances were lost at the same time. There is a possibility in such a rare case, that the services would be automatically restarted, but without the desired user configuration.

The steps for this persistence loss MOP are below: