Clustering

A message in the Rhino console log like the following indicates a JVM error.

If such a message is present, please see Java Virtual Machine Error.

First determine whether or not the segmentation is environmental. This means that the operating environment (see Operating Environment Issues) and the JVM Heap (see Java Virtual Machine Heap Issues) must be suitably configured. If these are configured suitably then the issue is likely to something with the network which interconnects the Rhino nodes. Typical network components to check for failure are machines, network interface cards, routers, switches, and cables.

If the cause of the symptoms are environmental, then refer to the workaround or resolution steps for these issues. If the cause is a network issue, then it needs to be repaired by replacing the failed components.

Check the current set of registered multicast groups with netstat -gn. These should include all the groups used by Savanna and the global IGMP membership reporting address 224.0.0.1. Compare the reported set between nodes and with the set present immediately after a node has started.

Check for "Martians" in syslog. Sometimes a peer will be a device on the multicast network that is in a different IP range from the hosts.

To enable logging of these on Linux run:

When multicast networking is in use a network device, commonly the switch or router, acts as an IGMP querier, sending regular messages to test for active multicast devices. The OS kernel receives and responds to these queries with membership reports. If no membership reports are received before the timeout configured on the querier it will stop forwarding multicast traffic.

To report IGMP traffic (Multicast group membership queries and reports) for all network interfaces on a host:

Use tcpdump or snoop to capture IGMP traffic on the interface used for multicast. Look for Query messages and Membership Reports.

The IGMP membership addresses 224.0.0.1 and 224.0.0.2 must be routed on the interface used for Savanna clustering as well as the configured Savanna group multicast addresses.

Configure the local switch to act as an IGMP querier. Especially important for virtualised environments

Sometimes you need to force the kernel to fall back to IGMPv2. Usually this will be because the switch does not support IGMPv3. To configure this, run:

Make this change permanent by adding net.ipv4.conf.eth0.force_igmp_version=2 to /etc/sysctl.conf

While uncommon for hosts in the cluster, switches are frequently in different IP ranges. The Linux kernel will drop these messages from "impossible" addresses by default as a DOS prevention measure. Also some switches will send IGMP queries on the All-Hosts group instead of the specific group Rhino joined. If your querier is on a different subnet from the nodes, disable the Martian packet safety check.

Fix this by configuring the rp_filter sysctl to accept packets from IPs that are not reachable from the cluster interface, by setting the rp_filter sysctl to 0 or 2 if the source address is on a network visible from another interface.

Make this change permanent by adding these to /etc/sysctl.conf:

You will also need to configure the firewall to accept IGMP packets sent to 224.0.0.1 and 224.0.0.2 if your rule does not include these implicitly.

Output is usually something like this:

If the cluster is configured to use scattercast follow the diagnostic procedures for Cluster Failing to Start and Scattercast endpoints out of sync. For multicast clusters follow the steps below:

The install script checks to ensure that a valid multicast range is used but this may be changed manually afterward. If the address reported is in the multicast range described above, check that your OS kernel supports multicast:

If the configured addresses are not all within the multicast range change the configuration to use only multicast addresses.

If the configured UDP multicast range is not used by other applications, it should work fine for Rhino. Failing that, any address range can be used in 224.0.0.0/4.

To change the addresses without reinstalling, edit the file $RHINO_BASE/node-101/config/config_variables and change the following lines to use multicast addresses not taken by other programs:

The master copy of this file (used when creating additional nodes) is $RHINO_BASE/etc/defaults/config/config_variables. If several nodes are to be deployed, this can be changed before nodes are created, to save extra editing at a later stage.

Output is something like this:

Also see Rhino Watchdog.

This problem can manifest in several ways, depending on how the scattercast endpoints are out of date. One example is shown below. Almost all examples of scattercast endpoints out of sync will result in the out of date nodes shutting down.

It is possible to create a situation where a node will boot with an out-of-date persistent endpoints set, and fail to shut down. If booted after a scattercast management command removes it from the endpoints set, a node will not go primary nor shut down.

This will trigger frequent cluster membership changes in the remaining nodes. The cluster membership change messages will not change the set of members.

Diagnostic and corrective steps If some nodes are operational, run the Rhino console command getscattercastendpoints to learn the current cluster configuration. This will print the live in-memory configuration; and if the scattercast.endpoints file on any running node contains a different configuration, it will print the memory state for each node and the differences from the disk state.

If all nodes have the same configuration, the output will be similar to the example below:

If the node configuration is consistent but does not contain the expected set of nodes, then the missing nodes should be added by running the addscattercastendpoints command.

If the disk mapping differs from the in-memory configuration, then the nodes with the differing configuration are likely to fail when restarted. An example output where node 101 has a malformed file that will cause a node to fail to reboot is below:

If some nodes have a different configuration on disk from the live memory configuration, then copy the scattercast.endpoints file from a node that has matching disk and memory configurations.