Below are troubleshooting steps — symptoms, diagnostic steps, and workarounds or resolutions — for clustering issues with Rhino.

Node failure

Rhino is designed to survive hardware and software failure by using a redundant cluster of nodes. There are many possible causes for node failure; below are symptoms and diagnostic steps for the most common.

In most deployments Rhino is configured to automatically restart failed nodes if the failure was caused by network or software faults. In all cases not caused by hardware faults contact your solution provider support, providing all the rhino.log and console.log files from the failed Rhino node. Restart the failed node if it did not restart automatically.

Configuration change messages appear in Rhino log output.

Symptoms

A Rhino node may determine that another node has left the primary component (due to failure of hardware, network, or software). When this happens, the output looks like this:

...
WARN [rhino.membership] Node(s) [102] left cluster admin operational member set
INFO [rhino.membership] Current membership set: [101]
...

Diagnosis and Resolution

Please see Cluster Segmentation.

An alarm indicates a node has failed.

Symptoms

An alarm appears in the console if a node has failed. To query active alarms run this command:

  ./client/bin/rhino-console -listActiveAlarms

This produces one line of log output for each active alarm. If the list of active alarms contains a line similar to the following, it indicates that a node has left the cluster due to hardware, network, or software failure.

...
Alarm ...(Node 101, 23-Nov-05 15:10:36.636): Major [rhino.cluster] Node 102 has left the cluster ...
...

Diagnosis and Resolution

Please see Cluster Segmentation.

A Rhino node exits the JVM.

Symptoms

A Rhino node may determine it is not in the cluster majority after a cluster segmentation. If this is the case it terminates.

An example of the log output from the terminated node is as follows.

INFO [rhino.main] Cluster backbone membership change: BboneCCM: Trans [103,30] - Memb [ 103 ]
INFO [rhino.main] Cluster backbone membership change: BboneCCM: Reg [103,32] - Memb [ 103 ]
...
WARN [rhino.membership] Cluster admin group has left primary component - exiting process
WARN [rhino.starter.loader] Node is no longer ready - exiting process
Tip For more information please see Cluster Segmentation.

Out of memory errors

Symptom

Out of memory errors in the console log (console.log).

JVM errors.

A message in the Rhino console log like the following indicates a JVM error.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f9c742ffe25, pid=16224, tid=0x00007f9c720c6700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5c2e25]  G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x45
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
...

If such a message is present, please see Java Virtual Machine Error.

Watchdog timeouts.

Symptoms

A message like the following in the Rhino logs indicates a watchdog timeout.

2005-09-06 10:18:13.484 ERROR [watchdog] Waiting for thread cleanup..
...
2005-09-06 10:18:14.494 ERROR [watchdog] ***** KILLING NODE *****

Diagnosis and Resolution

If such a message is present please see Rhino Watchdog.

Cluster segmentation

Cluster segmentation occurs when part of a Rhino cluster determines that a cluster node or a set of nodes is no longer reachable. Nodes that determine that they no longer make up a majority of the cluster (known as a quorum) automatically deactivate themselves.

Cluster segmentation can be caused by several things, the most common being a network failure such as a switch malfunctioning.

Symptoms

  • Configuration change messages appear in Rhino log output

  • A Rhino node exits the JVM

  • An alarm indicating a node has failed

  • Watchdog timeouts

Diagnostic steps

First determine whether or not the segmentation is environmental. This means that the operating environment (see Operating Environment Issues) and the JVM Heap (see Java Virtual Machine Heap Issues) must be suitably configured. If these are configured suitably then the issue is likely to something with the network which interconnects the Rhino nodes. Typical network components to check for failure are machines, network interface cards, routers, switches, and cables.

Tip See Node failure for other causes of node failure.

Workaround or resolution

If the cause of the symptoms are environmental, then refer to the workaround or resolution steps for these issues. If the cause is a network issue, then it needs to be repaired by replacing the failed components.

Cluster failing to start

Rhino nodes must be part of the “primary component” in order to perform any work. The primary component is the set of nodes sharing the same cluster state. When booting up, nodes will wait to join the primary component before being able to perform work. This section describes the likely issues which cause a node to wait on the primary component for an extended period of time (greater than several seconds).

If the following diagnostic steps do not resolve the problem, please ensure that the various cluster machines can reach each other. For example, the ping command should indicate whether or not nodes can reach each other. If nodes cannot ping each other, then the cause is likely to be network misconfiguration or hardware failure.

No Primary Component

Symptoms

Repeated messages in the Rhino log indicating that a node is waiting for the cluster to go primary.

Messages like this show up in the Rhino console and logs:

...
INFO [rhino.main] Waiting for cluster administration group to go primary; current primary members: []
INFO [rhino.main] Waiting for cluster administration group to go primary; current primary members: []
INFO [rhino.main] Waiting for cluster administration group to go primary; current primary members: []
INFO [rhino.main] Waiting for cluster administration group to go primary; current primary members: []
INFO [rhino.main] Waiting for cluster administration group to go primary; current primary members: []
...

Resolution

If this is occuring on all nodes, run the make-primary.sh script on one node.

Tip See the Rhino Production Getting Started Guide for more information about primary components.

Multicast Configuration Error

Symptoms

No configuration change messages in the Rhino logs.

A lack of configuration change messages in a clustered setup indicates that the Rhino node is not able to communicate with other Rhino nodes in the cluster.

Diagnostic and corrective steps This is caused by an inability for nodes to communicate. Diagnosis and correction depends on whether the cluster communication mode is Multicast or Scattercast.

Multicast

Ensure that a route for the multicast IP addresses being used by Rhino is present in the operating system.

Tip Use the route and netstat system tools for more information.

Ensure that any routers are configured to allow multicast IP packets with the configured multicast addresses to be passed to the appropriate machines.

Ensure that the following variables in the config/config_variables files on all Rhino nodes which are part of a cluster match:

SAVANNA_CLUSTER_ID
SAVANNA_CLUSTER_ADDR
SAVANNA_MCAST_START
SAVANNA_MCAST_END

Check that the host firewall is not configured to block the address/port range Savanna uses on the interface used for multicast. A simple test is to remove all firewall rules with iptables -F. Rhino uses UDP multicast to a range of IP addresses and ports set in node-<NODE_ID>/config/config_variables and node-<NODE_ID>/config/savanna/cluster.properties The IGMP membership groups 224.0.0.1 and 224.0.0.2 must also be allowed on this interface.

Verify whether or not multicast IP messages are being passed from one machine to another correctly. To see the IP traffic between cluster members, use tcpdump and/or snoop.

For example, to record packet traces for the network interface eth1 limiting to a maximum of two 500MB files:

tcpdump -C 500 -i eth1 -W 2 -w mcast_dump.pcap

The collected packet capture can be copied to the administrator’s workstation and opened with Wireshark.

Scattercast

Ensure that all scattercast endpoint addresses are routable from every host in the cluster. Check that the scattercast endpoints set is up to date on nodes waiting to go primary. If the node is sufficiently out of date to not receive from any node, it cannot detect that it is out of date and shutdown.

Check that the host firewall is not configured to block the address/port pairs and ranges used by Savanna for scattercast. A simple test is to remove all firewall rules with iptables -F. Rhino uses UDP unicast to a range of IP addresses and ports set in:

node-<NODE_ID>/config/config_variables
node-<NODE_ID>/config/savanna/cluster.properties
node-<NODE_ID>/config/savanna/scattercast.endpoints

Verify whether or not Savanna UDP IP messages are being passed from one machine to another correctly. To see the IP traffic between cluster members, use tcpdump and/or snoop.

For example, to record packet traces for the network interface eth1 limiting to a maximum of two 500MB files:

tcpdump -C 500 -i eth1 -W 2 -w mcast_dump.pcap

The collected packet capture can be copied to the administrator’s workstation and opened with Wireshark.

Cluster starts but stops after a few minutes

Note This affects multicast only.

Symptoms

  • Configuration change messages appearing in Rhino log output

  • Many or all Rhino nodes exit the JVM
    For example:

    2016-04-27 12:29:24.352  WARN    [rhino.membership.rhino-cluster]   <SavannaDelivery/rhino-cluster> Node has left primary component
  • Multiple alarms indicating a node has failed

  • Watchdog failures for the condition "GroupHeartbeat"
    For example:

    2016-04-27 12:29:24.207  ERROR   [watchdog]   <Global watchdog thread>   Failed watchdog condition: GroupHeartbeat for group rhino-monitoring (sent=33 received=23)

Diagnostic steps

Multicast registration timeout

Check the current set of registered multicast groups with netstat -gn. These should include all the groups used by Savanna and the global IGMP membership reporting address 224.0.0.1. Compare the reported set between nodes and with the set present immediately after a node has started.

Check for "Martians" in syslog. Sometimes a peer will be a device on the multicast network that is in a different IP range from the hosts.

To enable logging of these on Linux run:

echo 1 > /proc/sys/net/ipv4/conf/all/log_martians

When multicast networking is in use a network device, commonly the switch or router, acts as an IGMP querier, sending regular messages to test for active multicast devices. The OS kernel receives and responds to these queries with membership reports. If no membership reports are received before the timeout configured on the querier it will stop forwarding multicast traffic.

To report IGMP traffic (Multicast group membership queries and reports) for all network interfaces on a host:

tcpdump -i any igmp

Use tcpdump or snoop to capture IGMP traffic on the interface used for multicast. Look for Query messages and Membership Reports.

Workaround or resolution

The IGMP membership addresses 224.0.0.1 and 224.0.0.2 must be routed on the interface used for Savanna clustering as well as the configured Savanna group multicast addresses.

Configure the local switch to act as an IGMP querier. Especially important for virtualised environments

IGMP version

Sometimes you need to force the kernel to fall back to IGMPv2. Usually this will be because the switch does not support IGMPv3. To configure this, run:

/sbin/sysctl -w net.ipv4.conf.eth0.force_igmp_version=2`

Make this change permanent by adding net.ipv4.conf.eth0.force_igmp_version=2 to /etc/sysctl.conf

Martians

While uncommon for hosts in the cluster, switches are frequently in different IP ranges. The Linux kernel will drop these messages from "impossible" addresses by default as a DOS prevention measure. Also some switches will send IGMP queries on the All-Hosts group instead of the specific group Rhino joined. If your querier is on a different subnet from the nodes, disable the Martian packet safety check.

Fix this by configuring the rp_filter sysctl to accept packets from IPs that are not reachable from the cluster interface, by setting the rp_filter sysctl to 0 or 2 if the source address is on a network visible from another interface.

/sbin/sysctl -w net.ipv4.conf.default.rp_filter=0
/sbin/sysctl -w net.ipv4.conf.all.rp_filter=0

Make this change permanent by adding these to /etc/sysctl.conf:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
Tip For more information on rp_filter and other kernel multicast configuration, see https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt.

You will also need to configure the firewall to accept IGMP packets sent to 224.0.0.1 and 224.0.0.2 if your rule does not include these implicitly.

Rhino SLEE fails to start cluster groups

Typically, the failure to start a cluster group indicates a network configuration problem.

Unless configured to use scattercast, Rhino distributes state between cluster members using UDP multicast. The particular multicast addresses used are defined during installation. Multicast addresses are, by definition, in the range 224.0.0.0 through to 239.255.255.255, otherwise notated as 224.0.0.0/4.

A failure will occur if Rhino tries to create a multicast socket on a non-multicast address. The Rhino install script checks to ensure that a valid multicast address is used, so this would only occur if the Rhino configuration has been edited manually after installation.

It is possible to change the addresses used by the cluster manually, by editing the configuration on every node that has been created. Care should be taken to ensure all nodes in the cluster are configured to use the same multicast addresses.

To change these addresses, edit the file $RHINO_HOME/node-101/config/config_variables.

Symptoms

Output is usually something like this:

ERROR [savanna.stack.membership.manager] Failed to start cluster groups
java.lang.RuntimeException: Failed to change addresses to /224.0.24.0:45601

Diagnostic steps

If the cluster is configured to use scattercast follow the diagnostic procedures for Cluster Failing to Start and Scattercast endpoints out of sync. For multicast clusters follow the steps below:

The install script checks to ensure that a valid multicast range is used but this may be changed manually afterward. If the address reported is in the multicast range described above, check that your OS kernel supports multicast:

cat /boot/config-<kernel version> \grep CONFIG_IP_MULTICAST

Workaround or resolution

If the configured addresses are not all within the multicast range change the configuration to use only multicast addresses.

If the configured UDP multicast range is not used by other applications, it should work fine for Rhino. Failing that, any address range can be used in 224.0.0.0/4.

To change the addresses without reinstalling, edit the file $RHINO_BASE/node-101/config/config_variables and change the following lines to use multicast addresses not taken by other programs:

SAVANNA_CLUSTER_ADDR=224.0.24.1
SAVANNA_MCAST_START=224.0.24.1
SAVANNA_MCAST_END=224.0.24.8

The master copy of this file (used when creating additional nodes) is $RHINO_BASE/etc/defaults/config/config_variables. If several nodes are to be deployed, this can be changed before nodes are created, to save extra editing at a later stage.

Group heartbeat timeout

Rhino uses a heartbeat mechanism to ensure Savanna groups are alive and functioning correctly. This can cause a node to shut down when it detects abnormal group behaviour.

Symptoms

Output is something like this:

ERROR   [watchdog]   <Global watchdog thread> Watchdog Failure has been notified to all listeners because condition 'GroupHeartbeat for group domain-0-rhino-db-stripe-0 (sent=20 received=10)' failed
ERROR   [watchdog]   <Global watchdog thread> *** WATCHDOG TIMEOUT ***
ERROR   [watchdog]   <Global watchdog thread>   Failed watchdog condition: GroupHeartbeat for group domain-0-rhino-db-stripe-0 (sent=20 received=10)
ERROR   [watchdog]   <Global watchdog thread> ***** KILLING NODE *****
 Diagnostic and corrective steps
This is almost always caused by a networking issue that does not affect the backbone Savanna group used for cluster membership.
Diagnosis and correction is therefore identical to bxref:#cluster-failing-to-start[Cluster Failing to Start].

Also see Rhino Watchdog.

Scattercast endpoints out of sync

This issue can only occur on a cluster using scattercast communications mode. Scattercast does not permit autodiscovery of nodes. Therefore each node must a priori know the addresses of all nodes in the cluster. Whenever the persistent set of known endpoints gets out of sync on a node, it will not be able to successfully join the cluster.

Symptoms

This problem can manifest in several ways, depending on how the scattercast endpoints are out of date. One example is shown below. Almost all examples of scattercast endpoints out of sync will result in the out of date nodes shutting down.

ERROR   [savanna.stack.membership.manager]   <SavannaIO/pool-thread-1> Halting due to scattercast endpoint mismatch detection:Network advertised version: 2 does not match local version: 1
ERROR   [savanna.stack.membership.manager]   <SavannaIO/pool-thread-1> An up to date scattercast.endpoints file can be found on any running node
INFO    [rhino.exithandler]   <SavannaIO/pool-thread-1> Exiting process (Exiting due to fatal Savanna misconfiguration or conflict)

It is possible to create a situation where a node will boot with an out-of-date persistent endpoints set, and fail to shut down. If booted after a scattercast management command removes it from the endpoints set, a node will not go primary nor shut down.

This will trigger frequent cluster membership changes in the remaining nodes. The cluster membership change messages will not change the set of members.

Node 101 log fragment when Node102 booted with out of date endpoints set.
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Trans [101,10] - Memb [ 101 ]
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Reg [101,12] - Memb [ 101 ]
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Trans [101,14] - Memb [ 101 ]
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Reg [101,16] - Memb [ 101 ]
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Trans [101,18] - Memb [ 101 ]
INFO    [rhino.main]   <SavannaIO/pool-thread-1> Cluster backbone membership change: BboneCCM: Reg [101,20] - Memb [ 101 ]

Diagnostic and corrective steps If some nodes are operational, run the Rhino console command getscattercastendpoints to learn the current cluster configuration. This will print the live in-memory configuration; and if the scattercast.endpoints file on any running node contains a different configuration, it will print the memory state for each node and the differences from the disk state.


If all nodes have the same configuration, the output will be similar to the example below:

[Rhino@host (#0)] getscattercastendpoints
[Consensus] Disk Mapping   : Coherent
[Consensus] Memory Mapping :
      [101] Address  : 192.168.4.103:18000
      [102] Address  : 192.168.4.103:18001
      [103] Address  : 192.168.4.103:18002
      [104] Address  : 192.168.4.103:18003
      [105] Address  : 192.168.4.103:18004
      [106] Address  : 192.168.4.105:18005
      [107] Address  : 192.168.4.105:18006
      [108] Address  : 192.168.4.105:18007
      [109] Address  : 192.168.4.105:18008
      [110] Address  : 192.168.4.105:18009
      [111] Address  : 192.168.4.105:18010

If the node configuration is consistent but does not contain the expected set of nodes, then the missing nodes should be added by running the addscattercastendpoints command.


If the disk mapping differs from the in-memory configuration, then the nodes with the differing configuration are likely to fail when restarted. An example output where node 101 has a malformed file that will cause a node to fail to reboot is below:

[Rhino@host (#0)] getscattercastendpoints
[101] Disk Mapping   : 192.168.4.103:18001 present twice in endpoints file
[101] Memory Mapping :
      [101] Address  : 192.168.4.103:18000
      [102] Address  : 192.168.4.103:18001
      [103] Address  : 192.168.4.103:18002
      [104] Address  : 192.168.4.103:18003
      [105] Address  : 192.168.4.103:18004
      [106] Address  : 192.168.4.105:18005
      [107] Address  : 192.168.4.105:18006
      [108] Address  : 192.168.4.105:18007
      [109] Address  : 192.168.4.105:18008
      [110] Address  : 192.168.4.105:18009
      [111] Address  : 192.168.4.105:18010

[102] Disk Mapping   : Coherent
[102] Memory Mapping :
      [101] Address  : 192.168.4.103:18000
      [102] Address  : 192.168.4.103:18001
      [103] Address  : 192.168.4.103:18002
      [104] Address  : 192.168.4.103:18003
      [105] Address  : 192.168.4.103:18004
      [106] Address  : 192.168.4.105:18005
      [107] Address  : 192.168.4.105:18006
      [108] Address  : 192.168.4.105:18007
      [109] Address  : 192.168.4.105:18008
      [110] Address  : 192.168.4.105:18009
      [111] Address  : 192.168.4.105:18010
...

If some nodes have a different configuration on disk from the live memory configuration, then copy the scattercast.endpoints file from a node that has matching disk and memory configurations.


Note See also the Repair section of Scattercast Management in the Rhino Administration and Deployment Guide.
Previous page Next page