Environmental

Operating system utilities such as vmstat, mpstat, and iostat are particularly helpful in determining whether or not a problem is caused by resource contention. Run such tools for enough time to observe the problem, and then review the logged output immediately before and after. Also look for CPU, I/O, swap, and network use. You may also want to record these continuously, for analysis of long-term system performance and transient faults.

If possible, collect system statistics no less frequently than 10s intervals, and cover every major subsystem. The UNIX standard SAR monitor also collects a useful data; and other tools may offer higher resolution or better analysis support.

Some of the specific things to look for are listed below:

Please ensure that the system is sized such that no more than around 75% of CPU is used by Rhino under peak load with one failed node. If the machine is above this utilisation of CPU, then the machine has been incorrectly sized. If running a virtualised environment, this should be scaled to allow for failure of one physical host.

If other processes running on the same hardware compete for CPU for long enough, the responsiveness of a Rhino node may suffer. In our experience, operating system scheduling frequently does not provide Rhino with enough resources if there is competition for those resources. Rhino should be installed on dedicated hardware with no competing applications. Processes that can be especially demanding of CPU and I/O include databases and other Java servers.

Cron jobs can cause other system processes to run which may compete for system resources. Cron jobs which are not essential should be disabled. Particular note should be taken of the locate service which will rebuild its database as a daily or weekly task, an I/O intensive procedure.

CPU use of the machine should be monitored to ensure that there is more than about 25% of CPU available at all times, including with one node failed. Ideally data should be collected no less frequently than every 10s, preferably with 1s intervals.

High levels of disk I/O can result in increased CPU use, and can cause the Rhino JVM process to be scheduled infrequently. This can also cause writes to be scheduled synchronously, which may block threads during critical tasks such as GC. This is particularly likely on highly contended disks such as found in virtualised environments and SAN storage arrays.

Swap use is often the cause of disk I/O. Rhino should be installed on a machine with sufficient physical RAM for the workload of the machine.

Swap will also significantly impair GC performance. If even a small amount of the JVM heap is swapped out, GC pause times can increase by up to 300% and potentially much longer for greater swap usage. While the CMS collector will usually touch pages frequently enough to keep them in physical memory, the G1 collector can leave regions untouched for long enough to be swapped out preemptively on Linux.

Rhino’s cluster membership services use multicast UDP. If the IP network is loaded heavily, this can cause packets to be dropped, potentially causing cluster segmentation or failure if individual cluster nodes cannot reliably communicate.

See Cluster Segmentation, Cluster Failing to Start and Cluster Starts but Stops After a few Minutes for more details of multicast related problems.

Large systems (that is, four or more sockets per host have significant asymmetry in memory access speed and latency between CPU sockets. This can cause performance to be significantly less than optimum.

If any of the machines Rhino is running on are running over 75% CPU use, and this is not due to other system processes, then consider:

Processes causing a problem due to resource usage should not be used on a machine running Rhino. Move these to another host.

To prevent preemptive swapping slowing GC, reduce the value of the sysctl vm.swappiness from the default of 60 to something low enough to prevent this. 30 is a good starting point. If using G1 the vm.swappiness sysctl should be set no higher than 30 and preferably 10 or lower.

When running Rhino on NUMA hosts it is usually best to run multiple nodes on each host with each JVM bound to a separate NUMA domain. Configure this with numactl --cpunodebind=X --membind=X on Linux.

There are two common causes of heap issues when running Rhino:

For the latter, see Application or resource adaptor heap issues.

The heap usage of Rhino should be monitored. This includes both the Java heap and the native heap. The Java heap can be measured by Rhino’s statistics tool rhino-stats, or one of the Java monitoring tools — jstat or jvisualvm. The native heap size can be measured using system utilities such as ps or top.

Here’s an example of using the Rhino statistics tool to monitor the size of the Java heap:

The behaviour of the JVM process is that a well-sized heap should gradually decrease from, say, 1.5 gigabytes free to around, say, 800 megabytes free, and then increase up to its higher limit again. This will be overlaid by a shorter pattern of peaks of say 128 megabytes over the long pattern usage. The actual figures will depend on the services that are active and deployed; and the total heap used should never exceed 60% of the heap size allocated for sustained periods when using the default CMS collector.

If the JVM heap is sized appropriately for a given load, and a system appears to run stably at a given load and gradually degrades over a period of at least several hours, then it is possible that the JVM, a resource adaptor or Rhino has a memory leak.

Check the number of live activities and SBBs using the Rhino statistics monitoring tool. If the number of each are within expected values, then it is likely there is a memory leak. If there are too many, it indicates there may be a bug in the component. A strong indicator of an RA or service bug is activities that are more than ten times the age of the average call duration, or older than one whole day.

First raise the maximum heap size available to the JVM by modifying the HEAP_SIZE variable in the config/config_variables file. Ensure that there is enough physical memory on the machine such that the system will not swap if this extra memory is used by the JVM. Also set the NEW_SIZE based on the duration and memory usage of a typical session. The Rhino node must be restarted for this change to take effect. See to JVM documentation for more information on heap sizing and related overheads.

With some allocation patterns it may not be possible to find a heap size or new size that provides an acceptable balance between pause times and durations for the target load. If this is the case the only solutions are to use a different cluster size or contact your solution provider for an improved version of the service that uses memory more efficiently. This problem should only arise as a result of an error in capacity planning and testing. A workaround that is sometimes available if the host server has sufficient free RAM is to increase the heap size until it is large enough that the pauses can be avoided by restarting nodes during idle periods (however this is rarely possible).

Please contact your solution provider support to work through component memory leak issues.

When these symptoms occur it is often necessary to check both for application or resource adaptor related heap issues and for JVM heap issues.

Start the rhino-console, and query for activities that are older than expected for the system. For example, call activities typically last the duration of a phone call; so query for activities that are older than the majority of phone calls (say one hour). To perform this query use the findactivities command with -created-before as a flag.

Here’s an example of finding activities created before 2:30pm:

You can also query for activities older than a specified age, for example:

The resulting number of activities should be a low number. If it is a number larger than, say, 30 (depending on your application) then there may be an issue with some activities not being cleaned up properly by the service or resource adaptor. You should exclude ServiceActivities and ProfileActivities from the set counted.

If the number of activities is small, test for the number of SBB entities in Rhino. SBB entities are removed when they are no longer in use; so if you find a large number of SBB entities that were created a long time ago, it is likely that the service has a problem. The command to query SBB entities is the findsbbs command.

Here’s an example of finding SBB entities for the SIP proxy service created before 2:30pm:

We also recommend that you use the statistics client to examine the memory usage and number of activities in the running SLEE:

This indicates an unhandled case in the Services or resource adaptors being used. Activities and SBB entities known to be invalid may be removed through client/bin/rhino-console using the removeactivity and removesbb commands respectively. These commands may be used to stop a system from running out of memory before a patched application or resource adaptor is installed.

Firstly, check to see how much free memory is available to the JVMs, and that swap space is available. Rhino should never run using swap space (doing so will cause unpredictable performance); but nonetheless, sufficient swap space should be present to allow the OS to manage memory more effectively.

Check the amount of physical memory available. Under Linux, use the free command.

Compare this amount to the size of the JVM specified in the HEAP_SIZE variable in node-???/config/config_variables (for Rhino-production). Make sure that the JVM can fit comfortably in the physical memory on the computer.

Rhino versions earlier than 2.3.0.7 require about twice the amount of physical memory as the size of HEAP_SIZE to ensure that a fork() of the JVM during compilation of services will always be able to run in physical memory and not incur delays by swapping memory to and from disk. Linux systems use copy-on-write memory cloning when forking processes so the likelihood of swapping is much lower than on Solaris.

You may need to reduce the value of HEAP_SIZE by editing the file $RHINO_HOME/node-???/config/config_variables to make sure Rhino can fit in the available physical memory.

Also make sure there are no other running processes on the computer which may be using excessive system resources, and that there is sufficient swap space.

Rhino prints warnings at startup:

Ensure that the kernel parameters net.core.rmem_max and net.core.wmem_max are large enough.

See UDP buffer sizes in the Preparing your Network section of the Rhino Production Getting Started Guide for details.

When the JVM crashes, the Rhino node will fail and the cluster will reconfigure itself. On Unix systems, the node will die with either a SIGSEGV or a SIGBUS and perhaps dump core, depending on how your system is configured.

A message similar to the following appearing in logs and/or the Rhino console indicates a JVM error.

The following are examples of configuration change messages.

An alarm appears in the console if a node has failed. Use this command to query active alarms:

This command produces a block of output for each active alarm. A block in the list of active alarms similar to the following indicates that a node has left the cluster due to hardware, network, or software failure.

If any of these symptoms occur, then first check the logs of all Rhino cluster members (file console.log). If a JVM error message is part of the logs of any node, then a Java Virtual Machine error has occurred. If an error message cannot be found in the logs please refer to Cluster Segmentation, Cluster Failing to Start and Cluster Starts but Stops After a few Minutes

After checking Rhino’s logs, determine whether or not the hardware or operating system is causing the problem. Look at the logs of the local machine and determine whether the machine has had a history of restarts, kernel panics, process segmentation faults, and so forth. On Unix machines, system logs can be viewed using the dmesg command or by viewing logs in /var/logs.

When a JVM crash occurs, a file will be left with a name like hs_err_pid*.log. View this file and try to determine which part of the JVM caused the crash. This may provide clues as to how to resolve the situation and will be needed if a bug in the JVM is to be reported.

If the crash appears to be a one-off, the node can simply be restarted.

If the problem resides in faulty hardware, then that hardware will need to be fixed or replaced.

Metaswitch has experience with configuring the Sun Java Virtual Machine to be as stable as possible. Consider consulting Metaswitch support for information regarding known stable configurations of JVMs. Also ensure that thorough stability tests are performed on a new version of a Java Virtual Machine before it is deployed in a production system.

Also scan through the list of dynamic libraries in the hs_err_pid*.log file and see if they are all the latest version. Keep an eye out for libraries which should not be there, for example from other installations of older JVMs.

If it is a realistic option, upgrading the operating system to the latest version may also resolve this problem.

JVM errors that cannot be identified as being caused by system configuration or dynamic library problems should be reported to Oracle.

The host machines for Rhino have multiple network interfaces, and we need to control which interface is used for SLEE cluster traffic. Network monitoring shows this is currently not on the desired interface.

When the configuration differs between hosts, nodes may fail to go primary. This is indicated by repeated messages in the Rhino log indicating that a node is waiting for the cluster to go primary and alack of configuration change messages.

Messages like this show up in the Rhino console and logs:

Use a network monitoring tcpdump or snoop to detect multicast traffic on the groups configured for Rhino clustering.

Use netstat -rn, route, or ip route to examine the current multicast routing configuration.

The choice of interface is controlled solely by the machines' multicast configuration; it’s controlled at the OS level rather than by Rhino. Use route or ip route to modify the OS routing configuration. See Cluster Failing to Start for how to configure the interface used for multicast traffic.