Rhino Troubleshooting Guide

The stuck threads watchdog behaviour handles stuck or dying threads. Under some circumstances as discussed below, the node will leave the cluster and exit. If configured for restarting, the node will restart, rejoin the cluster and continue operating.

The timewarp watchdog behaviour handles the system clock suddenly changing from the JVM perspective. A warning is printed in the log if the system clock suddenly changes. A Critical alarm is raised whenever a timewarp is detected, and will not be automatically cleared. This alarm is considered critical as all timewarps indicate severe problems in the underlying system that threaten cluster stability and ability to process sessions in a timely manner.

A timewarp of 5s or greater is sufficient to break the clustering mechanism used in Rhino, and may cause the node to leave the cluster. This is done to prevent a split-brain situation, or data corruption. A single node cluster will not self shutdown due to timewarps, but this should still be considered a critical issue as the node is unavailable during the timewarps.

Rhino may fail if system time ever goes backwards on a single node while the cluster is running, or if it goes backwards to overlap with a time period when the cluster was previously running. This is because the unique IDs being generated may no longer be unique.

If the watchdog observes a sudden change in the system clock, the following will be printed on the console:

Several causes of timewarps exist.

A long-running transaction may be reported during installation of components during deployment or when a node is starting up. In this case, the error message is benign; and the warning will be cleared when the components have finished installing.

If this message appears only during start-up or deployment of components, then it can be safely ignored.

If such messages persist or occur during normal operation, then they should be reported to your solution provider. Please gather the output of the dumpthreads.sh command from the node directory. This output is written to the Rhino console and is not gathered as part of Rhino’s logging system. It can usually be found in the file console.log. Please provide this output to your solution provider support.

Below are examples of warning messages in Rhino logs or console that indicate lock timeouts.

The diagnostic process determines whether a failure has occurred due to environmental issues or hardware, network, or software related issues. A hardware, network, or software related issue is usually indicated by the presence of an alarm. In order to check for alarms, use the following rhino-console command.

This command produces one line of output for each active alarm. If the list of active alarms contains a line similar to the following, it indicates that a node has left the cluster due to hardware, network, or software failure.

If an alarm indicates node failure, follow the diagnostic and resolution procedures for Node Failure.

If the output of the command does not contain a line similar to above, or indicates no alarms are active (as below), then the symptoms are occurring because of environmental issues or software faults.

Overload conditions are likely to cause lock timeouts. CPU usage should never exceed 70% for an extended period of time. CPU usage can be measured on Linux using the top command.

Garbage collection, particularly under overload conditions can cause lock timeouts. The behaviour of the garbage collector can be analysed by examining the garbage collection logs. By default these are logged to the Rhino console or the console.log file in
node-???/work/log For further information on diagnosing GC related problems, please refer to Java Virtual Machine Heap Issues and Application or Resource Adaptor Heap Issues.

SLEE load can also be investigated with the use of the Statistics Client. For more information about usage of the Statistics Client, please see the rhino-stats section in the Rhino Administration and Deployment Guide. Of particular value are the time statistics for the lock manager, MemDB and Savanna subsystems. If any of these report times in the 10s of milliseconds or greater this is usually a symptom of overload.

To resolve problems caused by overload it is usually necessary to add more cluster nodes on new hardware, correct the configuration of the software or improve the efficiency of the service logic. Follow the diagnostic steps in the following sections Operating Environment Issues, Java Virtual Machine Heap Issues, and Application or Resource Adaptor Heap Issues. If this does not lead to a resolution contact your solution provider for assistance.

If a lock timeout is caused by overloading the cluster, then an interim resolution is to enable a user-specified rate limiter to limit the number of events processed by the cluster. Instructions for doing so are in the Rhino Administration and Deployment Guide.

Deadlock, lost transactions and stuck threads are typically caused by software bugs. These can be in Rhino, service logic, or RAs, and will often be unrelated to call rate. To assist in diagnosing these problems observe the statistics parameter sets StagingThreads and Events. Diagnosing the cause of these will usually require setting the log keys transaction.manager and transaction.instance to the debug level. The logs obtained should be sent to your solution provider.

The log key for these I/O exceptions contain any of the following phrases:

These messages typically occur if the network is unable to accept multicast traffic from Rhino. This can be caused by either inappropriate kernel IP routing configuration or problems with network elements such as routers, switches, ethernet cards, firewalls etc. Start diagnosis from Operating Environment Issues. If this does not produce a solution continue with bClustering. Finally contact your solution provider for support.

The log key for these I/O exceptions contains the following phrases:

These messages typically occur if the Rhino installation does not have appropriate disk space and/or permissions available or there is a hardware error.

The log key for these I/O exceptions contains the following phrases:

These messages should be provided to the solution provider unless they clearly indicate the cause.

If the log messages do not occur on any of the log keys mentioned in the cases, then they are likely to come from a resource adaptor or SBB. These messages should be provided to the solution provider and possibly the resource adaptor vendor.

A service fails with an error that looks like:

The above exception indicates that you need to activate the RA entity. Documentation on managing RA entities can be found in the Rhino Administration and Deployment Guide.

Rhino logs contain the message:

Compare the times these warnings are logged with garbage collection cycles. Cleanup of leaked OCIO buffers happens during CMS GC cycles and all G1 collections, and it writes one log entry per 100 buffers cleaned up. The log entry reports the accumulated total of all OCIO buffers cleaned up since the node started. If this message is occurring on every GC cycle, that may indicate an environment-related performance problem.

Continue diagnosis of the performance problem, starting at Operating Environment Issues.

If there are no signs of overload, then the warning is harmless and indicates that a section of code in the Rhino clustering implementation has cleaned up internal objects that would have otherwise leaked.

If the log key for the exception log matches the tracer for a resource adaptor entity e.g. trace.sipra the cause of the problem is likely to be a faulty resource adaptor. This is indicative of either a misconfiguration of the resource adaptors in question or a bug in their implementation. If this is occurring frequently then please contact your solution provider support. Contact your solution provider for support attaching the log files that contain the exception message.

Operating system utilities such as vmstat, mpstat, and iostat are particularly helpful in determining whether or not a problem is caused by resource contention. Run such tools for enough time to observe the problem, and then review the logged output immediately before and after. Also look for CPU, I/O, swap, and network use. You may also want to record these continuously, for analysis of long-term system performance and transient faults.

If possible, collect system statistics no less frequently than 10s intervals, and cover every major subsystem. The UNIX standard SAR monitor also collects a useful data; and other tools may offer higher resolution or better analysis support.

Some of the specific things to look for are listed below:

Please ensure that the system is sized such that no more than around 75% of CPU is used by Rhino under peak load with one failed node. If the machine is above this utilisation of CPU, then the machine has been incorrectly sized. If running a virtualised environment, this should be scaled to allow for failure of one physical host.

If other processes running on the same hardware compete for CPU for long enough, the responsiveness of a Rhino node may suffer. In our experience, operating system scheduling frequently does not provide Rhino with enough resources if there is competition for those resources. Rhino should be installed on dedicated hardware with no competing applications. Processes that can be especially demanding of CPU and I/O include databases and other Java servers.

Cron jobs can cause other system processes to run which may compete for system resources. Cron jobs which are not essential should be disabled. Particular note should be taken of the locate service which will rebuild its database as a daily or weekly task, an I/O intensive procedure.

CPU use of the machine should be monitored to ensure that there is more than about 25% of CPU available at all times, including with one node failed. Ideally data should be collected no less frequently than every 10s, preferably with 1s intervals.

High levels of disk I/O can result in increased CPU use, and can cause the Rhino JVM process to be scheduled infrequently. This can also cause writes to be scheduled synchronously, which may block threads during critical tasks such as GC. This is particularly likely on highly contended disks such as found in virtualised environments and SAN storage arrays.

Swap use is often the cause of disk I/O. Rhino should be installed on a machine with sufficient physical RAM for the workload of the machine.

Swap will also significantly impair GC performance. If even a small amount of the JVM heap is swapped out, GC pause times can increase by up to 300% and potentially much longer for greater swap usage. While the CMS collector will usually touch pages frequently enough to keep them in physical memory, the G1 collector can leave regions untouched for long enough to be swapped out preemptively on Linux.

Rhino’s cluster membership services use multicast UDP. If the IP network is loaded heavily, this can cause packets to be dropped, potentially causing cluster segmentation or failure if individual cluster nodes cannot reliably communicate.

See Cluster Segmentation, Cluster Failing to Start and Cluster Starts but Stops After a few Minutes for more details of multicast related problems.

Large systems (that is, four or more sockets per host have significant asymmetry in memory access speed and latency between CPU sockets. This can cause performance to be significantly less than optimum.

If any of the machines Rhino is running on are running over 75% CPU use, and this is not due to other system processes, then consider:

Processes causing a problem due to resource usage should not be used on a machine running Rhino. Move these to another host.

To prevent preemptive swapping slowing GC, reduce the value of the sysctl vm.swappiness from the default of 60 to something low enough to prevent this. 30 is a good starting point. If using G1 the vm.swappiness sysctl should be set no higher than 30 and preferably 10 or lower.

When running Rhino on NUMA hosts it is usually best to run multiple nodes on each host with each JVM bound to a separate NUMA domain. Configure this with numactl --cpunodebind=X --membind=X on Linux or Locality Groups on Solaris.

There are two common causes of heap issues when running Rhino:

For the latter, see Application or resource adaptor heap issues.

The heap usage of Rhino should be monitored. This includes both the Java heap and the native heap. The Java heap can be measured by Rhino’s statistics tool rhino-stats, or one of the Java monitoring tools — jstat or jvisualvm. The native heap size can be measured using system utilities such as ps or top.

Here’s an example of using the Rhino statistics tool to monitor the size of the Java heap:

The behaviour of the JVM process is that a well-sized heap should gradually decrease from, say, 1.5 gigabytes free to around, say, 800 megabytes free, and then increase up to its higher limit again. This will be overlaid by a shorter pattern of peaks of say 128 megabytes over the long pattern usage. The actual figures will depend on the services that are active and deployed; and the total heap used should never exceed 60% of the heap size allocated for sustained periods when using the default CMS collector.

If the JVM heap is sized appropriately for a given load, and a system appears to run stably at a given load and gradually degrades over a period of at least several hours, then it is possible that the JVM, a resource adaptor or Rhino has a memory leak.

Check the number of live activities and SBBs using the Rhino statistics monitoring tool. If the number of each are within expected values, then it is likely there is a memory leak. If there are too many, it indicates there may be a bug in the component. A strong indicator of an RA or service bug is activities that are more than ten times the age of the average call duration, or older than one whole day.

First raise the maximum heap size available to the JVM by modifying the HEAP_SIZE variable in the config/config_variables file. Ensure that there is enough physical memory on the machine such that the system will not swap if this extra memory is used by the JVM. Also set the NEW_SIZE based on the duration and memory usage of a typical session. The Rhino node must be restarted for this change to take effect. See to JVM documentation for more information on heap sizing and related overheads.

With some allocation patterns it may not be possible to find a heap size or new size that provides an acceptable balance between pause times and durations for the target load. If this is the case the only solutions are to use a different cluster size or contact your solution provider for an improved version of the service that uses memory more efficiently. This problem should only arise as a result of an error in capacity planning and testing. A workaround that is sometimes available if the host server has sufficient free RAM is to increase the heap size until it is large enough that the pauses can be avoided by restarting nodes during idle periods (however this is rarely possible).

Please contact your solution provider support to work through component memory leak issues.

When these symptoms occur it is often necessary to check both for application or resource adaptor related heap issues and for JVM heap issues.

Start the rhino-console, and query for activities that are older than expected for the system. For example, call activities typically last the duration of a phone call; so query for activities that are older than the majority of phone calls (say one hour). To perform this query use the findactivities command with -created-before as a flag.

Here’s an example of finding activities created before 2:30pm:

You can also query for activities older than a specified age, for example:

The resulting number of activities should be a low number. If it is a number larger than, say, 30 (depending on your application) then there may be an issue with some activities not being cleaned up properly by the service or resource adaptor. You should exclude ServiceActivities and ProfileActivities from the set counted.

If the number of activities is small, test for the number of SBB entities in Rhino. SBB entities are removed when they are no longer in use; so if you find a large number of SBB entities that were created a long time ago, it is likely that the service has a problem. The command to query SBB entities is the findsbbs command.

Here’s an example of finding SBB entities for the SIP proxy service created before 2:30pm:

We also recommend that you use the statistics client to examine the memory usage and number of activities in the running SLEE:

This indicates an unhandled case in the Services or resource adaptors being used. Activities and SBB entities known to be invalid may be removed through client/bin/rhino-console using the removeactivity and removesbb commands respectively. These commands may be used to stop a system from running out of memory before a patched application or resource adaptor is installed.

Firstly, check to see how much free memory is available to the JVMs, and that swap space is available. Rhino should never run using swap space (doing so will cause unpredictable performance); but nonetheless, sufficient swap space should be present to allow the OS to manage memory more effectively.

Check the amount of physical memory available. Under Linux, use the free command; under Solaris use /usr/sbin/prtconf \grep Memory.

Compare this amount to the size of the JVM specified in the HEAP_SIZE variable in node-???/config/config_variables (for Rhino-production). Make sure that the JVM can fit comfortably in the physical memory on the computer.

Rhino versions earlier than 2.3.0.7 require about twice the amount of physical memory as the size of HEAP_SIZE to ensure that a fork() of the JVM during compilation of services will always be able to run in physical memory and not incur delays by swapping memory to and from disk. Linux systems use copy-on-write memory cloning when forking processes so the likelihood of swapping is much lower than on Solaris.

You may need to reduce the value of HEAP_SIZE by editing the file $RHINO_HOME/node-???/config/config_variables to make sure Rhino can fit in the available physical memory.

Also make sure there are no other running processes on the computer which may be using excessive system resources, and that there is sufficient swap space.

Rhino prints warnings at startup:

Ensure that the kernel parameters net.core.rmem_max and net.core.wmem_max are large enough.

See UDP buffer sizes in the Preparing your Network section of the Rhino Production Getting Started Guide for details.

When the JVM crashes, the Rhino node will fail and the cluster will reconfigure itself. On Unix systems, the node will die with either a SIGSEGV or a SIGBUS and perhaps dump core, depending on how your system is configured.

A message similar to the following appearing in logs and/or the Rhino console indicates a JVM error.

The following are examples of configuration change messages.

An alarm appears in the console if a node has failed. Use this command to query active alarms:

This command produces a block of output for each active alarm. A block in the list of active alarms similar to the following indicates that a node has left the cluster due to hardware, network, or software failure.

If any of these symptoms occur, then first check the logs of all Rhino cluster members (file console.log). If a JVM error message is part of the logs of any node, then a Java Virtual Machine error has occured. If an error message cannot be found in the logs please refer to Cluster Segmentation, Cluster Failing to Start and Cluster Starts but Stops After a few Minutes

After checking Rhino’s logs, determine whether or not the hardware or operating system is causing the problem. Look at the logs of the local machine and determine whether the machine has had a history of restarts, kernel panics, process segmentation faults, and so forth. On Unix machines, system logs can be viewed using the dmesg command or by viewing logs in /var/logs.

When a JVM crash occurs, a file will be left with a name like hs_err_pid*.log. View this file and try to determine which part of the JVM caused the crash. This may provide clues as to how to resolve the situation and will be needed if a bug in the JVM is to be reported.

If the crash appears to be a one-off, the node can simply be restarted.

If the problem resides in faulty hardware, then that hardware will need to be fixed or replaced.

OpenCloud has experience with configuring the Sun Java Virtual Machine to be as stable as possible. Consider consulting OpenCloud support for information regarding known stable configurations of JVMs. Also ensure that thorough stability tests are performed on a new version of a Java Virtual Machine before it is deployed in a production system.

Also scan through the list of dynamic libraries in the hs_err_pid*.log file and see if they are all the latest version. Keep an eye out for libraries which should not be there, for example from other installations of older JVMs.

If it is a realistic option, upgrading the operating system to the latest version may also resolve this problem.

JVM errors that cannot be identified as being caused by system configuration or dynamic library problems should be reported to Oracle.

The host machines for Rhino have multiple network interfaces, and we need to control which interface is used for SLEE cluster traffic. Network monitoring shows this is currently not on the desired interface.

When the configuration differs between hosts, nodes may fail to go primary. This is indicated by repeated messages in the Rhino log indicating that a node is waiting for the cluster to go primary and alack of configuration change messages.

Messages like this show up in the Rhino console and logs:

Use a network monitoring tcpdump or snoop to detect multicast traffic on the groups configured for Rhino clustering.

Use netstat -rn, route, or ip route to examine the current multicast routing configuration.

The choice of interface is controlled solely by the machines' multicast configuration; it’s controlled at the OS level rather than by Rhino. Use route or ip route to modify the OS routing configuration. See Cluster Failing to Start for how to configure the interface used for multicast traffic.

A message in the Rhino console log like the following indicates a JVM error.

If such a message is present, please see Java Virtual Machine Error.

First determine whether or not the segmentation is environmental. This means that the operating environment (see Operating Environment Issues) and the JVM Heap (see Java Virtual Machine Heap Issues) must be suitably configured. If these are configured suitably then the issue is likely to something with the network which interconnects the Rhino nodes. Typical network components to check for failure are machines, network interface cards, routers, switches, and cables.

If the cause of the symptoms are environmental, then refer to the workaround or resolution steps for these issues. If the cause is a network issue, then it needs to be repaired by replacing the failed components.

Check the current set of registered multicast groups with netstat -gn. These should include all the groups used by Savanna and the global IGMP membership reporting address 224.0.0.1. Compare the reported set between nodes and with the set present immediately after a node has started.

Check for "Martians" in syslog. Sometimes a peer will be a device on the multicast network that is in a different IP range from the hosts.

To enable logging of these on Linux run:

When multicast networking is in use a network device, commonly the switch or router, acts as an IGMP querier, sending regular messages to test for active multicast devices. The OS kernel receives and responds to these queries with membership reports. If no membership reports are received before the timeout configured on the querier it will stop forwarding multicast traffic.

To report IGMP traffic (Multicast group membership queries and reports) for all network interfaces on a host:

Use tcpdump or snoop to capture IGMP traffic on the interface used for multicast. Look for Query messages and Membership Reports.

The IGMP membership addresses 224.0.0.1 and 224.0.0.2 must be routed on the interface used for Savanna clustering as well as the configured Savanna group multicast addresses.

Configure the local switch to act as an IGMP querier. Especially important for virtualised environments

Sometimes you need to force the kernel to fall back to IGMPv2. Usually this will be because the switch does not support IGMPv3. To configure this, run:

Make this change permanent by adding net.ipv4.conf.eth0.force_igmp_version=2 to /etc/sysctl.conf

While uncommon for hosts in the cluster, switches are frequently in different IP ranges. The Linux kernel will drop these messages from "impossible" addresses by default as a DOS prevention measure. Also some switches will send IGMP queries on the All-Hosts group instead of the specific group Rhino joined. If your querier is on a different subnet from the nodes, disable the Martian packet safety check.

Fix this by configuring the rp_filter sysctl to accept packets from IPs that are not reachable from the cluster interface, by setting the rp_filter sysctl to 0 or 2 if the source address is on a network visible from another interface.

Make this change permanent by adding these to /etc/sysctl.conf:

You will also need to configure the firewall to accept IGMP packets sent to 224.0.0.1 and 224.0.0.2 if your rule does not include these implicitly.

Output is usually something like this:

If the cluster is configured to use scattercast follow the diagnostic procedures for Cluster Failing to Start and Scattercast endpoints out of sync. For multicast clusters follow the steps below:

The install script checks to ensure that a valid multicast range is used but this may be changed manually afterward. If the address reported is in the multicast range described above, check that your OS kernel supports multicast:

If the configured addresses are not all within the multicast range change the configuration to use only multicast addresses.

If the configured UDP multicast range is not used by other applications, it should work fine for Rhino. Failing that, any address range can be used in 224.0.0.0/4.

To change the addresses without reinstalling, edit the file $RHINO_BASE/node-101/config/config_variables and change the following lines to use multicast addresses not taken by other programs:

The master copy of this file (used when creating additional nodes) is $RHINO_BASE/etc/defaults/config/config_variables. If several nodes are to be deployed, this can be changed before nodes are created, to save extra editing at a later stage.

Output is something like this:

Also see Rhino Watchdog.

This problem can manifest in several ways, depending on how the scattercast endpoints are out of date. One example is shown below. Almost all examples of scattercast endpoints out of sync will result in the out of date nodes shutting down.

It is possible to create a situation where a node will boot with an out-of-date persistent endpoints set, and fail to shut down. If booted after a scattercast management command removes it from the endpoints set, a node will not go primary nor shut down.

This will trigger frequent cluster membership changes in the remaining nodes. The cluster membership change messages will not change the set of members.

Diagnostic and corrective steps If some nodes are operational, run the Rhino console command getscattercastendpoints to learn the current cluster configuration. This will print the live in-memory configuration; and if the scattercast.endpoints file on any running node contains a different configuration, it will print the memory state for each node and the differences from the disk state.

If all nodes have the same configuration, the output will be similar to the example below:

If the node configuration is consistent but does not contain the expected set of nodes, then the missing nodes should be added by running the addscattercastendpoints command.

If the disk mapping differs from the in-memory configuration, then the nodes with the differing configuration are likely to fail when restarted. An example output where node 101 has a malformed file that will cause a node to fail to reboot is below:

If some nodes have a different configuration on disk from the live memory configuration, then copy the scattercast.endpoints file from a node that has matching disk and memory configurations.

The following steps are to be followed if the previous two stages of diagnosis were unsuccessful. All three steps use the output from client/bin/rhino-stats monitoring the Staging Threads parameter set.

The output for the Staging Threads should typically be monitored over at least a 30 minute period in order to ensure that the system characteristics are noticed through different garbage collection cycles.

An example command which connects to a Rhino cluster member on the local host and monitors the Staging Threads parameter set is as follows.

If the diagnostic steps show that there are not enough threads available, try doubling the number of staging threads. This can be achieved using the rhino-console command at runtime, or by editing the thread-count attribute of the staging queues element in config/config.xml and then restarting that node.

Note that it is possible to set the number of staging threads too high, and in doing so cause other performance problems. We do not recommend setting the number of staging threads higher than 200 on most systems or approximately 10x the number of CPU cores. Setting it too high will cause scheduler thrashing which will increase latency and reduce throughput.

If the problem persists then this indicates there is a potential issue with the Services, and/or Resource Adaptors installed in Rhino.

In the case of performance problems with services or suspected bugs in Rhino please contact your solution provider for support. If possible the problem should be reproduced in a test environment so additional logging for diagnosis does not impact live traffic.

If the exception is of the type AccessControlException, please refer to section Security Related Exceptions to solve the issue.

If the exceptions contain the text "Event router transaction failed to commit" it is likely that the in-memory database is not sized correctly for the load. Follow the diagnostic and resolution steps in Memory Database Full.

For other exceptions please contact your solution provider for support.

If lock timeout messages are occuring please refer to section Lock Timeouts. Two examples of lock timeout messages are as follows.

If lock timeout messages are occurring please refer to Lock Timeouts.

A rate limiter may be rejecting input if the incoming rate exceeds its specified rate.

The rate limiter raises an alarm and outputs to the Rhino logging system if it is actively rejecting input. An example of this is shown as follows.

A user rate limiter is designed to reject input above a certain threshold to avoid overloading Rhino. If this threshold is appropriately configured then the messages should occur very rarely. If they occur frequently then it is possible that the solution is not sized appropriately for the incoming load. In this case please contact your solution provider support.

This is difficult to diagnose in a general manner as it is specific to the solution running on Rhino. However the system administrator should be aware of any external systems which the solution communicates with. Examples may include rating/charging engines, HLRs, HSS, MSCs, IVRs, SMSCs, MMSCs, databases, etc.

Please refer to administrators of dependant systems and if necessary contact your solution provider support.

Java takes some time to settle during application startup due to Just in Time(JIT) compilation of code hotspots. For this reason, we recommend slowly introducing traffic to a newly booted node. Most of the hotspots will be optimised in the first minute, however it is better to ramp up load over the course of up to 15 minutes to allow JIT compilation and optimization to occur.

Cluster size should be set during provisioning to allow sufficient reserve capacity to handle peak-hour traffic with at least one failed node. This allows a lot of flexibility in bringing a failed node back into the cluster.

The recommended approach to bringing a node back into the cluster is to use a load balancing mechanism to limit traffic to the rebooted node. (Signalware has this mechanism built in) Usually a smooth increase from zero to a fair share over the course of 5 to 15 minutes (depending on total load) allows the JIT compiler to work without causing call failures or unacceptable latencies.

Another option, if sufficient reserve capacity exists, is to simply wait until cluster load drops to a level that a new node can cope with before rebooting the failed node. This works, as the reserve capacity ensures that the node failure is not visible to callers.

For Java 7 update 45 and newer enabling tiered compilation may result in some improvement at node boot time. This will require a much, much larger code cache as well due to the more aggressive compilation. This it does not avoid the initial very slow interpreted calls, it merely begins compilation more aggressively, which improves performance faster. This option is on by default in Java 8 where the default code cache size has been increased to handle the extra data.

Note that to avoid performance decreasing, the default code cache size must be set when using Java 7, as the default size of 48MB is not large enough. Java 8 defaults to 240MB and fixes some performance bugs in flushing old compilation data however you may still see benefit from increasing the code cache.

CPU load or memory usage is inconsistent across the cluster.

Examine the service usage statistics to rule out differences in applied load.

A small difference in CPU load is expected for nodes servicing management connections. These nodes will also consume slightly more heap and Java Permanent Generation memory. If the amount of PermGen memory used on a node is greater than 90% of the maximum a CMS GC cycle will be triggered to try to reclaim PermGen, When permgen remains above 90% CMS cycles will run almost continuously and cause a significant increase in CPU usage. Check the heap statistics for the node or the GC log entries in the console log to determine if this is the cause of the imbalance. Log entries will contain a section similar to the following with a high proportion of the Perm space used and only a small change after GC:

If the problem is memory related increase the allocated heap or PermGen so that there is sufficient free space for normal operation. For more information on heap configuration see Java Virtual Machine Heap Issues

If the problem is not memory related investigate the balance of load across threads. A small number of threads consuming a large percentage of CPU time may indicate a software bug or a load profile that is unusually serial in execution. On Linux the command top -ch will display the busiest threads and the process they belong to. Run the command $RHINO_HOME/dumpthreads.sh to trigger a Java thread dump. The thread IDs in the dump can be matched to the PIDs of the threads displayed in top by converting the PID to hexadecimal. Frequently the stack traces for busy threads will be similar. This can suggest the part of the code that contains the bug and may help identify the trigger conditions. Contact your solution provider for assistance, attaching the thread dump and list of busy threads.

Components inside Rhino run in a security context. These components include SBBs, Resource Adaptors, Profiles and Mlets. If a component attempts to perform some operation which requires a security permission that the component does not have, then an exception is generated. An example of this is when a Resource Adaptor attempts to write to the filesystem when it does not have permission to do so.

Typically this indicates that the component is not configured appropriately.

As a step in diagnosing security-related problems, the security manager can be disabled by commenting out the following line in the read-config-variables script:

This is not recommended as a permanent solution and should not be used on a production system. Disabling the security manager should only be used temporarily as a means to diagnose the problem in a test environment. To determine which permissions are required uncomment this line in $RHINO_HOME/read-config-variables:

Rhino will then print the permissions that are failing security checks to the console log. The required permissions should be added to the <security-permissions> section of the component deployment descriptor. For more information regarding granting additional permissions to an RA or SBB, refer to the sections on RA and SBB Deployment Descriptors in the Rhino Administration Manual.

If calls are not being setup successfully the failures may be caused by the size restriction of either or both of the LocalMemoryDatabase and ReplicatedMemoryDatabase installed in Rhino.

If the service does not use replicated transactions the Rhino log will contain messages similar to this:

Replicated services will produce log entries similar to this:

The general workaround for MemDB sizing problems is to make the appropriate Memory Database larger. To do this, edit the node-*/config/rhino-config.xml file and look for a <memdb> or <memdb-local> entry which has a <jndi-name> with the appropriate Database name (e.g. ProfileDatabase). Increase the <committed-size> of the database to increase its maximum committed size.

Increasing the size of a Memory Database also requires increasing the maximum heap size of the JVM to accomodate the larger database. To do this, edit the node-*/config/config_variables file and increase the HEAP_SIZE by the same amount.

These steps will need to be performed for all nodes, and the nodes will need to be restarted for the change to take effect.

Initially try doubling the size of a database and monitor usage to determine the relationship between load and usage. Assuming usage is proportional to call load you may then need to alter the configured size to accomodate the highest sustained load peak. If usage does not appear to be proportional to load or the problem is not solved after a second increase in size, please contact your solution provider for support.

Use the network diagnostic tools provided with your OS distribution to check the current connected and listening ports for running programs. One common tool for this purpose is netstat:

Check the firewall configuration on both the host running Rhino and the host providing the network function the RA connects to. Also check any intermediate network components that provide filtering functions.

Try adding`-Djava.net.preferIPv4Stack=true` as a command-line argument to Rhino. This option can be set by adding a line EXTRA_OPTIONS=-Djava.net.preferIPv4Stack=true to the RHINO_HOME/config_variables file. Other Java programs that connect to Rhino may also need this argument added. For other programs an RA connects to consult the documentation for that program to learn how to configure the address family to use. If a program cannot be configured appropriately you may need to disable support for IPv6 in the OS configuration.

If a firewall is configured to block the addresses used by the resource adaptor it must be reconfigured to allow connections.

First check that proper permissions have been given in the appropriate sections of $RHINO_HOME/config/mlet.conf (SDK) respectively $RHINO_HOME/etc/default/config/mlet-permachine.conf (Production).

Check that no matter whether you are trying to connect through the Standalone Webconsole or the Commandline Console application you get an exception similar to the one below:

Edit the contents of the file /etc/hosts. It should contain entries as shown below.

Entries that resolve the machine’s name to its local address should be commented out:

The management clients show the following error when attempting to connect to the SLEE:

A management client hangs for a long period of time on start-up as it tries to read from /dev/random.

The ideal problem resolution is to create more system entropy. This can be done by wiggling the mouse, or on a remote server by logging in and running top or other system utilities. Refer also to the operating system’s documentation, on Linux this is the random(4) man page: man 4 random.

When gathering statistics, the following may appear in the logs:

This is a benign problem and can be safely ignored. The reported sample statistics will be slightly inaccurate. To prevent it, reduce the sampling rate.

The statistics client will fail with an Out of Memory Exception.

The user is recommended to use the -k option of the statistics client when running in graphical mode to limit the number of hours of statistics kept. If it is required that statistics be kept for a longer period, it is recommended that the statistics client be run in command-line mode and the output be piped to a text file for later analysis.

For more information, run client/bin/rhino-stats without any parameters. This will print a detailed usage description of the statistics client.

The following error would appear in Rhino’s logs:

For the case where a SyslogAppender is required, the createsyslogappender command of the rhino-console provides a much easier user interface to achieve this task.

Replacing the entry org.apache.log4j.net.SyslogAppender above with com.opencloud.rhino.logging.RhinoSyslogAppender will also fix this problem. The Open Cloud version of the SyslogAppender is a simple wrapper around the Log4J version which wraps the append(LoggingEvent event) method in a “doPrivileged” block. For custom appenders not provided with Rhino, the same method can be used:

The presence of an alarm may be viewed by the output of the following command.

The alarm type used in notifications that are reporting an alarm has cleared is the original alarm type plus .clear, for example rhino.node-failure.clear.

Alarms with an alarm source of ThresholdAlarms indicate that the system is receiving more input that it has been configured to receive.

Alarms with an alarm source of LicenseManager indicate that a Rhino installation is not licensed appropriately. Other alarms are either user defined licenses, or defined by an Application or Resource Adaptor.

Alarms with an alarm source of MLetStarter or some other non-obvious key usually indicate a software issue, such as a misconfiguration of the installed cluster.

In most of these cases, the remedy is to contact your solution support provider for a new license or for instructions on how to remedy the situation.

DeploymentException when trying to deploy a component.

Each time you deploy the RA, it happens in a new classloader (because the code may have changed). If no class GC has happened, or if something is holding a reference to the old classloader and keeping it alive, the old library will still be loaded as well. See http://java.sun.com/docs/books/jni/html/design.html#8628

modifying the deployable-unit.xml file within it and the output of jar -tvf looks similar to the below:

Jars always use forward slashes ("/") as a path separator. Repackage the DU jar with a different file archiver, preferrably the jar tool.

You are deploying Rhino using a script that creates and deploys components to multiple nodes asynchronously. The deployment fails with one of the following exceptions on each node. When deploying the nodes serially, one after the other, no exceptions are reported.

Rhino provides single system image for management. You do not need to deploy a DU on each node in a cluster. Installing a deployable unit on any node in Rhino cluster propagates that DU to all nodes in the cluster, so if the DU had already deployed via node 102, it can’t also be deployed via node 101.

In addition, if a new node is created and joins a running cluster, it will be automatically synchronised with the active cluster members (i.e DUs installed, service states, log levels, trace levels, alarms etc).

A Rhino cluster will only allow one management operation that modifes internal state to be executed at any one time, so you can’t, for example, install a DU on node 101 and a DU on node 102 at the same time. One of the install operations will block until the other has finished. You can run multiple read-only operations simultaneously, though.

Create all nodes from the same base install, optionally starting the nodes in parallel. Wait for the nodes to start then run the component installation script against only one node.

You are trying to use rhino-console to talk to multiple rhino instances however it will not connect to the second instance.

Unfortunately it is not possible to store keys for multiple Rhino instances in the client’s keystore, they are stored using fixed aliases. With the current implementation, there are two ways to connect to multiple Rhino instances from a single management client:

If doing this you may also want to parameterise the keystore password to restrict access to authorised users.

Deployment fails with Unable to prepare due to size limits of the DB

Installation fails with an error like:

Run Ant with debugging output to check which version of Ant is being used and the classpath

Add the missing libraries to your Ant lib directory

We recommend you use the Ant version shipped with Rhino to avoid this problem e.g.

Performing a management operation, e.g. activating an RA entity, fails with the following error:

The message is sometimes seen when Rhino is under load and JMX operations are slow to return. Check the CPU load on the Rhino servers.

REM Exceptions of this type can also occur when stopping or starting the whole cluster.

When REM auto-refresh interval is set to low value (default is 30 seconds) there is high likelihood of a lock collision happening. With higher auto-refresh intervals the likelihood drops down. With auto-refresh interval set to "Off" the exception may not occur at all.

If the rem.interceptor.connection log key is set to DEBUG in REM’s log4j.properties, then the logs will show which operations could not acquire the JMX connection lock.

If the CPU load on the Rhino server is high then follow the resolution advice in Operating environment issues.

If the auto-refresh interval is low then increase it until the problem stops.

For further diagnostic and resolution assistance contact Open Cloud or your solution provider, providing the REM logs.

Trying to export Rhino configuration with rhino-export fails with an error like:

Run rhino-export with the -J option to use JMX for exporting the profile data. This is slightly less efficient but can handle multiple profile storage locations.

After installing multiple versions of a service on Rhino listlogkeys reports a number of obsolete keys. How can these be removed?

The unused log keys are marked for removal - they should disappear when the Rhino JVM restarts. Until then they are harmless.

Any management operation done on the cluster fails with:

The management lock is acquired and held for the duration of all management operations to prevent concurrent modification of Rhino state and on-disk data. To find out more information about the transaction holding the lock use the gettransactioninfo console command. To find out what method is blocking release of the lock use jstack $(cat node-???/work/rhino.pid) or kill -QUIT $(cat node-???/work/rhino.pid) to dump the current thread state to the Rhino console log.

Contact your solution provider with the Rhino logs showing the problem and a list of the management operations that were performed immediately prior to the one that timed out. If the management operation is permanently blocked, e.g. by an infinite loop in the raStopping() callback of an RA, the cluster will need to be restarted to interrupt the stuck operation. If it is not permanently blocked you must wait until the operation has finished.

Setting the log level for a logger trace.??? does not change the level of information logged under this key. The log level was set using the setloglevel command.

The log keys trace.??? are special. These are SLEE tracers which have their own level configuration that they feed into the logging subsystem. Use the settracerlevel command to set tracer levels for RA entities and SBBs.

After updating or reinstalling REM access fails with an error in the REM log similar to:

The most common cause for these errors is that the browser-hosted part of REM does not match the server code. Refresh the browser tab to reload the client code. You may need to clear the browser cache.

Warning messages in the system console from the log key memdb.storer

Messages similar to the following indicate that a PostgreSQL Database Server has terminated either due to administrative action or failure.

The management database server should be restarted as soon as possible. Once the database has been restarted and the Rhino cluster has reconnected, messages similar to the following should appear in the Rhino logs.

To reduce the risk of this causing data loss Rhino can be configured to replicate persisted data across multiple PostgreSQL instances. When using Oracle as the persistence database, replication is provided by the Oracle RAC clustering mechanism.

If, when activating the resource adaptor (or starting the Rhino SLEE with the resource adaptor active) the following message immediately appears in the Rhino logs and an alarm is raised stating that a backend connection has been lost, then this indicates that the backend process is not reachable by the resource adaptor. Generally this would mean it is either not running or, due to resource adaptor or network misconfiguration, the resource adaptor is unable to establish a TCP connection to it.

If the resource adaptor successfully established a connection to the backend but later one or both of the following messages appears in the logs this indicates that the connection to the backend process was setup successfully but lost due to backend or network failure.

First determine whether the backend process is running. If the backend process is not running then either it or Signalware may be incorrectly configured.

If the backend process is running check that the host and port it is listening on is correctly configured in the resource adaptor entity configuration properties. Also verify that the host running the backend is reachable from the Rhino host and that the port is not blocked by a firewall on either host.

If a connection is established but gets dropped this indicates either a software failure in the backend or a network level failure. Check that the backend process is running and if not, restart it. Any error messages from a failed backend should be sent to your solution provider to determine the cause of failure. If the backend process is still running check for any network connectivity problems between the Rhino host and the host running the backend processes.

If the backend process has reached its maximum number of active dialogs an warning similar to the following will be logged by the Resource Adaptor:

Additionally an error similar to the following will be visible in the output of the backend process:

By default each backend process supports up to 32,000 simultaneous dialogs. If the number of simultaneous dialogs exceeds this limit the above errors will occur. There are two approaches to working around this issue:

If the backend process has reached its maximum number of active dialogs the Signalware backend process will log something similar to the following:

The root cause and solution for this is identical to that for CGIN RA Cannot Create Outgoing Dialogs. Please see that section for details.

No known exit codes other than 0 (Normal exit)

Exit codes above 128 occur as a result of the JVM receiving a signal from the OS. These are always a result of either OS process management or a JVM bug that has triggered an OS level protection mechanism. In many cases the console log will have information on the cause. There may also be information in the system log. On Linux these are the following, other OSes differ: