Management

The management clients show the following error when attempting to connect to the SLEE:

A management client hangs for a long period of time on start-up as it tries to read from /dev/random.

The ideal problem resolution is to create more system entropy. This can be done by wiggling the mouse, or on a remote server by logging in and running top or other system utilities. Refer also to the operating system’s documentation, on Linux this is the random(4) man page: man 4 random.

When gathering statistics, the following may appear in the logs:

This is a benign problem and can be safely ignored. The reported sample statistics will be slightly inaccurate. To prevent it, reduce the sampling rate.

The statistics client will fail with an Out of Memory Exception.

The user is recommended to use the -k option of the statistics client when running in graphical mode to limit the number of hours of statistics kept. If it is required that statistics be kept for a longer period, it is recommended that the statistics client be run in command-line mode and the output be piped to a text file for later analysis.

For more information, run client/bin/rhino-stats without any parameters. This will print a detailed usage description of the statistics client.

The following error would appear in Rhino’s logs:

For the case where a SyslogAppender is required, the createsyslogappender command of the rhino-console provides a much easier user interface to achieve this task.

Replacing the entry org.apache.log4j.net.SyslogAppender above with com.opencloud.rhino.logging.RhinoSyslogAppender will also fix this problem. The Open Cloud version of the SyslogAppender is a simple wrapper around the Log4J version which wraps the append(LoggingEvent event) method in a “doPrivileged” block. For custom appenders not provided with Rhino, the same method can be used:

The presence of an alarm may be viewed by the output of the following command.

The alarm type used in notifications that are reporting an alarm has cleared is the original alarm type plus .clear, for example rhino.node-failure.clear.

Alarms with an alarm source of ThresholdAlarms indicate that the system is receiving more input that it has been configured to receive.

Alarms with an alarm source of LicenseManager indicate that a Rhino installation is not licensed appropriately. Other alarms are either user defined licenses, or defined by an Application or Resource Adaptor.

Alarms with an alarm source of MLetStarter or some other non-obvious key usually indicate a software issue, such as a misconfiguration of the installed cluster.

In most of these cases, the remedy is to contact your solution support provider for a new license or for instructions on how to remedy the situation.

DeploymentException when trying to deploy a component.

Each time you deploy the RA, it happens in a new classloader (because the code may have changed). If no class GC has happened, or if something is holding a reference to the old classloader and keeping it alive, the old library will still be loaded as well. See http://java.sun.com/docs/books/jni/html/design.html#8628

modifying the deployable-unit.xml file within it and the output of jar -tvf looks similar to the below:

Jars always use forward slashes ("/") as a path separator. Repackage the DU jar with a different file archiver, preferrably the jar tool.

You are deploying Rhino using a script that creates and deploys components to multiple nodes asynchronously. The deployment fails with one of the following exceptions on each node. When deploying the nodes serially, one after the other, no exceptions are reported.

Rhino provides single system image for management. You do not need to deploy a DU on each node in a cluster. Installing a deployable unit on any node in Rhino cluster propagates that DU to all nodes in the cluster, so if the DU had already deployed via node 102, it can’t also be deployed via node 101.

In addition, if a new node is created and joins a running cluster, it will be automatically synchronised with the active cluster members (i.e DUs installed, service states, log levels, trace levels, alarms etc).

A Rhino cluster will only allow one management operation that modifes internal state to be executed at any one time, so you can’t, for example, install a DU on node 101 and a DU on node 102 at the same time. One of the install operations will block until the other has finished. You can run multiple read-only operations simultaneously, though.

Create all nodes from the same base install, optionally starting the nodes in parallel. Wait for the nodes to start then run the component installation script against only one node.

You are trying to use rhino-console to talk to multiple rhino instances however it will not connect to the second instance.

Unfortunately it is not possible to store keys for multiple Rhino instances in the client’s keystore, they are stored using fixed aliases. With the current implementation, there are two ways to connect to multiple Rhino instances from a single management client:

If doing this you may also want to parameterise the keystore password to restrict access to authorised users.

Deployment fails with Unable to prepare due to size limits of the DB

Installation fails with an error like:

Run Ant with debugging output to check which version of Ant is being used and the classpath

Add the missing libraries to your Ant lib directory

We recommend you use the Ant version shipped with Rhino to avoid this problem e.g.

Performing a management operation, e.g. activating an RA entity, fails with the following error:

The message is sometimes seen when Rhino is under load and JMX operations are slow to return. Check the CPU load on the Rhino servers.

REM Exceptions of this type can also occur when stopping or starting the whole cluster.

When REM auto-refresh interval is set to low value (default is 30 seconds) there is high likelihood of a lock collision happening. With higher auto-refresh intervals the likelihood drops down. With auto-refresh interval set to "Off" the exception may not occur at all.

If the rem.interceptor.connection log key is set to DEBUG in REM’s log4j.properties, then the logs will show which operations could not acquire the JMX connection lock.

If the CPU load on the Rhino server is high then follow the resolution advice in Operating environment issues.

If the auto-refresh interval is low then increase it until the problem stops.

For further diagnostic and resolution assistance contact Open Cloud or your solution provider, providing the REM logs.

Trying to export Rhino configuration with rhino-export fails with an error like:

Run rhino-export with the -J option to use JMX for exporting the profile data. This is slightly less efficient but can handle multiple profile storage locations.

After installing multiple versions of a service on Rhino listlogkeys reports a number of obsolete keys. How can these be removed?

The unused log keys are marked for removal - they should disappear when the Rhino JVM restarts. Until then they are harmless.

Any management operation done on the cluster fails with:

The management lock is acquired and held for the duration of all management operations to prevent concurrent modification of Rhino state and on-disk data. To find out more information about the transaction holding the lock use the gettransactioninfo console command. To find out what method is blocking release of the lock use jstack $(cat node-???/work/rhino.pid) or kill -QUIT $(cat node-???/work/rhino.pid) to dump the current thread state to the Rhino console log.

Contact your solution provider with the Rhino logs showing the problem and a list of the management operations that were performed immediately prior to the one that timed out. If the management operation is permanently blocked, e.g. by an infinite loop in the raStopping() callback of an RA, the cluster will need to be restarted to interrupt the stuck operation. If it is not permanently blocked you must wait until the operation has finished.

Setting the log level for a logger trace.??? does not change the level of information logged under this key. The log level was set using the setloglevel command.

The log keys trace.??? are special. These are SLEE tracers which have their own level configuration that they feed into the logging subsystem. Use the settracerlevel command to set tracer levels for RA entities and SBBs.

After updating or reinstalling REM access fails with an error in the REM log similar to:

The most common cause for these errors is that the browser-hosted part of REM does not match the server code. Refresh the browser tab to reload the client code. You may need to clear the browser cache.