Rhino Production Getting Started Guide

Other documentation for the Rhino TAS can be found on the Rhino TAS product page.

Rhino requires a process ulimit of at least 4096 processes to function reliably.

The general hardware requirements below are for a Rhino production system, used for:

Performance measures and targets vary, based on the application deployed.

The Rhino SLEE requires an DBMS database for persisting the main working memory to non-volatile memory (the main working memory in Rhino contains the runtime state, deployments, profiles, resource adaptor entity configuration state, and so on). The Rhino SLEE remains available whether or not the database is available.

The database does not affect or limit how Rhino SLEE applications are written or operate — it provides a backup of the working memory only, so that the cluster can be restored if it has entirely failed and needs to be restarted.

The database can be installed on any network-reachable host, and only a single database is required for the entire Rhino SLEE cluster (the Rhino SLEE can replicate the main working memory across multiple servers).

Internal performance testing results show that on a 2 socket machine, NUMA may be safely ignored but does offer quite small benefits to maximum and 99th percentile latencies.

For larger machines (4 socket and up) ignoring NUMA architecture was not possible. It is impossible to size a Rhino node such that it can exploit all sockets without crashing or exhibiting unacceptable latencies under load.

Running multiple Rhino nodes on a multi-socket server should in theory be sufficient for production Rhino, as the default policy for all supported OSs is to attempt to keep threads from one process on the same CPU, and balance load equally amongst CPUs. Under low cluster load and during cluster startup this is not reliable, and may not remain stable over time with daily load cycles.

Using NUMA binding tools to restrict each Rhino node to a single CPU(using local memory) guarantees that the nodes will never migrate between CPUs, and is considered safer in a production environment where sudden performance changes are undesireable.

Each time you run the rhino-install.sh script, it generates a matching set of server and client keys, for authenticating SSL connections. For a client to connect to a server, their keystores must match. If you have multiple nodes for a cluster, for different clients to connect to different nodes, you will need to copy over their keys.

The keys are stored in:

To allow a single Rhino client to connect to multiple Rhino nodes, copy rhino-client.keystore, from the Rhino base directory of the node on which rhino-install.sh was run with that client, to the Rhino base directory on the other nodes to which you want that client to be able to connect.

To view the keys in each keystore, and to check that the keyEntry in a keystore matches the trustCertEntry in another, use the commands keytool -keystore rhino-client.keystore -list and keytool -keystore rhino-server.keystore -list.

The install script has the following options:

You’ll use:

For example, to create the answer file:

And then to install, unattended, based on that answer file:

Below is an example of an answer file:

Rhino is configured to use the Postgres database by default. To initialise it, execute the init-management-db.sh shell script from a node directory (see Create New Nodes).

For example:

The init-management-db.sh script produces the following console output:

To use Oracle as Rhino’s persistent datastore, reconfigure before initialising:

The rhino.sh script is used to start and stop Rhino nodes on a host. It can be used to manage individual nodes or all the nodes on the local host.

If starting a cluster for the first time (i.e. where no previous cluster state exists) and the 'make primary' option to create-node.sh was not specified for one of the nodes, then the cluster must be manually made primary (by using the make-primary.sh script in a node directory).

The slee.sh script is used to control SLEE state for a cluster. It can be used to manage individual nodes or the whole cluster.

Every node contains an init.d script for managing itself. This script can be found at ${NODE_HOME}/init.d/rhino-node-xxx

If node xxx is intended to operate as a quorum node, the rhino-node-xxx script will need to be modified before use to replace the '-s' argument with '-q'. The command line options used during Rhino start can be found in the script in the RHINO_START_ARGUMENTS variable.

If configuring a cluster for the first time (i.e. where no previous cluster state exists) and the 'make primary' option to create-node.sh was not specified for one of the nodes, then the cluster must be manually made primary.

The recommended initial setup procedure is to start each node via its associated init.d script and then run 'make-primary.sh' on one (and only one) of them. This additional setup step should only be performed once to initialise the cluster state.

There is an init.d script that manages multiple nodes, in ${RHINO_HOME}/init.d/rhino.

The script requires two variables to be set.

It will detect nodes automatically, but it is recommended to set the RHINO_SLEE_NODES and RHINO_QUORUM_NODES variables explicitly (this is typically done in $RHINO_BASE/rhino.env which is also used by the rhino.sh control script).

If starting a cluster for the first time (i.e. where no previous cluster state exists) and the 'make primary' option to create-node.sh was not specified for one of the nodes, then the cluster must be manually made primary (by using the make-primary.sh script in a node directory).

There is a sample Systemd service control file equivalent to the host-wide init script for use on RHEL 7 and similar systems.

The service file assumes a Rhino installation in /opt/opencloud/rhino. It delegates to rhino.sh and expects RHINO_SLEE_NODES and RHINO_QUORUM_NODES to be set in rhino.env. This service file must be modified to match the Rhino install path on the system and should have the dependency on PostgreSQL removed if using Oracle.

Like the host-wide init script it does not cause a new cluster to become primary automatically. This must be done on node creation or by using the make-primary.sh script in a node directory.

To start a node, run the start-rhino.sh shell script ($RHINO_HOME/node-NNN/start-rhino.sh), which causes the following sequence of events:

The start-rhino.sh script supports the following arguments:

The -s, -x, -d, and -c options cannot be used in conjunction with the -q option. The -s and -x options are also mutually exclusive and cannot be used together. The -d and -c options must be used together if the starting node already has per-node activation state and you want that state replaced with the state from another node.

You can start and stop SLEE event-routing functions on each individual cluster node. To transition the SLEE on a node to the RUNNING state:

To start a cluster for the first time and create the primary component, the system administrator typically starts the first node with the -p option and all nodes with the -s option, as follows.

On the first machine:

On the second machine:

On the last machine:

You can stop a node using the $RHINO_HOME/node-NNN/stop-rhino.sh shell script. This script has the following options:

For example:

This terminates the node process, while leaving the remainder of the cluster running.

Use the following command to stop and shutdown the cluster.

This transitions the Rhino SLEE to the STOPPED state on every node in the cluster, and then terminates them all.

Use the following command to restart a node.

This will first stop the SLEE on the node then shut it down. The node will automatically restart to the state it was in before the command was invoked.

Use the following command to restart the cluster.

This will first stop the SLEE on every node in the cluster then shut them down. The nodes will automatically restart to the state each was in before the command was invoked.

After installation, you can modify the default configuration variables if needed, for each node, by editing node-NNN/config/config_variables. This file includes the following entries:

The ports chosen during installation time can be changed at a later stage by editing the file $RHINO_HOME/etc/defaults/config/config_variables.

The default usernames and passwords for remote JMX access can be changed by editing the file $RHINO_HOME/etc/defaults/config/rhino.passwd. For example,

The watchdog thread is a lightweight thread which monitors the Rhino SLEE for undesirable behaviour. Currently, the only user-configurable settings for the watchdog thread relate to its behaviour when dealing with stuck worker threads. A stuck worker thread is a thread which has taken more than a reasonable period of time to execute the service logic associated with an event. The cause for this may be faulty service logic, or service logic which blocks while waiting on an external resource (such as a database).

The period (in milliseconds) after which a worker thread is presumed to be stuck can be configured by editing the RHINO_WATCHDOG_THREADS_THRESHOLD variable in $RHINO_HOME/etc/defaults/config/config_variables, for example:

If too many worker threads become stuck, there can be a performance impact on the Rhino SLEE, and in extreme cases can prevent all future event processing entirely. The watchdog thread can be configured to terminate a node in the event that a certain percentage of its worker threads have become stuck by modifying the variable;

The value specified for RHINO_WATCHDOG_THREADS_THRESHOLD in $RHINO_HOME/etc/defaults/config/config_variables is the percentage of worker threads which must remain alive (unstuck) before a node will self-terminate. If RHINO_WATCHDOG_THREADS_THRESHOLD is set to 100, it means that if any of the worker threads become stuck, the node will terminate itself. If this setting is set to 0, it means that the node will never terminate itself due to stuck worker threads. This provides a mechanism for cluster nodes which have stuck worker threads to free up those threads by terminating the JVM and restarting (assuming the nodes have been configured to restart automatically). By default, the watchdog thread will kill a node in which less than half (50) of the worker threads are still alive.

A typical Rhino installation includes the following files.

Creating a new Rhino node (by running the create-node.sh script) involves making a directory for that node. This directory contains the following files, which that node uses to store state, including configuration, logs and temporary files. The following table summarises the files for a node with id 101.

The Rhino SLEE uses the Apache log4j libraries for logging. In the default configuration, it sends logging output to both the standard error stream (the user’s console) and also the following log files in the work/log directory: