Rhino Administration and Deployment Guide

Rhino extension

public void createNamespace(String name, NamespaceOptions options)
  throws NullPointerException, InvalidArgumentException,
    NamespaceAlreadyExistsException, ManagementException;

Removing a Namespace

To remove an existing user-defined namespace, use the following rhino-console command or related MBean operation.

The default namespace cannot be removed.

All deployable units (other than the deployable unit containing the standard JAIN SLEE-defined types) must be uninstalled and all profile tables removed from a namespace before that namespace can be removed.

Console command: `removenamespace`

Command	removenamespace <name> Description Remove an existing deployment namespace
Example	$ ./rhino-console removenamespace testnamespace Namespace testnamespace removed

Command

removenamespace <name>
  Description
    Remove an existing deployment namespace

Example

$ ./rhino-console removenamespace testnamespace
Namespace testnamespace removed

MBean operation: `removeNamespace`

MBean	`Namespace Management`
Rhino extension	public void removeNamespace(String name) throws NullPointerException, UnrecognizedNamespaceException, InvalidStateException, ManagementException;

MBean

Rhino extension

public void removeNamespace(String name)
  throws NullPointerException, UnrecognizedNamespaceException,
    InvalidStateException, ManagementException;

Listing Namespaces

To list all user-defined namespaces in a SLEE, use the following rhino-console command or related MBean operation.

Console command: `listnamespaces`

Command	listnamespaces [-v] Description List all deployment namespaces. If the -v (verbose) option is given then the options that each namespace was created with is also given
Example	$ ./rhino-console listnamespaces testnamespace

Command

listnamespaces [-v]
  Description
    List all deployment namespaces.  If the -v (verbose) option is given then the
    options that each namespace was created with is also given

Example

$ ./rhino-console listnamespaces
testnamespace

MBean operation: `getNamespaces`

MBean	`Namespace Management`
Rhino extension	public String[] getNamespaces() throws ManagementException; This operation returns the names of the user-defined namespaces that have been created.

MBean

Rhino extension

public String[] getNamespaces()
  throws ManagementException;

This operation returns the names of the user-defined namespaces that have been created.

Setting the Active Namespace

Each individual authenticated client connection to Rhino is associated with a namespace.

This setting, known as the active namespace, controls which namespace is affected by management commands such as those that install deployable units or change operational states.

To change the active namespace for a client connection, use the following rhino-console command or related MBean operation.

Console command: `setactivenamespace`

Command and command-line option	Interactive mode In interactive mode, the `setActiveNamespace` command can be used to set the active namespace for future management operations. setactivenamespace <-default\|name> Description Set the active namespace Non-interactive mode In non-interactive mode, the `-n` command-line option can be used to select the namespace that the executed command is processed against.
Example	Interactive mode $ ./rhino-console Interactive Rhino Management Shell Rhino management console, enter 'help' for a list of commands [Rhino@localhost (#0)] setactivenamespace testnamespace The active namespace is now testnamespace [Rhino@localhost [testnamespace] (#1)] setactivenamespace -default The active namespace is now the default namespace [Rhino@localhost (#2)] Non-interactive mode $ ./rhino-console -n testnamespace start The active namespace is now testnamespace Starting SLEE on node(s) [101] SLEE transitioned to the Starting state on node 101

Command and command-line option

Interactive mode

In interactive mode, the setActiveNamespace command can be used to set the active namespace for future management operations.

setactivenamespace <-default|name>
  Description
    Set the active namespace

Non-interactive mode

In non-interactive mode, the -n command-line option can be used to select the namespace that the executed command is processed against.

Example

Interactive mode

$ ./rhino-console
Interactive Rhino Management Shell
Rhino management console, enter 'help' for a list of commands
[Rhino@localhost (#0)] setactivenamespace testnamespace
The active namespace is now testnamespace
[Rhino@localhost [testnamespace] (#1)] setactivenamespace -default
The active namespace is now the default namespace
[Rhino@localhost (#2)]

Non-interactive mode

$ ./rhino-console -n testnamespace start
The active namespace is now testnamespace
Starting SLEE on node(s) [101]
SLEE transitioned to the Starting state on node 101

MBean operation: `setActiveNamespace`

MBean	`Namespace Management`
Rhino extension	public void setActiveNamespace(String name) throws NoAuthenticatedSubjectException, UnrecognizedNamespaceException, ManagementException; This operation sets the active namespace for the client connection. A `null` parameter value can be used to specify that the default namespace should be made active.

MBean

Rhino extension

public void setActiveNamespace(String name)
  throws NoAuthenticatedSubjectException, UnrecognizedNamespaceException,
    ManagementException;

This operation sets the active namespace for the client connection. A null parameter value can be used to specify that the default namespace should be made active.

Getting the Active Namespace

To get the active namespace for a client connection, use the following rhino-console information and related MBean operation.

Console:

Command prompt information	The currently active namespace is reported in the command prompt within square brackets. If no namespace is reported, then the default namespace is active.
Example	$ ./rhino-console Interactive Rhino Management Shell Rhino management console, enter 'help' for a list of commands [Rhino@localhost (#0)] setactivenamespace testnamespace The active namespace is now testnamespace [Rhino@localhost [testnamespace] (#1)] setactivenamespace -default The active namespace is now the default namespace [Rhino@localhost (#2)]

Command prompt information

The currently active namespace is reported in the command prompt within square brackets.

If no namespace is reported, then the default namespace is active.

Example

$ ./rhino-console
Interactive Rhino Management Shell
Rhino management console, enter 'help' for a list of commands
[Rhino@localhost (#0)] setactivenamespace testnamespace
The active namespace is now testnamespace
[Rhino@localhost [testnamespace] (#1)] setactivenamespace -default
The active namespace is now the default namespace
[Rhino@localhost (#2)]

MBean operation: `getActiveNamespace`

MBean	`Namespace Management`
Rhino extension	public String getActiveNamespace() throws NoAuthenticatedSubjectException, ManagementException; This operation returns the name of the namespace currently active for the client connection.

MBean

Retrieving the state of nodes

Rhino extension

public String getActiveNamespace()
  throws NoAuthenticatedSubjectException, ManagementException;

This operation returns the name of the namespace currently active for the client connection.

Operational State

As well as an overview of SLEE operational states, this section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples and links to related javadocs:

Procedure rhino-console command MBean → Operation

Procedure	rhino-console command	MBean → Operation
Starting and Stopping the SLEE	setsleedesiredstate	SLEE Management → `setdesiredstate`
Retrieving the state of nodes
Retrieving the basic operational state of nodes	getsleeactualstate, getsleedesiredstate	SLEE Management → `getActualState`
Retrieving detailed information for every node in the cluster	getClusterState	Rhino Housekeeping → `getClusterState`
Terminating and rebooting nodes
Gracefully shutting nodes down and, optionally, rebooting them	shutdown	SLEE Management → `shutdown`
Forcefully terminating nodes	kill	SLEE Management → `kill`
Managing per-node desired state
Listing nodes with per-node desired state	getnodeswithpernodedesiredstate	Node Housekeeping → `getNodesWithPerNodeActivationState`
Copying per-node desired state to another node	copypernodedesiredstate	Node Housekeeping → `copyPerNodeActivationState`
Removing per-node desired state	removepernodedesiredstate	Node Housekeeping → `removePerNodeActivationState`

Starting and Stopping the SLEE

 setsleedesiredstate

SLEE Management → setdesiredstate

Retrieving the basic operational state of nodes

 getsleeactualstate, getsleedesiredstate

SLEE Management → getActualState

Retrieving detailed information for every node in the cluster

 getClusterState

Rhino Housekeeping → getClusterState

Terminating and rebooting nodes

Gracefully shutting nodes down and, optionally, rebooting them

 shutdown

SLEE Management → shutdown

Forcefully terminating nodes

 kill

SLEE Management → kill

Managing per-node desired state

Listing nodes with per-node desired state

 getnodeswithpernodedesiredstate

Node Housekeeping → getNodesWithPerNodeActivationState

Copying per-node desired state to another node

 copypernodedesiredstate

Node Housekeeping → copyPerNodeActivationState

Removing per-node desired state

 removepernodedesiredstate

Node Housekeeping → removePerNodeActivationState

About SLEE Operational States

The SLEE specification defines the operational lifecycle of a SLEE — illustrated, defined, and summarised below.

Figure 1. The SLEE lifecycle

SLEE lifecycle states

The SLEE lifecycle states are:

State	Definition
STOPPED	The SLEE has been configured and initialised, and is ready to be started. Active resource adaptor entities have been loaded and initialised, and SBBs corresponding to active services have been loaded and are ready to be instantiated. The entire event-driven subsystem, however, is idle: resource adaptor entities and the SLEE are not actively producing events, the event router is not processing work, and the SLEE is not creating SBB entities.
STARTING	Resource adaptor entities in the SLEE that have been recorded in the management database as being in the ACTIVE state are started. The SLEE still does not create SBB entities. The node automatically transitions from this state to the RUNNING state when all startup tasks are complete, or to the STOPPING state if a startup task fails.
RUNNING	Activated resource adaptor entities in the SLEE can fire events, and the SLEE creates SBB entities and delivers events to them as needed.
STOPPING	Identical to the RUNNING state, except resource adaptor entities do not create (and the SLEE does not accept) new activity objects. Existing activity objects can end (according to the resource adaptor specification). The node automatically transitions out of this state, returning to the STOPPED state, when all SLEE activities have ended. The node can transition to this state directly from the STARTING state, effective immediately, if it has no activity objects.

State

Definition

 STOPPED

The SLEE has been configured and initialised, and is ready to be started. Active resource adaptor entities have been loaded and initialised, and SBBs corresponding to active services have been loaded and are ready to be instantiated. The entire event-driven subsystem, however, is idle: resource adaptor entities and the SLEE are not actively producing events, the event router is not processing work, and the SLEE is not creating SBB entities.

 STARTING

Resource adaptor entities in the SLEE that have been recorded in the management database as being in the ACTIVE state are started. The SLEE still does not create SBB entities.

The node automatically transitions from this state to the RUNNING state when all startup tasks are complete, or to the STOPPING state if a startup task fails.

 RUNNING

Activated resource adaptor entities in the SLEE can fire events, and the SLEE creates SBB entities and delivers events to them as needed.

 STOPPING

Identical to the RUNNING state, except resource adaptor entities do not create (and the SLEE does not accept) new activity objects. Existing activity objects can end (according to the resource adaptor specification).

The node automatically transitions out of this state, returning to the STOPPED state, when all SLEE activities have ended. The node can transition to this state directly from the STARTING state, effective immediately, if it has no activity objects.

Independent SLEE states

Each namespace in each event-router node in a Rhino cluster maintains its own SLEE-lifecycle state machine, independent from other namespaces on the same or other nodes in the cluster. For example:

the default namespace on one node in a cluster might be in the RUNNING state
while a user-defined namespace on the same node is in the STOPPED state
while the default namespace on another node is in the STOPPING state
and the user-defined namespace on that node is in the RUNNING state.

The operational state of each namespace on each cluster node persists to the disk-based database.

Bootup SLEE state

After completing bootup and initialisation, a namespace on a node will enter the STOPPED state if:

the database has no persistent operational state information for that namespace on that node;
the namespace’s persistent operational state is STOPPED on that node; or
the node was started with the -x option (see Start Rhino in the Rhino Getting Started Guide).

Otherwise, the namespace will return to the same operational state that it was last in, as recorded in persistent storage.

Changing a namespace’s operational state

You can change the operational state of any namespace on any node at any time, as long as least one node in the cluster is available to perform the management operation (regardless of whether or not the node whose operational state being changed is a current cluster member). For example, you might set the operational state of the default namespace on node 103 to RUNNING before node 103 is started — then, when node 103 is started, after it completes initialising, the default namespace will enter the RUNNING state.

Changing a quorum node’s operational state

You can also change the operational state of a node which is a current member of the cluster as a quorum node… but quorum nodes make no use of operational state information stored in the database, and will not respond to operational state changes. (A node only uses operational state information if it starts as a regular event-router node.)

Starting the SLEE

To start a SLEE on one or more nodes, use the following rhino-console command or related MBean operations.

If executed without a list of nodes, all per-node desired state for the SLEE is removed and the default desired state of the SLEE is set to running (if it is not already).

Console command: `start`

Command	start [-nodes node1,node2,...] [-ifneeded] Description Start the SLEE (on the specified nodes)
Example	To start nodes 101 and 102: $ ./rhino-console start -nodes 101,102 Starting SLEE on node(s) [101,102] SLEE transitioned to the Starting state on node 101 SLEE transitioned to the Starting state on node 102

Command

start [-nodes node1,node2,...] [-ifneeded]
  Description
    Start the SLEE (on the specified nodes)

Example

To start nodes 101 and 102:

$ ./rhino-console start -nodes 101,102
Starting SLEE on node(s) [101,102]
SLEE transitioned to the Starting state on node 101
SLEE transitioned to the Starting state on node 102

MBean operation: `setPerNodeDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on specific nodes public void setPerNodeDesiredState(int[] nodeIDs, SleeDesiredState desiredState) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to set the desired state for a SLEE on a set of nodes.

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(int[] nodeIDs, SleeDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to set the desired state for a SLEE on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on nodes that do not have per-node SLEE state configured public void setDefaultDesiredState(SleeDesiredState desiredState) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to set the desired state for a SLEE on nodes that do not have a per-node desired state configured.

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node SLEE state configured

public void setDefaultDesiredState(SleeDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to set the desired state for a SLEE on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on nodes that have per-node SLEE state configured that is different from the default state public void removePerNodeDesiredState(int[] nodeIDs) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to clear the desired state for a SLEE on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean

Rhino extension

Activate or deactivate on nodes that have per-node SLEE state configured that is different from the default state

public void removePerNodeDesiredState(int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to clear the desired state for a SLEE on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `start`

MBean	`SLEE Management`
SLEE-defined	Start all nodes public void start() throws InvalidStateException, ManagementException; Rhino’s implementation of the SLEE-defined `start` operation attempts to transition all event-router nodes in the primary component from the STOPPED to the STARTING state. For this to work, at least one node must be in the STOPPED state.
Rhino extension	Start specific nodes public void start(int[] nodeIDs) throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException; Rhino provides an extension that adds an argument which lets you control which nodes to start (by specifying node IDs). For this to work, the specified nodes must be in the STOPPED state.

MBean

SLEE-defined

Start all nodes

public void start()
  throws InvalidStateException, ManagementException;

Rhino’s implementation of the SLEE-defined start operation attempts to transition all event-router nodes in the primary component from the STOPPED to the STARTING state. For this to work, at least one node must be in the STOPPED state.

Rhino extension

Start specific nodes

public void start(int[] nodeIDs)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

Rhino provides an extension that adds an argument which lets you control which nodes to start (by specifying node IDs). For this to work, the specified nodes must be in the STOPPED state.

Stopping the SLEE

To stop SLEE event-routing functions on one or more nodes, use the following rhino-console command or related MBean operations.

If executed without a list of nodes, all per-node desired state for the SLEE is removed and the default desired state of the SLEE is set to stopped (if it is not already).

Console command: `stop`

Command

stop [-nodes node1,node2,...] [-reassignto -node3,node4,...] [-ifneeded]
  Description
    Stop the SLEE (on the specified nodes (reassigning replicated activities to the
    specified nodes))

Examples

To stop nodes 101 and 102:

$ ./rhino-console stop -nodes 101,102
Stopping SLEE on node(s) [101,102]
SLEE transitioned to the Stopping state on node 101
SLEE transitioned to the Stopping state on node 102

To stop only node 101 and reassign replicated activities to node 102:

$ ./rhino-console stop -nodes 101 -reassignto 102
Stopping SLEE on node(s) [101]
SLEE transitioned to the Stopping state on node 101
Replicated activities reassigned to node(s) [102]

To stop node 101 and distribute replicated activities of each replicating resource adaptor entity to all other eligible nodes (those on which the resource adaptor entity is in the ACTIVE state and the SLEE is in the RUNNING state), specify an empty (zero-length) argument for the -reassignto option:

$ ./rhino-console stop -nodes 101 -reassignto ""
Stopping SLEE on node(s) [101]
SLEE transitioned to the Stopping state on node 101
Replicated activities reassigned to node(s) [102,103]

See also Reassigning a Resource Adaptor Entity’s Activities to Other Nodes, particularly the Requirements tab.

MBean operation: `setPerNodeDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on specific nodes public void setPerNodeDesiredState(int[] nodeIDs, SleeDesiredState desiredState) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to set the desired state for a SLEE on a set of nodes.

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(int[] nodeIDs, SleeDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to set the desired state for a SLEE on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on nodes that do not have per-node SLEE state configured public void setDefaultDesiredState(SleeDesiredState desiredState) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to set the desired state for a SLEE on nodes that do not have a per-node desired state configured.

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node SLEE state configured

public void setDefaultDesiredState(SleeDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to set the desired state for a SLEE on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean	`SLEE Management`
Rhino extension	Activate or deactivate on nodes that have per-node state configured that is different from the default state public void removePerNodeDesiredState(int[] nodeIDs) throws NullPointerException, InvalidArgumentException, SLEEManagementException; Rhino provides an extension to clear the desired state for a SLEE on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean

Rhino extension

Activate or deactivate on nodes that have per-node state configured that is different from the default state

public void removePerNodeDesiredState(int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        SLEEManagementException;

Rhino provides an extension to clear the desired state for a SLEE on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `stop`

MBean	`SLEE Management`
SLEE-defined	Stop all nodes public void stop() throws InvalidStateException, ManagementException; Rhino’s implementation of the SLEE-defined `stop` operation attempts to transition all event-router nodes in the primary component from the RUNNING to the STOPPING state. For this to work, at least one node must begin in the RUNNING state.
Rhino extensions	Stop specific nodes public void stop(int[] nodeIDs) throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException; Rhino provides an extension that adds an argument which lets you control which nodes to stop (by specifying node IDs). For this to work, specified nodes must begin in the RUNNING state. Reassign activities to other nodes public void stop(int[] stopNodeIDs, int[] reassignActivitiesToNodeIDs) throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException; Rhino also provides an extension that adds another argument, which lets you reassign ownership of replicated activities (from replicating resource adaptor entities) from the stopping nodes, distributing the activities of each resource adaptor entity equally among other event-router nodes where the resource adaptor entity is eligible to adopt them. With a smaller set of activities, the resource adaptor entities on the stopping nodes can more quickly return to the INACTIVE state (which is required for the SLEE to transition from the STOPPING to the STOPPED state). This only works for resource adaptor entities that are replicating activity state (see the description of the "Rhino-defined configuration property" on the MBean tab on Creating a Resource Adaptor Entity). See also Reassigning a Resource Adaptor Entity’s Activities to Other Nodes, in particular the Requirements tab.

MBean

SLEE-defined

Stop all nodes

public void stop()
  throws InvalidStateException, ManagementException;

Rhino’s implementation of the SLEE-defined stop operation attempts to transition all event-router nodes in the primary component from the RUNNING to the STOPPING state. For this to work, at least one node must begin in the RUNNING state.

Rhino extensions

Stop specific nodes

public void stop(int[] nodeIDs)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

Rhino provides an extension that adds an argument which lets you control which nodes to stop (by specifying node IDs). For this to work, specified nodes must begin in the RUNNING state.

Reassign activities to other nodes

public void stop(int[] stopNodeIDs, int[] reassignActivitiesToNodeIDs)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

Rhino also provides an extension that adds another argument, which lets you reassign ownership of replicated activities (from replicating resource adaptor entities) from the stopping nodes, distributing the activities of each resource adaptor entity equally among other event-router nodes where the resource adaptor entity is eligible to adopt them. With a smaller set of activities, the resource adaptor entities on the stopping nodes can more quickly return to the INACTIVE state (which is required for the SLEE to transition from the STOPPING to the STOPPED state). This only works for resource adaptor entities that are replicating activity state (see the description of the "Rhino-defined configuration property" on the MBean tab on Creating a Resource Adaptor Entity). See also Reassigning a Resource Adaptor Entity’s Activities to Other Nodes, in particular the Requirements tab.

Retrieving the State of Nodes

Rhino includes operations to retrieve:

Basic Operational State of a Node
Detailed Information for Every Node in the Cluster

Basic Operational State of a Node

Retrieving actual state
- Console command: getsleeactualstate
- MBean operation: getActualState
Retrieving desired state
Retrieving SLEE-defined state
- Console command: state
- MBean operation: getState

Retrieving actual state

To retrieve the actual operational state of a node, use the following rhino-console command or related MBean operation. For an explanation of the terms "actual state" and "desired state" see bxref:concepts-and-terminology.

Console command: `getsleeactualstate`

Command	getsleeactualstate <-all\|-nodes node1,node2,...> Description Get the actual SLEE state for the specified nodes. If -all is specified, query the state of all current event router cluster members.
Output	The `rhino-console` client displays the actual operational state of the specified node(s), or every event-router node in the primary component if `-all` is specified.
Examples	To display the actual state of only node 101: $ ./rhino-console getsleeactualstate -nodes 101 Node 101: Stopped To display the actual state of every event-router node: $ ./rhino-console getsleeactualstate -all Getting desired SLEE state for node(s) [101,102] Node 101: Stopped Node 102: Running

Command

getsleeactualstate <-all|-nodes node1,node2,...>
  Description
    Get the actual SLEE state for the specified nodes. If -all is specified, query
    the state of all current event router cluster members.

Output

The rhino-console client displays the actual operational state of the specified node(s), or every event-router node in the primary component if -all is specified.

Examples

To display the actual state of only node 101:

$ ./rhino-console getsleeactualstate -nodes 101
Node 101: Stopped

To display the actual state of every event-router node:

$ ./rhino-console getsleeactualstate -all
Getting desired SLEE state for node(s) [101,102]
Node 101: Stopped
Node 102: Running

MBean operation: `getActualState`

MBean	`SLEE Management`
Rhino extension	Return actual state of a set of nodes public SleeActualState getActualState(int[] nodeIDs) throws ManagementException;

MBean

Rhino extension

Return actual state of a set of nodes

public SleeActualState getActualState(int[] nodeIDs)
  throws ManagementException;

Retrieving desired state

To retrieve the desired operational state of a node, use the following rhino-console command or related MBean operation.

Console command: `getsleedesiredstate`

Command	getsleedesiredstate <-default\|-all\|-nodes node1,node2,...> Description Get the default or per-node desired SLEE state. If -all is specified, query the state of all current event router nodes as well as all nodes with saved per-node state.
Output	The `rhino-console` client displays the desired state of the specified node(s), or every node with configured state and every event-router node in the primary component if `-all` is specified.
Examples	To display the desired state of only node 101: $ ./rhino-console getsleedesiredstate -nodes 101 Node 101: Stopped To display the desired state of every event-router node and configured node: $ ./rhino-console getsleedesiredstate -all Node 101: Stopped Node 102: Running (default) Node 103: Running To display the default desired state that unconfigured event router nodes will inherit: $ ./rhino-console getsleedesiredstate -default Getting default SLEE state Default SLEE state is: running

Command

getsleedesiredstate <-default|-all|-nodes node1,node2,...>
  Description
    Get the default or per-node desired SLEE state. If -all is specified, query the
    state of all current event router nodes as well as all nodes with saved per-node
    state.

Output

The rhino-console client displays the desired state of the specified node(s), or every node with configured state and every event-router node in the primary component if -all is specified.

Examples

To display the desired state of only node 101:

$ ./rhino-console getsleedesiredstate -nodes 101
Node 101: Stopped

To display the desired state of every event-router node and configured node:

$ ./rhino-console getsleedesiredstate -all
Node 101: Stopped
Node 102: Running (default)
Node 103: Running

To display the default desired state that unconfigured event router nodes will inherit:

$ ./rhino-console getsleedesiredstate -default
Getting default SLEE state
Default SLEE state is: running

MBean operation: `getPerNodeDesiredState`

MBean	`SLEE Management`
Rhino extension	Return desired state of a set of nodes public SleeActualState getPerNodeDesiredState(int[] nodeIDs) throws ManagementException;

MBean

Rhino extension

Return desired state of a set of nodes

public SleeActualState getPerNodeDesiredState(int[] nodeIDs)
  throws ManagementException;

MBean operation: `getDefaultDesiredState`

MBean	`SLEE Management`
Rhino extension	Return the default desired state used by nodes that do not have a configured per-node state public SleeActualState getDefaultDesiredState() throws ManagementException;

MBean

Rhino extension

Return the default desired state used by nodes that do not have a configured per-node state

public SleeActualState getDefaultDesiredState()
  throws ManagementException;

Retrieving SLEE-defined state

To retrieve the basic operational state of a node in a form compatible with the JAIN SLEE specification, use the following rhino-console command or related MBean operation.

Console command: `state`

Command	state [-nodes node1,node2,...] Description Get the state of the SLEE (on the specified nodes)
Output	The `rhino-console` client displays the operational state of the specified node(s), or every event-router node in the primary component if none are specified.
Examples	To display the state of only node 101: $ ./rhino-console state -nodes 101 Node 101 is Stopped To display the state of every event-router node: $ ./rhino-console state Node 101 is Stopped Node 102 is Running

Command

state [-nodes node1,node2,...]
  Description
    Get the state of the SLEE (on the specified nodes)

Output

The rhino-console client displays the operational state of the specified node(s), or every event-router node in the primary component if none are specified.

Examples

To display the state of only node 101:

$ ./rhino-console state -nodes 101
Node 101 is Stopped

To display the state of every event-router node:

$ ./rhino-console state
Node 101 is Stopped
Node 102 is Running

MBean operation: `getState`

MBean

SLEE-defined

Return state of current node

public SleeState getState()
  throws ManagementException;

Rhino’s implementation of the SLEE-defined getState operation returns the SLEE defined state most closely representative of the actual state the SLEE on the node the invoking client is connected to. When using the Rhino client library with a list of hosts this will usually be the node on the first host in the list. When multiple nodes are running on the same host, the oldest node on the host will usually expose the management interface and thus be the target of this query.

Since Rhino 3.0.0 the actual state of components on each node can update asynchronously. This differs from symmetric activation state mode in earlier Rhino versions in that the value returned by getState() is not representative of the state on other cluster nodes. Users of this method who previously configured symmetric activation state mode should switch to checking the state of all nodes using the method getState(int[] nodeIDs) or one of the new getDesiredState(int[] nodeIDs) or getActualState(int[] nodeIDs) depending on the purpose of the state query. A list of event router node IDs can be obtained using RhinoHousekeepingMBean.getEventRouterNodes(). For example, to wait until the SLEE is active on all nodes:

RhinoHousekeepingMBean rhinoHousekeeping =  RhinoManagement.getRhinoHousekeepingMBean(client);
SleeManagementMBean sleeManagement =  RhinoManagement.getSleeManagementMBean(client);
boolean active = false;
while (!active) {
    SleeState[] nodeStates = sleeManagement.getActualState(rhinoHousekeeping.getEventRouterNodes());
    active = Arrays.stream(nodeStates).filter(s -> s != SleeActualState.ACTIVE).count() == 0;
}

Rhino extension

Return state of specific nodes

public SleeState[] getState(int[] nodeIDs)
  throws NullPointerException, InvalidArgumentException,
    ManagementException;

Rhino provides an extension that adds an argument which lets you control which nodes to examine (by specifying node IDs).

Detailed Information for Every Node in the Cluster

To retrieve detailed information for every node in the cluster (including quorum nodes), use the following rhino-console command or related MBean operation.

Console command: `getclusterstate`

Command	getclusterstate Description Display the current state of the Rhino Cluster
Output	For every node in the cluster, retrieves detailed information on the: node ID type (event-router or quorum) host name of the machine the node is running on time the node was started, and how long it has been alive operational state number of alarms currently raised on the node.
Example	$ ./rhino-console getclusterstate node-id active-alarms host node-type slee-state start-time up-time -------- -------------- ----------------- ------------- ----------- ------------------ ----------------- 101 0 host1.domain.com event-router Stopped 20080327 12:16:26 0days,2h,40m,3s 102 0 host2.domain.com event-router Running 20080327 12:16:30 0days,2h,39m,59s 103 0 host3.domain.com quorum n/a 20080327 14:36:25 0days,0h,20m,4s

Command

getclusterstate
  Description
    Display the current state of the Rhino Cluster

Output

For every node in the cluster, retrieves detailed information on the:

node ID
type (event-router or quorum)
host name of the machine the node is running on
time the node was started, and how long it has been alive
operational state
number of alarms currently raised on the node.

Example

$ ./rhino-console getclusterstate
node-id   active-alarms   host               node-type      slee-state   start-time          up-time
--------  --------------  -----------------  -------------  -----------  ------------------  -----------------
     101               0   host1.domain.com   event-router      Stopped   20080327 12:16:26    0days,2h,40m,3s
     102               0   host2.domain.com   event-router      Running   20080327 12:16:30   0days,2h,39m,59s
     103               0   host3.domain.com         quorum          n/a   20080327 14:36:25    0days,0h,20m,4s

MBean operation: `getClusterState`

MBean	`Rhino Housekeeping`
Rhino extension	public TabularData getClusterState() throws ManagementException; (Refer to the `javadoc` for the structure of the `TabularData` returned by this operation.)

MBean

Rhino extension

public TabularData getClusterState()
throws ManagementException;

(Refer to the javadoc for the structure of the TabularData returned by this operation.)

Terminating Nodes

To terminate cluster nodes, you can:

Shut Down Gracefully
Forcefully Terminate.

What’s the difference between "stopping", "shutting down" and "killing" a node?

You can stop functions on nodes and nodes themselves, by:

Stopping — stops event-routing functions on the node, but the node remains alive and a member of the primary component.
Shutting down — terminates the node (so that it leaves the primary component).
Killing — terminates the node regardless of its operational state. Killing a node is not recommended unless the node cannot be shut down normally (for example, the node becomes stuck in the STOPPING state for some reason).

See also Stop Rhino in the Getting Started Guide, which details using the stop-rhino.sh script (which uses the rhino-console commands described in this section) to shut down or kill nodes or clusters.

Shut Down Gracefully

To gracefully shut down one or more nodes, use the following rhino-console command or related MBean operation.

Console command: `shutdown`

Command	shutdown [-nodes node1,node2,...] [-timeout timeout] [-restart] Description Gracefully shutdown and terminate the cluster (or the specified nodes). If the SLEE is running in any namespace on any target node, event routing functions are allowed to complete before termination without affecting existing desired state. The optional timeout is specified in seconds. Optionally restart the node(s) after shutdown
Examples	To shut down the entire cluster: $ ./rhino-console shutdown Shutting down the SLEE Shutdown successful To shut down only node 102: $ ./rhino-console shutdown -nodes 102 Shutting down node(s) [102] Shutdown successful

Command

shutdown [-nodes node1,node2,...] [-timeout timeout] [-restart]
  Description
    Gracefully shutdown and terminate the cluster (or the specified nodes). If the
    SLEE is running in any namespace on any target node, event routing functions are
    allowed to complete before termination without affecting existing desired state.
    The optional timeout is specified in seconds. Optionally restart the node(s)
    after shutdown

Examples

To shut down the entire cluster:

$ ./rhino-console shutdown
Shutting down the SLEE
Shutdown successful

To shut down only node 102:

$ ./rhino-console shutdown -nodes 102
Shutting down node(s) [102]
Shutdown successful

Since Rhino 3.0.0 the shutdown console command will shut down the specified nodes regardless of the desired SLEE state. If the SLEE is running in any namespace on any target node, event routing functions are allowed to complete before termination without affecting existing desired state.

MBean operation: `shutdownCluster`

MBean	`SLEE Management`
Rhino extension	Shut down all nodes public void shutdownCluster(boolean restart) throws InvalidStateException, ManagementException; The shutdownCluster operation terminates every node in the cluster. If the restart flag is set, the nodes will be restarted to the currently configured desired state.
Rhino extension	Shut down specified nodes public void shutdown(boolean restart, long timeout) throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException; The shutdownCluster operation terminates every node in the cluster. If the restart flag is set, the nodes will be restarted to the currently configured desired state. If the timeout argument is greater than zero, any nodes that still have live activities will be shutdown anyway. This may result in call failures.

MBean

Rhino extension

Shut down all nodes

public void shutdownCluster(boolean restart)
  throws InvalidStateException, ManagementException;

The shutdownCluster operation terminates every node in the cluster. If the restart flag is set, the nodes will be restarted to the currently configured desired state.

Rhino extension

Shut down specified nodes

public void shutdown(boolean restart, long timeout)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

The shutdownCluster operation terminates every node in the cluster. If the restart flag is set, the nodes will be restarted to the currently configured desired state. If the timeout argument is greater than zero, any nodes that still have live activities will be shutdown anyway. This may result in call failures.

MBean operation: `shutdownNodes`

MBean	`SLEE Management`
Rhino extension	Shut down specific nodes public void shutdownNodes(int[] nodeIDs, boolean restart) throws InvalidStateException, ManagementException; The shutdownNodes operation terminates the specified set of nodes. If the restart flag is set, the nodes will be restarted to the currently configured desired state.
Rhino extension	Shut down specific nodes public void shutdown(int[] nodeIDs, boolean restart, long timeout) throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException; The shutdownNodes operation terminates the specified set of nodes. If the restart flag is set, the nodes will be restarted to the currently configured desired state. If the timeout argument is greater than zero, any nodes that still have live activities will be shutdown anyway. This may result in call failures.

MBean

Rhino extension

Shut down specific nodes

public void shutdownNodes(int[] nodeIDs, boolean restart)
  throws InvalidStateException, ManagementException;

The shutdownNodes operation terminates the specified set of nodes. If the restart flag is set, the nodes will be restarted to the currently configured desired state.

Rhino extension

Shut down specific nodes

public void shutdown(int[] nodeIDs, boolean restart, long timeout)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

The shutdownNodes operation terminates the specified set of nodes. If the restart flag is set, the nodes will be restarted to the currently configured desired state. If the timeout argument is greater than zero, any nodes that still have live activities will be shutdown anyway. This may result in call failures.

MBean operation: `shutdown`

MBean

SLEE-defined

Shut down all nodes

public void shutdown()
  throws InvalidStateException, ManagementException;

Rhino’s implementation of the SLEE-defined shutdown operation terminates every node in the cluster.

Rhino extension

Shut down specific nodes

public void shutdown(int[] nodeIDs)
  throws NullPointerException, InvalidArgumentException,
    InvalidStateException, ManagementException;

Rhino provides an extension that adds an argument which lets you control which nodes to shut down (by specifying node IDs).

MBean operation: `reboot`

MBean

Rhino extension

Reboot all nodes

public void reboot(SleeState[] states)
  throws InvalidArgumentException, InvalidStateException, ManagementException;

Reboots every node in the cluster to the state specified.

Rhino extension

Reboot specific nodes

public void reboot(int[] nodeIDs, SleeState[])
  throws NullPointerException, InvalidArgumentException, InvalidStateException, ManagementException;

Extension to reboot that adds an argument which lets you control which nodes to shut down (by specifying node IDs).

Event-router nodes can restart to either the RUNNING state or the STOPPED state. Quorum nodes must have a state provided but do not use this in operation.

Forcefully Terminate

To forcefully terminate a cluster node that is in any state where it can respond to management operations, use the following rhino-console command or related MBean operation.

Console command: `kill`

Command

kill -nodes node1,node2,...
  Description
    Forcefully terminate the specified nodes (forces them to become non-primary)

Example

To forcefully terminate nodes 102 and 103:

$ ./rhino-console kill -nodes 102,103
Terminating node(s) [102,103]
Termination successful

MBean operation: `kill`

MBean

Rhino operation

public void kill(int[] nodeIDs)
  throws NullPointerException, InvalidArgumentException,
    ManagementException;

Rhino’s kill operation immediately and forcefully terminates specified nodes. It requires an argument to select which nodes to terminate (by specifying node IDs).

Application state may be lost

Killing a node is not recommended — forcibly terminated nodes lose all non-replicated application state.

Activation State

This section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples, and links to related javadocs:

Procedure rhino-console command MBean → Operation

Listing all default and per-node desired states

 listdesiredstates

Node Housekeeping → getDesiredStates

Listing nodes with per-node desired state

 getnodeswithpernodedesiredstate

Node Housekeeping → getNodesWithPerNodeActivationState

Copying per-node desired state to another node

 copypernodedesiredstate

Node Housekeeping → copyPerNodeActivationState

Removing per-node desired state

 removePerNodeActivationState

Node Housekeeping → removePerNodeActivationState

It also describes the deprecated activation state modes that have been functionally replaced by default and per-node desired state.

About Activation State Modes

Rhino versions prior to 3.0.0 had two modes of operation for managing the activation state of services and resource adaptor entities: per-node and symmetric. From Rhino 3.0.0 these two modes were combined and have been superseded by default desired state which can be overridden by per-node desired state. Per-node desired state overrides default desired state if present. Default desired state is effective if no per-node desired state exists.

The actual state for all functions is always maintained on a per-node basis.

Per-node activation state

In per-node activation state mode, Rhino maintained activation state for the installed services and created resource adaptor entities in a namespace on a per-node basis. That is, the SLEE recorded separate activation state information for each individual cluster node.

The per-node activation state mode was the default mode in a newly installed Rhino cluster.

Symmetric activation state

In the symmetric activation state mode, Rhino maintained a single cluster-wide activation state view for each installed service and created resource adaptor entity. So, for example, if a service was activated, then it was simultaneously activated on every cluster node. If a new node joined the cluster, then the services and resource adaptor entities on that node each entered the same operational state as for existing cluster nodes.

Default and per-node desired state and actual state

In Rhino 3.0.0, a default activation state for the SLEE, an installed service, or a created resource adaptor entity is configured for all nodes in the cluster with optional overrides configured on a per-node basis. The effective desired state for a node is the per-node state, or the default state if no per-node state exists for a given function. If it is desired to manage the state of a cluster in the way previously served by symmetric activation state mode, the default state should be used and per-node state left unconfigured. Commands for managing per-node desired state can be found under the topic Per-Node Desired State.

In operation, Rhino nodes have an actual state that is the current operational state. The actual state follows the desired state with a per-node convergence subsystem managing transitions between actual states as the lifecycle rules of system functions allow.

These terms are defined under Declarative Configuration Concepts and Terminology.

Listing All Desired States

To obtain a report detailing all the default and per-node desired states for the SLEE, services, and resource adaptor entities, use the following rhino-console command or related MBean operation.

Console command: `listdesiredstates`

Command

listdesiredstates [-o filename]
  Description
    List all default and per-node desired states for the SLEE, services, and
    resource adaptor entities.  The -o option will output the raw json-formatted
    report to the specified file instead of a human-readable report being output to
    the console.

Examples

$ ./rhino-console listdesiredstates
SLEE desired state:
  Default desired state: running
  Per-node desired states:
    node 103: stopped

Service desired states:
  Service: ServiceID[name=SIS-IN Test Service Composition Selector Service,vendor=OpenCloud,version=0.3]
    Default desired state: active
    Per-node desired states:
      node 103: inactive

  Service: ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.3]
    Default desired state: active
    Per-node desired states:
      node 103: inactive

  Service: ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.3]
    Default desired state: active
    Per-node desired states:
      node 103: inactive

  Service: ServiceID[name=Call Duration Logging Service,vendor=OpenCloud,version=0.3]
    Default desired state: active
    Per-node desired states:
      node 103: inactive

  Service: ServiceID[name=VPN Service,vendor=OpenCloud,version=0.3]
    Default desired state: active
    Per-node desired states:
      node 103: inactive

Resource adaptor entity desired states:
  Resource adaptor entity: insis-ptc-1a
    Default desired state: active

  Resource adaptor entity: insis-ptc-1b
    Default desired state: active

  Resource adaptor entity: insis-ptc-external
    Default desired state: active

To save the report to a file in JSON format:

$ ./rhino-console listdesiredstates -o desired-states.json
Output written to file: desired-states.json

MBean operation: `getDesiredStates`

MBean

Listing nodes with per-node desired state

Rhino operation

public String getDesiredStates()
    throws ManagementException;

This operation returns a JSON-formatted string that reports the default desired state and any per-node desired state, where it exists, for the SLEE and each service and resource adaptor entity.

Per-Node Desired State

This section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples, and links to related javadocs.

Procedure rhino-console command MBean → Operation

 getnodeswithpernodedesiredstate

Node Housekeeping → getNodesWithPerNodeActivationState

Copying per-node desired state to another node

 copypernodedesiredstate

Node Housekeeping → copyPerNodeActivationState

Removing per-node desired state

 removepernodedesiredstate

Node Housekeeping → removePerNodeActivationState

Listing Nodes with Per-Node Desired State

To get a list of nodes with per-node desired state, use the following rhino-console command or related MBean operation.

Console command: `getnodeswithpernodedesiredstate`

Command

getnodeswithpernodedesiredstate
  Description
    Get the set of nodes for which per-node desired state exists

Example

$ ./rhino-console getnodeswithpernodedesiredstate
Nodes with per-node desired state: [101,102,103]

MBean operation: `getNodesWithPerNodeActivationState`

MBean

Rhino operation

public int[] getNodesWithPerNodeActivationState()
    throws ManagementException;

This operation returns an array, listing the cluster node IDs for nodes that have per-node desired state recorded in the database).

The term activation state is used in this method name due to functional equivalence to per-node activation state configuration in versions of Rhino prior to 3.0.0.

Copying Per-Node Desired State to Another Node

To copy per-node desired state from one node to another, use the following rhino-console command or related MBean operation. This replaces any configured desired state for the node and triggers state convergence to update the actual state for the SLEE and all Services and Resource Adaptor Entities. Copying the state from a node that does not have per-node desired state configured will remove the state configuration for the target node. When a node has no per-node desired state configured it uses the default desired state.

Console command: `copypernodedesiredstate`

Command

copypernodedesiredstate <from-node-id> <to-node-id>
  Description
    Copy per-node desired state from one node to another

Example

To copy the per-node desired state from node 101 to node 102:

$ ./rhino-console copypernodedesiredstate 101 102
Per-node desired state copied from 101 to 102

MBean operation: `copyPerNodeActivationState`

MBean

Rhino operation

public boolean copyPerNodeActivationState(int targetNodeID)
    throws UnsupportedOperationException, InvalidArgumentException,
           InvalidStateException, ManagementException;

This operation:

copies the per-node desired state recorded in the database for the node for which the Node Housekeeping MBean was created (see Finding Housekeeping MBeans) to the given target node
returns the value true if it found and copied per-node desired state, or false if the two nodes had identical per-node desired state.

The term activation state is used in this method name due to functional equivalence to per-node activation state configuration in versions of Rhino prior to 3.0.0.

The start-rhino.sh command with the Production version of Rhino also includes an option (-c) to copy per-node desired state from another node to the booting node as it initialises. (See Start Rhino in the Getting Started Guide.)

Removing Per-Node Desired State

To remove per-node desired state, use the following rhino-console command or related MBean operation. This removes any configured desired state for the node and triggers state convergence to update the actual state for the SLEE and all Services and Resource Adaptor Entities. When a node has no per-node desired state configured it uses the default desired state.

Console command: `removepernodedesiredstate`

Command

removepernodedesiredstate <-all|-nodes node1,node2,...>
  Description
    Removes all per-node desired state from either all nodes (with -all), or
    specific nodes (with -nodes). This can remove per-node desired state from
    offline nodes.

Example

To remove per-node desired state from node 103:

$ ./rhino-console removepernodedesiredstate 103
Per-node desired state removed from 103

MBean operation: `removePerNodeActivationState`

MBean

Rhino operation

public boolean removePerNodeActivationState()
    throws UnsupportedOperationException, InvalidStateException,
           ManagementException;

This operation:

removes the per-node desired state recorded in the database for node for which this Node Housekeeping MBean was created (see Finding Housekeeping MBeans)
returns the value true if it found and removed per-node desired state, or false if it found no per-node desired state to remove.

The start-rhino.sh command with the Production version of Rhino also includes an option (-d) to remove per-node desired state from the booting node as it initialises. (See Start Rhino in the Getting Started Guide.)

Startup and Shutdown Priority

Startup and shutdown priorities should be set when resource adaptors and services need to be activated or deactivated in a particular order when the SLEE is started or stopped. For example, the resource adaptors responsible for writing Call Detail Records often need to be deactivated last.

Valid priorities are between -128 and 127. Startup and shutdown occur from highest to lowest priority.

Console commands

Console command: `getraentitystartingpriority`

Command

getraentitystartingpriority <entity-name>
  Description
    Get the starting priority for a resource adaptor entity

Examples

./rhino-console getraentitystartingpriority sipra
Resource adaptor entity sipra activation priority is currently 0

Console command: `getraentitystoppingpriority`

Command

getraentitystoppingpriority <entity-name>
  Description
    Get the stopping priority for a resource adaptor entity

Examples

./rhino-console getraentitystoppingpriority sipra
Resource adaptor entity sipra deactivation priority is currently 0

Console command: `getservicestartingpriority`

Command

getservicestartingpriority <service-id>
  Description
    Get the starting priority for a service

Examples

./rhino-console getservicestartingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] activation priority is currently 0

Console command: `getservicestoppingpriority`

Command

getservicestoppingpriority <service-id>
  Description
    Get the stopping priority for a service

Examples

./rhino-console getservicestoppingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] deactivation priority is currently 0

Console command: `setraentitystartingpriority`

Command

setraentitystartingpriority <entity-name> <priority>
  Description
    Set the starting priority for a resource adaptor entity.  The priority must be
    between -128 and 127 and higher priority values have precedence over lower
    priority values

Examples

./rhino-console setraentitystartingpriority sipra 127
Resource adaptor entity sipra activation priority set to 127

./rhino-console setraentitystartingpriority sipra -128
Resource adaptor entity sipra activation priority set to -128

Console command: `setraentitystoppingpriority`

Command

setraentitystoppingpriority <entity-name> <priority>
  Description
    Set the stopping priority for a resource adaptor entity.  The priority must be
    between -128 and 127 and higher priority values have precedence over lower
    priority values

Examples

./rhino-console setraentitystoppingpriority sipra 127
Resource adaptor entity sipra deactivation priority set to 127

./rhino-console setraentitystoppingpriority sipra -128
Resource adaptor entity sipra deactivation priority set to -128

Console command: `setservicestartingpriority`

Command

setservicestartingpriority <service-id> <priority>
  Description
    Set the starting priority for a service.  The priority must be between -128 and
    127 and higher priority values have precedence over lower priority values

Examples

./rhino-console setservicestartingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1 127
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] activation priority set to 127

./rhino-console setservicestartingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1 -128
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] activation priority set to -128

Console command: `setservicestoppingpriority`

Command

setservicestoppingpriority <service-id> <priority>
  Description
    Set the stopping priority for a service.  The priority must be between -128 and
    127 and higher priority values have precedence over lower priority values

Examples

./rhino-console setservicestoppingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1 127
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] deactivation priority set to 127

./rhino-console setservicestoppingpriority name=SIP\ Presence\ Service,vendor=OpenCloud,version=1.1 -128
Service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] deactivation priority set to -128

MBean operations

Services

MBean

Rhino extensions

getStartingPriority

byte getStartingPriority(ServiceID service)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

getStartingPriorities

Byte[] getStartingPriorities(ServiceID[] services)
    throws NullPointerException, ManagementException;

getStoppingPriority

byte getStoppingPriority(ServiceID service)
   throws NullPointerException, UnrecognizedServiceException, ManagementException;

getStoppingPriorities

Byte[] getStoppingPriorities(ServiceID[] services)
    throws NullPointerException, ManagementException;

setStartingPriority

void setStartingPriority(ServiceID service, byte priority)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

setStoppingPriority

void setStoppingPriority(ServiceID service, byte priority)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

Resource Adaptors

MBean

Installing deployable units

Rhino extensions

getStartingPriority

byte getStartingPriority(String entityName)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException, ManagementException;

getStartingPriorities

Byte[] getStartingPriorities(String[] entityNames)
    throws NullPointerException, ManagementException;

getStoppingPriority

byte getStoppingPriority(String entityName)
   throws NullPointerException, UnrecognizedResourceAdaptorEntityException, ManagementException;

getStoppingPriorities

Byte[] getStoppingPriorities(String[] entityNames)
    throws NullPointerException, ManagementException;

setStartingPriority

void setStartingPriority(String entityName, byte priority)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException, ManagementException;

setStoppingPriority

void setStoppingPriority(String entityName, byte priority)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException, ManagementException;

Deployable Units

As well as an overview of deployable units, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean → Operation

 install
installlocaldu

DeploymentMBean → install

Uninstalling deployable units

 uninstall

DeploymentMBean → uninstall

Listing installed deployable units

 listdeployableunits

DeploymentMBean → getDeployableUnits

About Deployable Units

Below are a definition, preconditions for installing and uninstalling, and an example of a deployable unit.

What is a deployable unit?

A deployable unit is a jar file that can be installed in the SLEE. It contains:

a deployment descriptor
constituent jar files, with Java class files and deployment descriptors for components such as:
- SBBs
- events
- profile specifications
- resource adaptor types
- resource adaptors
- libraries
XML files for services.

The JAIN SLEE 1.1 specification defines the structure of a deployable unit.

Installing and uninstalling deployable units

You must install and uninstall deployable units in a particular order, according to the dependencies of the SLEE components they contain. You cannot install a deployable unit unless either it contains all of its dependencies, or they are already installed. For example, if your deployable unit contains an SBB which depends on a library jar, the library jar must either already be installed in the SLEE, or be included in that same deployable unit jar.

Pre-conditions

A deployable unit cannot be installed if any of the following is true:

A deployable unit with the same URL has already been installed in the SLEE.
The deployable unit contains a component with the same name, vendor and version as a component of the same type that is already installed in the SLEE.
The deployable unit contains a component that references other components that are not yet installed in the SLEE and are not included in the deployable unit jar. (For example, an SBB component may reference event-type components and profile-specification components that are not included or pre-installed.)

A deployable unit cannot be uninstalled if either of the following is true:

There are any dependencies on any of its components from components in other installed deployable units. For example, if a deployable unit contains an SBB jar that depends on a profile-specification jar contained in a second deployable unit, the deployable unit containing the profile-specification jar cannot be uninstalled while the deployable unit containing the SBB jar remains installed.
There are "instances" of components contained in the deployable unit. For example, a deployable unit containing a resource adaptor cannot be uninstalled if the SLEE includes resource adaptor entities of that resource adaptor.

Deployable unit example

The following example illustrates the deployment descriptor for a deployable unit jar file:

<deployable-unit>
<description> ... </description>
...
<jar> SomeProfileSpec.jar </jar>
<jar> BarAddressProfileSpec.jar </jar>
<jar> SomeCustomEvent.jar </jar>
<jar> FooSBB.jar </jar>
<jar> BarSBB.jar </jar>
...
<service-xml> FooService.xml </service-xml>
...
</deployable-unit>

The content of the deployable unit jar file is as follows:

META-INF/deployable-unit.xml
META-INF/MANIFEST.MF
...
SomeProfileSpec.jar
BarAddressProfileSpec.jar
SomeCustomEvent.jar
FooSBB.jar
BarSBB.jar
FooService.xml
...

Installing Deployable Units

To install a deployable unit, use the following rhino-console command or related MBean operation.

Console commands: `install`, `installlocaldu`

Commands

Installing from a URL

install <url> [-type <type>] [-installlevel <level>]
  Description
    Install a deployable unit jar or other artifact.  To install something other
    than a deployable unit, the -type option must be specified.  The -installlevel
    option controls to what degree the deployable artifact is installed

Installing from a local file

installlocaldu <file url> [-type <type>] [-installlevel <level>] [-url url]
  Description
    Install a deployable unit or other artifact. This command will attempt to
    forward the file content (by reading the file) to rhino if the management client
    is on a different host.  To install something other than a deployable unit, the
    -type option must be specified.  The -installlevel option controls to what
    degree the deployable artifact is installed.  The -url option allows the
    deployment unit to be installed with an alternative URL identifier

Examples

To install a deployable unit from a given URL:

$ ./rhino-console install file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar
installed: DeployableUnitID[url=file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar]

To install a deployable unit from the local file system of the management client:

$ ./rhino-console installlocaldu file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar
installed: DeployableUnitID[url=file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar]

MBean operation: `install`

MBean

SLEE-defined

Install a deployable unit from a given URL

public DeployableUnitID install(String url)
  throws NullPointerException, MalformedURLException,
    AlreadyDeployedException, DeploymentException,
      ManagementException;

Installs the given deployable unit jar file into the SLEE. The given URL must be resolvable from the Rhino node.

Rhino extension

Install a deployable unit from a given byte array

public DeployableUnitID install(String url, byte[] content)
  throws NullPointerException, MalformedURLException,
      AlreadyDeployedException, DeploymentException,
          ManagementException;

Installs the given deployable unit jar file into the SLEE. The caller passes the actual file contents of the deployable unit in a byte array as a parameter to this method. The SLEE then installs the deployable unit as if it were from the URL.

Uninstalling Deployable Units

To uninstall a deployable unit, use the following rhino-console command or related MBean operation.

A deployable unit cannot be uninstalled if it contains any components that any other deployable unit installed in the SLEE depends on.

Console command: `uninstall`

Command

uninstall <url>
  Description
    Uninstall a deployable unit jar

Examples

To uninstall a deployable unit which was installed with the given URL:

$ ./rhino-console uninstall file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar
uninstalled: DeployableUnitID[url=file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar]

Console command: `cascadeuninstall`

Command

cascadeuninstall <type> <url|component-id> [-force] [-s]
  Description
    Cascade uninstall a deployable unit or copied component. The optional -force
    argument prevents the command from prompting for confirmation before the
    uninstall occurs. The -s argument removes the shadow from a shadowed component
    and is not valid for deployable units

Examples

To uninstall a deployable unit which was installed with the given URL and all deployable units that depend on this:

  $ ./rhino-console cascadeuninstall du file:du/ocsip-ra-2.3.1.17.du.jar
Cascade removal of deployable unit file:du/ocsip-ra-2.3.1.17.du.jar requires the following operations to be performed:
  Deployable unit file:jars/sip-registrar-service.jar will be uninstalled
    SBB with SbbID[name=RegistrarSbb,vendor=OpenCloud,version=1.8] will be uninstalled
    Service with ServiceID[name=SIP Registrar Service,vendor=OpenCloud,version=1.8] will be uninstalled
      This service will first be deactivated
  Deployable unit file:jars/sip-presence-service.jar will be uninstalled
    SBB with SbbID[name=EventStateCompositorSbb,vendor=OpenCloud,version=1.0] will be uninstalled
    SBB with SbbID[name=NotifySbb,vendor=OpenCloud,version=1.1] will be uninstalled
    SBB with SbbID[name=PublishSbb,vendor=OpenCloud,version=1.0] will be uninstalled
    Service with ServiceID[name=SIP Notification Service,vendor=OpenCloud,version=1.1] will be uninstalled
      This service will first be deactivated
    Service with ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1] will be uninstalled
      This service will first be deactivated
    Service with ServiceID[name=SIP Publish Service,vendor=OpenCloud,version=1.0] will be uninstalled
      This service will first be deactivated
  Deployable unit file:jars/sip-proxy-service.jar will be uninstalled
    SBB with SbbID[name=ProxySbb,vendor=OpenCloud,version=1.8] will be uninstalled
    Service with ServiceID[name=SIP Proxy Service,vendor=OpenCloud,version=1.8] will be uninstalled
      This service will first be deactivated
  Deployable unit file:du/ocsip-ra-2.3.1.17.du.jar will be uninstalled
    Resource adaptor with ResourceAdaptorID[name=OCSIP,vendor=OpenCloud,version=2.3.1] will be uninstalled
      Resource adaptor entity sipra will be removed
        This resource adaptor entity will first be deactivated
        Link name OCSIP bound to this resource adaptor entity will be removed

Continue? (y/n): y
Deactivating service ServiceID[name=SIP Registrar Service,vendor=OpenCloud,version=1.8]
Deactivating service ServiceID[name=SIP Notification Service,vendor=OpenCloud,version=1.1]
Deactivating service ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1]
Deactivating service ServiceID[name=SIP Publish Service,vendor=OpenCloud,version=1.0]
Deactivating service ServiceID[name=SIP Proxy Service,vendor=OpenCloud,version=1.8]
All necessary services are inactive
Deactivating resource adaptor entity sipra
All necessary resource adaptor entities are inactive
Uninstalling deployable unit file:jars/sip-registrar-service.jar
Uninstalling deployable unit file:jars/sip-presence-service.jar
Uninstalling deployable unit file:jars/sip-proxy-service.jar
Unbinding resource adaptor entity link name OCSIP
Removing resource adaptor entity sipra
Uninstalling deployable unit file:du/ocsip-ra-2.3.1.17.du.jar

Utility: cascade-uninstall

cascade-uninstall

Uninstalls a deployable unit along with everything that depends on it

MBean operation: `uninstall`

MBean

SLEE-defined

public void uninstall(DeployableUnitID id)
  throws NullPointerException, UnrecognizedDeployableUnitException,
         DependencyException, InvalidStateException, ManagementException;

Uninstalls the given deployable unit jar file (along with all the components it contains) out of the SLEE.

Listing Deployable Units

To list the installed deployable units, use the following rhino-console command or related MBean operation.

Console command: `listdeployableunits`

Command

listdeployableunits
  Description
    List the current installed deployable units

Example

To list the currently installed deployable units:

$ ./rhino-console listdeployableunits
DeployableUnitID[url=file:/home/rhino/rhino/examples/sip-examples-2.0/lib/jsip-library-du-1.2.jar]
DeployableUnitID[url=file:/home/rhino/rhino/lib/javax-slee-standard-types.jar]

MBean operation: `getDeployableUnits`

MBean

Listing all available services

SLEE-defined

  public DeployableUnitID[] getDeployableUnits()
    throws ManagementException;

Returns the set of deployable unit identifiers that identify all the deployable units installed in the SLEE.

Services

As well as an overview of SLEE services, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean → Operation(s)

 listservices

Deployment → getServices

Retrieving a Service’s State

 getserviceactualstate, getservicedesiredstate

Service Management → getServiceState

Listing services in a specified state

 listservicesbystate

Service Management → getServices

Activating services

 activateservice

Service Management → activate

Dectivating services

 deactivateservice

Service Management → deactivate

Upgrading (activating & dectivating) services

 deactivateandactivateservice +

Service Management → deactivateAndActivate

Getting link bindings required by a service

 listserviceralinks

Deployment → getServices

 listsbbs

Deployment → getSbbs

Deployment → getDescriptors

Configuring service metrics recording status

 getservicemetricsrecordingenabled

ServiceManagementMBean → getServiceMetricsRecordingEnabled

 setservicemetricsrecordingenabled

ServiceManagementMBean → setServiceMetricsRecordingEnabled

Configuring service replication

 getservicereplicationselectors

ServiceManagementMBean → getReplicationSelectors

 setservicereplicationselectors

ServiceManagementMBean → setReplicationSelectors

About Services

The SLEE specification presents the operational lifecycle of a SLEE service — illustrated, defined and summarised below.

What are SLEE services?

Services are SLEE components that provide the application logic to act on input from resource adaptors.

Service lifecycle

Service lifecycle states

State

Definition

 INACTIVE

The service has been installed successfully and is ready to be activated, but not yet running. The SLEE will not create SBB entities of the service’s root SBB, to process events.

 ACTIVE

The service is running. The SLEE will create SBB entities, of the service’s root SBB, to process initial events. The SLEE will also deliver events to SBB entities of the service’s SBBs, as appropriate.

 STOPPING

The service is deactivating. Existing SBB entities of the service continue running and may complete their processing. But the SLEE will not create new SBB entities of the service’s root SBB, for new activities.

When the SLEE can reclaim all of a service’s SBB entities, it transitions out of the STOPPING state and returns to the INACTIVE state.

Independent operational states

As explained in About SLEE Operational States, each event-router node in a Rhino cluster maintains its own lifecycle state machine, independent of other nodes in the cluster. This is also true for each service: one service might be INACTIVE on one node in a cluster, ACTIVE on another, and STOPPING on a third. The operational state of a service on each cluster node also persists to the disk-based database.

A service will enter the INACTIVE state, after node bootup and initialisation completes, if the database’s persistent operational state information for that service is missing, or is set to INACTIVE or STOPPING.

And, like node operational states, you can change the operational state of a service at any time, as long as least one node in the cluster is available to perform the management operation (regardless of whether or not the node whose operational state being changed is a current cluster member). For example, you might activate a service on node 103 before node 103 is booted — then, when node 103 boots, and after it completes initialisation, that service will transition to the ACTIVE state.

Configuring services

An administrator can configure a service before deployment by modifying its service-jar.xml deployment descriptor (in its deployable unit). This includes specifying:

the address profile table to use when a subscribed address selects initial events for the service’s root SBB
the default event-router priority for the SLEE to give to root SBB entities of the service when processing initial events.

Individual SBBs used in a service can also have configurable properties or environment entries. Values for these environment entries are defined in the sbb-jar.xml deployment descriptor included in the SBB’s component jar. Administrators can set or adjust the values for each environment entry before the SBB is installed in the SLEE.

The SLEE only reads the configurable properties defined for a service or SBB deployment descriptor at deployment time. If you need to change the value of any of these properties, you’ll need to:

uninstall the related component (service or SBB whose properties you want to configure) from the SLEE
change the properties
reinstall the component
uninstall and reinstall other components (as needed) to satisfy dependency requirements enforced by the SLEE.

Retrieving a Service’s State

Retrieving actual state
- Console command: getserviceactualstate
- MBean operation: getActualState
Retrieving desired state
Retrieving SLEE-defined state
- Console command: getservicestate
- MBean operation: getState

Retrieving actual state

To retrieve the actual operational state of a Service, use the following rhino-console command or related MBean operation. For an explanation of the terms "actual state" and "desired state" see bxref:concepts-and-terminology.

Console command: `getserviceactualstate`

Command

getserviceactualstate <service-id> <-all|-nodes node1,node2,...>
  Description
    Get the actual service state for the specified nodes. If -all is specified,
    query the state of all current event router cluster members.

Output

The rhino-console client displays the actual operational state of the specified node(s), or every event-router node in the primary component if -all is specified.

Examples

To display the actual state of the service with the ServiceID name=SimpleService1,vendor=Open Cloud,version=1.0 only node 101:

$ ./rhino-console getserviceactualstate name=SimpleService1,vendor=Open Cloud,version=1.0 -nodes 101
Getting actual service state for node(s) [101]
Node 101: Stopped

To display the actual state of the service name=SimpleService1,vendor=Open Cloud,version=1.0 on every event-router node:

$ ./rhino-console getserviceactualstate name=SimpleService1,vendor=Open Cloud,version=1.0 -all
Getting actual service state for node(s) [101,102]
Node 101: Stopped
Node 102: Running

MBean operation: `getActualState`

MBean

Rhino extension

Return actual state of a set of nodes

public ServiceActualState getActualState(ServiceID service ID, int[] nodeIDs)
  throws ManagementException;

Retrieving desired state

To retrieve the desired operational state of a Service, use the following rhino-console command or related MBean operation.

Console command: `getservicedesiredstate`

Command

getservicedesiredstate <service-id> <-default|-all|-nodes node1,node2,...>
  Description
    Get the default or per-node desired service state. If -all is specified, query
    the state of all current event router nodes as well as all nodes with saved
    per-node state.

Output

The rhino-console client displays the desired state of the specified node(s), or every node with configured state and every event-router node in the primary component if -all is specified.

Examples

To display the desired state of only node 101:

$ ./rhino-console getservicedesiredstate -nodes 101
Node 101: Stopped

To display the desired state of the service name=SimpleService1,vendor=Open Cloud,version=1.0 every event-router node and configured node:

$ ./rhino-console getservicedesiredstate -all
Node 101: Stopped
Node 102: Running (default)
Node 103: Running

To display the default desired state that unconfigured event router nodes will inherit:

$ ./rhino-console getservicedesiredstate -default
Getting default service state
Default service state is: running

MBean operation: `getPerNodeDesiredState`

MBean

Rhino extension

Return desired state of a set of nodes

public ServiceDesiredState getPerNodeDesiredState(ServiceID service ID, int[] nodeIDs)
  throws ManagementException;

MBean operation: `getDefaultDesiredState`

MBean

Rhino extension

Return the default desired state used by nodes that do not have a configured per-node state

public ServiceDesiredState getDefaultDesiredState()
  throws ManagementException;

Retrieving SLEE-defined state

To retrieve the operational state of a service in a form compatible with the JAIN SLEE specification, use the following rhino-console command or related MBean operation.

Console command: `getservicestate`

Command

getservicestate <service-id> [-nodes node1,node2,...]
  Description
    Get the state of a service (on the specified nodes)

Output

The rhino-console client displays the operational state of the specified node(s), or every event-router node in the primary component if none are specified.

Examples

To display the state of the service with the ServiceID name=SimpleService1,vendor=Open Cloud,version=1.0 on every event-router node:

$ ./rhino-console getservicestate name=SimpleService1,vendor=Open Cloud,version=1.0
Service is Inactive on node 101
Service is Active on node 102

To display the state of the service on only node 101:

$ ./rhino-console getservicestate name=SimpleService1,vendor=Open Cloud,version=1.0 -nodes 101
Service is Inactive on node 101

MBean operation: `getState`

MBean

SLEE-defined

Return state of service on current node

public ServiceState getState(ServiceID id)
    throws NullPointerException,
    UnrecognizedServiceException,
    ManagementException;

Rhino’s implementation of the SLEE-defined getState operation returns the SLEE-defined state most closely representative of the actual state of a service on the node the invoking client is connected to. When using the Rhino client library with a list of hosts this will usually be the node on the first host in the list. When multiple nodes are running on the same host, the oldest node on the host will usually expose the management interface and thus be the target of this query.

Since Rhino 3.0.0 the actual state of components on each node can update asynchronously. This differs from symmetric activation state mode in earlier Rhino versions in that the value returned by getState() is not representative of the state on other cluster nodes. Users of this method who previously configured symmetric activation state mode should switch to checking the state of all nodes using the method getState(int[] nodeIDs) or one of the new getDesiredState(int[] nodeIDs) or getActualState(int[] nodeIDs) depending on the purpose of the state query. A list of event router node IDs can be obtained using RhinoHousekeepingMBean.getEventRouterNodes(). For example, to verify that a service is configured to be active on all nodes:

RhinoHousekeepingMBean rhinoHousekeeping =  RhinoManagement.getRhinoHousekeepingMBean(client);
ServiceManagementMBean serviceManagement =  RhinoManagement.getServiceManagementMBean(client);
ServiceState[] nodeStates = serviceManagement.getDesiredState(serviceID, rhinoHousekeeping.getEventRouterNodes());
boolean active = Arrays.stream(nodeStates).filter(s -> s != ServiceDesiredState.ACTIVE).count() == 0;

Rhino extension

Return state of service on specified node(s)

public ServiceState[] getState(ServiceID id, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
    UnrecognizedServiceException, ManagementException;

Rhino provides an extension that adds an argument which lets you control the nodes on which to return the state of the service (by specifying node IDs).

Listing Services

You can list:

All Available Services
Services by State

All Available Services

To list all available services installed in the SLEE, use the following rhino-console command or related MBean operation.

Console command: `listservices`

Command

listservices
  Description
    List the current installed services

Example

$ ./rhino-console listservices
ServiceID[name=SIP AC Location Service,vendor=OpenCloud,version=1.7]
ServiceID[name=SIP Proxy Service,vendor=OpenCloud,version=1.8]
ServiceID[name=SIP Registrar Service,vendor=OpenCloud,version=1.8]

MBean operation: `getServices`

MBean

SLEE-defined

public ServiceID[] getServices()
  throws ManagementException;

This operation returns an array of service component identifiers, identifying the services installed in the SLEE.

Services by State

To list the services in a particular operational state, use the following rhino-console command or related MBean operation.

Console command: `listservicesbystate`

Command

listservicesbystate <state> [-node node]
  Description
    List the services that are in the specified state (on the specified node)

Output

The operational state of a service is node-specific. If the -node argument is not provided, this command returns the services in the given operational state on the node that rhino-console is connected to for management. (Otherwise, the command returns the services in the given operational state on the specified node.)

Example

To list the services in the ACTIVE state on node 102:

$ ./rhino-console listservicesbystate Active -node 102
Services in Active state on node 102:
  ServiceID[name=SIP Proxy Service,vendor=OpenCloud,version=1.8]
  ServiceID[name=SIP Registrar Service,vendor=OpenCloud,version=1.8]

MBean operation: `getServices`

MBean

SLEE-defined

Get services on all nodes

public ServiceID[] getServices(ServiceState state)
  throws NullPointerException, ManagementException;

Rhino’s implementation of the SLEE-defined getServices operation returns an array identifying all the services in the requested state on the node where you invoke the operation.

Rhino extension

Get services on specific nodes

public ServiceID[] getServices(ServiceState state, int nodeID)
  throws NullPointerException, InvalidArgumentException,
    ManagementException;

Rhino provides an extension that adds an argument that lets you control the nodes on which to list services in a particular state (by specifying node IDs).

Activating and Deactivating Services

Rhino provides procedures for:

Activating Services
Deactivating Services
Upgrading (Activating & Deactivating) Services

Activating Services

To activate one or more services, use the following rhino-console command or related MBean operations.

If executed without a list of nodes, all per-node desired state for the service is removed and the default desired state of the service is set to active (if it is not already).

Console command: `activateservice`

Command

activateservice <service-id>* [-nodes node1,node2,...] [-ifneeded]
  Description
    Activate a service (on the specified nodes)

Example

To activate the Call Barring and Call Forwarding services on nodes 101 and 102:

$ ./rhino-console activateservice \
  "name=Call Barring Service,vendor=OpenCloud,version=0.2" \
  "name=Call Forwarding Service,vendor=OpenCloud,version=0.2" \
  -nodes 101,102
Activating services [ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2],
  ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.2]] on node(s) [101,102]
Services transitioned to the Active state on node 101
Services transitioned to the Active state on node 102

MBean operation: `setPerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(ServiceID id, int[] nodeIDs, ServiceDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to set the desired state for a service on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node state configured for the specified service

public void setDefaultDesiredState(ServiceID id, ServiceDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to set the desired state for a service on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that have per-node service state configured that is different from the default state

public void removePerNodeDesiredState(ServiceID id, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to clear the desired state for a service on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `activate`

MBean

SLEE-defined

Activate on all nodes

public void activate(ServiceID id)
    throws NullPointerException, UnrecognizedServiceException,
        InvalidStateException, InvalidLinkNameBindingStateException,
        ManagementException;

public void activate(ServiceID[] ids)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, InvalidStateException,
        InvalidLinkNameBindingStateException, ManagementException;

Rhino’s implementation of the SLEE-defined activate operation will attempt to activate particular services on all current event-router nodes in the primary component. For this to work, the specified services must be in the INACTIVE state on at least one node.

Rhino extension

Activate on specific nodes

public void activate(ServiceID id, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, InvalidStateException,
        ManagementException;

public void activate(ServiceID[] ids, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, InvalidStateException,
        ManagementException;

Rhino provides an extension that adds an argument to let you control the nodes on which to activate the specified services (by specifying node IDs). For this to work, the specified services must be in the INACTIVE state on the specified nodes.

A service may require resource adaptor entity link names to be bound to appropriate resource adaptor entities before it can be activated. (See Getting Link Bindings Required by a Service and Managing Resource Adaptor Entity Link Bindings.)

Deactivating Services

To deactivate one or more services on one or more nodes, use the following rhino-console command or related MBean operations.

If executed without a list of nodes, all per-node desired state for the service is removed and the default desired state of the service is set to inactive (if it is not already).

Console command: deactivateservice
MBean operation: setPerNodeDesiredState
MBean operation: setDefaultDesiredState
MBean operation: removePerNodeDesiredState
MBean operation: deactivate
Console command: waittilserviceisinactive

Console command: `deactivateservice`

Command

deactivateservice <service-id>* [-nodes node1,node2,...] [-ifneeded]
  Description
    Deactivate a service (on the specified nodes)

Example

To deactivate the Call Barring and Call Forwarding services on nodes 101 and 102:

$ ./rhino-console deactivateservice \
    "name=Call Barring Service,vendor=OpenCloud,version=0.2" \
    "name=Call Forwarding Service,vendor=OpenCloud,version=0.2" \
    -nodes 101,102
Deactivating services [ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2],
  ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.2]] on node(s) [101,102]
Services transitioned to the Stopping state on node 101
Services transitioned to the Stopping state on node 102

MBean operation: `setPerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(ServiceID id, int[] nodeIDs, ServiceDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to set the desired state for a service on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node state configured for the specified service

public void setDefaultDesiredState(ServiceID id, ServiceDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to set the desired state for a service on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that have per-node state configured that is different from the default state

public void removePerNodeDesiredState(ServiceID id, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, ManagementException;

Rhino provides an extension to clear the desired state for a service on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `deactivate`

MBean

SLEE-defined

Deactivate on all nodes

public void deactivate(ServiceID id)
    throws NullPointerException, UnrecognizedServiceException,
          InvalidStateException, ManagementException;

public void deactivate(ServiceID[] ids)
    throws NullPointerException, InvalidArgumentException,
          UnrecognizedServiceException, InvalidStateException,
          ManagementException;

Rhino’s implementation of the SLEE-defined deactivate operation attempts to deactivate particular services on all current event-router nodes in the primary component. For this to work, the specified services must be in the ACTIVE state on at least one node.

Rhino extension

Deactivate on specific nodes

public void deactivate(ServiceID id, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, InvalidStateException,
        ManagementException;

public void deactivate(ServiceID[] ids, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedServiceException, InvalidStateException,
        ManagementException;

Rhino provides an extension that adds an argument that lets you control the nodes on which to deactivate the specified services (by specifying node IDs). For this to work, the specified services must be in the ACTIVE state on the specified nodes.

Console command: `waittilserviceisinactive`

Command

waittilserviceisinactive <service-id> [-timeout timeout] [-nodes node1,node2,...]
    Wait for a service to finish deactivating (on the specified nodes) (timing out after N seconds)

Example

To wait for the Call Barring and Call Forwarding services on nodes 101 and 102:

$ ./rhino-console waittilserviceisinactive \
    "name=Call Barring Service,vendor=OpenCloud,version=0.2" \
    "name=Call Forwarding Service,vendor=OpenCloud,version=0.2" \
    -nodes 101,102
Service ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2] is in the Inactive state on node(s) [101,102]
Service ServiceID[Call Forwarding Service,vendor=OpenCloud,version=0.2] is in the Inactive state on node(s) [101,102]

Upgrading (Activating & Deactivating) Services

To activate some services and deactivate others, use the following rhino-console command or related MBean operation.

Activating and deactivating in one operation

The SLEE specification defines the ability to deactivate some services and activate other services in a single operation. As one set of services deactivates, the existing activities being processed by those services continue to completion, while new activities (started after the operation is invoked) are processed by the activated services. The intended use of this is to upgrade a service or services with new versions (however the operation does not have to be used strictly for this purpose).

Console command: `deactivateandactivateservice`

Command

deactivateandactivateservice Deactivate <service-id>* Activate <service-id>*
[-nodes node1,node2,...]
  Description
    Deactivate some services and Activate some other services (on the specified
    nodes)

Example

To deactivate version 0.2 of the Call Barring and Call Forwarding services and activate version 0.3 of the same services on nodes 101 and 102:

$ ./rhino-console deactivateandactivateservice \
    Deactivate "name=Call Barring Service,vendor=OpenCloud,version=0.2" \
               "name=Call Forwarding Service,vendor=OpenCloud,version=0.2" \
    Activate   "name=Call Barring Service,vendor=OpenCloud,version=0.3" \
               "name=Call Forwarding Service,vendor=OpenCloud,version=0.3" \
    -nodes 101,102
On node(s) [101,102]:
    Deactivating service(s) [ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2],
      ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.2]]
    Activating service(s) [ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.3],
      ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.3]]
Deactivating service(s) transitioned to the Stopping state on node 101
Activating service(s) transitioned to the Active state on node 101
Deactivating service(s) transitioned to the Stopping state on node 102
Activating service(s) transitioned to the Active state on node 102

MBean operation: `deactivateAndActivate`

MBean

SLEE-defined

Deactivate and activate on all nodes

public void deactivateAndActivate(ServiceID deactivateID, ServiceID activateID)
    throws NullPointerException, InvalidArgumentException,
           UnrecognizedServiceException, InvalidStateException,
           InvalidLinkNameBindingStateException, ManagementException;

public void deactivateAndActivate(ServiceID[] deactivateIDs, ServiceID[] activateIDs)
    throws NullPointerException, InvalidArgumentException,
           UnrecognizedServiceException, InvalidStateException,
           InvalidLinkNameBindingStateException, ManagementException;

Rhino’s implementation of the SLEE-defined deactivateAndActivate operation attempts to deactivate specified services and activate others on all current event-router nodes in the primary component. For this to work, the services to deactivate must be in the ACTIVE state, and the services to activate must be in the INACTIVE state, on those nodes.

Rhino extension

Deactivate and activate on specific nodes

public void deactivateAndActivate(ServiceID deactivateID, ServiceID activateID, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
           UnrecognizedServiceException, InvalidStateException,
           ManagementException;

public void deactivateAndActivate(ServiceID[] deactivateIDs, ServiceID[] activateIDs, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
           UnrecognizedServiceException, InvalidStateException,
           ManagementException;

Rhino provides an extension that adds an argument that lets you control the nodes on which to activate and deactivate services (by specifying node IDs). For this to work, the services to deactivate must be in the ACTIVE state, and the services to activate must be in the INACTIVE state, on the specified nodes.

Getting Link Bindings Required by a Service

To find the resource adaptor entity link name bindings needed for a service, and list the service’s SBBs, use the following rhino-console commands or related MBean operations.

Console commands

`listserviceralinks`

Command

listserviceralinks service-id
  Description
    List resource adaptor entity links required by a service

Example

To list the resource adaptor entity links that the JCC VPN service needs:

$ ./rhino-console listserviceralinks "name=JCC 1.1 VPN,vendor=Open Cloud,version=1.0"
In service ServiceID[name=JCC 1.1 VPN,vendor=Open Cloud,version=1.0]:
    SBB SbbID[name=AnytimeInterrogation sbb,vendor=Open Cloud,version=1.0] requires entity link bindings: slee/resources/map
    SBB SbbID[name=JCC 1.1 VPN sbb,vendor=Open Cloud,version=1.0] requires entity link bindings: slee/resources/cdr

`listsbbs`

Command

listsbbs [service-id]
  Description
    List the current installed SBBs.  If a service identifier is specified only the
    SBBs in the given service are listed

Example

To list the SBBs in the JCC VPN service:

$ ./rhino-console listsbbs "name=JCC 1.1 VPN,vendor=Open Cloud,version=1.0"
SbbID[name=AnytimeInterrogation sbb,vendor=Open Cloud,version=1.0]
SbbID[name=JCC 1.1 VPN sbb,vendor=Open Cloud,version=1.0]
SbbID[name=Proxy route sbb,vendor=Open Cloud,version=1.0]

MBean operations: `getServices`, `getSbbs`, and `getDescriptors`

MBean

Finding RA configuration properties

SLEE-defined

Get all services in the SLEE

public ServiceID[] getServices()
    throws ManagementException;

getServices returns an array containing the component identifiers of all services installed in the SLEE.

Get all SBBs in a service

public SbbID[] getSbbs(ServiceID service)
    throws NullPointerException, UnrecognizedServiceException,
        ManagementException;

getSbbs returns an array containing the component identifiers of all SBBs included in the given service.

Get the component descriptor for a component

public ComponentDescriptor[] getDescriptors(ComponentID[] ids)
    throws NullPointerException, ManagementException;

getDescriptors returns the component descriptor for each given component.

Getting entity link information for an SBB

To find the entity link names for an individual SBB, you can:

cast a ComponentDescriptor object for the SBB to an SbbDescriptor
retrieve an array of entity link names required by the SBB (from the SbbDescriptor), using the getResourceAdaptorEntityLinks operation. The array will be zero-length if the SBB does not require any entity link bindings.

Configuring service metrics recording status

To check and configure the status for recording service metrics, use the following rhino-console commands or related MBean operations.

The details for metrics stats are listed in Metrics.Services.cmp and Metrics.Services.lifecycle.

The default is set to disabled for performance consideration.

Console commands

`getservicemetricsrecordingenabled`

Command

getservicemetricsrecordingenabled <service-id>
  Description
    Determine if metrics recording for a service has been enabled

Example

To check the status for recording metrics:

$ ./rhino-console getservicemetricsrecordingenabled name=service1,vendor=OpenCloud,version=1.0
Metrics recording for ServiceID[name=service1,vendor=OpenCloud,version=1.0] is currently disabled

`setservicemetricsrecordingenabled`

Command

setservicemetricsrecordingenabled <service-id> <true|false>
  Description
    Enable or disable the recording of metrics for a service

Example

To enable the recording metrics:

$ ./rhino-console setservicemetricsrecordingenabled name=service1,vendor=OpenCloud,version=1.0 true
Metrics recording for ServiceID[name=service1,vendor=OpenCloud,version=1.0] has been enabled

MBean operations: `getServiceMetricsRecordingEnabled` and `setServiceMetricsRecordingEnabled`

MBean

ServiceManagementMBean

Rhino extension

Determine if the recording of metrics for a service is currently enabled or disabled.

public boolean getServiceMetricsRecordingEnabled(ServiceID service)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

Enable or disable the recording of metrics for a service.

public void setServiceMetricsRecordingEnabled(ServiceID service, boolean enabled)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

Configuring service replication

The default replication behaviour of a service is defined by the service in its deployment descriptor, but may be overridden by an administrator after the service has been installed into the SLEE.

Default replication behaviour

Default replication behaviour is specified by a service in its oc-service.xml extension service deployment descriptor. The service can specify the conditions under which the application state of the service will be replicated by using the following replication selectors:

Savanna — Service replication will occur if the namespace the service is installed in replicates application state over the traditional savanna framework.
KeyValueStore — Service replication will occur if the namespace the service is installed in utilises a key/value store to persist application state.
Always — The service will always be replicated regardless of any underlying replication mechanism.

Zero or more replication selectors can be specified by the service. If any condition for replication is matched at deployment time then the service application state will be replicated. If not, no replication will take place for that service.

Configuring replication behaviour

The default replication selectors specified by a service can be changed by an administrator after the service is installed, but before it is deployed, using the following rhino-console commands or related MBean operations.

Console commands

`getservicereplicationselectors`

Command

getservicereplicationselectors <service-id>
  Description
    Get the replication selectors for a service

Example

To check the current replication selectors for a service:

$ ./rhino-console getservicereplicationselectors name=service1,vendor=OpenCloud,version=1.0
Service ServiceID[name=service1,vendor=OpenCloud,version=1.0] current replication selectors are: [KEYVALUESTORE]

`setservicereplicationselectors`

Command

setservicereplicationselectors <service-id> -none|selector*
  Description
    Set the replication selectors for a service, valid selectors are: [ALWAYS,
    SAVANNA, KEYVALUESTORE]

Example

The change the replication selectors for a service:

$ ./rhino-console setservicereplicationselectors name=service1,vendor=OpenCloud,version=1.0 SAVANNA KEYVALUESTORE
Service ServiceID[name=service1,vendor=OpenCloud,version=1.0] replication selectors set to [SAVANNA, KEYVALUESTORE]

MBean operations: `getReplicationSelectors` and `setReplicationSelectors`

MBean

ServiceManagementMBean

Rhino extension

Get the current replication selectors for a service.

public ReplicationSelector[] getReplicationSelectors(ServiceID id)
    throws NullPointerException, UnrecognizedServiceException, ManagementException;

Set the replication selectors for a service.

    public void setReplicationSelectors(ServiceID id, ReplicationSelector[] selectors)
        throws NullPointerException, UnrecognizedServiceException, InvalidStateException, ManagementException;

Resource Adaptor Entities

As well as an overview of resource adaptor entities, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command MBean → Operation

 listraconfigproperties

Resource Management → getConfigurationProperties

Creating a resource adaptor entity

 createraentity

Resource Management → createResourceAdaptorEntity

Removing a resource adaptor entity

 removeraentity

Resource Management → removeResourceAdaptorEntity

Listing configuration properties for a resource adaptor entity

 listraentityconfigproperties

Resource Management → getConfigurationProperties

Updating configuration properties for a resource adaptor entity

 updateraentityconfigproperties

Resource Management → updateConfigurationProperties

Activating a resource adaptor entity

 activateraentity

Resource Management → activateResourceAdaptorEntity

Deactivating a resource adaptor entity

 deactivateraentity

Resource Management → deactivateResourceAdaptorEntity

Reassigning a resource adaptor entity’s activities to other nodes

 reassignactivities

Resource Management → reassignActivities

Retrieving a resource adaptor entity’s state

 getraentityactualstate, getraentitydesiredstate

Resource Management → getState

Listing resource adaptor entities by state

 listraentitiesbystate

Resource Management → getResourceAdaptorEntities

Binding a resource adaptor entity to a link name

 bindralinkname

Resource Management → bindLinkName

Unbinding link names

 unbindralinkname

Resource Management → unbindLinkName

Listing link name bindings

 listralinknames

Resource Management → getLinkNames

About Resource Adaptor Entities

Resource adaptors (RAs) are SLEE components which let particular network protocols or APIs be used in the SLEE.

They typically include a set of configurable properties (such as address information of network endpoints, URLs to external systems, or internal timer-timeout values). These properties may include default values. A resource adaptor entity is a particular configured instance of a resource adaptor, with defined values for all of that RA’s configuration properties.

The resource adaptor entity lifecycle

The SLEE specification presents the operational lifecycle of a resource adaptor entity — illustrated, defined, and summarised below.

Resource adaptor entity lifecycle states

The SLEE lifecycle states are:

State

Definition

 INACTIVE

The resource adaptor entity has been configured and initialised. It is ready to be activated, but may not yet create activities or fire events to the SLEE. Typically, it is not connected to network resources.

 ACTIVE

The resource adaptor entity is connected to the resources it needs to function (assuming they are available), and may create activities and fire events to the SLEE.

 STOPPING

The resource adaptor entity may not create new activities in the SLEE, but may fire events to the SLEE on already existing activities. A resource adaptor entity transitions out of the STOPPING state, returning to the INACTIVE state, when all activities it owns have either ended or been assigned to another node for continued processing.

Creating activities in the STOPPING state

By default, Rhino 3.1 prevents a resource adaptor from creating an activity in the STOPPING state.

This behaviour is controlled by the rhino.skip_lifecycle_checks system property, which defaults to false.

When set to true, Rhino does not enforce this restriction. Resource adaptors should check the state before creating an activity, to avoid a situation where a resource adaptor entity never deactivates because new activities are being created.

The default value in earlier versions of Rhino was true.

Independent lifecycle state machines

As explained in About SLEE Operational States, each event-router node in a Rhino cluster maintains its own lifecycle state machine, independent of other nodes in the cluster. This is also true for each resource adaptor entity: one resource adaptor entity might be INACTIVE on one node in a cluster, ACTIVE on another, and STOPPING on a third. The operational state of a resource adaptor entity on each cluster node also persists to the disk-based database.

A resource adaptor entity will enter the INACTIVE state, after node bootup and initialisation completes, if the database’s persistent operational state information for that resource adaptor entity is missing, or is set to INACTIVE or STOPPING.

And, like node operational states, you can change the operational state of a resource adaptor entity at any time, as long as least one node in the cluster is available to perform the management operation (regardless of whether or not the node whose operational state being changed is a current cluster member). For example, you might activate a resource adaptor entity on node 103 before node 103 is booted — then, when node 103 boots, and after it completes initialisation, that resource adaptor entity will transition to the ACTIVE state.

Creating and Removing Resource Adaptor Entities

Rhino includes procedures for:

Finding RA Configuration Properties
Creating a Resource Adaptor Entity
Removing a Resource Adaptor Entity

Finding RA Configuration Properties

To determine resource adaptor configuration properties (which you need to know when Creating a Resource Adaptor Entity) use the following rhino-console command or related MBean operation.

Console command: `listraconfigproperties`

Command

listraconfigproperties <resource-adaptor-id>
  Description
    List the configuration properties (and any default values) for a resource
    adaptor

Example

To list the configuration properties of the Metaswitch SIP Resource Adaptor:

$ ./rhino-console listraconfigproperties name=OCSIP,vendor=OpenCloud,version=2.1
Configuration properties for resource adaptor name=OCSIP,vendor=OpenCloud,version=2.1:
  Automatic100TryingSupport (java.lang.Boolean): true
  CRLLoadFailureRetryTimeout (java.lang.Integer): 900
  CRLNoCRLLoadFailureRetryTimeout (java.lang.Integer): 60
  CRLRefreshTimeout (java.lang.Integer): 86400
  CRLURL (java.lang.String):
  ...

MBean operation: `getConfigurationProperties`

MBean

SLEE-defined

public ConfigProperties getConfigurationProperties(ResourceAdaptorID id)
    throws NullPointerException, UnrecognizedResourceAdaptorException,
        ManagementException

Output

This operation returns a ConfigProperties object. A ConfigProperties object contains a set of ConfigProperty.Property objects, each of which identifies one configuration property defined by the RA. If the RA has defined a default value for the configuration property, the ConfigProperty.Property object will include it.

Creating a Resource Adaptor Entity

To create a resource adaptor entity use the following rhino-console command or related MBean operation.

Console command: `createrantity`

Command

createraentity <resource-adaptor-id> <entity-name>
[<config-params>|(<property-name> <property-value)*]
  Description
    Create a resource adaptor entity with the given name.  Optionally configuration
    properties can be specified, either as a single comma-separated string of
    name=value pairs, or as a series of separate name and value argument pairs

Example

To create an instance of the Metaswitch SIP resource adaptor, called sipra, with the following configuration property values replacing the defaults (if any) for the IPAddress (192.168.0.100), Port (5160) and SecurePort (5161) :

$ ./rhino-console createraentity name=OCSIP,vendor=OpenCloud,version=2.1 sipra \
    IPAddress=192.168.0.100,Port=5160,SecurePort=5161
Created resource adaptor entity sipra

Notes

Entering configuration properties

When creating a resource adaptor entity, determine its configuration properties and then enter them in rhino-console as a comma-separated list of property-name=value pairs.

White space, commas, quotes

If a configuration-property value contains white space or a comma, you must quote the value. For example:

$ ./rhino-console createraentity name=MyRA,vendor=Me,version=1.0 myra Value="The quick brown fox",Colour=brown

If the value requires quotes, you must escape them using a backslash ``' (to avoid them being removed by the parser). For example:

$ ./rhino-console createraentity name=MyRA,vendor=Me,version=1.0 myra Value="\"The quick brown fox\"",Colour=brown

MBean operation: `createResourceAdaptorEntity`

MBean

SLEE-defined

public void createResourceAdaptorEntity(ResourceAdaptorID id, String entityName, ConfigProperties properties)
    throws NullPointerException, InvalidArgumentException,
          UnrecognizedResourceAdaptorException,
          ResourceAdaptorEntityAlreadyExistsException,
          InvalidConfigurationException, ManagementException;

Arguments

This operation requires that you specify the resource adaptor entity’s:

ResourceAdaptorID — identifier of the resource adaptor from which to create the resource adaptor entity
entityName — an assigned name
ConfigProperties — configuration properties.

You only need to specify configuration properties that have no defined default, or have a default other than what the resource adaptor entity requires. (Rhino uses the default value if it is not specified within the properties argument.)

Rhino-defined configuration property

When creating a resource adaptor entity, you may specify the Rhino-defined configuration property: slee-vendor:com.opencloud.rhino_replicate_activities. This property describes the resource adaptor entity’s activity-replication behaviour (assuming it has been specifically designed to support activity-state replication in Rhino). Possible values are:

none — the resource adaptor entity will not generate replicated activities
mixed — the resource adaptor entity will generate a mix of replicated and non-replicated activities
all — all activities generated by the resource adaptor entity will be replicated.

The default value is none. (You can specify an alternative default by defining a configuration property in the deployment descriptor with this name but with a different default value.)

Removing a Resource Adaptor Entity

To remove a resource adaptor entity use the following rhino-console command or related MBean operation.

You can only remove a resource adaptor entity from the SLEE when it is in the INACTIVE state on all event-router nodes currently in the primary component.

Console command: `removeraentity`

Command

removeraentity <entity-name>
  Description
    Remove a resource adaptor entity

Example

To remove the resource adaptor entity named sipra:

$ ./rhino-console removeraentity sipra
Removed resource adaptor entity sipra

MBean operation: `removeResourceAdaptorEntity`

MBean

SLEE-defined

public void removeResourceAdaptorEntity(String entityName)
    throws NullPointerException,
          UnrecognizedResourceAdaptorEntityException,
          InvalidStateException, DependencyException, ManagementException;

Configuring Resource Adaptor Entities

Rhino includes procedures for:

Listing configuration properties for a resource adaptor entity
Updating configuration properties for a resource adaptor entity

Listing configuration properties for a Resource Adaptor Entity

To list the configuration properties for a resource adaptor entity use the following rhino-console command or related MBean operation.

Console command: `listraentityconfigproperties`

Command

listraentityconfigproperties <entity-name>
  Description
    List the configuration property values for a resource adaptor entity

Example

To list the resource adaptor entity called sipra:

$ ./rhino-console listraentityconfigproperties sipra
Configuration properties for resource adaptor entity sipra:
 Automatic100TryingSupport (java.lang.Boolean): true
 AutomaticOptionsResponses (java.lang.Boolean): true
 CRLLoadFailureRetryTimeout (java.lang.Integer): 900
 CRLNoCRLLoadFailureRetryTimeout (java.lang.Integer): 60
 CRLRefreshTimeout (java.lang.Integer): 86400
 CRLURL (java.lang.String):
 ClientAuthentication (java.lang.String): NEED
 EnableDialogActivityTests (java.lang.Boolean): false
 EnabledCipherSuites (java.lang.String):
 ExtensionMethods (java.lang.String):
 IPAddress (java.lang.String): AUTO
 Keystore (java.lang.String): sip-ra-ssl.keystore
 KeystorePassword (java.lang.String):
 KeystoreType (java.lang.String): jks
 MaxContentLength (java.lang.Integer): 131072
 OffsetPorts (java.lang.Boolean): false
 Port (java.lang.Integer): 5060
 PortOffset (java.lang.Integer): 101
 ReplicatedDialogSupport (java.lang.Boolean): false
 RetryAfterInterval (java.lang.Integer): 5
 SecurePort (java.lang.Integer): 5061
 TCPIOThreads (java.lang.Integer): 1
 Transports (java.lang.String): udp,tcp
 Truststore (java.lang.String): sip-ra-ssl.truststore
 TruststorePassword (java.lang.String):
 TruststoreType (java.lang.String): jks
 UseVirtualAddressInURIs (java.lang.Boolean): true
 ViaSentByAddress (java.lang.String):
 VirtualAddresses (java.lang.String):
 WorkerPoolSize (java.lang.Integer): 4
 WorkerQueueSize (java.lang.Integer): 50
 slee-vendor:com.opencloud.rhino_max_activities (java.lang.Integer): 0
 slee-vendor:com.opencloud.rhino_replicate_activities (java.lang.String): mixed

MBean operation: `getConfigurationProperties`

MBean

SLEE-defined

public ConfigProperties getConfigurationProperties(String entityName)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException, ManagementException;

Output

This operation returns a ConfigProperties object. A ConfigProperties object contains a set of ConfigProperty.Property objects, each of which identifies one configuration property defined by the RA. If the RA has defined a default value for the configuration property, the ConfigProperty.Property object will include it.

Updating configuration properties for a Resource Adaptor Entity

To update configuration properties for a resource adaptor entity use the following rhino-console command or related MBean operation.

When is it appropriate to update configuration properties?

A resource adaptor may elect to support reconfiguration when its resource adaptor entities are active using the supports-active-reconfiguration attribute of the <resource-adaptor-class> deployment descriptor element.

If the value of the supports-active-reconfiguration attribute is False, the updateraentityconfigproperties command and related MBean operation may only be invoked to reconfigure a resource adaptor entity when it is in the Inactive state, or when the SLEE is in the Stopped state.

If the value of the supports-active-reconfiguration attribute is True, then a resource adaptor entity may be reconfigured when it, and the SLEE, are in any state, i.e. reconfiguration is possible while the resource adaptor entity is creating activities and firing events in the SLEE.

Console command: `updateraentityconfigproperties`

Command

updateraentityconfigproperties <entity-name> [<config-params>|(<property-name>
<property-value)*]
  Description
    Update configuration properties for a resource adaptor entity. Properties can be
    specified either as a single comma-separated string of name=value pairs or as a
    series of separate name and value argument pairs

Example

To update the Port and SecurePort in resource adaptor entity called sipra:

$ ./rhino-console updateraentityconfigproperties sipra Port 5061 SecurePort 5062
Updated configuration parameters for resource adaptor entity sipra

MBean operation: `updateConfigurationProperties`

MBean

SLEE-defined

public void updateConfigurationProperties(String entityName, ConfigProperties properties)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException,
           InvalidStateException, InvalidConfigurationException,
           ManagementException;

Input

This operation requires a ConfigProperties object. A ConfigProperties object contains a set of ConfigProperty.Property objects, each of which identifies one configuration property defined by the RA.

Activating and Deactivating Resource Adaptor Entities

Rhino includes procedures for:

Activating a Resource Adaptor Entity
Deactivating a Resource Adaptor Entity
Reassigning a Resource Adaptor Entity’s Activities to Other Nodes

Activating a Resource Adaptor Entity

To activate a resource adaptor entity on one or more nodes use the following rhino-console command or related MBean operations.

If executed without a list of nodes, all per-node desired state for the resource adaptor entity is removed and the default desired state of the resource adaptor entity is set to active (if it is not already).

Console command: `activateraentity`

Command

activateraentity <entity-name> [-nodes node1,node2,...] [-ifneeded]
  Description
    Activate a resource adaptor entity (on the specified nodes)

Example

To activate the resource adaptor entity called sipra on nodes 101 and 102:

$ ./rhino-console activateraentity sipra -nodes 101,102
Activating resource adaptor entity sipra on node(s) [101,102]
Resource adaptor entity transitioned to the Active state on node 101
Resource adaptor entity transitioned to the Active state on node 102

MBean operation: `setPerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(String entityName, int[] nodeIDs, ResourceAdaptorEntityDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to set the desired state for a resource adaptor entity on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node state configured for the specified resource adaptor entity

public void setDefaultDesiredState(String entityName, ResourceAdaptorEntityDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to set the desired state for a resource adaptor entity on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that have per-node state configured that is different from the default state

public void removePerNodeDesiredState(String entityName, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to clear the desired state for a resource adaptor entity on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `activateResourceAdaptorEntity`

MBean

SLEE-defined

Activate on all nodes

public void activateResourceAdaptorEntity(String entityName)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException,
          InvalidStateException, ManagementException;

Rhino’s implementation of the SLEE-defined activateResourceAdaptorEntity operation attempts to activate a resource adaptor entity on all current event-router nodes in the primary component. For this to work, the resource adaptor entity must be in the INACTIVE state on at least one node.

Rhino extension

Activate on specific nodes

public void activateResourceAdaptorEntity(String entityName, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
          UnrecognizedResourceAdaptorEntityException,
          InvalidStateException, ManagementException;

Rhino provides an extension that adds an argument that lets you control the nodes on which to activate the resource adaptor entity (by specifying node IDs). For this to work, the resource adaptor entity must be in the INACTIVE state on the specified nodes.

Deactivating a Resource Adaptor Entity

To deactivate a resource adaptor entity on one or more nodes use the following rhino-console command or related MBean operation.

If executed without a list of nodes, all per-node desired state for the resource adaptor entity is removed and the default desired state of the resource adaptor entity is set to inactive (if it is not already).

See also Reassigning a resource adaptor entity’s Activities to Other Nodes, particularly the Requirements tab.

Console command: `deactivateraentity`

Command

deactivateraentity <entity-name> [-nodes node1,node2,... [-reassignto
node3,node4,...]] [-ifneeded]
  Description
    Deactivate a resource adaptor entity (on the specified nodes (reassigning
    replicated activities to the specified nodes))

Examples

To deactivate the resource adaptor entity named sipra on nodes 101 and 102:

$ ./rhino-console deactivateraentity sipra -nodes 101,102
Deactivating resource adaptor entity sipra on node(s) [101,102]
Resource adaptor entity transitioned to the Stopping state on node 101
Resource adaptor entity transitioned to the Stopping state on node 102

To deactivate the resource adaptor entity named sipra on node 101, and reassign replicated activities to node 102:

$ ./rhino-console deactivateraentity sipra -nodes 101 -reassignto 102
Deactivating resource adaptor entity sipra on node(s) [101]
Resource adaptor entity transitioned to the Stopping state on node 101
Replicated activities reassigned to node(s) [102]

To deactivate the resource adaptor entity named sipra on node 101, and distribute replicated activities equally among all other eligible nodes (those on which the resource adaptor entity is in the ACTIVE state and the SLEE is in the RUNNING state), specify an empty (zero-length) argument for the -reassignto option:

$ ./rhino-console deactivateraentity sipra -nodes 101 -reassignto ""
Deactivating resource adaptor entity sipra on node(s) [101]
Resource adaptor entity transitioned to the Stopping state on node 101
Replicated activities reassigned to node(s) [102,103]

MBean operation: `setPerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on specific nodes

public void setPerNodeDesiredState(String entityName, int[] nodeIDs, ResourceAdaptorEntityDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to set the desired state for a resource adaptor entity on a set of nodes.

MBean operation: `setDefaultDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that do not have per-node state configured for the specified resource adaptor entity

public void setDefaultDesiredState(String entityName, ResourceAdaptorEntityDesiredState desiredState)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to set the desired state for a resource adaptor entity on nodes that do not have a per-node desired state configured.

MBean operation: `removePerNodeDesiredState`

MBean

Rhino extension

Activate or deactivate on nodes that have per-node state configured that is different from the default state

public void removePerNodeDesiredState(String entityName, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension to clear the desired state for a resource adaptor entity on a set of nodes. Nodes that do not have a per-node desired state configured use the default desired state.

MBean operation: `deactivateResourceAdaptorEntity`

MBean

SLEE-defined

Deactivate on all nodes

public void deactivateResourceAdaptorEntity(String entityName)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException,
          InvalidStateException, ManagementException;

Rhino’s implementation of the SLEE-defined deactivateResourceAdaptorEntity operation attempts to deactivate a resource adaptor entity on all current event-router nodes in the primary component. For this to work, the resource adaptor entity must be in the ACTIVE state on at least one node.

Rhino extensions

Deactivate on specific nodes

public void deactivateResourceAdaptorEntity(String entityName, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, InvalidStateException,
        ManagementException;

Rhino provides an extension that adds an argument that lets you control the nodes on which to deactivate the resource adaptor entity (by specifying node IDs). For this to work, the resource adaptor entity must be in the ACTIVE state on the specified nodes.

Reassign deactivating activities to other nodes

public void deactivateResourceAdaptorEntity(String entityName, int[] nodeIDs, int[] reassignActivitiesToNodeIDs)
    throws NullPointerException, InvalidArgumentException,
        UnrecognizedResourceAdaptorEntityException, InvalidStateException,
        ManagementException;

Rhino also provides an extension that adds another argument, that lets you reassign ownership of replicated activities (from a replicating resource adaptor entity), distributing them equally among other available event-router nodes. This reduces the set of activities on the nodes with the deactivating resource adaptor entity, so the resource adaptor entity can return to the INACTIVE state on those nodes quicker. This only works for resource adaptor entities that are replicating activity state (see the description of the "Rhino-defined configuration property" for the MBean on Creating a Resource Adaptor Entity).

Reassigning a Resource Adaptor Entity’s Activities to Other Nodes

To reassign activities from a resource adaptor entity to a different node, use the following rhino-console command or related MBean operation, noting the requirements.

Why reassign replicating activities?

A resource adaptor entity in the STOPPING state cannot return to the INACTIVE state until all the activities that it owns have ended. You can let a deactivating resource adaptor entity return to the INACTIVE state quicker by reassigning its replicating activities to other eligible nodes.

Console command: `reassignactivities`

Command

reassignactivities <entity-name> -from node1,node2,... -to node3,node4,...
  Description
    Reassign replicated activities of a resource adaptor entity from the specified
    nodes to other nodes

Examples

To reassign activities owned by the resource adaptor entity named sipra from node 101 to node 102 and 103:

$ ./rhino-console reassignactivities sipra -from 101 -to 102,103
Replicated activities for sipra reassigned to node(s) [102,103]

Reassigning to all available nodes

You can also specify an empty (zero-length) argument for the -to option. This reassigns replicated activities, distributing them equally among all other nodes that can adopt them (nodes on which the resource adaptor entity is in the ACTIVE state and the SLEE is in the RUNNING state).

To reassign activities owned by the resource adaptor entity named sipra from node 101 to all other eligible nodes:

$ ./rhino-console reassignactivities sipra -from 101 -to ""
Replicated activities for sipra reassigned to node(s) [102,103]

MBean operation: `reassignActivities`

MBean

Rhino extension

public void reassignActivities(String entityName, int[] fromNodeIDs, int[] toNodeIDs)
      throws NullPointerException, InvalidArgumentException,
            UnrecognizedResourceAdaptorEntityException,
            InvalidStateException, ManagementException;

This operation reassigns replicated activities owned by the named resource adaptor entity, on the nodes specified, using Rhino’s standard failover algorithm, to the nodes specified by the toNodeIDs argument. (If toNodeIDs is a zero-length array, the operation reassigns activities to any remaining eligible nodes.)

Requirements for reassigning activities

You can only reassign replicated activities from a resource adaptor entity to other nodes if the all the following conditions are satisfied:

The node is a current member of the primary component.
The node is an event-router node (not a quorum node).
The operational state of the SLEE on the node is RUNNING or STOPPING.
The operational state of the resource adaptor entity on the node is ACTIVE or STOPPING.

Further, a node can only take ownership of replicated activities if it satisfies all the following conditions:

The node is a current member of the primary component.
The node is an event-router node (not a quorum node).
The operational state of the SLEE on the node is RUNNING.
The operational state of the resource adaptor entity on the node is ACTIVE.

Also, non-replicated activities cannot be reassigned to other nodes, and a resource adaptor entity must end any non-replicated activities it created itself.

You can choose to forcefully remove activities if a resource adaptor entity fails to end them in a timely manner.

Retrieving the State of Resource Adaptor Entities

Rhino includes procedures for:

Retrieving a Resource Adaptor Entity’s State
Listing Resource Adaptor Entities by State

Retrieving a Resource Adaptor Entity’s State

Retrieving actual state
- Console command: getraentityactualstate
- MBean operation: getActualState
Retrieving desired state
Retrieving SLEE-defined state
- Console command: getraentitystate
- MBean operation: getState

Retrieving actual state

To retrieve the actual operational state of a Resource Adaptor Entity, use the following rhino-console command or related MBean operation. For an explanation of the terms "actual state" and "desired state" see bxref:concepts-and-terminology.

Console command: `getraentityactualstate`

Command

getraentityactualstate <entity-name> <-all|-nodes node1,node2,...>
  Description
    Get the actual resource adaptor entity state for the specified nodes. If -all is
    specified, query the state of all current event router cluster members.

Output

The rhino-console client displays the actual operational state of the specified node(s), or every event-router node in the primary component if -all is specified.

Examples

To display the actual state of the Resource Adaptor Entity sipra only node 101:

$ ./rhino-console getraentityactualstate sipra -nodes 101
Getting actual Resource Adaptor Entity state for node(s) [101]
Node 101: Stopped

To display the actual state of the Resource Adaptor Entity sipra on every event-router node:

$ ./rhino-console getraentityactualstate sipra -all
Getting actual Resource Adaptor Entity state for node(s) [101,102]
Node 101: Stopped
Node 102: Running

MBean operation: `getActualState`

MBean

Rhino extension

Return actual state of a set of nodes

public ResourceAdaptorEntityActualState getActualState(String entityName, int[] nodeIDs)
  throws ManagementException;

Retrieving desired state

To retrieve the desired operational state of a Resource Adaptor Entity, use the following rhino-console command or related MBean operation.

Console command: `getraentitydesiredstate`

Command

getraentitydesiredstate <entity-name> <-default|-all|-nodes node1,node2,...>
  Description
    Get the default or per-node desired resource adaptor entity state. If -all is
    specified, query the state of all current event router nodes as well as all
    nodes with saved per-node state.

Output

The rhino-console client displays the desired state of the specified node(s), or every node with configured state and every event-router node in the primary component if -all is specified.

Examples

To display the desired state of only node 101:

$ ./rhino-console getraentitydesiredstate -nodes 101
Node 101: Stopped

To display the desired state of the Resource Adaptor Entity sipra every event-router node and configured node:

$ ./rhino-console getraentitydesiredstate -all
Node 101: Stopped
Node 102: Running (default)
Node 103: Running

To display the default desired state that unconfigured event router nodes will inherit:

$ ./rhino-console getraentitydesiredstate -default
Getting default Resource state
Default Resource state is: running

MBean operation: `getPerNodeDesiredState`

MBean

Rhino extension

Return the desired state of a set of nodes

public ResourceAdaptorEntityDesiredState getPerNodeDesiredState(String entityName, int[] nodeIDs)
  throws ManagementException;

MBean operation: `getDefaultDesiredState`

MBean

Rhino extension

Return the default desired state used by nodes that do not have a configured per-node state

public ResourceAdaptorEntityDesiredState getDefaultDesiredState(String entityName)
  throws ManagementException;

Retrieving SLEE-defined state

To retrieve the operational state of a Resource Adaptor Entity in a form compatible with the JAIN SLEE specification, use the following rhino-console command or related MBean operation.

Console command: `getraentitystate`

Command

getraentitystate <entity-name> [-nodes node1,node2,...]
  Description
    Get the state of a resource adaptor entity (on the specified nodes)

Output

The rhino-console client displays the operational state of the specified node(s), or every event-router node in the primary component if none are specified.

Examples

To display the state of the resource adaptor entity with the name sipra on every event-router node:

$ ./rhino-console getraentitystate sipra
Resource Adaptor Entity is Inactive on node 101
Resource Adaptor Entity is Active on node 102

To display the state of the Resource Adaptor Entity on only node 101:

$ ./rhino-console getraentitystate sipra -nodes 101
Resource Adaptor Entity is Inactive on node 101

MBean operation: `getState`

MBean

SLEE-defined

Return state of Resource Adaptor Entity on current node

public ResourceAdaptorEntityState getState(String entityName)
    throws NullPointerException,
    UnrecognizedResourceAdaptorEntityException,
    ManagementException;

Rhino’s implementation of the SLEE-defined getState operation returns the SLEE-defined state most closely representative of the actual state of a Resource Adaptor Entity on the node the invoking client is connected to. When using the Rhino client library with a list of hosts this will usually be the node on the first host in the list. When multiple nodes are running on the same host, the oldest node on the host will usually expose the management interface and thus be the target of this query.

Since Rhino 3.0.0 the actual state of components on each node can update asynchronously. This differs from symmetric activation state mode in earlier Rhino versions in that the value returned by getState() is not representative of the state on other cluster nodes. Users of this method who previously configured symmetric activation state mode should switch to checking the state of all nodes using the method getState(int[] nodeIDs) or one of the new getDesiredState(int[] nodeIDs) or getActualState(int[] nodeIDs) depending on the purpose of the state query. A list of event router node IDs can be obtained using RhinoHousekeepingMBean.getEventRouterNodes(). For example, to verify that a Resource Adaptor Entity is configured to be active on all nodes:

RhinoHousekeepingMBean rhinoHousekeeping =  RhinoManagement.getRhinoHousekeepingMBean(client);
ResourceManagementMBean resourceManagement =  RhinoManagement.getResourceManagementMBean(client);
ResourceAdaptorEntityState[] nodeStates = resourceManagement.getDesiredState(entityName, rhinoHousekeeping.getEventRouterNodes());
boolean active = Arrays.stream(nodeStates).filter(s -> s != ResourceAdaptorEntityDesiredState.ACTIVE).count() == 0;

Rhino extension

Return state of Resource Adaptor Entity on specified node(s)

public ResourceAdaptorEntityState[] getState(String entityName, int[] nodeIDs)
    throws NullPointerException, InvalidArgumentException,
    UnrecognizedResourceAdaptorEntityException, ManagementException;

Rhino provides an extension that adds an argument which lets you control the nodes on which to return the state of the Resource Adaptor Entity (by specifying node IDs).

Listing Resource Adaptor Entities by State

To list resource adaptor entities in a particular operational state, use the following rhino-console command or related MBean operation.

Console command: `listraentitiesbystate`

Command

listraentitiesbystate <state> [-node node]
  Description
    List the resource adaptor entities that are in the specified state (on the
    specified node)

Examples

To list the resource adaptor entities on the node where rhino-console is connected:

$ ./rhino-console listraentitiesbystate Active
No resource adaptor entities in Active state on node 101

To list the resource adaptor entities that are active on node 102:

$ ./rhino-console listraentitiesbystate Active -node 102
Resource adaptor entities in Active state on node 102:
sipra

MBean operation: `getResourceAdaptorEntities`

MBean

SLEE-defined

Return names of resource adaptor entities in specified state on current node

public String[] getResourceAdaptorEntities(ResourceAdaptorEntityState state)
    throws NullPointerException, ManagementException;

Rhino’s implementation of the SLEE-defined getResourceAdaptorEntities operation returns the names of resource adaptor entities in a specified state on the node where you invoke the operation.

Rhino extension

Return names of resource adaptor entities in specified state on specified node

public String[] getResourceAdaptorEntities(ResourceAdaptorEntityState state, int nodeID)
    throws NullPointerException, InvalidArgumentException,
    ManagementException;

Rhino provides an extension that lets you specify the nodes (by specifying node IDs) on which to return the names of resource adaptor entities in the specified state.

Managing Resource Adaptor Entity Link Bindings

What are resource adaptor entity link name bindings?

When an SBB needs access to a resource adaptor entity, it uses JNDI to get references to Java objects that implement the resource adaptor interface (provided by the resource adaptor entity). The SBB declares (in its deployment descriptor) the resource adaptor type it expects, and an arbitrary link name. Before activating a service using the SBB, an administrator must bind a resource adaptor entity (of the type expected) to the specified link name.

Rhino includes procedures for:

Binding a Resource Adaptor Entity to a Link Name
Unbinding Link Names
Listing Link Name Bindings

Binding a Resource Adaptor Entity to a Link Name

To bind a resource adaptor entity to a link name, use the following rhino-console command or related MBean operation.

Only one resource adaptor entity can be bound to a link name at any time.

Console command: `bindralinkname`

Command

bindralinkname <entity-name> <link-name>
  Description
    Bind a resource adaptor entity to a link name

Example

To bind the resource adaptor entity with the name sipra to the link name sip:

$ ./rhino-console bindralinkname sipra sip
Bound sipra to link name sip

MBean operation: `bindLinkName`

MBean

SLEE-defined

public void bindLinkName(String entityName, String linkName)
    throws NullPointerException, InvalidArgumentException,
            UnrecognizedResourceAdaptorEntityException,
            LinkNameAlreadyBoundException, ManagementException;

Unbinding Link Names

To unbind a resource adaptor entity from a link name, use the following rhino-console command or related MBean operation.

Console command: `unbindralinkname`

Command

unbindralinkname <link-name>
  Description
    Unbind a resource adaptor entity from a link name

Example

To unbind the link name sip:

$ ./rhino-console unbindralinkname sip
Unbound link name sip

MBean operation: `unbindLinkName`

MBean

SLEE-defined

public void unbindLinkName(String linkName)
    throws NullPointerException, UnrecognizedLinkNameException,
            DependencyException,ManagementException;

Listing Link Name Bindings

To list resource adaptor entity link names that have been bound in the SLEE, use the following rhino-console command or related MBean operation.

Console command: `listralinknames`

Command

listralinknames [entity name]
  Description
    List the bound link names (for the specified resource adaptor entity)

Examples

To list all resource adaptor entity link name bindings:

$ ./rhino-console listralinknames
slee/resources/cdr -> cdrra
slee/resources/map -> mapra

To list all link name bindings for the resource adaptor entity named mapra:

$ ./rhino-console listralinknames mapra
slee/resources/map

MBean operation: `getLinkNames`

MBean

Listing profile attributes

SLEE-defined

List all bound link names

public String[] getLinkNames()
    throws ManagementException;

Rhino’s implementation of the SLEE-defined getLinkNames operation returns an array of all link names that have been bound in the SLEE.

List link names to which a specific resource adaptor entity has been bound

public String[] getLinkNames(String entityName)
    throws NullPointerException,
            UnrecognizedResourceAdaptorEntityException,
            ManagementException;

The SLEE-defined operation also includes an argument for returning just link names to which a specified resource adaptor entity has been bound. If the resource adaptor entity has not been bound to any link names, the returned array is zero-length.

Profile Tables and Profiles

As well as an overview of SLEE profiles, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Creating profile tables

 createprofiletable

Profile Provisioning → createProfileTable

Creating profiles

 createprofile

Profile Provisioning → createProfile

Listing profile tables

 listprofiletables

Profile Provisioning → getProfileTables

Listing profiles

 listprofiles

Profile Provisioning → getProfiles

 listprofileattributes

Profile Provisioning, Profile → getProfile

Setting profile attributes

 setprofileattributes

Profile Provisioning, Profile → getProfile

Finding profiles

  listprofilesbyattribute +
listprofilesbyindexedattribute

Profile Provisioning → getProfilesByAttribute
Profile Provisioning → getProfilesByIndexedAttribute
Profile Provisioning → getProfilesByStaticQuery
Profile Provisioning → getProfilesByDynamicQuery

Exporting and importing profiles

 exportall +
importprofiles

Profile Provisioning → exportProfiles
Profile Provisioning → importProfiles

About Profiles

What are profiles? profile tables? profile specifications?

A profile is an entry in a profile table. It has a name, may have values (called "attributes") and may have indexed fields. It’s like a row in SQL, but may also include business and management logic.

A profile table is a "container" for profiles. Its specification schema, the profile specification deployment descriptor, may define queries for the profile table. The SLEE specification defines the format and structure of profile specification schemas.

A profile table’s default profile is the initial set of profile attribute values for newly created profiles within that table (if not specified explicitly with the profile-creation command).

Before deploying a profile into the SLEE, an administrator can configure its profile specification. You do this by modifying values in the profile’s profile-spec-jar.xml deployment descriptor (in its deployable unit). For example, you can specify:

static queries available to SLEE components, and administrators using the management interface
profile specification environment entries
indexing hints for profile attributes.

For more on profile static queries, environment entires and indexing, see the SLEE 1.1 specification.

Creating Profile Tables

To create a new profile table based on an already-deployed profile specification, use the following rhino-console command or related MBean operation.

Name character restriction

The profile table name cannot include the / character.

Console command: `createprofiletable`

Command

createprofiletable <profile-spec-id> <table-name>
  Description
    Create a profile table

Example

$ ./rhino-console createprofiletable name=AddressProfileSpec,vendor=javax.slee,version=1.1 testtable
Created profile table testtable

MBean operation: `createProfileTable`

MBean

SLEE-defined

public void createProfileTable(javax.slee.profile.ProfileSpecificationID id, String newProfileTableName)
    throws NullPointerException, UnrecognizedProfileSpecificationException,
          InvalidArgumentException, ProfileTableAlreadyExistsException,
          ManagementException;

Arguments

This operation requires that you specify the profile table’s:

id — component identifier of the profile specification from which to create the profile table
newProfileTableName — name of the profile table to create.

Creating Profiles

To create a profile in an existing profile table, use the following rhino-console command or related MBean operation.

Console command `createprofile`

Command

createprofile <table-name> <profile-name> (<attr-name> <attr-value>)*
  Description
    Add a profile to a table, optionally setting attributes (see
    -setProfileAttributes option)

Add a profile to a table, optionally setting attributes (See Setting Profile attributes)

Example

$ ./rhino-console createprofile testtable testprofile
Profile testtable/testprofile created

Notes

Setting profile attributes

When creating a profile, decide the profile’s attribute names and then enter them in rhino-console as a space-separated list of property-name value pairs.

White space, commas, quotes

If a profile or profile table name or an attribute name or value contains white space or a comma, you must quote the string. For example:

$ ./rhino-console createprofile "testtable 2" "testprofile 2" SubscriberAddress "my address" forwarding true

If the value requires quotes, you must escape them using a backslash "\" (to avoid them being removed by the parser). For example:

$ ./rhino-console createprofile testtable testprofile attrib "\"The quick brown fox\""

Name uniqueness

The profile name must be unique within the scope of the profile table.

MBean operation: `createProfile`

MBean

SLEE-defined

public javax.management.ObjectName createProfile(String profileTableName, String newProfileName)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          InvalidArgumentException, ProfileAlreadyExistsException,
          ManagementException;

Arguments

This operation requires that you specify the profile’s:

profileTableName — name of the profile table to create the profile in
newProfileName — name of the new profile.

Notes

Profile MBean commit state

This operation returns an ObjectName, which the management client can use to access a Profile MBean for the new profile. This MBean will be in the read-write state, so the management client can configure initial values for profile attributes before the SLEE adds the profile to the profile table. You cannot see the new profile in the profile table until you commit the Profile MBean’s state, using the ProfileMBean.commitProfile() operation.

Name uniqueness

The profile name must be unique within the scope of the profile table.

Listing Profile Tables

To list all profile tables in a SLEE, use the following rhino-console command or related MBean operation.

Console command: `listprofiletables`

Command

listprofiletables
  Description
    List the current created profile tables

Example

$ ./rhino-console listprofiletables
callbarring
callforwarding

MBean operation: `getProfileTables`

MBean

SLEE-defined

public Collection getProfileTables()
    throws ManagementException;

Listing Profiles

To list all profiles of a specific profile table, use the following rhino-console command or related MBean operation.

Console command: `listprofiles`

Command

listprofiles <table-name>
  Description
    List the profiles in a table

Example

$ ./rhino-console listprofiles testtable
testprofile

MBean operation: `getProfiles`

MBean

SLEE-defined

public Collection getProfiles(String profileTableName)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          ManagementException;

Arguments

This operation requires that you specify the profile table’s:

profileTableName — name of the profile table.

Listing Profile Attributes

To list a profile’s attributes (names and current values), use the following rhino-console command or related MBean operation.

Console command: `listprofileattributes`

Command

listprofileattributes <table-name> [profile-name]
  Description
    List the current values of a profile, or if no profile is specified the current
    values of the default profile are listed

Example

$ ./rhino-console listprofileattributes testtable testprofile
Address={null}

MBean operation: `getProfile`

MBean

SLEE-defined

public javax.management.ObjectName getProfile(String profileTableName,String profileName)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          UnrecognizedProfileNameException, ManagementException;

Arguments

This operation requires that you specify the profile table’s:

profileTableName — name of the profile table to get the profile from.
profileName — name of the profile.

Notes

Profile MBean state

This operation returns an ObjectName, which the management client can use to access a Profile MBean for this specific profile. This MBean will be in the read-only state, so the management client can only read the profile attributes. (To change profile attributes, see Setting Profile Attributes.)

For more about Profile MBeans, their requirements and restrictions, please see chapter 10.26 "Profile MBean" in the SLEE 1.1 Specification.

Setting Profile Attributes

To set a profile’s attribute values, use the following rhino-console command or related MBean operation.

Console command: `setprofileattributes`

Command

setprofileattributes <table-name> <profile-name> (<attr-name> <attr-value>)*
  Description
    Set the current values of a profile (use "" for default profile). The
    implementation supports only a limited set of attribute types that it can
    convert from strings to objects

Example

$ ./rhino-console setprofileattributes testtable testprofile Address IP:192.168.0.1
Set attributes in profile testtable/testprofile

Notes

White space, commas, quotes

If a profile or profile table name or an attribute name or value contains white space or a comma, you must quote the string. For example:

$ ./rhino-console setprofileattributes "testtable 2" "testprofile 2" SubscriberAddress "my address" forwarding true

If the value requires quotes, you must escape them using a backslash "\" (to avoid them being removed by the parser). For example:

$ ./rhino-console setprofileattributes testtable testprofile attrib "\"The quick brown fox\""

MBean operation: `getProfile`

MBean

SLEE-defined

public javax.management.ObjectName getProfile(String profileTableName,String profileName)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          UnrecognizedProfileNameException, ManagementException;

Arguments

This operation requires that you specify the profile table’s:

profileTableName — name of the profile table to get the profile from.
profileName — name of the profile.

Notes

Profile MBean state

This operation returns an ObjectName, which the management client can use to access a Profile MBean for this specific profile. This MBean will be in the read-only state, so the management client can only read the profile attributes.

To put the MBean into the read-write state, invoke ProfileMBean.editProfile(). This will give you access to the profile’s attributes using the MBean’s getter and setter methods. You cannot see the profile’s new values until you commit the Profile MBean’s state, using the ProfileMBean.commitProfile() operation.

For more about Profile MBeans, their requirements and restrictions, please see chapter 10.26 "Profile MBean" in the SLEE 1.1 Specification.

Finding Profiles

Rhino includes procedures for:

Finding Profiles by Attribute Value
Finding Profiles Using Static Queries
Finding Profiles Using Dynamic Queries

Finding Profiles by Attribute Value

To retrieve all profiles with a specific attribute value, use the following rhino-console commands or related MBean operations:

Console command: `listprofilesbyattribute`

Command

listprofilesbyattribute <table-name> <attr-name> <attr-value>
[display-attributes (true/false)]
  Description
    List the profile which have an attribute <attr-name> equal to <attr-value>. The
    implementation supports only a limited set of attribute types that it can
    convert from strings to objects

Example

$ ./rhino-console listprofilesbyattribute testtable Address IP:192.168.0.1
1 profiles returned
ProfileID[table=testtable,profile=testprofile]

Notes

SLEE 1.1- & SLEE 1.0-specific commands

Between SLEE 1.0 and SLEE 1.1, the underlying profile specification schema changed significantly. According to the SLEE 1.1 Specification, profile attributes no longer have to be indexed to be legally used by a find-by-attribute-value query. Therefore, the listprofilesbyattribute command can only be used for profiles and profile tables that are based on a SLEE 1.1-compliant profile specification. For running a find-by-attribute-value search on a SLEE 1.0-compliant profile table, use the listprofilesbyindexedattribute command.

Backwards compatibility

SLEE 1.1 demands backwards compatibility for SLEE 1.0-compliant profiles, which means a SLEE 1.0 -compliant profile specification can be deployed into the SLEE; and profile tables and profiles can be successfully created and managed.

Console command: `listprofilesbyindexedattribute`

Command

listprofilesbyindexedattribute <table-name> <attr-name> <attr-value>
[display-attributes (true/false)]
  Description
    List the profiles which have an indexed attribute <attr-name> equal to
    <attr-value>. The implementation supports only a limited set of attribute types
    that it can convert from strings to objects

Example

$ ./rhino-console listprofilesbyindexedattribute testtable indexedAttrib someValue
1 profiles returned
ProfileID[table=testtable,profile=testprofile]

MBean operation: `getProfilesByAttribute`

MBean

SLEE-defined

public Collection getProfilesByAttribute(String profileTableName, String attributeName, Object attributeValue)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          UnrecognizedAttributeException, InvalidArgumentException,
          AttributeTypeMismatchException, ManagementException;

Arguments

This operation requires that you specify the:

profileTableName — name of the profile table
attributeName — name of the profile’s attribute to check
attributeValue — value to compare the attribute with.

Notes

SLEE 1.1- & SLEE 1.0-specific commands

Between SLEE 1.0 and SLEE 1.1, the underlying profile specification schema changed significantly. According to the SLEE 1.1 Specification, profile attributes no longer have to be indexed to be legally used by a find-by-attribute-value query. Therefore, the getProfilesByAttribute operation can only be used for profiles and profile tables that are based on a SLEE 1.1-compliant profile specification. For running a find-by-attribute-value search on a SLEE 1.0-compliant profile table, use the getProfilesByIndexedAttribute operation.

Backwards compatibility

SLEE 1.1 demands backwards compatibility for SLEE 1.0-compliant profiles, which means a SLEE 1.0 compliant profile specification can be deployed into the SLEE; and profile tables and profiles can be successfully created and managed.

MBean operation: `getProfilesByIndexedAttribute`

MBean

Profile Management

SLEE-defined

public Collection getProfilesByIndexedAttribute(String profileTableName, String attributeName, Object attributeValue)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          UnrecognizedAttributeException, AttributeNotIndexedException,
          AttributeTypeMismatchException, ManagementException;

Arguments

This operation requires that you specify the:

profileTableName — name of the profile table
attributeName — name of the profile’s attribute to check
attributeValue — value to compare the attribute with.

Finding Profiles Using Static Queries

To retrieve all profiles match a static query (pre-defined in a profile table’s profile specification schema), use the following MBean operation.

The Rhino SLEE does not use a rhino-console command for this function.

MBean operation: `getProfilesByStaticQuery`

MBean

SLEE-defined

public Collection getProfilesByStaticQuery(String profileTableName, String queryName, Object[] parameters)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          UnrecognizedQueryNameException, InvalidArgumentException,
          AttributeTypeMismatchException, ManagementException;

Arguments

This operation requires that you specify the:

profileTableName — name of the profile table
queryName — name of a static query defined in the profile table’s profile specification deployment descriptor
parameters — an array of parameter values, to apply to parameters in the query (may only be null if the static query takes no arguments).

For more about static query methods, please see chapter 10.8.2 "Static query methods" in the SLEE 1.1 specification.

Finding Profiles Using Dynamic Queries

To retrieve all profiles match a dynamic query (an expression the administrator constructs at runtime) , use the following MBean operation.

The Rhino SLEE does not use a rhino-console command for this function.

MBean operation: `getProfilesByDynamicQuery`

MBean

SLEE-defined

public Collection getProfilesByDynamicQuery(String profileTableName, QueryExpression expr)
    throws NullPointerException, UnrecognizedProfileTableNameException,
        UnrecognizedAttributeException, AttributeTypeMismatchException,
        ManagementException;

Arguments

This operation requires that you specify the:

profileTableName — name of the profile table
expr — query expression to apply to profiles in the profile table.

For more about dynamic query methods, please see chapter 10.20.3 "Dynamic Profile queries" in the SLEE 1.1 specification.

Export and Import

Rhino includes procedures for:

Exporting Profiles
Importing Profiles

Exporting Profiles

To export SLEE profiles, use the following rhino-console command or related MBean operation.

Console command: `exportall`

The Rhino command console currently does not have a command specific to profile exports. Instead you use a more general export function, which (apart from SLEE profiles) also exports deployable units for services and RAs currently installed in the SLEE.

Command

exportall <zip|directory>
  Description
    Export the internal state of the SLEE including deployable units, profile
    tables, and other component state as an imperative-style configuration export.
    Uses JMX to export profiles. Use of the standalone rhino-export utility is
    encouraged for deployments involving large profile sets.

Example

$ ./rhino-console exportall /home/userXY/myexport
Exporting file:jars/incc-callbarring-service.jar...
Exporting file:jars/incc-callforwarding-service.jar...
Taking snapshot for callforwarding
Saving callforwarding.jar (183kb)
Streaming profile table 'callforwarding' snapshot to callforwarding.data (2 entries)
[################################################################################] 2/2 entries

Taking snapshot for callbarring
Saving callbarring.jar (177kb)
Streaming profile table 'callbarring' snapshot to callbarring.data (2 entries)
[################################################################################] 2/2 entries

Extracted 4 of 4 entries (157 bytes)
Snapshot timestamp 2008-05-07 15:17:42.325 (1210130262325)
   Critical region time     : 0.002 s
   Request preparation time : 0.053 s
   Data extraction time     : 0.302 s
   Total time               : 0.355 s

Converting 2 profile table snapshots...
Converting callforwarding...
bean class=class com.opencloud.deployed.Profile_Table_2.ProfileOCBB_Bean
[###########################################################################] converted 2 of 2
[###########################################################################] converted 2 of 2
Converted 2 records

Converting callbarring...
bean class=class com.opencloud.deployed.Profile_Table_1.ProfileOCBB_Bean
[###########################################################################] converted 2 of 2
[###########################################################################] converted 2 of 2
Converted 2 records
Export complete

Exported profile files

After the export, you will find the exported profiles as .xml files in the profiles subfolder of the chosen export directory (in the above example, /home/userXY/myexport/profiles).

Exporting "snapshots"

See also Profile Snapshots, to export profile snapshots in binary format and convert them into xml format for later imports.

Exporting a SLEE

See also Exporting a SLEE, to export all deployed components and configuration of a Rhino SLEE.

MBean operation: `exportProfiles`

MBean

Rhino extension

com.opencloud.rhino.management.profile.ProfileDataCollection exportProfiles(String profileTableName, String[] profileNames)
    throws NullPointerException, UnrecognizedProfileTableNameException,
    ManagementException;

Arguments

This operation requires that you specify the profile table’s:

profileTableName — name of the profile table to export from
profileNames — an array listing the profile names to export (elements corresponding to unknown profile names are ignored).

Exporting the default profile

To export the default profile, enter a null element in the profileNames array.

Importing Profiles

To import SLEE profiles, use the following rhino-console command or related MBean operation.

Console command: `importprofiles`

Use the importprofiles command to import profile data from an xml file that has previously been created (for example, using the exportall command).

Command

importprofiles <filename.xml> [-table table-name] [-replace] [-max
profiles-per-transaction] [-noverify]
  Description
    Import profiles from xml data

Example

$ ./rhino-console exportall /home/userXY/myexport
...
./rhino-console importprofiles /home/userXY/myexport/profiles/testtable.xml
Importing profiles into profile table: testtable
2 profile(s) processed: 1 created, 0 replaced, 0 removed, 1 skipped

Notes

Referenced profile table must exist

For the profile import to run successfully, the profile table the xml data refers to must exist before invoking the importprofiles command. (The importprofiles command will not create the profile table if it does not exist. Instead it will complete successfully — but without importing anything.)

MBean operation: `importProfiles`

MBean

Clearing individual alarms

Rhino extension

com.opencloud.rhino.management.profile.ProfileImportResult importProfiles(com.opencloud.rhino.management.profile.ProfileDataCollection profileData)
    throws NullPointerException, UnrecognizedProfileTableNameException,
          InvalidArgumentException, ProfileAlreadyExistsException,
          UnrecognizedProfileNameException, ManagementException;

Arguments

This operation requires that you specify the profile table’s:

profileData — the profile data collection containing the exported profiles.

Importing the default profile

To import the default profile, include a profile with a null name in the profile data collection.

Alarms

As well as an overview and list of alarms, this section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples and links to related javadocs.

Procedure rhino-console command MbBean → Operations

Viewing active alarms

 listactivealarms

Alarm → getAlarms
Alarm → getDescriptors

 clearalarm

Alarm → clearAlarm

Clearing alarms raised by a particular notification source

 clearalarms

Alarm → clearAlarms

Configuring alarm log period

 setalarmlogperiod
getalarmlogperiod

Logging Management → SetAlarmLogPeriod
Logging Management → GetAlarmLogPeriod

Creating and removing threshold-alarm rules

 createthresholdrule
removethresholdrule

Threshold Rule Management → createRule
Threshold Rule Management → removeRule

Listing all threshold-alarm rules

 listthresholdrules

Threshold Rule Management → getRules

Configuring threshold-alarm rules

Viewing a current threshold-alarm rule
Saving a threshold-alarm rule configuration to a file for editing
Importing a modified threshold-alarm rule
Configuring trigger conditions for a threshold-alarm rule (adding, getting, removing, getting and setting operators and periods)
Configuring reset conditions for a threshold-alarm rule (adding, getting, removing, getting and setting operators and periods)
Setting a threshold rule alarm
Getting a threshold rule alarm’s level, type and message

 getconfig
exportconfig
importconfig

Threshold Rule → addTriggerCondition
Threshold Rule → getTriggerConditions
Threshold Rule → removeTriggerCondition
Threshold Rule → getTriggerConditionsOperator
Threshold Rule → setTriggerConditionsOperator
Threshold Rule → getTriggerPeriod
Threshold Rule → setTriggerPeriod
Threshold Rule → addResetCondition
Threshold Rule → getResetConditions
Threshold Rule → removeResetCondition
Threshold Rule → getResetConditionsOperator
Threshold Rule → setResetConditionsOperator
Threshold Rule → getResetPeriod
Threshold Rule → setResetPeriod
Threshold Rule → setAlarm
Threshold Rule → getAlarmLevel
Threshold Rule → getAlarmType
Threshold Rule → getAlarmMessage

Activating and deactivating threshold-alarm rules

  activatethresholdrule

Threshold Rule → activateRule
Threshold Rule → deactivateRule

Setting and getting the threshold-alarm rule-scan periods

 getthresholdrulescanperiod
setthresholdrulescanperiod

Threshold Rule Management → getScanPeriod
Threshold Rule Management → setScanPeriod

About Alarms

Alarms in Rhino alert the SLEE administrator to exceptional conditions.

Application components in the SLEE raise them, as does Rhino itself (upon detecting an error condition). Rhino clears some alarms automatically when the error conditions are resolved. The SLEE administrator must clear others manually.

When an alarm is raised or cleared, Rhino generates a JMX notification from the Alarm MBean. Management clients may attach a notification listener to the Alarm MBean, to receive alarm notifications. Rhino logs all alarm notifications.

What’s new in SLEE 1.1?

While only SBBs could generate alarms in SLEE 1.0, other types of application components can also generate alarms in SLEE 1.1.

In SLEE 1.1, alarms are stateful — between being raised and cleared, an alarm persists in the SLEE, where an administrator may examine it. (In SLEE 1.0, alarms could be generated with a severity level that indicated a cleared alarm, but the fact that an error condition had occurred did not persist in the SLEE beyond the initial alarm generation.)

Sample log file messages

SLEE 1.1

SLEE 1.0

For both SLEE 1.1 and 1.0, if the cause of an alarm is an Java exception, the log includes the exception and its stack trace (following the alarm description message).

Configuring alarm log period

To set and get the interval between periodic active alarm logs, use the following rhino-console commands or related MBean operations.

Rhino periodically logs active alarms and the default interval is 60 seconds.

setalarmlogperiod

Command

setalarmlogperiod <seconds>
  Description
    Sets the interval between periodic active alarm logs.
  Required Arguments
    seconds  The interval between periodic alarm logs. Setting to 0 will disable
    logging of periodic alarms.

Example

To set log period to 30 seconds:

$ ./rhino-console setalarmlogperiod 30
  Active alarm logging period set to 30 seconds.

getalarmlogperiod

Command

getalarmlogperiod
  Description
    Returns the interval between periodic active alarm logs.

Example

To get alarm log period:

$ ./rhino-console getalarmlogperiod
  Active alarm logging period is currently 30 seconds.

MBean operations: `setAlarmLogPeriod`

MBean

Logging Management MBean

SLEE-defined

Set the interval between periodic active alarm logs

public void setAlarmLogPeriod(int period) throws IllegalArgumentException, ConfigurationException;

Sets the interval between periodic active alarm logs. Setting the period to 0 will disable periodic alarm logging.

Get the interval between periodic active alarm logs

public int getAlarmLogPeriod() throws ConfigurationException;

Returns the interval between periodic active alarm logs.

Viewing Active Alarms

To view active alarms, use the following rhino-console command or related MBean operation.

Console command: `listactivealarms`

Command

listactivealarms [<type> <notif-source>] [-stack]
  Description
    List the alarms currently active in the SLEE (for a specific notification if
    provided).  Use -stack to display stack traces for alarm cause exceptions.

Example

To list all active alarms in the SLEE:

$ ./rhino-console listactivealarms
1 alarm:

Alarm 101:193215480667648 [diameter.peer.connectiondown]
  Level      : Warning
  InstanceID : diameter.peer.hss-instance
  Source     : (RA Entity) sh-cache-ra
  Timestamp  : 20161019 14:02:58 (active 15m 30s)
  Message    : Connection to hss-instance:3868 is down

The number value on the first line "101:193215480667648" is the alarmid.

The value in the square brackets "diameter.peer.connectiondown" is the alarm-type.

MBean operations: `getAlarms` and `getDescriptors`

MBean

Alarm MBean

SLEE-defined

Get identifiers of all active alarms in the SLEE

public String[] getAlarms()
    throws ManagementException;

Rhino’s implementation of the SLEE-defined getAlarms operation returns an array containing the identifiers of all alarms currently raised in the SLEE, regardless of which cluster node the alarm was raised on.

Get identifiers of active alarms raised by a specific notification source

public String[] getAlarms(NotificationSource notificationSource)
    throws NullPointerException, UnrecognizedNotificationSourceException,
           ManagementException;

This variant of getAlarms returns an array containing the identifiers of the current alarms that were raised by the specified notification source, on any node in the cluster. If there are currently no active alarms raised by this notification source, the operation returns a zero-length array.

Get alarm descriptor for an alarm identifier

public Alarm[] getDescriptors(String[] alarmIDs)
    throws NullPointerException, ManagementException;

This operation returns the alarm descriptor for each given alarm identifier.

Clearing Alarms

Rhino includes operations to:

Clear Individual Alarms
Clear Alarms Raised by a Particular Notification Source

Clear Individual Alarms

To clear an alarm using its alarm identifier, use the following rhino-console command or related MBean operation.

Console command: `clearalarm` console command

Command

clearalarm <alarmid>
  Description
    Clear the specified alarm.

Example

To clear the alarm with the identifier 101:102916243593:1:

$ ./rhino-console clearalarm 101:102916243593:1
Alarm 101:102916243593:1 cleared

MBean operation: `clearAlarm`

MBean

Alarm MBean

SLEE-defined

public boolean clearAlarm(String alarmID)
  throws NullPointerException, ManagementException;

Rhino’s implementation of the SLEE-defined clearAlarm operation clears the alarm with the given identifier, regardless of the cluster node the alarm was raised on. It returns a value of either true or false, depending on whether or not the SLEE found and cleared the alarm.

Clear Alarms Raised by a Particular Notification Source

To clear alarms raised by a particular notification source, use the following rhino-console command or related MBean operation.

Console command: `clearalarms`

Command

clearalarms <type> <notification-source> [<alarm type>]
  Description
    Clear all alarms raised by the notification source (of the specified alarm type)

This command clears all alarms of the specified alarm type (or all alarms if no alarm-type is specified), that have been raised by the specified notification source.

Example

To clear all alarms raised by a resource adaptor entity named insis-cap:

$ ./rhino-console clearalarms resourceadaptorentity insis-cap
2 alarms cleared

To clear only "noconnection" alarms raised by the resource adaptor entity named insis-cap:

$ ./rhino-console clearalarms resourceadaptorentity insis-cap noconnection
1 alarm cleared

MBean operation: `clearAlarms`

MBean

Alarm MBean

SLEE-defined

Clear all active alarms raised by a notification source

public int clearAlarms(NotificationSource notificationSource)
    throws NullPointerException, UnrecognizedNotificationSourceException,
        ManagementException

Rhino’s implementation of the SLEE-defined clearAlarms operation clears all active alarms that were raised by the given notification source on any cluster node. It returns the number of alarms that were cleared.

Clear active alarms of a specified type raised by a notification source

public int clearAlarms(NotificationSource notificationSource, String alarmType)
    throws NullPointerException, UnrecognizedNotificationSourceException,
        ManagementException;

This variant of clearAlarms clears only active alarms of the given alarm type raised by the given notification source on any cluster node. It also returns the number of alarms that were cleared.

Threshold Alarms

To supplement standard alarms (which Rhino and installed components raise), an administrator may configure custom alarms (which Rhino will raise or clear automatically based on SLEE Statistics.

These are known as threshold alarms, and you manage them using the Threshold Rule Management MBean.

Threshold rules

You configure the threshold rules governing each threshold alarm using a Threshold Rule MBean.

Each threshold rule consists of:

a unique name identifying the rule
one or more trigger conditions
an alarm level, type and message text
and optionally:
- one or more reset conditions
- how long (in milliseconds) the trigger conditions must remain before Rhino raises the alarm
- how long (in milliseconds) the reset conditions must remain before Rhino clears the alarm.

You can combine condition sets using either an AND or an OR operator. (AND means all conditions must be satisfied, whereas OR means any one of the conditions may be satisfied — to raise or clear the alarm.)

Parameter sets

Threshold rules use the same parameter sets as the statistics client. You can discover them either by using the statistics client graphically or by using its command-line mode from a command shell as shown below.

To list all available parameter sets:

$ ./rhino-stats -l
The following root parameter sets are available for monitoring:
Activities, ActivityHandler, ByteArrayBuffers, CGIN, DatabaseQuery, Diameter,
EndpointLimiting, EventRouter, Events, HTTP, JDBCDatasource, JVM, LicenseAccounting,
Limiters, LockManagers, MemDB-Local, MemDB-Replicated, MemDB-Timestamp, Metrics,
ObjectPools, SIP, SIS-SIP, SLEE-Usage, Services, StagingThreads, StagingThreads-Misc,
TimerFacility, TransactionManager

For parameter set type descriptions and a list of available parameter sets use the
-l <root parameter set name> option

To list the statistics collected by the JVM parameter set:

$ ./rhino-stats -l JVM
Connecting to localhost:1199
Parameter Set: JVM
Parameter Set Type: JVM
Description: JVM Statistics

Counter type statistics:
  Id: Name:                 Label:      Description:
  0   heapUsed              husd        Used heap memory
  1   heapCommitted         hcomm       Committed heap memory
  2   heapInitial           hinit       Initial heap memory
  3   heapMaximum           hmax        Maximum heap memory
  4   nonHeapUsed           nhusd       Used non-heap memory
  5   nonHeapCommitted      nhcomm      Committed non-heap memory
  6   nonHeapInitial        nhinit      Initial non-heap memory
  7   nonHeapMaximum        nhmax       Maximum non-heap memory
  8   classesCurrentLoaded  cLoad       Number of classes currently loaded
  9   classesTotalLoaded    cTotLoad    Total number of classes loaded since JVM start
  10  classesTotalUnloaded  cTotUnload  Total number of classes unloaded since JVM start

Sample type statistics: (none defined)

Found 1 parameter sets under 'JVM':
    ->  "JVM"

How Rhino evaluates threshold rules

Rhino periodically evaluates the trigger conditions of each configured rule. When a trigger condition is satisfied and its trigger period has been met or exceeded, Rhino raises the corresponding alarm. If the rule has reset conditions, Rhino evaluates those too, and when the reset condition is satisfied and the reset trigger period has been met or exceeded, clears the alarm. If the rule does not have reset conditions, an administrator must clear the alarm manually.

You can configure the frequency of threshold alarm rule evaluation, using the Threshold Rule Management MBean. An administrator can specify a polling frequency in milliseconds, or enter 0 to disable rule evaluation. The Rhino default is 0 (which must be changed to enable threshold-rule evaluation). Ideal polling frequency is highly dependent on the nature of alarms configured.

Simple and relative rule conditions

There are two types of threshold rule conditions, explained in the tables below.

Simple rule conditions

What it compares Operators for comparison Conditions Example

The value of a counter-type Rhino statistic against a constant value.

 >, >=, <, <=, ==, !=

The constant value to compare against may be any floating-point number. The condition can either compare against the absolute value of the statistic (suitable for gauge-type statistics), or against the observed difference between successive samples (suitable for pure counter-type statistics).

A condition that selects the statistic rolledBack from the Transactions parameter set, and evaluates to true when the number of transactions rolled back is > 100.

Relative rule conditions

What it compares Operators for comparison Conditions Example

The ratio between two monitored statistics against a constant value.

 >, >=, <, <=, ==, !=

The constant value to compare against may be any floating-point number.

A condition that selects the statistics freeMemory and totalMemory from the SystemInfo parameter set, and evaluates to true when free heap space is less than 20% of total heap space. (Using the < operator and a constant value of 0.2, the condition would evaluate to true when the value of freeMemory / totalMemory is less than 0.2.)

For definitions of counter, guage and sample type statistics, see About Rhino Statistics.

Creating and Removing Rules

To create or remove a threshold-alarm rule, use the following rhino-console commands or related MBean operations.

Console command: `createthresholdrule`

Command

createthresholdrule <name>
  Description
    Create a threshold alarm rule

Example

To create a rule named "low memory":

$ ./rhino-console createthresholdrule "low memory"
Threshold rule low memory created

MBean operation: `createRule`

MBean

Rhino operation

public ObjectName createRule(String ruleName)
    throws ConfigurationException, ValidationException;

This operation creates a rule with the name given, and returns the JMX object name of a Threshold Rule MBean (which you can use to configure that rule).

Console command: `removethresholdrule`

Command

removethresholdrule <name>
  Description
    Remove a threshold alarm rule

Example

To remove a rule named "low memory":

$ ./rhino-console removethresholdrule "low memory"
Threshold rule low memory removed

MBean operation: `removeRule`

MBean

Example

public void removeRule(String ruleName)
    throws ConfigurationException, ValidationException;

This operation removes the rule with the name given.

Listing Rules

To list all threshold alarm rules, use the following rhino-console command or related MBean operation.

Console command: `listthresholdrules`

Command

listthresholdrules
  Description
    List threshold alarm rules

Example

To list all threshold alarm rules, with their activation states:

$ ./rhino-console listthresholdrules
Current threshold rules:
    low memory (active)
    low disk (inactive)
    testrule (inactive)

MBean operation: `getRules`

MBean

addTriggerCondition
getTriggerConditions
removeTriggerCondition
getTriggerConditionsOperator
setTriggerConditionsOperator
getTriggerPeriod
setTriggerPeriod

Rhino operation

public String[] getRules()
    throws ConfigurationException;

Configuring Rules

To configure a threshold alarm rule:

use the following rhino-console commands to view available rules, export a rule to an XML file, edit the rule file, and then re-import the edited file into the SLEE
or use Threshold Rule MBean operations.

View rules

To view a current threshold alarm rule., use the getconfig console command:

Command

getconfig [-namespace] <configuration type> [configuration key]
  Description
    Extract and display content of a container configuration key.  The optional
    -namespace argument must be used to get the config of a namespace-specific key.
    If no key is specified the configs of all keys of the given type are shown

Example

To display the threshold alarm rule named "low_memory":

$ ./rhino-console getconfig threshold-rules "rule/low_memory"
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rhino-threshold-rules-config PUBLIC "-//Open Cloud Ltd.//DTD Rhino Threshold Rules Config 2.3//EN" "rhino-threshold-rules-config-2.3.dtd">
<rhino-threshold-rules-config config-version="2.3" rhino-version="Rhino (version='2.5', release='1.0', build=xxx, revision=xxx)" timestamp="xxx">
    <!-- Generated Rhino configuration file: xxxx-xx-xx xx:xx:xx.xxx -->
    <threshold-rules active="false" name="low_memory">
        <trigger-conditions name="Trigger conditions" operator="OR" period="0">
            <relative-threshold operator="&lt;=" value="0.2">
                <first-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="freePhysicalMemorySize"/>
                <second-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="totalPhysicalMemorySize"/>
            </relative-threshold>
        </trigger-conditions>
        <reset-conditions name="Reset conditions" operator="OR" period="0"/>
        <trigger-actions>
            <raise-alarm-action level="Major" message="Low on memory" type="MEMORY"/>
        </trigger-actions>
        <reset-actions>
            <clear-raised-alarm-action/>
        </reset-actions>
    </threshold-rules>
</rhino-threshold-rules-config>

Export rules

To save a threshold rule configuration to a file for editing, use the exportconfig console command:

Command

exportconfig [-namespace] <configuration type> [configuration key] <filename>
  Description
    Extract content of a container configuration key and save it to a file.  The
    optional -namespace argument must be used to export the config of a
    namespace-specific key

Example

To export the threshold alarm rule named "low memory" to the file rule_low_memory.xml:

$ ./rhino-console exportconfig threshold-rules "rule/low memory" rule_low_memory.xml
Export threshold-rules: (rule/low memory) to rule_low_memory.xml
Wrote rule_low_memory.xml

The structure of the exported data in the XML file is identical to that displayed by the getconfig command.

Edit rules

You can modify a rule using a text editor. In the following example, a reset condition has been added to a rule previously exported, so that the alarm raised will automatically clear when free memory becomes greater than 30%. (Previously the reset-conditions element in this rule had no conditions.)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rhino-threshold-rule PUBLIC "-//Open Cloud Ltd.//DTD Rhino Threshold Rule 2.0//EN"
  "http://www.opencloud.com/dtd/rhino-config-threshold-rules_2_0.dtd">
<rhino-threshold-rule config-version="2.3" rhino-version="Rhino (version='2.5', release='00',
  build=xxx, revision=xxx)" timestamp=xxx>
    <!-- Generated Rhino configuration file: xxxx-xx-xx xx:xx:xx.xxx -->
    <threshold-rules active="false" name="low memory">
        <trigger-conditions name="Trigger conditions" operator="OR" period="0">
            <relative-threshold operator="&lt;=" value="0.2">
                <first-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="freePhysicalMemorySize"/>
                <second-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="totalPhysicalMemorySize"/>
            </relative-threshold>
        </trigger-conditions>
        <reset-conditions name="Reset conditions" operator="OR" period="0">
            <relative-threshold operator="&gt;" value="0.3">
                <first-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="freePhysicalMemorySize"/>
                <second-statistic calculate-delta="false" parameter-set="JVM.OperatingSystem" statistic="totalPhysicalMemorySize"/>
            </relative-threshold>
        </reset-conditions>
        <trigger-actions>
            <raise-alarm-action level="Major" message="Low on memory" type="MEMORY"/>
        </trigger-actions>
        <reset-actions>
            <clear-raised-alarm-action/>
        </reset-actions>
    </threshold-rules>
</rhino-threshold-rule>

Import rules

To import the modified threshold alarm rule file, use the importconfig console command:

Command

importconfig [-namespace] <configuration type> <filename> [-replace]
  Description
    Import a container configuration key.  The optional -namespace argument must be
    used to import a config for a namespace-specific key

Example

To import the threshold alarm rule from the file rule_low_memory.xml:

$ ./rhino-console importconfig threshold-rules rule_low_memory.xml -replace
Configuration successfully imported.

The -replace option is required when importing a rule with the same name as an existing rule, as there can be only one rule configuration with a given name present at any one time.

Threshold Rule MBean Operations

To configure a threshold alarm rule, use the following MBean operations (defined on the Threshold Rule MBean interface), for:

adding, removing and getting trigger conditions, and getting and setting their operators and periods
adding, removing and getting reset conditions, and getting and setting their operators and periods
setting the alarm
getting an alarm’s level, type, and message.

Trigger conditions

To add, remove and get threshold alarm trigger conditions, and get and set their operators and periods, use the following MBean operations:

Operations Usage

To add a trigger condition to the rule:

public void addTriggerCondition(String parameterSetName, String statistic, String operator, double value)
    throws ConfigurationException, UnknownStatsParameterSetException,
          UnrecognizedStatisticException, ValidationException;
public void addTriggerCondition(String parameterSetName1, String statistic1, String parameterSetName2,
              String statistic2, String operator, double value
    throws ConfigurationException, UnknownStatsParameterSetException,
          UnrecognizedStatisticException, ValidationException;

The first operation adds a simple trigger condition to the rule. The second operation adds a relative condition between two parameter set statistics (see Simple and relative rule conditions).

To get the current trigger conditions:

public String[] getTriggerConditions()
    throws ConfigurationException;

To remove a trigger condition:

public void removeTriggerCondition(String key)
    throws ConfigurationException, ValidationException;

To get or set the trigger condition operator:

public String getTriggerConditionsOperator()
    throws ConfigurationException;
public void setTriggerConditionsOperator(String operator)
    throws ConfigurationException, ValidationException;

The operator must be one of the logical operators AND or OR (the operator is ignored if the rule has only one trigger condition).

To get or set the trigger condition period:

public long getTriggerPeriod()
    throws ConfigurationException;
public void setTriggerPeriod(long period)
    throws ConfigurationException, ValidationException;

The trigger period is measured in milliseconds. If it is 0, the SLEE raises the alarm whenever the trigger conditions are true (and the alarm is not already raised). Otherwise, the SLEE raises the alarm once the trigger conditions have held true for at least the amount of time specified.

Reset conditions

To add, remove and get threshold alarm reset conditions, and get and set their operators and periods, use the following MBean operations:

Operations Usage

addResetCondition
getResetConditions
removeResetCondition
getResetConditionsOperator
setResetConditionsOperator
getResetPeriod
setResetPeriod

To add a reset condition to the rule:

public void addResetCondition(String parameterSetName, String statistic, String operator, double value)
    throws ConfigurationException, UnknownStatsParameterSetException,
           UnrecognizedStatisticException, ValidationException;
public void addResetCondition(String parameterSetName1, String statistic1, String parameterSetName2,
                              String statistic2, String operator, double value)
    throws ConfigurationException, UnknownStatsParameterSetException,
           UnrecognizedStatisticException, ValidationException;

The first operation adds a simple reset condition to the rule. The second operation adds a relative condition between two parameter set statistics (see bxfref:threshold-alarms[Simple and relative rule conditions]).

To get the current reset conditions:

public String[] getResetConditions()
    throws ConfigurationException;

To remove a reset condition:

public void removeResetCondition(String key)
    throws ConfigurationException, ValidationException;

To get or set the reset condition operator:

public String getResetConditionsOperator()
    throws ConfigurationException;
public void setResetConditionsOperator(String operator)
    throws ConfigurationException, ValidationException;

The operator must be one of the logical operators AND or OR (the operator is ignored if the rule has only one reset condition).

To get or set the reset condition period:

public long getResetPeriod()
    throws ConfigurationException;
public void setResetPeriod(long period)
    throws ConfigurationException, ValidationException;

The reset period is measured in milliseconds. If it is 0, the SLEE clears the alarm whenever the reset conditions are true (and the alarm is raised). Otherwise, the SLEE clears the alarm once the reset conditions have held true for at least the amount of time specified.

Setting alarms

To set the alarm to be raised by a threshold rule, use the following MBean operation:

Operations Usage

setAlarm

public void setAlarm(AlarmLevel level, String type, String message)
    throws ConfigurationException, ValidationException;

The alarm level may be any level other than AlarmLevel.CLEAR

Getting alarm information

To get a threshold alarm’s level, type, and message, use the following MBean operations:

Operations Usage

getAlarmLevel
getAlarmType
getAlarmMessage

public AlarmLevel getAlarmLevel()
    throws ConfigurationException;
public String getAlarmType()
    throws ConfigurationException;
public String getAlarmMessage()
    throws ConfigurationException;

Activating and Deactivating Rules

To activate or deactivate a threshold-alarm rule, use the following rhino-console commands or related MBean operations.

Activate Rules

Console command: `activatethresholdrule`

Command

activatethresholdrule <name>
  Description
    Activate a threshold alarm rule

Example

To activate the rule with the name "low memory":

$ ./rhino-console activatethresholdrule "low memory"
Threshold rule low memory activated

You can also activate a rule by exporting it, modifying the XML, and then reimporting it (assuming the active parameter in the rule is set to true — see Configuring Rules).

MBean operation: `activateRule`

MBean

ThresholdRuleMBean

Rhino operation

public void activateRule()
    throws ConfigurationException;

This operation activates the threshold-alarm rule represented by the ThresholdRuleMBean.

threshold rule scan period must be configured to a non-zero value before Rhino will evaluate active threshold-alarm rules.

Deactivate rules

Console command: `deactivatethresholdrule`

Command

deactivatethresholdrule <name>
  Description
    Deactivate a threshold alarm rule

Example

To deactivate the rule with the name "low memory":

$ ./rhino-console deactivatethresholdrule "low memory"
Threshold rule low memory deactivated

MBean operation: `deactivateRule`

MBean

ThresholdRuleMBean

Rhino operation

public void deactivateRule()
    throws ConfigurationException;

This operation deactivates the threshold-alarm rule represented by the ThresholdRuleMBean.

Setting and Getting Rule-Scan Periods

To set or get the threshold rule scan period, use the following rhino-console commands or MBean operations.

What is a rule-scan period?

A threshold-alarm rule-scan period determines when Rhino’s threshold-rule scanner evaluates active threshold-alarm rules.

The scan period must be set to a valid non-zero value for Rhino to evaluate the rules. At the beginning of each scan period, Rhino evaluates each active threshold-alarm rule as follows:

If the rule’s trigger condition is true and the trigger period is 0, the rule triggers and raises its alarm.
The first time the threshold-rule scanner finds a rule’s trigger condition to be true, with a trigger period greater than 0, it records the time. Thereafter, when it evaluates the rule, if the trigger condition continues to be true, when the accumulated time exceeds the rule’s trigger period, the rule triggers and raises its alarm. (If the rule evaluates to false at any time, the rule scanner discards any accumulated time from while it was true.)

(The same process applies to the reset conditions once a rule has been triggered.)

Console command: `setthresholdrulescanperiod`

Command

setthresholdrulescanperiod <period>
  Description
    Set the threshold alarm rule scan period, measured in ms.  Must be > 500 or 0 to
    disable rule checking

Example

To set the threshold rule scan period to 30000ms (30s):

$ ./rhino-console  setthresholdrulescanperiod 30000
Threshold rule scan period set to 30000ms

To disable threshold rule scanning:

$ ./rhino-console  setthresholdrulescanperiod 0
Threshold rule scanning disabled

MBean operation: `setScanPeriod`

MBean

Rhino operation

public void setScanPeriod(int scanPeriod)
    throws ConfigurationException, ValidationException;

The scan period is measured in milliseconds.

Console command: `getthresholdrulescanperiod`

Command

getthresholdrulescanperiod
  Description
    Get the threshold alarm rule scan period

Example

$ ./rhino-console  getthresholdrulescanperiod
Threshold rule scan period set to 30000ms

MBean operation: `getScanPeriod`

MBean

rhino.cassandra-kv-store.no-nodes-available

Rhino operation

public int getScanPeriod()
  throws ConfigurationException;

The scan period is measured in milliseconds.

Runtime Alarm List

To list all alarms that may be raised by Rhino and installed components (including their messages, and when raised and cleared), use the following rhino-console command.

Console command: `alarmcatalog`

Command

alarmcatalog [-v]
  Description
    List the alarms that may be raised by Rhino and installed components. Using the
    -v flag will display more detail.

Example

$ ./rhino-console alarmcatalog

  Rhino Alarms
  ============

    Source                 Category                  Level     Alarm Type and Message
    ------                 --------                  -----     ----------------------
    Abnormal Execution     AbnormalExecution         WARNING   rhino.uncaught-exception  "Uncaught exception thrown by thread %s: %s"
    Activity Handler       Activity Handler          WARNING   rhino.ah.snapshot-age  "Oldest activity handler snapshot is older than %s, snapshot is %s (from %d), creating thread: %s"
    Cluster State          Clustering                MAJOR     rhino.node-failure  "Node %d has left the cluster"

... edited for brevity ...

And this displays more detail:

$ ./rhino-console alarmcatalog -v
Rhino Alarms
============

Source: Abnormal Execution

  Category: AbnormalExecution (Alarms raised as a result of an abnormal execution condition being detected)

    Alarm Type:  rhino.uncaught-exception
    Level:       WARNING
    Message:     "Uncaught exception thrown by thread %s: %s"
    Description: An uncaught exception has been detected.
    Raised:      When an uncaught exception has been thrown.
    Cleared:     Never, must be cleared manually or Rhino restarted with the source of the uncaught exception corrected.


Source: Activity Handler

  Category: Activity Handler (Alarms raised by Rhino activity handler)

    Alarm Type:  rhino.ah.snapshot-age
    Level:       WARNING
    Message:     "Oldest activity handler snapshot is older than %s, snapshot is %s (from %d), creating thread: %s"
    Description: The oldest activity handler snapshot is too old.
    Raised:      When the age of the oldest activity handler snapshot is greater than the threshold set by the rhino.ah.snapshot_age_warn system property (30s default).
    Cleared:     When the age of the oldest snapshot is less than or equal to the threshold.


Source: Cluster State

  Category: Clustering (Alarms raised by Rhino cluster state changes)

    Alarm Type:  rhino.node-failure
    Level:       MAJOR
    Message:     "Node %d has left the cluster"
    Description: A node left the cluster for some reason other than a management-initiated shutdown.
    Raised:      When the cluster state listener detects a node has left the cluster unexpectedly.
    Cleared:     When the failed node returns to the cluster.

... edited for brevity ...

Rhino Alarm List

This is a list of all alarms raised by this version of Rhino. For the management command that lists all alarms that may be raised by Rhino and installed components see Runtime Alarm List.

Alarm Type

Description

Category: AbnormalExecution (Alarms raised as a result of an abnormal execution condition being detected)

rhino.uncaught-exception

An uncaught exception has been detected.

Category: Activity Handler (Alarms raised by Rhino activity handler)

rhino.ah.snapshot-age

The oldest activity handler snapshot is too old.

Category: Cassandra Key/Value Store (Alarms raised by the Cassandra key/value store)

All database nodes for all persistence instances have failed or are otherwise unreachable.

rhino.cassandra-kv-store.connection-error

The local database driver cannot connect to the configured persistence instance.

rhino.cassandra-kv-store.db-node-failure

The local database driver cannot connect to a database node.

rhino.cassandra-kv-store.pending-size-limit-reached

The volume of committed but not yet persisted application state has exceeded the configured pending size limit threshold. State generated for new transactions will be ignored by the key/value store and not buffered for persisting until sufficient state has been persisted to reduce the pending size volume below the limit again

rhino.cassandra-kv-store.scan-persist-time-threshold-reached

The allowed pending transaction scan or persist time has exceeded the configured thresholds due to overload. State generated for new transactions will be ignored by the key/value store and not buffered for persisting until sufficient state has been persisted to reduce the load on the pending transaction scanner

Category: Cassandra Session Ownership Store (Alarms raised by the Cassandra session ownership store)

rhino.cassandra-session-ownership-store.no-nodes-available

All database nodes for all persistence instances have failed or are otherwise unreachable.

rhino.cassandra-session-ownership-store.connection-error

The local database driver cannot connect to the configured persistence instance.

rhino.cassandra-session-ownership-store.db-node-failure

The local database driver cannot connect to a database node.

Category: Cluster Clock Synchronisation (Alarms raised by the cluster clock synchronisation monitor)

rhino.monitoring.clocksync

Another cluster node is reporting a system clock deviation relative to the local node beyond the maximum permitted threshold. The status of external processes maintaining the system clock on that node (eg. NTP) should be checked.

Category: Clustering (Alarms raised by Rhino cluster state changes)

rhino.node-failure

A node left the cluster for some reason other than a management-initiated shutdown.

Category: Configuration Management (Alarms raised by the Rhino configuration manager)

rhino.config.save-error

An error occurred while trying to write the file-based configuration for the configuration type specified in the alarm instance.

rhino.config.read-error

An error occurred while trying to read the file-based configuration for the configuration type specified in the alarm instance. Rhino will use defaults from defaults.xml, move the broken configuration aside, and overwrite the config file.

rhino.config.activation-failure

An error occurred while trying to activate the file-based configuration for the configuration type specified in the alarm instance. Rhino will use defaults from defaults.xml, move the broken configuration aside, and overwrite the config file.

Category: Database (Alarms raised during database communications)

rhino.database.no-persistence-config

A persistence resource configuration referenced in rhino-config.xml has been removed at runtime.

rhino.database.no-persistence-instances

A persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.

rhino.database.persistence-instance-instantiation-failure

Rhino requires a backing database for persistence of state for failure recovery purposes. A persistent instance defines a connection to a database backend. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.

rhino.database.connection-lost

Rhino requires a backing database for persistence of state for failure recovery purposes. If no connection to the database backend is available, state cannot be persisted.

rhino.jdbc.persistence-instance-instantiation-failure

A persistent instance defines the connection to the database backend. If the persistent instance cannot be instantiated then JDBC connections cannot be made.

Category: Event Router State (Alarms raised by event router state management)

rhino.state.unlicensed-slee

A licensing problem was detected during SLEE start.

rhino.state.unlicensed-service

A licensing problem was detected during service activation.

rhino.state.unlicensed-raentity

A licensing problem was detected during resource adaptor entity activation.

rhino.state.convergence-failure

A component reported an unexpected error during convergence

rhino.state.convergence-timeout

A component has not transitioned to the effective desired state after the timeout period

rhino.state.raentity.active-reconfiguration

A resource adaptor entity is of a type that does not support active reconfiguration but has a desired state that contains configuration properties different from those in the actual state

Category: GroupRMI (Alarms raised by the GroupRMI server)

rhino.group-rmi.dangling-transaction

A group RMI invocation completed without committing or rolling back a transaction that it started. The dangling transaction will be automatically rolled back by the group RMI server to prevent future issues but these occurrences are software bugs that should be reported.

Category: Key/Value Store (Alarms raised by key/value store persistence resource managers)

rhino.kv-store.no-persistence-config

A persistence resource configuration referenced in rhino-config.xml has been removed at runtime.

rhino.kv-store.no-persistence-instances

A persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.

rhino.kv-store.persistence-instance-instantiation-failure

A persistence instance used by a key/value store cannot be instantiated. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.

Category: Licensing (Alarms raised by Rhino licensing)

rhino.license.over-limit

Rate limiter throttling is active. This throttling and hence this alarm only happens in SDK versions of Rhino, not production versions.

rhino.license.expired

A license installed in Rhino has passed its expiry time.

rhino.license.pending-expiry

A license installed in Rhino is within seven days of its expiry time.

rhino.license.partially-licensed-host

The hardware addresses listed in a host-based license only partially match those on the host.

rhino.license.unlicensed-host

The hardware addresses listed in a host-based license do not match those on the host.

rhino.license.unlicensed-rhino

Rhino does not have a valid license installed.

rhino.license.over-licensed-capacity

The work done by a function exceeds licensed capacity.

rhino.license.unlicensed-function

A particular function is not licensed.

Category: Limiting (Alarms raised by Rhino limiting)

rhino.limiting.below-negative-capacity

A rate limiter is below negative capacity.

rhino.limiting.stat-limiter-misconfigured

A stat limiter is misconfigured.

Category: Logging (Alarms raised by Rhino logging)

rhino.logging.appender-error

An appender has thrown an exception when attempting to pass log messages from a logger to it.

Category: M-lets Startup (Alarms raised by the M-let starter)

rhino.mlet.loader-failure

The M-Let starter component could not register itself with the platform MBean server. This normally indicates a serious JVM misconfiguration.

rhino.mlet.registration-failure

The M-Let starter component could not register an MBean for a configured m-let. This normally indicates an error in the m-let configuration file.

Category: REM Startup (Alarms raised by embedded REM starter)

rhino.rem.missing

This version of Rhino is supposed to contain an embedded instance of REM but it was not found, most likely due to a packaging error.

rhino.rem.startup

There was an unexpected problem while starting the embedded REM. This could be because of a port conflict or packaging problem.

Category: Remote Timer Server (Alarms raised by the remote timer server)

rhino.remote-timer-server.bind-error

The remote timer server is unable to bind a socket listener within the configured range of ports. The Rhino configuration may need to be checked and an alternate or wider range of ports allowed.

Category: Runtime Environment (Alarms related to the runtime environment)

rhino.runtime.unsupported.jvm

This JVM is not a supported JVM.

rhino.runtime.slee

SLEE event-routing functions failed to start after node restart

rhino.runtime.long-filenames-unsupported

Filenames with the maximum length expected by Rhino are unsupported on this filesystem. Unexpected deployment errors may occur as a result

Category: SAS facility (Alarms raised by Rhino SAS Facility)

rhino.sas.connection.lost

Attempting to reconect to SAS server

rhino.sas.queue.full

SAS message queue is full. Some events have not been reported to SAS

Category: SLEE State (Alarms raised by SLEE state management)

rhino.state.slee-start

An unexpected exception was caught during SLEE start.

Category: SNMP (Alarms raised by Rhino SNMP)

rhino.snmp.no-bind-addresses

The SNMP agent listens for requests received on all network interfaces that match the requested SNMP configuration. If no suitable interfaces can be found that match the requested configuration, then the SNMP agent cannot process any SNMP requests.

rhino.snmp.bind-failure

The SNMP agent attempts to bind a UDP port on each configured SNMP interface to receive requests. If no ports could be bound, the SNMP agent cannot process any SNMP requests.

rhino.snmp.partial-failure

The SNMP agent attempts to bind a UDP port on each configured SNMP interface to receive requests. If this succeeds on some (but not all) interfaces, the SNMP agent can only process requests received via the interfaces that succeeded.

rhino.snmp.general-failure

This is a catchall alarm for unexpected failures during agent startup. If an unexpected failure occurs, the state of the SNMP agent is unpredictable and requests may not be successfully processed.

rhino.snmp.notification-address-failure

This alarm represents a failure to determine an address from the notification target configuration. This can occur if the notification hostname is not resolvable, or if the specified hostname is not parseable.

rhino.snmp.duplicate-oid-mapping

Multiple parameter set type configurations for in-use parameter set types map to the same OID. All parameter set type mappings will remain inactive until the conflict is resolved.

rhino.snmp.duplicate-counter-mapping

Multiple counters in the parameter set type configuration map to the same index. The parameter set type mappings will remain inactive until the conflict is resolved.

Category: Scattercast Management (Alarms raised by Rhino scattercast management operations)

rhino.scattercast.update-reboot-required

Reboot needed to make scattercast update active.

Category: Service State (Alarms raised by service state management)

rhino.state.service-activation

The service threw an exception during service activation, or an unexpected exception occurred while attempting to activate the service.

Category: Session Ownership Store (Alarms raised by session ownership store persistence resource managers)

rhino.session-ownership-store.no-persistence-config

The persistence resource configuration referenced in rhino-config.xml has been removed at runtime.

rhino.session-ownership-store.no-persistence-instances

The persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.

rhino.session-ownership-store.persistence-instance-instantiation-failure

A persistence instance used by the session ownership store cannot be instantiated. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.

Category: Threshold Rules (Alarms raised by the threshold alarm rule processor)

rhino.threshold-rules.rule-failure

A threshold rule trigger or reset rule failed.

rhino.threshold-rules.unknown-parameter-set

A threshold rule trigger or reset rule refers to an unknown statistics parameter set.

Category: Watchdog (Alarms raised by the watchdog)

rhino.watchdog.no-exit

The system property watchdog.no_exit is set, enabling override of default node termination behaviour on failed watchdog conditions. This can cause catastrophic results and should never be used.

rhino.watchdog.forward-timewarp

A forward timewarp was detected.

rhino.watchdog.reverse-timewarp

A reverse timewarp was detected.

Category: AbnormalExecution

Alarms raised as a result of an abnormal execution condition being detected

rhino.uncaught-exception
Alarm Type	rhino.uncaught-exception
Level	WARNING
Message	Uncaught exception thrown by thread %s: %s
Description	An uncaught exception has been detected.
Raised	When an uncaught exception has been thrown.
Cleared	Never, must be cleared manually or Rhino restarted with the source of the uncaught exception corrected.

Category: Activity Handler

Alarms raised by Rhino activity handler

rhino.ah.snapshot-age
Alarm Type	rhino.ah.snapshot-age
Level	WARNING
Message	Oldest activity handler snapshot is older than %s, snapshot is %s (from %d), creating thread: %s
Description	The oldest activity handler snapshot is too old.
Raised	When the age of the oldest activity handler snapshot is greater than the threshold set by the rhino.ah.snapshot_age_warn system property (30s default).
Cleared	When the age of the oldest snapshot is less than or equal to the threshold.

Category: Cassandra Key/Value Store

Alarms raised by the Cassandra key/value store

rhino.cassandra-kv-store.connection-error
Alarm Type	rhino.cassandra-kv-store.connection-error
Level	CRITICAL
Message	Connection error for persistence instance %s
Description	The local database driver cannot connect to the configured persistence instance.
Raised	When communication with the database fails, for example because no host is available to execute a query.
Cleared	When the connection error is resolved.

rhino.cassandra-kv-store.db-node-failure
Alarm Type	rhino.cassandra-kv-store.db-node-failure
Level	MAJOR
Message	Connection lost to database node %s in persistence instance %s
Description	The local database driver cannot connect to a database node.
Raised	When communication with the database node fails.
Cleared	When the connection error is resolved or the node is removed from the cluster.

rhino.cassandra-kv-store.no-nodes-available
Alarm Type	rhino.cassandra-kv-store.no-nodes-available
Level	CRITICAL
Message	No database node in any persistence instance is available to execute queries
Description	All database nodes for all persistence instances have failed or are otherwise unreachable.
Raised	When an attempted database query execution fails because no node is available to accept it in any persistence instance.
Cleared	When one or more nodes become available to accept queries.

rhino.cassandra-kv-store.pending-size-limit-reached
Alarm Type	rhino.cassandra-kv-store.pending-size-limit-reached
Level	WARNING
Message	Not-yet-persisted application state has exceeded the configured pending size limit, newly committed state is being discarded
Description	The volume of committed but not yet persisted application state has exceeded the configured pending size limit threshold. State generated for new transactions will be ignored by the key/value store and not buffered for persisting until sufficient state has been persisted to reduce the pending size volume below the limit again
Raised	When the pending size volume exceeds the pending size limit.
Cleared	When the pending size volume falls below the pending size limit again.

rhino.cassandra-kv-store.scan-persist-time-threshold-reached
Alarm Type	rhino.cassandra-kv-store.scan-persist-time-threshold-reached
Level	WARNING
Message	Pending transaction scan or persist time has exceeded the configured maximum thresholds, newly committed state is being discarded
Description	The allowed pending transaction scan or persist time has exceeded the configured thresholds due to overload. State generated for new transactions will be ignored by the key/value store and not buffered for persisting until sufficient state has been persisted to reduce the load on the pending transaction scanner
Raised	When the pending transaction scan or persist times exceed the configured maximum thresholds.
Cleared	When the pending transaction scan and persist times fall below the configured maximum thresholds again.

Category: Cassandra Session Ownership Store

Alarms raised by the Cassandra session ownership store

rhino.cassandra-session-ownership-store.connection-error
Alarm Type	rhino.cassandra-session-ownership-store.connection-error
Level	CRITICAL
Message	Connection error for persistence instance %s
Description	The local database driver cannot connect to the configured persistence instance.
Raised	When communication with the database fails, for example because no host is available to execute a query.
Cleared	When the connection error is resolved.

rhino.cassandra-session-ownership-store.db-node-failure
Alarm Type	rhino.cassandra-session-ownership-store.db-node-failure
Level	MAJOR
Message	Connection lost to database node %s in persistence instance %s
Description	The local database driver cannot connect to a database node.
Raised	When communication with the database node fails.
Cleared	When the connection error is resolved or the node is removed from the cluster.

rhino.cassandra-session-ownership-store.no-nodes-available
Alarm Type	rhino.cassandra-session-ownership-store.no-nodes-available
Level	CRITICAL
Message	No database node in any persistence instance is available to execute queries
Description	All database nodes for all persistence instances have failed or are otherwise unreachable.
Raised	When an attempted database query execution fails because no host is available to accept it in any persistence instance.
Cleared	When one or more nodes become available to accept queries.

Category: Cluster Clock Synchronisation

Alarms raised by the cluster clock synchronisation monitor

rhino.monitoring.clocksync
Alarm Type	rhino.monitoring.clocksync
Level	WARNING
Message	Node %d is reporting a local clock time deviation beyond the maximum expected threshold of %dms
Description	Another cluster node is reporting a system clock deviation relative to the local node beyond the maximum permitted threshold. The status of external processes maintaining the system clock on that node (eg. NTP) should be checked.
Raised	When a cluster node reports a local clock time deviation relative to the local node beyond the maximum permitted threshold.
Cleared	When the clock deviation returns to a value at or below the threshold.

Category: Clustering

Alarms raised by Rhino cluster state changes

rhino.node-failure
Alarm Type	rhino.node-failure
Level	MAJOR
Message	Node %d has left the cluster
Description	A node left the cluster for some reason other than a management-initiated shutdown.
Raised	When the cluster state listener detects a node has left the cluster unexpectedly.
Cleared	When the failed node returns to the cluster.

Category: Configuration Management

Alarms raised by the Rhino configuration manager

rhino.config.activation-failure
Alarm Type	rhino.config.activation-failure
Level	MAJOR
Message	Error activating configuration from file %s. Configuration was replaced with defaults and old configuration file was moved to %s.
Description	An error occurred while trying to activate the file-based configuration for the configuration type specified in the alarm instance. Rhino will use defaults from defaults.xml, move the broken configuration aside, and overwrite the config file.
Raised	When an exception occurs while activating a file-based configuration.
Cleared	Never, must be cleared manually.

rhino.config.read-error
Alarm Type	rhino.config.read-error
Level	MAJOR
Message	Error reading configuration from file %s. Configuration was replaced with defaults and old configuration file was moved to %s.
Description	An error occurred while trying to read the file-based configuration for the configuration type specified in the alarm instance. Rhino will use defaults from defaults.xml, move the broken configuration aside, and overwrite the config file.
Raised	When an exception occurs while reading a configuration file.
Cleared	Never, must be cleared manually.

rhino.config.save-error
Alarm Type	rhino.config.save-error
Level	MAJOR
Message	Error saving file based configuration: %s
Description	An error occurred while trying to write the file-based configuration for the configuration type specified in the alarm instance.
Raised	When an exception occurs while writing to a configuration file.
Cleared	Never, must be cleared manually.

Category: Database

Alarms raised during database communications

rhino.database.connection-lost
Alarm Type	rhino.database.connection-lost
Level	MAJOR
Message	Connection to %s database failed: %s
Description	Rhino requires a backing database for persistence of state for failure recovery purposes. If no connection to the database backend is available, state cannot be persisted.
Raised	When the connection to a database backend is lost.
Cleared	When the connection is restored.

rhino.database.no-persistence-config
Alarm Type	rhino.database.no-persistence-config
Level	CRITICAL
Message	Persistence resource config for %s has been removed
Description	A persistence resource configuration referenced in rhino-config.xml has been removed at runtime.
Raised	When an in-use persistence resource configuration is removed by a configuration update.
Cleared	When the persistence resource configuration is restored.

rhino.database.no-persistence-instances
Alarm Type	rhino.database.no-persistence-instances
Level	CRITICAL
Message	Persistence resource config for %s has no active persistence instances
Description	A persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.
Raised	When an in-use persistence resource configuration has no active persistence instances.
Cleared	When at least one active persistence instance exists for the persistence resource configuration.

rhino.database.persistence-instance-instantiation-failure
Alarm Type	rhino.database.persistence-instance-instantiation-failure
Level	MAJOR
Message	Unable to instantiate persistence instance %s for database %s
Description	Rhino requires a backing database for persistence of state for failure recovery purposes. A persistent instance defines a connection to a database backend. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.
Raised	When a persistent instance configuration change occurs but instantiation of that persistent instance fails.
Cleared	When a correct configuration is installed.

rhino.jdbc.persistence-instance-instantiation-failure
Alarm Type	rhino.jdbc.persistence-instance-instantiation-failure
Level	MAJOR
Message	Unable to instantiate persistence instance %s for JDBC configuration with JNDI name %s
Description	A persistent instance defines the connection to the database backend. If the persistent instance cannot be instantiated then JDBC connections cannot be made.
Raised	When a persistent instance configuration change occurs but instantiation of that persistent instance fails.
Cleared	When a correct configuration is installed.

Category: Event Router State

Alarms raised by event router state management

rhino.state.convergence-failure
Alarm Type	rhino.state.convergence-failure
Level	MAJOR
Message	State convergence failed for "%s". The component remains in the "%s" state.
Description	A component reported an unexpected error during convergence
Raised	When a configuration change requiring a component to change state does not complete convergence due to an error.
Cleared	When the component transitions to the configured desired state.

rhino.state.convergence-timeout
Alarm Type	rhino.state.convergence-timeout
Level	MINOR
Message	State convergence timed out for "%s". The component remains in the "%s" state. Convergence will be retried periodically until it reaches the desired state.
Description	A component has not transitioned to the effective desired state after the timeout period
Raised	When a configuration change requiring a component to change state does not complete convergence in the expected time.
Cleared	When the component transitions to the configured desired state.

rhino.state.raentity.active-reconfiguration
Alarm Type	rhino.state.raentity.active-reconfiguration
Level	MINOR
Message	Resource adaptor entity "%s" does not support active reconfiguration. Configuration changes will not take effect until the resource adaptor entity is deactivated and reactivated
Description	A resource adaptor entity is of a type that does not support active reconfiguration but has a desired state that contains configuration properties different from those in the actual state
Raised	When a configuration change requiring a resource adaptor entity to be reconfigured and the resource adaptor does not support active reconfiguration.
Cleared	When the resource adaptor entity is deactivated and convergence has updated the configuration properties.

rhino.state.unlicensed-raentity
Alarm Type	rhino.state.unlicensed-raentity
Level	MAJOR
Message	No valid license for resource adaptor entity "%s" found. The resource adaptor entity has not been activated.
Description	A licensing problem was detected during resource adaptor entity activation.
Raised	When a node attempts to transition a resource adaptor entity from an actual state of INACTIVE to an actual state of ACTIVE but absence of a valid license prevents that transition.
Cleared	When a valid license is installed.

rhino.state.unlicensed-service
Alarm Type	rhino.state.unlicensed-service
Level	MAJOR
Message	No valid license for service "%s" found. The service has not been activated.
Description	A licensing problem was detected during service activation.
Raised	When a node attempts to transition a service from an actual state of INACTIVE to an actual state of ACTIVATING but absence of a valid license prevents that transition.
Cleared	When a valid license is installed.

rhino.state.unlicensed-slee
Alarm Type	rhino.state.unlicensed-slee
Level	CRITICAL
Message	No valid license for the SLEE found. The SLEE has not been started.
Description	A licensing problem was detected during SLEE start.
Raised	When a node attempts to transition its SLEE from an actual state of STOPPED state to an actual state of STARTING but absence of a valid license prevents that transition.
Cleared	When a valid license is installed.

Category: GroupRMI

Alarms raised by the GroupRMI server

rhino.group-rmi.dangling-transaction
Alarm Type	rhino.group-rmi.dangling-transaction
Level	WARNING
Message	Group RMI invocation %s completed leaving an active transaction dangling: %s. Please report this bug to Metaswitch support.
Description	A group RMI invocation completed without committing or rolling back a transaction that it started. The dangling transaction will be automatically rolled back by the group RMI server to prevent future issues but these occurrences are software bugs that should be reported.
Raised	When a group RMI invocation completes leaving an active transaction dangling.
Cleared	Never, must be cleared manually.

Category: Key/Value Store

Alarms raised by key/value store persistence resource managers

rhino.kv-store.no-persistence-config
Alarm Type	rhino.kv-store.no-persistence-config
Level	CRITICAL
Message	Persistence resource config for %s has been removed
Description	A persistence resource configuration referenced in rhino-config.xml has been removed at runtime.
Raised	When an in-use persistence resource configuration is removed by a configuration update.
Cleared	When the persistence resource configuration is restored.

rhino.kv-store.no-persistence-instances
Alarm Type	rhino.kv-store.no-persistence-instances
Level	CRITICAL
Message	Persistence resource config for %s has no active persistence instances
Description	A persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.
Raised	When an in-use persistence resource configuration has no active persistence instances.
Cleared	When at least one active persistence instance exists for the persistence resource configuration.

rhino.kv-store.persistence-instance-instantiation-failure
Alarm Type	rhino.kv-store.persistence-instance-instantiation-failure
Level	MAJOR
Message	Unable to instantiate persistence instance %s for key/value store %s
Description	A persistence instance used by a key/value store cannot be instantiated. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.
Raised	When a persistent instance configuration change occurs but instantiation of that persistent instance fails.
Cleared	When a correct configuration is installed.

Category: Licensing

Alarms raised by Rhino licensing

rhino.license.expired
Alarm Type	rhino.license.expired
Level	MAJOR
Message	License with serial "%s" has expired
Description	A license installed in Rhino has passed its expiry time.
Raised	When a license expires and there is no superseding license installed.
Cleared	When the license is removed or a superseding license is installed.

rhino.license.over-licensed-capacity
Alarm Type	rhino.license.over-licensed-capacity
Level	MAJOR
Message	Over licensed capacity for function "%s".
Description	The work done by a function exceeds licensed capacity.
Raised	When the amount of work processed by the named function exceeds the licensed capacity.
Cleared	When the amount of work processed by the function becomes less than or equal to the licensed capacity.

rhino.license.over-limit
Alarm Type	rhino.license.over-limit
Level	MAJOR
Message	Rate limiter throttling active, throttled to %d events/second
Description	Rate limiter throttling is active. This throttling and hence this alarm only happens in SDK versions of Rhino, not production versions.
Raised	When there is more incoming work than allowed by the licensed limit so Rhino starts rejecting some.
Cleared	When the total input rate (both accepted and rejected work) drops below the licensed limit.

rhino.license.partially-licensed-host
Alarm Type	rhino.license.partially-licensed-host
Level	MINOR
Message	Host "%s" is not fully licensed. Not all hardware addresses on this host match those licensed. Please request a new license for host "%s".
Description	The hardware addresses listed in a host-based license only partially match those on the host.
Raised	When a host-based license with invalid host addresses is installed.
Cleared	When the license is removed, or a superseding license is installed.

rhino.license.pending-expiry
Alarm Type	rhino.license.pending-expiry
Level	MAJOR
Message	License with serial "%s" is due to expire on %s
Description	A license installed in Rhino is within seven days of its expiry time.
Raised	Seven days before a license will expire and there is no superseding license installed.
Cleared	When the license expires, the license is removed, or a superseding license is installed.

rhino.license.unlicensed-function
Alarm Type	rhino.license.unlicensed-function
Level	MAJOR
Message	There are no valid licenses installed for function "%s" and version "%s".
Description	A particular function is not licensed.
Raised	When a unit of an unlicensed function is requested.
Cleared	When a license is installed that licenses a particular function, and another unit is requested.

rhino.license.unlicensed-host
Alarm Type	rhino.license.unlicensed-host
Level	MINOR
Message	"%s" is not licensed. Hardware addresses on this host did not match those licensed, or hostname has changed. Please request a new license for host "%s".
Description	The hardware addresses listed in a host-based license do not match those on the host.
Raised	When a host-based license with invalid host addresses is installed.
Cleared	When the license is removed, or a superseding license is installed.

rhino.license.unlicensed-rhino
Alarm Type	rhino.license.unlicensed-rhino
Level	MAJOR
Message	Rhino platform is no longer licensed
Description	Rhino does not have a valid license installed.
Raised	When a license expires or is removed leaving Rhino in an unlicensed state.
Cleared	When an appropriate license is installed.

Category: Limiting

Alarms raised by Rhino limiting

rhino.limiting.below-negative-capacity
Alarm Type	rhino.limiting.below-negative-capacity
Level	WARNING
Message	Token count in rate limiter "%s" capped at negative saturation point on node %d. Too much work has been forced. Alarm will clear once token count >= 0.
Description	A rate limiter is below negative capacity.
Raised	By a rate limiter when a very large number of units have been forcibly used and the internal token counter has reached the biggest possible negative number (-2,147,483,648).
Cleared	When the token count becomes greater than or equal to zero.

rhino.limiting.stat-limiter-misconfigured
Alarm Type	rhino.limiting.stat-limiter-misconfigured
Level	MAJOR
Message	Stat limiter "%s" is misconfigured: %s. All unit requests will be allowed by this limiter until the error is resolved.
Description	A stat limiter is misconfigured.
Raised	By a stat limiter that has been asked for one or more units and has been unable to find the configured parameter set or statistic name.
Cleared	When the stat limiter is reconfigured or the configured parameter set that was missing is deployed.

Category: Logging

Alarms raised by Rhino logging

rhino.logging.appender-error
Alarm Type	rhino.logging.appender-error
Level	MAJOR
Message	An error occurred logging to an appender: %s
Description	An appender has thrown an exception when attempting to pass log messages from a logger to it.
Raised	When an appender throws an AppenderLoggingException when a logger tries to log to it.
Cleared	When the problem with the given appender has been resolved and the logging configuration is updated.

Category: M-lets Startup

Alarms raised by the M-let starter

rhino.mlet.loader-failure
Alarm Type	rhino.mlet.loader-failure
Level	MAJOR
Message	Error registering MLetLoader MBean
Description	The M-Let starter component could not register itself with the platform MBean server. This normally indicates a serious JVM misconfiguration.
Raised	During Rhino startup if an error occurred registering the m-let loader component with the MBean server.
Cleared	Never, must be cleared manually or Rhino restarted.

rhino.mlet.registration-failure
Alarm Type	rhino.mlet.registration-failure
Level	MINOR
Message	Could not create or register MLet: %s
Description	The M-Let starter component could not register an MBean for a configured m-let. This normally indicates an error in the m-let configuration file.
Raised	During Rhino startup if an error occurred starting a m-let configured.
Cleared	Never, must be cleared manually or Rhino restarted with updated configuration.

Category: REM Startup

Alarms raised by embedded REM starter

rhino.rem.missing
Alarm Type	rhino.rem.missing
Level	MINOR
Message	Rhino Element Manager classes not found, embedded REM is disabled.
Description	This version of Rhino is supposed to contain an embedded instance of REM but it was not found, most likely due to a packaging error.
Raised	During Rhino startup if the classes could not be found to start the embedded REM.
Cleared	Never, must be cleared manually.

rhino.rem.startup
Alarm Type	rhino.rem.startup
Level	MINOR
Message	Could not start embedded Rhino Element Manager
Description	There was an unexpected problem while starting the embedded REM. This could be because of a port conflict or packaging problem.
Raised	During Rhino startup if an error occurred starting the embedded REM.
Cleared	Never, must be cleared manually or Rhino restarted with updated configuration.

Category: Remote Timer Server

Alarms raised by the remote timer server

rhino.remote-timer-server.bind-error
Alarm Type	rhino.remote-timer-server.bind-error
Level	CRITICAL
Message	Unable to bind remote timer server to %s %s
Description	The remote timer server is unable to bind a socket listener within the configured range of ports. The Rhino configuration may need to be checked and an alternate or wider range of ports allowed.
Raised	If the remote timer server cannot bind a socket listener within the configured range of ports.
Cleared	Never, must be cleared manually.

Category: Runtime Environment

Alarms related to the runtime environment

rhino.runtime.long-filenames-unsupported
Alarm Type	rhino.runtime.long-filenames-unsupported
Level	WARNING
Message	Filenames with a length of %s characters are unsupported on this filesystem. Unexpected deployment errors may occur as a result
Description	Filenames with the maximum length expected by Rhino are unsupported on this filesystem. Unexpected deployment errors may occur as a result
Raised	During Rhino startup if the long filename check fails.
Cleared	Never, must be cleared manually or Rhino restarted after being installed on a filesystem supporting long filenames.

rhino.runtime.slee
Alarm Type	rhino.runtime.slee
Level	CRITICAL
Message	SLEE event-routing functions failed to start after node restart
Description	SLEE event-routing functions failed to start after node restart
Raised	During Rhino startup if SLEE event-routing functions fail to restart.
Cleared	Never, must be cleared manually or the node restarted.

rhino.runtime.unsupported.jvm
Alarm Type	rhino.runtime.unsupported.jvm
Level	WARNING
Message	This JVM (%s) is not supported. Supported JVMs are: %s
Description	This JVM is not a supported JVM.
Raised	During Rhino startup if an unsupported JVM was detected.
Cleared	Never, must be cleared manually or Rhino restarted with a supported JVM.

Category: SAS facility

Alarms raised by Rhino SAS Facility

rhino.sas.connection.lost
Alarm Type	rhino.sas.connection.lost
Level	MAJOR
Message	Connection to SAS server at %s:%d is down
Description	Attempting to reconect to SAS server
Raised	When SAS client loses connection to server
Cleared	On reconnect

rhino.sas.queue.full
Alarm Type	rhino.sas.queue.full
Level	WARNING
Message	SAS message queue is full
Description	SAS message queue is full. Some events have not been reported to SAS
Raised	When SAS facility outgoing message queue is full
Cleared	When the queue is not full for at least sas.queue_full_interval

Category: SLEE State

Alarms raised by SLEE state management

rhino.state.slee-start
Alarm Type	rhino.state.slee-start
Level	CRITICAL
Message	The SLEE failed to start successfully.
Description	An unexpected exception was caught during SLEE start.
Raised	When a node attempts to transition its SLEE from an actual state of STOPPED state to an actual state of STARTING but an unexpected exception occurred while fulfilling that request.
Cleared	After the desired state of the SLEE is reset to STOPPED.

Category: SNMP

Alarms raised by Rhino SNMP

rhino.snmp.bind-failure
Alarm Type	rhino.snmp.bind-failure
Level	MAJOR
Message	The SNMP agent could not be started on node %d: no addresses were successfully bound.
Description	The SNMP agent attempts to bind a UDP port on each configured SNMP interface to receive requests. If no ports could be bound, the SNMP agent cannot process any SNMP requests.
Raised	When the SNMP Agent attempts to start listening for requests, but no port in the configured range on any configured interface could be used.
Cleared	When the SNMP Agent is stopped.

rhino.snmp.duplicate-counter-mapping
Alarm Type	rhino.snmp.duplicate-counter-mapping
Level	WARNING
Message	Duplicate counter mappings in parameter set type %s
Description	Multiple counters in the parameter set type configuration map to the same index. The parameter set type mappings will remain inactive until the conflict is resolved.
Raised	When a in-use parameter set type has a configuration with duplicate counter mappings.
Cleared	When the conflict is resolved, either by changing the relevant counter mappings, or if the parameter set type is removed from use.

rhino.snmp.duplicate-oid-mapping
Alarm Type	rhino.snmp.duplicate-oid-mapping
Level	WARNING
Message	Duplicate parameter set type mapping configurations for OID %s
Description	Multiple parameter set type configurations for in-use parameter set types map to the same OID. All parameter set type mappings will remain inactive until the conflict is resolved.
Raised	When multiple in-use parameter set types have a configuration that map to the same OID.
Cleared	When the conflict is resolved, either by changing the OID mappings in the relevant parameter set type configurations, or if a parameter set type in conflict is removed from use.

rhino.snmp.general-failure
Alarm Type	rhino.snmp.general-failure
Level	MINOR
Message	The SNMP agent encountered an error during startup: %s
Description	This is a catchall alarm for unexpected failures during agent startup. If an unexpected failure occurs, the state of the SNMP agent is unpredictable and requests may not be successfully processed.
Raised	When the SNMP Agent attempts to start listening for requests, but there is an unexpected failure not covered by other alarms.
Cleared	When the SNMP Agent is stopped.

rhino.snmp.no-bind-addresses
Alarm Type	rhino.snmp.no-bind-addresses
Level	MAJOR
Message	The SNMP agent could not be started on node %d: no suitable bind addresses available.
Description	The SNMP agent listens for requests received on all network interfaces that match the requested SNMP configuration. If no suitable interfaces can be found that match the requested configuration, then the SNMP agent cannot process any SNMP requests.
Raised	When the SNMP Agent attempts to start listening for requests, but no suitable network interface addresses can be found to bind to.
Cleared	When the SNMP Agent is stopped.

rhino.snmp.notification-address-failure
Alarm Type	rhino.snmp.notification-address-failure
Level	MAJOR
Message	Failed to create notification target for address "%s".
Description	This alarm represents a failure to determine an address from the notification target configuration. This can occur if the notification hostname is not resolvable, or if the specified hostname is not parseable.
Raised	During SNMP agent start if a notification target address cannot be determined (e.g. due to a hostname resolution failing).
Cleared	When the SNMP Agent is stopped.

rhino.snmp.partial-failure
Alarm Type	rhino.snmp.partial-failure
Level	MINOR
Message	The SNMP agent failed to bind to the following addresses: %s
Description	The SNMP agent attempts to bind a UDP port on each configured SNMP interface to receive requests. If this succeeds on some (but not all) interfaces, the SNMP agent can only process requests received via the interfaces that succeeded.
Raised	When the SNMP Agent attempts to start listening for requests, and only some of the configured interfaces successfully bound a UDP port.
Cleared	When the SNMP Agent is stopped.

Category: Scattercast Management

Alarms raised by Rhino scattercast management operations

rhino.scattercast.update-reboot-required
Alarm Type	rhino.scattercast.update-reboot-required
Level	CRITICAL
Message	Scattercast endpoints have been updated. A cluster reboot is required to apply the update. An automatic reboot has been triggered, Manual intervention required if the reboot fails.
Description	Reboot needed to make scattercast update active.
Raised	When scattercast endpoints are updated.
Cleared	On node reboot.

Category: Service State

Alarms raised by service state management

rhino.state.service-activation
Alarm Type	rhino.state.service-activation
Level	MAJOR
Message	Service "%s" failed to activate successfully.
Description	The service threw an exception during service activation, or an unexpected exception occurred while attempting to activate the service.
Raised	When a node attempts to transition a service from an actual state of INACTIVE to an actual state of ACTIVATING but the service rejected the activation request or an unexpected exception occurred while fulfilling that request.
Cleared	After the desired state of the service is reset to INACTIVE.

Category: Session Ownership Store

Alarms raised by session ownership store persistence resource managers

rhino.session-ownership-store.no-persistence-config
Alarm Type	rhino.session-ownership-store.no-persistence-config
Level	CRITICAL
Message	Persistence resource config has been removed
Description	The persistence resource configuration referenced in rhino-config.xml has been removed at runtime.
Raised	When an in-use persistence resource configuration is removed by a configuration update.
Cleared	When the persistence resource configuration is restored.

rhino.session-ownership-store.no-persistence-instances
Alarm Type	rhino.session-ownership-store.no-persistence-instances
Level	CRITICAL
Message	Persistence resource config has no active persistence instances
Description	The persistence resource configuration referenced in rhino-config.xml has no persistence instances configured, or no configured persistence instances could be instantiated.
Raised	When an in-use persistence resource configuration has no active persistence instances.
Cleared	When at least one active persistence instance exists for the persistence resource configuration.

rhino.session-ownership-store.persistence-instance-instantiation-failure
Alarm Type	rhino.session-ownership-store.persistence-instance-instantiation-failure
Level	MAJOR
Message	Unable to instantiate persistence instance %s
Description	A persistence instance used by the session ownership store cannot be instantiated. If the persistent instance cannot be instantiated then that connection cannot be made and state cannot be persisted to that instance.
Raised	When a persistent instance configuration change occurs but instantiation of that persistent instance fails.
Cleared	When a correct configuration is installed.

Category: Threshold Rules

Alarms raised by the threshold alarm rule processor

rhino.threshold-rules.rule-failure
Alarm Type	rhino.threshold-rules.rule-failure
Level	WARNING
Message	Threshold rule %s trigger or reset condition failed to run
Description	A threshold rule trigger or reset rule failed.
Raised	When a threshold rule condition cannot be evaluated, for example it refers to a statistic that does not exist.
Cleared	When the threshold rule condition is corrected.

rhino.threshold-rules.unknown-parameter-set
Alarm Type	rhino.threshold-rules.unknown-parameter-set
Level	WARNING
Message	Threshold rule %s refers to unknown statistics parameter set '%s'
Description	A threshold rule trigger or reset rule refers to an unknown statistics parameter set.
Raised	When a threshold rule condition cannot be evaluated because it refers to a statistics parameter set that does not exist.
Cleared	When the threshold rule condition is corrected.

Category: Watchdog

Alarms raised by the watchdog

rhino.watchdog.forward-timewarp
Alarm Type	rhino.watchdog.forward-timewarp
Level	WARNING
Message	Forward timewarp of %sms detected at %s
Description	A forward timewarp was detected.
Raised	When the system clock is detected to have progressed by an amount exceeding the sum of the watchdog check interval and the maximum pause margin.
Cleared	Never, must be cleared manually.

rhino.watchdog.no-exit
Alarm Type	rhino.watchdog.no-exit
Level	CRITICAL
Message	System property watchdog.no_exit is set, watchdog will be terminated rather than killing the node if a failed watchdog condition occurs
Description	The system property watchdog.no_exit is set, enabling override of default node termination behaviour on failed watchdog conditions. This can cause catastrophic results and should never be used.
Raised	When the watchdog.no_exit system property is set.
Cleared	Never, must be cleared manually.

rhino.watchdog.reverse-timewarp
Alarm Type	rhino.watchdog.reverse-timewarp
Level	WARNING
Message	Reverse timewarp of %sms detected at %s
Description	A reverse timewarp was detected.
Raised	When the system clock is detected to have progressed by an amount less than the difference between the watchdog check interval and the reverse timewarp margin.
Cleared	Never, must be cleared manually.

Usage

As well as an overview of usage, this section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples and links to related javadocs:

Procedure rhino-console command MBean → Operations

Viewing Usage Parameters

 dumpusagestats

Usage → get<usage-parameter-name>

Enabling and Disabling Usage Notifications

 setusagenotificationsenabled

UsageNotificationManager → set<usage-parameter-name> NotificationsEnabled

Viewing Usage-Notification Status

 listusagenotificationsenabled

UsageNotificationManager → get<usage-parameter-name> NotificationsEnabled

Creating Usage Parameter Sets

 createusageparameterset

ServiceUsage → createUsageParameterSet
ProfileTableUsage → createUsageParameterSet
ResourceUsage → createUsageParameterSet

Listing Usage Parameter Sets

 listusageparametersets

ServiceUsage → getUsageParameterSets
ProfileTableUsage → getUsageParameterSets
ResourceUsage → getUsageParameterSets

Removing Usage Parameter Sets

 removeusageparameterset

ServiceUsage → removeUsageParameterSet
ProfileTableUsage → removeUsageParameterSet
ResourceUsage → removeUsageParameterSet

About Usage

A usage parameter is a parameter that an object in the SLEE can update, to provide usage information.

There are two types:

Counter-type usage parameters have values that can be incremented or decremented.
Sample-type usage parameters accumulate sample data.

Accessing usage parameters

Administrators can access usage parameters through the SLEE’s management interface.

Management clients can access usage parameters through the usage parameters interface declared in an SBB, resource adaptor or profile specification. Usage parameters cannot be created through the management interface. Instead, a usage parameters interface must be declared in the SLEE component. For example, an SBB declares an sbb-usage-parameters-interface element in the SBB deployment descriptor (similar procedures apply for resource adaptors and profile specifications).

You can also use notifications to output usage parameters to management clients.

Creating named usage parameter sets

By default, the SLEE creates unnamed usage parameter sets for a notification source. You can also create named usage parameter sets, for example to hold multiple values of usage parameters for the same notification source.

Rhino usage extensions

To alleviate the limitations of the SLEE-defined usage mechanism, Rhino provides a usage extension mechanism that allows an SBB or resource adaptor to declare multiple usage parameters interfaces, and defines a Usage facility with which SBBs and resource adaptors can manage and access their own usage parameter sets.

Viewing Usage Parameters

To view the current value of a usage parameter, use the following rhino-console command or related MBean operation.

Whereas the MBean operation below can only get individual usage parameter values, the console command outputs current values of all usage parameters for a specified notification source.

Console command: `dumpusagestats`

Command

dumpusagestats <type> <notif-source> [param-set-name] [reset]
  Description
    Dump the current values of the usage parameters for the specified notification
    source.  The usage parameter set name is optional and if not specified the
    values for the unnamed (or root) parameter set are returned.  If [reset] is
    specified, the values of the usage parameters are reset after being obtained

Example

$ ./rhino-console  dumpusagestats sbb \
  "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]"
parameter-name       counter-value   sample-stats   type
-------------------  --------------  -------------  --------
       callAttempts               0                  counter
  missingParameters               0                  counter
        offNetCalls               0                  counter
         onNetCalls               0                  counter
   unknownShortCode               0                  counter
 unknownSubscribers               0                  counter
6 rows

MBean operation: `get<usage-parameter-name>`

MBean

Usage

SLEE-defined

Counter-type usage parameters

public long get<usage-parameter-name>(boolean reset)
        throws ManagementException;

Sample-type usage parameters

public SampleStatistics get<usage-parameter-name>(boolean reset)
        throws ManagementException;

Arguments

This operation requires that you specify whether the values are to be reset after being read:

reset — boolean value to reset the usage parameter’s value after being read

Return value

Operations for counter-type usage parameters return the current value of the counter. Operations for sample-type usage parameters return a SampleStatistics object.

Usage Notifications

You can enable or disable usage notifications, and list which usage notifications are enabled:

Enabling and Disabling Usage Notifications
Viewing Usage-Notification Status

Enabling and Disabling Usage Notifications

To enable or disable usage notifications, use the following rhino-console command or related MBean operation.

The notifications-enabled flag

To enable notifications to output usage parameters to management clients, set the usage notifications-enabled flag, and an appropriate debug level for the SLEE component’s relevant tracer. To disable notifications, unset the notifications-enabled flag.

Console command: `setusagenotificationsenabled`

Command

setusagenotificationsenabled <type> <notif-source> [upi-type] <param-name>
<flag>
  Description
    Set the usage notifications-enabled flag for specified usage notification
    source's usage parameter.  The usage parameters interface type is optional and
    if not specified the root usage parameters interface type is used

Example

$ ./rhino-console setusagenotificationsenabled sbb \
    "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]" \
    callAttempts true
Usage notifications for usage parameter callAttempts for
SbbNotification[service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]]
have been enabled

MBean operation: `set<usage-parameter-name>NotificationsEnabled`

MBean

UsageNotificationManager

SLEE-defined

public void set<usage-parameter-name>NotificationsEnabled(boolean enabled)
    throws ManagementException;

Arguments

enabled — a flag to enable or disable notifications for this usage parameter.

Notes

Enabling usage notification

Usage notifications are enabled or disabled on a per-usage-parameter basis for each notification source. That means that if usage notifications are enabled for a particular usage parameter, if that usage parameter is updated in any usage parameter set belonging to the notification source, a usage notification will be generated by the SLEE.

Viewing Usage Notification Status

To list usage parameter status, use the following rhino-console command or related MBean operation.

To see which usage parameters management clients are receiving through notifications, you can list usage parameter status.

Console command: `listusagenotificationsenabled`

Command

listusagenotificationsenabled <type> <notif-source> [upi-type]
  Description
    List the usage notification manager flags for the specified notification source.
     The usage parameters interface type is optional and if not specified the flags
    for the root usage parameters interface type are returned

Example

$ ./rhino-console listusagenotificationsenabled sbb \
    "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]"
parameter-name       notifications-enabled
-------------------  ----------------------
       callAttempts                    true
  missingParameters                   false
        offNetCalls                   false
         onNetCalls                   false
   unknownShortCode                   false
 unknownSubscribers                   false
6 rows

MBean operation: `get<usage-parameter-name>NotificationsEnabled`

MBean

UsageNotificationManager

SLEE-defined

public boolean get<usage-parameter-name>NotificationsEnabled()
        throws ManagementException;

Arguments

return — a flag to indicate whether notifications are enabled or disabled for this usage parameter.

Named Usage Parameter Sets

By default, the SLEE creates unnamed usage parameter sets for a notification source. You can also create named usage parameter sets, for example to hold multiple values of usage parameters for the same notification source.

Rhino includes facilities for creating, listing and removing named usage parameter sets for services, resource adaptor entities and profile tables.

This section includes the following procedures:

Creating Usage Parameter Sets
Listing Usage Parameter Sets
Removing Usage Parameter Sets

Usage parameter sets for internal subsystems (not listed using console command)

The SLEE specification also includes usage parameter sets for "internal subsystems". You can list these, but not create or remove them, since they are part of the SLEE implementation. However, Rhino uses its own statistics API to collect statistics from internal subsystems — so if you try to list usage parameter set names for an internal subsystem using rhino-console, it will always returns an empty list.

Creating Usage Parameter Sets

To create a named usage parameter set for services, resource adaptor entities or profile tables, use the following rhino-console or related MBean operations.

Services

Console command: `createusageparameterset`

Command

createusageparameterset <type> <notif-source> <param-set-name>
  Description
    Create a new usage parameter set with the specified name for the specified
    notification source

Example

$ ./rhino-console createusageparameterset sbb \
    "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]" \
    firstLook
created usage parameter set firstLook for
SbbNotification[service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]]

MBean operation: `createUsageParameterSet`

MBean

ServiceUsage

SLEE-defined

public void createUsageParameterSet(SbbID id, String paramSetName)
        throws NullPointerException, UnrecognizedSbbException,
               InvalidArgumentException, UsageParameterSetNameAlreadyExistsException,
               ManagementException;

Arguments

id — the component identifier of an SBB, which must be used in a service whose usage information this MBean manages.
paramSetName — the usage parameter set name.

Resource adaptor entities

Console command: `createusageparameterset`

Command

createusageparameterset <type> <notif-source> <param-set-name>
  Description
    Create a new usage parameter set with the specified name for the specified
    notification source

Example

$ ./rhino-console createusageparameterset resourceadaptorentity \
    "entity=cdr" \
    cdr-usage
created usage parameter set cdr-usage for RAEntityNotification[entity=cdr]

MBean operation: `createUsageParameterSet`

MBean

ResourceUsage

SLEE-defined

public void createUsageParameterSet(String paramSetName)
        throws NullPointerException, InvalidArgumentException,
               UsageParameterSetNameAlreadyExistsException,
               ManagementException;

Arguments

paramSetName — the usage parameter set name.

Profile tables

Console command: `createusageparameterset`

Command

createusageparameterset <type> <notif-source> <param-set-name>
  Description
    Create a new usage parameter set with the specified name for the specified
    notification source

Example

$ ./rhino-console createusageparameterset profiletable \
    "table=PostpaidChargingPrefixTable" \
    ppprefix-usage
created usage parameter set ppprefix-usage for ProfileTableNotification[table=PostpaidChargingPrefixTable]

MBean operation: `createUsageParameterSet`

MBean

ProfileTableUsage

SLEE-defined

public void createUsageParameterSet(String paramSetName)
        throws NullPointerException, InvalidArgumentException,
               UsageParameterSetNameAlreadyExistsException,
               ManagementException;

Arguments

paramSetName — the usage parameter set name.

Listing Usage Parameter Sets

To list named usage parameter sets for services, resource adaptor entities or profile tables, use the following rhino-console or related MBean operations.

Services

Console command: `listusageparametersets`

Command

listusageparametersets <type> <notif-source>
  Description
    List the usage parameter sets for the specified notification source.  The
    unnamed (or root) parameter set is not included in this list

Example

$ ./rhino-console listusageparametersets sbb \
      "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]"
firstLook
secondLook

MBean operation: `getUsageParameterSets`

MBean{cth}

ServiceUsage

SLEE-defined

public String[] getUsageParameterSets(SbbID id)
        throws NullPointerException, UnrecognizedSbbException,
              InvalidArgumentException, ManagementException

Arguments

id — the component identifier of an SBB, which must be used in a service whose usage information this MBean manages.

Resource adaptor entities

Console command: `listusageparametersets`

Command

listusageparametersets <type> <notif-source>
  Description
    List the usage parameter sets for the specified notification source.  The
    unnamed (or root) parameter set is not included in this list

Example

$ ./rhino-console listusageparametersets resourceadaptorentity \
      "entity=cdr"
cdr-usage

MBean operation: `getUsageParameterSets`

MBean{cth}

ResourceUsage

SLEE-defined

public String[] getUsageParameterSets()
        throws ManagementException

Profile tables

Console command: `listusageparametersets`

Command

listusageparametersets <type> <notif-source>
  Description
    List the usage parameter sets for the specified notification source.  The
    unnamed (or root) parameter set is not included in this list

Example

$ ./rhino-console listusageparametersets profiletable \
      "table=PostpaidChargingPrefixTable"
ppprefix-usage

MBean operation: `getUsageParameterSets`

MBean{cth}

ProfileTableUsage

SLEE-defined

public String[] getUsageParameterSets()
        throws ManagementException

Removing Usage Parameter Sets

To removed a named usage parameter set for services, resource adaptor entities or profile tables, use the following rhino-console or related MBean operations.

Services

Console command: `removeusageparameterset`

Command

removeusageparameterset <type> <notif-source> <param-set-name>
  Description
    Remove the existing usage parameter set with the specified name from the
    specified notification source

Example

$ ./rhino-console  removeusageparameterset sbb \
      "service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]" \
      secondLook
removed usage parameter set secondLook for
SbbNotification[service=ServiceID[name=VPN Service,vendor=OpenCloud,version=0.2],sbb=SbbID[name=VPN SBB,vendor=OpenCloud,version=0.2]]

MBean operation: `removeUsageParameterSet`

MBean

ServiceUsage

SLEE-defined

public void removeUsageParameterSet(SbbID id, String paramSetName)
        throws NullPointerException, UnrecognizedSbbException,
              InvalidArgumentException, UnrecognizedUsageParameterSetNameException,
              ManagementException;

Arguments

id — the component identifier of an SBB, which must be used in a service whose usage information this MBean manages.
paramSetName — the usage parameter set name.

Resource adaptor entities

Console command: `removeusageparameterset`

Command

removeusageparameterset <type> <notif-source> <param-set-name>
  Description
    Remove the existing usage parameter set with the specified name from the
    specified notification source

Example

$ ./rhino-console  removeusageparameterset resourceadaptorentity \
      "entity=cdr" \
      cdr-usage
removed usage parameter set cdr-usage for RAEntityNotification[entity=cdr]

MBean operation: `removeUsageParameterSet`

MBean

ResourceUsage

SLEE-defined

public void removeUsageParameterSet(String paramSetName)
        throws NullPointerException,
              InvalidArgumentException, UnrecognizedUsageParameterSetNameException,
              ManagementException;

Argument

paramSetName — the usage parameter set name.

Profile tables

Console command: `removeusageparameterset`

Command

removeusageparameterset <type> <notif-source> <param-set-name>
  Description
    Remove the existing usage parameter set with the specified name from the
    specified notification source

Example

$ ./rhino-console removeusageparameterset profiletable \
      "table=PostpaidChargingPrefixTable" \
      ppprefix-usage
removed usage parameter set ppprefix-usage for ProfileTableNotification[table=PostpaidChargingPrefixTable]

MBean operation: `removeUsageParameterSet`

MBean

ProfileTableUsage

SLEE-defined

public void removeUsageParameterSet(String paramSetName)
        throws NullPointerException,
                InvalidArgumentException, UnrecognizedUsageParameterSetNameException,
                ManagementException;

Argument

paramSetName — the usage parameter set name.

User Transactions

As well as an overview of user transactions, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Starting user transactions

 `startusertransaction`

User Transaction Management → startUserTransaction

Committing user transactions

 `commitusertransaction`

User Transaction Management → commitUserTransaction

Rolling back user transactions

 `rollbackusertransaction`

User Transaction Management → rollbackUserTransaction

About User Transactions

Using the User Transaction Management MBean, a client can demarcate transaction boundaries for a subset of profile-management operations, by:

starting a user transaction
performing some profile-management operations, across a number of different profiles (in the context of that transaction)
then committing the transaction — resulting in an atomic update of profile state.

Binding user transactions with authenticated subjects

The SLEE binds user transactions to the java.security.auth.Subject associated with the invoking thread. For all user-transaction management, the thread invoking the management operation must therefore be associated with an authenticated subject. The command console interface handles this task as part of the client-login process. (Other user-provided m-lets installed in the Rhino SLEE will need to take care of this requirement in their own way.)

Executing Profile Provisioning operations in a user transaction

The following operations on the Profile Provisioning MBean support execution in a user transaction: createProfileTable, removeProfileTable, getProfileTables, getProfileTables(ProfileSpecificationID) createProfile, createProfiles, removeProfile, getDefaultProfile, getProfile, getProfilesByIndexedAttribute, getProfilesByAttribute, getProfilesByStaticQuery getProfilesByDynamicQuery, and importProfiles.

Furthermore, accessing a Profile MBean while a user transaction is active:

enlists that MBean into that user transaction
changes that MBean to the read/write state
puts any changes to the profile in context of the user transaction.

Committing or rolling back profiles enlisted in user transactions

You cannot invoke the ProfileMBean.commitProfile() or ProfileMBean.restoreProfile() operations on a Profile MBean enlisted in a user transaction. Any changes made to such a profile will be committed or rolled back when the user transaction is committed or rolled back (respectively).

Starting User Transactions

To start a user transaction, use the following rhino-console command or related MBean operation.

Console command: `startusertransaction`

Command

startusertransaction
  Description
    Start a client-demarcated transaction.  Note that only a limited set of Rhino
    management operations support user transactions

Example

$ ./rhino-console startusertransaction

MBean operation: `startUserTransaction`

MBean

User Transaction Management

Rhino extension

void startUserTransaction()
    throws com.opencloud.rhino.management.usertx.NoAuthenticatedSubjectException,
          NotSupportedException, ManagementException;

Committing User Transactions

To commit a user transaction, use the following rhino-console command or related MBean operation.

Console command: `commitusertransaction`

Command

commitusertransaction
  Description
    Commit a client-demarcated transaction

Example

$ ./rhino-console commitusertransaction

MBean operation: `commitUserTransaction`

MBean

User Transaction Management

Rhino extension

void commitUserTransaction()
    throws com.opencloud.rhino.management.usertx.NoAuthenticatedSubjectException,
           InvalidStateException, ProfileVerificationException,
           HeuristicMixedException, HeuristicRollbackException,
           RollbackException, ManagementException;

Rolling Back User Transactions

To rollback a user transaction, use the following rhino-console command or related MBean operation.

Console command: `rollbackusertransaction`

Command

rollbackusertransaction
  Description
    Roll back a client-demarcated transaction

Example

$ ./rhino-console rollbackusertransaction

MBean operation: `rollbackUserTransaction`

MBean

User Transaction Management

Rhino extension

void rollbackUserTransaction()
    throws com.opencloud.rhino.management.usertx.NoAuthenticatedSubjectException,
           InvalidStateException, ManagementException;

Auditing Management Operations

Rhino logs all management operations to two log files in the working directory of each Rhino node (work/logs by default):

management.csv, a plain text CSV file
encrypted.management.csv, a second, encrypted copy of the logs.

You can use the encrypted version to detect tampering with the plain text copy — decode with the decrypt-management-log script, in the client/bin directory.

The management audit logs roll over once they reach 100MB, with an unlimited number of backup files. This logging configuration is currently hard-coded.

The format of management audit log can be chosen via system property rhino.audit.log_format. See more detail in system properties.

Below are descriptions of:

each field logged
available operation types
commands for getting and setting the audit level

and two examples:

Deploying an RA, and creating and activating a resource adaptor entity for it
A GET-ATTRIBUTES operation called on the Licensing MBean.

What’s in the log file?

Rhino management operations logs include the following fields:

Field Description

 date

A timestamp in the form 2010-05-11 14:55:33.692.

 uniqueID

An identifier used to correlate a set of log lines for a single management operation. All of the log lines from the same operation will have the same uniqueID.

 txID

The transaction ID associated with the operation, used to correlate a set of log lines scoped to a single transactional update. This field only has a value:

for operations invoked while a user (externally demarcated) transaction is active; or
when logging internal state changes that occur as a result of a declarative configuration import operation.

 opcode

Uniquely identifies the type of operation.

 user

The name of the user invoking the management operation, or unknown¹ if there is no authenticated user.

 roles

Any roles associated with the user.

 access

Identifies whether the operation results in a state change of some sort. May be read or write. ²

 client address

The IP address of the client invoking the management operation.

 namespace

The namespace where the management operation invoked. Empty if it is the default namespace.

 MBean name

ObjectName of the MBean invoked.

 operation type

The general type of operation.

 operation name

The name of the invoked method or get/set attribute.

 arguments

The contents of all arguments passed to the management operation. Byte array arguments display converted to a length and a hash.

 duration

How long (in milliseconds) the operation took.

 result

Either ok or failed.³ ⁴

 failure reason

A text string indicating why an operation failed. (Only present for failed results.) ²

All management operations except for AUTHENTICATION type operations come in pairs with the first entry indicating the start of an operation, and the second entry indicating success or failure, as well as how long the operation took. Only the result lines make use of the duration, result, and failure reason fields.

For a list of all operations currently recognised by the auditing subsystem, run the getopcodexml command from the command-line console. It will return the complete XML representation of all known management operations.

¹ This usually only happens if unauthenticated access has been enabled in Rhino.
² By default, users with the view permission may only perform read operations.
³ This field is only set for operation results.
⁴ A failed management operation is one which did not return successfully.

Operation types

The operation type field may contain one of the following values:

Type Result type Description

 AUTHENTICATION

n/a

A successful or failed authentication attempt.

 INVOKE

 INVOKE (RESULT)

An MBean invoke operation.

GET

 GET (RESULT)

An MBean attribute get operation.

SET

 SET (RESULT)

An MBean attribute set operation.

 GET-ATTRIBUTES

 GET-ATTRIBUTES (RESULT)

An MBean bulk-attributes GET operation. Log lines with these markers denote a series of related GET requests.

 SET-ATTRIBUTES

 SET-ATTRIBUTES (RESULT)

An MBean bulk-attributes SET operation. Log lines with these markers denote a series of related SET requests.

Managing the audit level

The auditing subsystem provides two console commands to manage what gets logged to the management audit log:

getmanagementauditlevel
  Description
    Returns the current level of management operation auditing.

setmanagementauditlevel <none \| writes \| all>
  Description
    Sets the current level of management operation auditing.

Writes is useful, for example, to avoid receiving excessive messages from an automated management client that continually polls Rhino state using JMX.

Rhino always logs changes to the audit level (irrespective of the current level).

Example 1: Resource adaptor deployment and activation

The following example shows management logs from deploying a resource adaptor, creating a resource adaptor entity for it, and activating that resource adaptor entity.

The log shows the resource adaptor activated twice in a row, the second operation failing (because the RA was already activated) — see the result and failure fields.

date

uniqueID

txID

opcode

user

roles

access

client address

MBean name

operation type

operation name

arguments

duration

result

failure reason

namespace

 2010-06-08 14:22:06.850

 101:176452077447:22

 admin

  192.168.0.7

 AUTHENTICATION

ok

 2010-06-08 14:22:35.622

 101:176452077447:29

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=Deployment

 INVOKE

 install

 [file:/home/alex/simple/simple-ra-ha.jar,
[byte array, length=65164,
  md5sum=96322071e6128333bdee3364a224b48c]

 2010-06-08 14:22:38.961

 101:176452077447:29

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=Deployment

 INVOKE (RESULT)

 install

 [file:/home/alex/simple/simple-ra-ha.jar,
[byte array, length=65164,
  md5sum=96322071e6128333bdee3364a224b48c]
]

 3339ms

ok

 2010-06-08 14:22:53.356

 101:176452077447:36

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE

 getConfigurationProperties

 [ResourceAdaptorID
[name=Simple,vendor=OpenCloud,version=1.0]
]

 2010-06-08 14:22:53.359

 101:176452077447:36

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE (RESULT)

 getConfigurationProperties

 [ResourceAdaptorID
[name=Simple,vendor=OpenCloud,version=1.0]
]

3ms

ok

 2010-06-08 14:22:53.369

 101:176452077447:39

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE

 createResourceAdaptorEntity

[ResourceAdaptorID
[name=Simple,vendor=OpenCloud,version=1.0],
 simplera,
[(Host:java.lang.String=localhost),
  (Port:java.lang.Integer=14477),
  (slee-vendor:
    com.opencloud.rhino_replicate_activities:
    java.lang.String=none)
 ]
]

 2010-06-08 14:22:53.536

 101:176452077447:39

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE (RESULT)

 createResourceAdaptorEntity

[ResourceAdaptorID
[name=Simple,vendor=OpenCloud,version=1.0],
 simplera,
[(Host:java.lang.String=localhost),
  (Port:java.lang.Integer=14477),
  (slee-vendor:
    com.opencloud.rhino_replicate_activities:
    java.lang.String=none)
 ]
]

 167ms

ok

 2010-06-08 14:23:11.987

 101:176452077447:47

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE

 activateResourceAdaptorEntity

[simplera,[101]]

 2010-06-08 14:23:12.029

 101:176452077447:47

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE (RESULT)

 activateResourceAdaptorEntity

[simplera,[101]]

 42ms

ok

 2010-06-08 14:23:30.802

 101:176452077447:52

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE

 activateResourceAdaptorEntity

[simplera,[101]]

 2010-06-08 14:23:30.820

 101:176452077447:52

 admin

 admin

 write

 192.168.0.7

 javax.slee.management:
 name=ResourceManagement

 INVOKE (RESULT)

 activateResourceAdaptorEntity

[simplera,[101]]

 18ms

 failed

 simplera not in INACTIVE state on node(s)[101]

Example 2: Bulk `GET` operation on `Licensing` MBean

The example below shows a GET-ATTRIBUTES operation called on the Licensing MBean. It includes queries on four separate attributes: LicenseSummary, LicensedFunctions, LicensedVersions, and Licenses. The result of the bulk-attribute query operation are in the last line.

Note that the uniqueID field is the same for all lines representing the GET-ATTRIBUTES operation.

date

uniqueID

txID

opcode

user

roles

access

client address

MBean name

operation type

operation name

arguments

duration

result

failure reason

namespace

 2010-05-28 14:07:11.223

 101:175500674962:292

 admin

 admin

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET-ATTRIBUTES

 2010-05-28 14:07:11.223

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

GET

 LicenseSummary

 2010-05-28 14:07:11.223

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

GET

 LicensedFunctions

 2010-05-28 14:07:11.223

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

GET

 LicensedVersions

 2010-05-28 14:07:11.223

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

GET

 Licenses

 2010-05-28 14:07:11.226

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET (RESULT)

 LicenseSummary

3ms

ok

 2010-05-28 14:07:11.226

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET (RESULT)

 LicensedFunctions

3ms

ok

 2010-05-28 14:07:11.226

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET (RESULT)

 LicensedVersions

3ms

ok

 2010-05-28 14:07:11.226

 101:175500674962:292

 admin

 admin

 read

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET (RESULT)

 Licenses

3ms

ok

 2010-05-28 14:07:11.226

 101:175500674962:292

 admin

 admin

 192.168.0.7

 com.opencloud.rhino:
 type=Licensing

 GET-ATTRIBUTES (RESULT)

3ms

ok

The durations listed for the individual GET (RESULT) lines correspond to the duration of the entire GET-ATTRIBUTES operation and not the individual GET components. In the example above, the entire GET-ATTRIBUTES operation took 3ms.

Linked and Shadowed Components

There can be times when creating component dependencies in a deployable unit where a specific dependency target may not be known. For example, the particular version of a dependent library may be variable. At other times, some already installed component may need to be replaced with another, possibly a new version with a bug fix, and reinstalling all dependent components with updated deployment descriptors is undesirable.

Bindings can help with this problem, to some degree, however bindings can introduce other issues. Bindings always operate on virtual copies of the original components, and keeping track of copied components can be difficult if many binding operations are made.

Rhino provides a solution to these problems with support for linked and shadowed components.

Linked components

A linked component is a virtual component that provides an alias for some other component. Incoming references to the linked component are redirected to the link target. A linked component’s component type, for example SBB, profile specification, library, and so on, is the same as the component that it is linked to; and, like all other components, has a unique identity represented by the (name, vendor, version) tuple.

A linked component identifier can be used anywhere where a regular component identifier is required.

Shadowed components

A shadowed component is an existing component that has been "shadowed" or replaced by a link to another component of the same type. Incoming references to the shadowed component are redirected to the link target rather than using the original component.

Conceptually, linked and shadowed component perform the same function: to redirect an incoming reference to another component. The difference is that a linked component is a new virtual component with a unique identity, whereas a shadow replaces a component that is already installed in the SLEE.

Components supporting links and shadows

The following types of components currently support links and shadows:

services
SBBs
SBB parts
profile specifications
resource adaptor types
resource adaptors
event types
libraries.

Managing linked components

Below are overviews of the procedures to create, remove, and view the metadata for linked components.

Creating a linked component

You create linked components using the createLinkedComponent management operation. For example, using rhino-console:

[Rhino@localhost:2199 (#0)] createlinkedcomponent sbb name=MySBB,vendor=OpenCloud,version=1.0 MySBBLink OpenCloud 1.0
Component SbbID[name=MySBBLink,vendor=OpenCloud,version=1.0] linked to SbbID[name=MySBB,vendor=OpenCloud,version=1.0]

The first two arguments identify the component type and identifier of the link target. The target component must already exist in the SLEE. The last three arguments define the name, vendor, and version strings for the new linked component identifier.

Removing a linked component

You remove a linked component using the removeLinkedComponent management operation. For example, using rhino-console:

[Rhino@localhost:2199 (#0)] removelinkedcomponent sbb name=MySBBLink,vendor=OpenCloud,version=1.0
Linked component SbbID[name=MySBBLink,vendor=OpenCloud,version=1.0] removed

A linked component cannot be removed if:

another component with an install level of VERIFIED or DEPLOYED references it;
another linked component specifies this linked component as its target; or
another component is shadowed by this linked component.

Viewing linked component metadata

The getDescriptor management operation returns a SLEE ComponentDescriptor object for any component that exists in the SLEE. A ComponentDescriptor object for a linked component has the following properties:

its deployable unit is the same as the deployable unit of the link target
its source component jar is the same as the source component jar of the link target
it contains a vendor-specific data object of type LinkedComponentDescriptorExtensions.

Linked component descriptor extensions

The LinkedComponentDescriptorExtensions class defines Rhino-specific component metadata extensions for linked components. Here’s what it looks like:

package com.opencloud.rhino.management.deployment;

import java.io.Serializable;
import java.util.Date;
import javax.slee.ComponentID;

public class LinkedComponentDescriptorExtensions implements Serializable {
    public LinkedComponentDescriptorExtensions(...) { ... }

    public ComponentID getLinkTarget() { ... }

    public Date getLinkDate() { ... }

    public InstallLevel getInstallLevel() { ... }

    public ComponentID[] getIncomingLinks() { ... }

    public ComponentID[] getShadowing() { ... }

    ...
}

The getLinkTarget method returns the component identifier of the link target.
The getLinkDate method returns a Date object that specifies the date and time the linked component was created.
The getInstallLevel method returns the current install level of the linked component. The install level of a linked component is immaterial, and changing it has no effect on the linked component itself; however, since an install level is a property of all components installed in Rhino, a linked component must have one by definition.
The getIncomingLinks method returns the component identifiers of any other linked components that have this linked component as a target.
The getShadowing method returns the component identifiers of any other component that has been shadowed by this linked component.

Managing component shadows

Shadowing or unshadowing a component effectively changes the definition of the component; therefore a component can only undergo these transitions if it has an install level of INSTALLED. This ensures that any components that depend on the affected component also have an install level of INSTALLED, and thus will need (re)verifying against the updated component before further use. Rhino will allow a component with an install level of VERIFIED to be shadowed or unshadowed, but will automatically transition the component (and any upstream dependencies) to the INSTALLED install level first. A component with an install level of DEPLOYED must be manually undeployed before a shadow can be created or removed.

Below are overviews of the procedures to shadow, unshadow, and view the shadow metadata for a component.

Shadowing a component

You shadow one component with another using the shadowComponent management operation. For example, using rhino-console:

[Rhino@localhost:2199 (#0)] shadowcomponent sbb name=MySBB,vendor=OpenCloud,version=1.0 name=MySBB,vendor=OpenCloud,version=1.0.2
Component SbbID[name=MySBB,vendor=OpenCloud,version=1.0] shadowed by SbbID[name=MySBB,vendor=OpenCloud,version=1.0.2]

The first two arguments identify the component type and identifier of the component to be shadowed. The last argument identifies the component that this component will be shadowed by. Both the shadowed and shadowing components must already exist in the SLEE.

Link cycles won’t work

Using shadows, you might try to create a link cycle. For example, if component A is a link to component B, component B is a link to component C, and component C is shadowed by component A, then a link cycle would exist which cannot be resolved to any concrete component. Rhino forbids such cycles to be created. An attempt to shadow a component with another component that would result in a link cycle will fail with an appropriate error.

Unshadowing a component

You unshadow a component using the unshadowComponent management operation. For example, using rhino-console:

[Rhino@localhost:2199 (#0)] unshadow sbb name=MySBB,vendor=OpenCloud,version=1.0
Shadow removed from component SbbID[name=MySBB,vendor=OpenCloud,version=1.0]

Viewing shadowed component metadata

The getDescriptor management operation returns a SLEE ComponentDescriptor object for any component that exists in the SLEE. The component descriptor for a shadowed component continues to describe the original unshadowed component, but contains a vendor-specific data object of type com.opencloud.rhino.management.deployment.ComponentDescriptorExtensions that includes the following information relevant to shadowing:

The getShadowedBy method returns the component identifier of the component that shadows this component. This target component will be used in place of the described component.
The getShadowDate method returns a Date object that specifies the date and time the shadow was established.
The getShadowing method returns the component identifiers of any other component that has in turn been shadowed by this shadowed component.

Linked and shadowed component resolution

In most cases where a component identifier is specified, Rhino will follow a chain of links and shadows to resolve the component identifier to a concrete target component. Typical cases where this occurs are as follows:

wherever a component references another component in its deployment descriptor or in a binding descriptor
if a service component is activated or deactivated
when a profile table is created from a profile specification
(though Rhino will report that the profile table was created from the specified component rather than the resolved target)
when a resource adaptor entity is created from a resource adaptor
(though again Rhino will report that the resource adaptor entity was created from the specified component rather than the resolved target)
when interrogating or updating a component’s security policy.

Specific cases where a management operation applies directly to a linked or shadowed component rather than its resolved target are as follows:

when requesting a component’s metadata descriptor
when copying a shadowed component
(The shadowed component itself is copied, rather than the shadowing component. Linked components are still resolved though when determining the actual component to copy; so an attempt to copy a linked component will result in a copy of the resolved target component being copied.)

Additional notes

Creating a link to a service component automatically adds a clone of the resolved target service’s statistics with the linked component identifier to the stats manager. For example, if service component A is linked to service component B, then the stats for B can be accessed from the stats manager using either component identifier A or B. The same result will be obtained in each case. Listing the available stats parameter sets will include both A and B.
The actual and desired states reported for a linked or shadowed service component are the states of the service component that the link or shadow resolves to. Activating or deactivating the linked or shadowed component has the same effect as activating or deactivating the resolved component.
If a resource adaptor entity generates events that may be consumed by a given service component, and a link to that service component is created, then the resource adaptor entity will also be notified about a change to the lifecycle state for the linked component when the state of the target service component changes.
A resource adaptor entity may fire an event targeted at a linked service component, and Rhino will deliver the event to the resolved target service component. If an SBB in the service invokes a resource adaptor interface API method while handling that event, then the value returned by the ResourceAdaptorContext.getInvokingService() method will equal the target service component identifier specified by the resource adaptor entity when the event was fired; that is, it will be the linked component identifier. However if an SBB in the service invokes a resource adaptor interface API method while handling an event that had no specific service target, then the value returned by the same getInvokingService() method will be the service component identifier of the resolved service that is actually processing the event.

Component Activation Priorities

Rhino versions 2.4 and above allow configuration of the activation order of SLEE components. This ordering is controlled separately for activating and deactivating components.

Introduction to priorities

In Rhino 2.3.1 and older, RAs and services started in effectively random order. The startup order was based on the indexing hash order in the system.

The priority system added in Rhino 2.4 allows operator control of this order.

Priorities are values between -128 and 127. If a component (service or resource adaptor entity), X, has a numerically higher priority value than another component, Y, then X will be started before Y. Components with the same priority may be started in an arbitrary order, or may be started concurrently. The same rule applies for component stopping priorities; i.e., highest priority stops first.

If you have assigned startup priorities to; RAs A=100, B=20, C=10; and service S=15, they will start up in the following order:

activate RA entity A
activate RA entity B
activate service S
activate RA entity C

Note that a service can still potentially receive an event from an RA before it receives a ServiceStartedEvent on the ServiceActivity. That’s a completely different problem to activation ordering, and given the asynchronous nature of event delivery not something we can readily do anything about. A service that depends on the ServiceStartedEvent may be able to use the service activation callbacks in Rhino 2.4 instead. You may notice that services of the same priority level as RA entities start before the RA entities and stop after them. This ordering is not part of the priority system definition. It is possible that they will be started concurrently in the future, so always use different priorities if you need a specific order.

Managing priorities

Below are overviews of the procedures to manage service priorities, manage RA entity priorities, and list priorities.

Managing service priorities

You get priorities for services using the getStartingPriority and getStoppingPriority management operations.

You set priorities for services using the setStartingPriority and setStoppingPriority management operations.

For example, using rhino-console:

[Rhino@localhost:2199 (#0)] setservicestartingpriority name=MyService,vendor=OpenCloud,version=1.0 10
Service ServiceID[name=MyService,vendor=OpenCloud,version=1.0] activation priority set to 10

Managing RA entity priorities

You get priorities for RA entities using the getStartingPriority and getStoppingPriority management operations.

You set priorities for RA entities using the setStartingPriority and setStoppingPriority management operations.

For example, using rhino-console:

[Rhino@localhost:2199 (#0)] setraentitystartingpriority sipra 80
Resource adaptor entity sipra activation priority set to 80

Listing priorities

You list priorities for services using the getStartingPriorities and getStoppingPriorities management operations.

You list priorities for RA entities using the getStartingPriorities and getStoppingPriorities management operations.

You get a full combined listing using the liststartingpriorities and liststoppingpriorities commands in rhino-console.

For example:

[Rhino@localhost:2199 (#0)] liststartingpriorities
Starting priorities of installed services and resource adaptor entities:
  80 : resource adaptor entity sipra
  20 : ServiceID[name=SIP Presence Service,vendor=OpenCloud,version=1.1]
  10 : ServiceID[name=SIP Proxy Service,vendor=OpenCloud,version=1.8]
   0 : ServiceID[name=SIP Notification Service,vendor=OpenCloud,version=1.1]
       ServiceID[name=SIP Profile Location Service,vendor=OpenCloud,version=1.0]
       ServiceID[name=SIP Publish Service,vendor=OpenCloud,version=1.0]
  -5 : ServiceID[name=SIP Registrar Service,vendor=OpenCloud,version=1.8]
Note: Components with the same priority may be started in any order

Declarative Configuration

Declarative configuration decouples the configuration of components and their actual activation state from their declared desired state. This allows the platform to be configured with an intended configuration, with Rhino asynchronously making the required component state transitions to achieve the new configuration. This applies to services, resource adaptor entities, and the activation state of the SLEE.

The expected users of Rhino declarative configuration are tools that manage a cluster (or clusters) of Rhino nodes providing a service. For this use case, declarative configuration bundles replace the role previously held by SLEE management Ant tasks or the use of console commands in scripts. It is possible to use the import of partial declarative configuration bundles for manual SLEE management, but this does not offer any significant benefits over other methods. Notably, the Rhino Element Manager continues to function as an easy to use administrative interface for common maintenance tasks.

In this section...

Concepts and Terminology
Configuration Bundles
Declarative Configuration Commands
Convergence
Developing With Configuration Bundles
Version Control
Common Use Cases
Common Problems

Concepts and Terminology

Concept Meaning

Declarative configuration

Configuration that describes the intended state of the system, rather than the changes needed to achieve that state. A traditional example is a configuration file. In the case of Rhino, this is represented by configuration fragments and configuration bundles.

Configuration fragment

A YAML document file that contains intended configuration information. This could be about the whole system or only part of the system.

Configuration bundle

A .zip file containing one or more configuration fragments plus metadata that describes the type and structure of the configuration bundle.

Desired state

The intended target state of the system or a part of the system. This includes things such as desired configuration properties (environment entries) and activation state of services, which resource adaptor entities should be present and their configuration and activation state, the activation state of the SLEE, and so on.

Default desired state

The intended target state of the system or a part of the system. This applies to any nodes that do not have a desired state configured and provides similar functionality to the symmetric activation state mode in Rhino releases prior to 3.0.0.

Per-node desired state

The intended target state of the system or a part of the system on a node.

Effective desired state

The computed target state of a part of the system on a node. This is the per-node desired state if configured or the default desired state if no per-node desired state exists for the part of the system.

Actual state

The current operational state of the system or a part of the system.

Convergence

The internally driven process of transitioning the actual state of the system towards the desired state until they are equal.

Complete configuration or complete configuration bundle

A configuration bundle that contains configuration fragments that represent the entire state of the system. A complete configuration bundle is denoted by the configuration bundle containing a format value of complete. Complete configuration bundles replace current system configuration when applied.

Partial configuration or partial configuration bundle

A configuration bundle that contains configuration fragments that represent changes to the state of the system. A partial configuration bundle is denoted by the configuration bundle containing a format value of partial. Partial configuration bundles supplement and alter current system configuration when applied.

Configuration Bundles

A configuration bundle is a zip archive containing one or more configuration fragments as YAML files, each containing a single YAML document. Individual configuration fragments are combined and normalized by Rhino during a declarative import. The content of an individual configuration fragment does not need to be structurally complete, however the combination of all configuration fragments in a configuration bundle must be both structurally valid and free from contradictory configuration settings.

Configuration fragment YAML files within a configuration bundle must have a .yaml filename extension, but other than this the names of these YAML files and any directory structure imposed on them within a configuration bundle are immaterial. Rhino will scan a configuration bundle and consider only *.yaml files as part of the desired configuration. This means it’s possible to include other content, such as documentation, in a configuration bundle using files with other filename extensions without interfering with the validity of the configuration bundle. Collectively, the configuration YAML files in a configuration bundle must adhere to the configuration bundle schema.

A configuration bundle must include a YAML document which contains a top-level object with the following structure:

The examples given here assume the content is given in a single configuration fragment, however as previously stated, it is possible for this content to be split across configuration fragments within a configuration bundle.

rhino-config:config-bundle:

  format: complete

  schema-version: '1.0'

The schema version defines the structure of the configuration fragment documents. Currently only one schema version is supported and this field must be set to 1.0.

Rhino supports two types of configuration bundles - complete and partial.

A complete configuration bundle includes everything that needs to be configured for the deployed application to function, including profiles, resource adaptor entities, tracer configuration, usage parameter sets and desired activation states for the SLEE, services and resource adaptor entities.
A partial configuration bundle only includes configuration for some aspects of some of these. For example, it might create a resource adaptor entity link name and set the level for a tracer for a service to FINEST.

Complete Configuration Bundles

A complete configuration bundle includes all intended configuration states for the deployed application. Anything already present in a Rhino instance that is not specifically included in a complete configuration bundle when it’s imported will either be removed from Rhino if it’s removable, or be reverted to a default state otherwise.

The table below illustrates the effect a complete configuration import would have on state not defined in the configuration bundle:

State type

Effect of non-declaration in complete configuration bundle

Profile table

Removed

Profile

Removed

Usage parameter set

Removed

Tracer level

Removed

SBB environment entry

Reverts to default value as defined in the SBB’s deployment descriptor.

Per-node desired state

Removed

Default desired state

Reverts to the unconfigured default state – STOPPED for SLEE state and INACTIVE for services and resource adaptor entities.

Resource adaptor entity

Removed

Resource adaptor entity configuration property

Reverts to the default value as defined in the resource adaptor’s deployment descriptor.

Security permission specification

Reverts to the default permissions as defined in the component’s component jar.

Partial Configuration Bundles

A partial configuration bundle includes a subset of the configuration state for the deployed application. Configuration not described in the partial configuration bundle will not be modified. In a partial configuration bundle, configuration that is to be removed is marked with the attribute present: false. It is not possible to remove the default desired state for a component.

Importing a Configuration Bundle

Two user interfaces are provided for importing configuration bundles. Both use the same underlying method. A Java Management Bean method - ConfigManagementMBean.importConfiguration() and rhino-console command - importdeclarativeconfig.

When a configuration is imported, Rhino returns a JSON document describing the changes that will be made to the state of the system or a list of validation errors that prevented the import from succeeding. If using rhino-console, the document is parsed and a human-readable status output is printed. You can also save the returned document to a file using the –o <output.json> command parameter.

Version control of configuration bundles imported into Rhino simplifies change management and rollback.

Declarative Configuration Commands

Managing Rhino declarative configuration typically consists of tasks such as importing and exporting configuration bundles, checking and setting the desired state of components and querying whether the actual state of the system has converged to the currently configured desired state.

This section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples and links to related javadocs:

Exporting Configuration
Importing Configuration
Waiting For Convergence
Retrieving a Convergence Status Report

Exporting Configuration

To export configuration, use the following rhino-console command or associated MBean operation.

Console command: `exportdeclarativeconfig`

Command

exportdeclarativeconfig <zip|directory>
  Description
    Export a declarative configuration bundle which represents the current desired
    component configuration. Outputs the configuration bundle to a directory by
    default, but will instead output a zip file if a .zip suffix is used for the
    destination argument.

Example

To export the configuration bundle site-1-config-backup.zip:

$ ./rhino-console exportdeclarativeconfig site-1-config-backup.zip
Export written to: site-1-config-backup.zip

MBean operation: `exportConfiguration`

MBean

Rhino operation

public byte[] exportConfiguration()
    throws ManagementException;

This operation:

returns a Zip archive containing a single complete config bundle describing the configuration for all namespaces in the cluster.

Importing Configuration

To import configuration, use the following rhino-console command or associated MBean operation providing a configuration bundle. After importing a configuration bundle it is useful to wait for convergence before directing traffic to the system.

The status document returned by the MBean operation can be saved to disk by providing the -o <output file> option to the console command. The format of this document is described at Declarative Config Import Result Schema.

Console command: `importdeclarativeconfig`

Command

importdeclarativeconfig <zip|directory> [-dryrun] [-o resultfile.json] [-v]
[-reconfigurationaction
<validateEntityState|deactivateAndReactivate|doNotDeactivate>
  Description
    Import a declarative configuration bundle into the SLEE.  Source may be either a
    config bundle zip file or a path to a directory containing an unzipped config
    bundle. Specifying -o will output the resulting import status json to a file
    instead of the console. Specifying -v will include verbose elements such as
    stacktraces in the console output. Specifying a reconfiguration action other
    than the default of "doNotDeactivate" will change how resource adaptor entities
    that do not support active reconfiguration are managed during this import.

Example

To import the configuration bundle site-1.zip:

$ ./rhino-console importdeclarativeconfig site-1.zip
Importing configuration...

Result
=======
 Status: Success


Import result reported success; all changes have been applied.

MBean operation: `importConfiguration`

MBean

Rhino operation

public String importConfiguration(byte[] configBundle, ImportOptions options)
    throws NullPointerException, MalformedConfigBundleException,
           ManagementException;

This operation:

validates and imports the configuration provided in the configBundle argument
initiates a convergence check in all namespaces affected by the imported configuration
returns a status document containing the overall status of the import operation (Success or Failure) and detailed results describing the validation errors found or state changes that resulted from the import.

Waiting For Convergence

To check the current convergence status of a Rhino cluster use the following rhino-console commands or associated MBean operation:

Console command: isconverged
Console command: waittilconverged
MBean operation: isConvergedToDesiredState

Console command: `isconverged`

Command

isconverged [-nodes node1,node2,...]
  Description
    Check if Rhino's actual state is currently equal to desired state.  If a node
    list is provided, only the specified nodes are checked.

Examples

To check if the actual states of all components match the desired states:

$ ./rhino-console isconverged
Rhino is currently converged to desired state

To check if the actual states of all components match the desired states on node 101:

$ ./rhino-console isconverged -nodes 101
Node [101] is currently converged to desired state

Console command: `waittilconverged`

Command

waittilconverged [-timeout timeout]
  Description
    Wait for actual state to converge to desired state. The optional timeout is
    specified in seconds

Example

To wait for up to one minute for the actual states of all components to match the desired states:

$ ./rhino-console waittilconverged -timeout 60
Covergence reached. Actual state has converged to desired state.

MBean operation: `isConvergedToDesiredState`

MBean

Rhino operations

public boolean isConvergedToDesiredState()
    throws ManagementException;

public boolean isConvergedToDesiredState(int[] nodes)
    throws ManagementException;

These operations return true if the actual state of all components in all namespaces matches the desired state. The first method checks all event router nodes in the cluster. The second method only checks the specified cluster nodes.

Retrieving a Convergence Status Report

To retrieve a detailed report on the current convergence status of a Rhino cluster use the following rhino-console command or associated MBean operation:

Console command: `reportconvergencestatus`

Command

reportconvergencestatus [-nodes node1,node2] [-diff] [-o filename]
  Description
    Report on the current convergence status.  If a node list is provided, only the
    specified nodes are included in the report.  The -diff option will limit the
    report to include only entities where the actual state differs from the desired
    state.  The -o option will output the raw json-formatted report to the specified
    file instead of a human-readable report being output to the console.

Examples

To retrieve a full convergence report for the cluster:

$ ./rhino-console reportconvergencestatus
In the following report, any desired state that is followed by an asterisk (*)
indicates that that desired state is a per-node override from the default.

In the default namespace:
SLEE status:
  Node   Desired state   Actual state
  101    running         running
  102    running         running
  103    stopped *       stopped

Service status:
  service: ServiceID[name=SIS-IN Test Service Composition Selector Service,vendor=OpenCloud,version=0.3]
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    inactive *      inactive

  service: ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.3]
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    inactive *      inactive

  service: ServiceID[name=Call Forwarding Service,vendor=OpenCloud,version=0.3]
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    inactive *      inactive

  service: ServiceID[name=Call Duration Logging Service,vendor=OpenCloud,version=0.3]
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    inactive *      inactive

  service: ServiceID[name=VPN Service,vendor=OpenCloud,version=0.3]
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    inactive *      inactive

Resource adaptor entity status:
  entity name: insis-ptc-1a
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    active          active

  entity name: insis-ptc-1b
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    active          active

  entity name: insis-ptc-external
    Node   Desired state   Actual state
    101    active          active
    102    active          active
    103    active          active

To report only on where convergence has not been met, you can use the -diff option:

$ ./rhino-console reportconvergencestatus -diff
Rhino is currently converged to desired state, no differences to report

To save the report to a file in JSON format:

$ ./rhino-console reportconvergencestatus -o convergence-report.json
Output written to file: convergence-report.json

MBean operation: `getConvergenceStatus`

MBean

getMaximumSize
setMaximumSize

Rhino operations

public String getConvergenceStatus(boolean differencesOnly)
    throws ManagementException;

public String getConvergenceStatus(int[] nodeIDs, boolean differencesOnly)
    throws InvalidArgumentException, ManagementException;

These operations return a JSON-formatted string that reports the desired state and actual state for each service and resource adaptor entity in the SLEE, along with the desired state and actual state of the SLEE itself, across all namespaces.

Convergence

When a change is made to the desired state of components or another configuration change allows a component that has failed to activate to be retried, Rhino will initiate a series of changes to the actual state that will converge this to match the desired state. First, Rhino will calculate the changes required to make the actual state match the desired state. Once the required actions have been calculated, a series of tasks will run that update the state of individual components towards the desired state.

Convergence of components proceeds stepwise with actions ordered roughly by the desired activation state. Typically, deactivation of components will occur first, followed by reconfiguration, then activation. Sometimes components will be unable to transition to the desired state immediately. For example, a service may remain in the Stopping state for some time while SBBs drain. Tasks that depend on the state transition completing, such as removing a resource adaptor entity link name binding, will wait for a short period, then retry. Other tasks that do not depend on a state transition will still execute.

Sometimes change to the configuration of a component requires additional state transitions to occur first. The two main cases where this is true are reconfiguration of a resource adaptor entity that does not support active reconfiguration, and removing a resource adaptor entity link name binding. In both cases, any components that must be deactivated for the configuration change to complete will first be deactivated, then the configuration change will be made. Finally, Rhino will attempt to reactivate the components. If the new configuration is missing data required for reactivation to succeed the components may fail to activate.

Rhino periodically checks that there are no outstanding differences between the actual state and the desired state. If any are found, it will start converging the actual state towards the desired state.

The management interface for importing configuration provides a single method which can be polled until the state has converged – isConvergedToDesiredState(). In rhino-console this polling is contained in the command waittilconverged.

Monitoring Convergence

Most users of the Rhino declarative configuration interfaces will want to monitor the progress of the system towards convergence. Rhino provides some basic interfaces to support this.

Checking convergence status

Through rhino-console

The following console commands can be used to monitor convergence status:

 isconverged

Reports whether convergence has been achieved.

 reportconvergencestatus

Provides a detailed report on the convergence status of every stateful entity in the SLEE.

 waittilconverged

Wait until the state of all components has converged.

Via JMX

Poll the system using the ConfigManagementMBean method isConvergedToDesiredState().

Obtain a convergence report using the ConfigManagementMBean method getConvergenceStatus(boolean).

Checking the status of individual components

Through rhino-console

The Rhino console provides commands for querying the desired and actual state of components. There are commands for checking the desired and actual state of a SLEE, resource adaptor entities and services. You can list these in the console by typing help declarative.

Via JMX

The ResourceManagementMBean, ServiceManagementMBean and SLEEManagementMBean contain methods to get the desired and actual state of components. Tools that create and import configuration bundles can use these methods to retrieve the states and present them to the system administrator.

Developing With Configuration Bundles

There are two major development use cases that use configuration bundles: development on the Rhino platform such as TAS services, Sentinel services, and development of management tools.

When developing a new service, it is useful to write or export a configuration bundle that can be used for testing. For automated testing this may be templated with the test setup tool substituting values as required for the test environment. Performing test setup in this manner is particularly useful for container-based deployment.

Management tools can loosely be divided into two categories: ones that manipulate configuration bundles to provide a service-specific configuration interface, and ones that operate on the state of the Rhino cluster using the declarative configuration management operations. Some examples of tools are:

Sentinel VoLTE config-converter – a tool that generates configuration bundles for Rhino from simplified high-level configuration documents describing the operator and site-specific attributes of a Sentinel VoLTE deployment.
Initconf – a daemon for managing cloud-based Rhino clusters. Initconf is responsible for ensuring that newly started Rhino VMs are correctly configured, and for performing controlled shutdown of nodes.

Configuration bundle manipulation tools are often task specific and frequently work with configuration fragments, creating, modifying and combining them into a configuration bundle for import into Rhino. They use the configuration-bundle YANG schema to structure and validate the generated fragments.

Rhino management tools can be task specific, such as a simple importer that only uses the ConfigManagementMBean methods to import a configuration bundle and wait until the system state has converged or general such as the rhino-console. Management tools should use the configuration operations operating on desired and actual state in preference to the imperative operations, such as ServiceManagementMBean.activate().

Import tools should parse the JSON document returned by ConfigManagementMBean.importConfiguration() to provide useful feedback to the user. Validation errors, in particular, need to be printed with the ID of the associated component. Feedback on the progress of convergence can be provided by using the component management MBeans to get the status of components having an activation state delta in the returned success document.

Version Control

It is advisable to store the complete configuration for operational Rhino clusters in a version control system (VCS). There are several ways to organize a version control repository; a general guideline is to use branches to separate variants and avoid storing multiple copies of files. A simple method is described below.

Store common configuration on a master branch, and the complete configuration for each site on a separate branch derived from this master branch. Store partial configurations alongside the complete configuration they modify in a branch derived from the site that is affected. By doing this, it is easy to follow the history of changes to the configuration as the branch records the history of reconfigurations and makes it easy to identify the prior version if a change needs to be rolled back. Name branches for change rollout according to the nature of the changes that they contain, for example, “ocs-server-migration".

When making a permanent change using partial configuration, update the site master with the changes so future maintenance work has a clear starting reference. Update the site master and, if appropriate, the master branch once the system configuration has been tested as functioning correctly. After applying partial configuration and verifying, export the configuration, unzip it, and add it to the site master branch. This avoids the need to run multiple changes in sequence to restore the configuration of a cluster after data loss.

version control branching

Common Use Cases

The use cases below illustrate the tasks that declarative configuration supports. Many operators and developers will have different approaches to these tasks. Use of declarative configuration simplifies management of variants and migration between versions of the system configuration.

Deploying a new application

A system integrator (SI) creates an application on Rhino to deploy to multiple sites in a customer’s network.

The SI creates a single VM image with the components needed for the application deployed into Rhino and a configuration fragment describing the configuration common to all sites. For each site, the SI creates a configuration fragment describing the configuration of the components specific to that site. They combine this configuration fragment with the common fragment to create a site-specific configuration bundle.

Using the VM image, the SI starts a fleet of virtual machines on each site. They import the site-specific configuration bundle using the rhino-console command importdeclarativeconfig <config-bundle.zip> -o output.json and save the resulting output file. Rhino applies the configuration and converges the actual state of the updated components to match the newly imported desired state.

Upgrade

A developer creates a new version of the application on Rhino to deploy in a customer’s network. The developer has created a VM image with the components needed for the application deployed into Rhino.

The developer uses the rhino-console command exportdeclarativeconfig <old-config-bundle.zip> to write the state of the operational cluster to a file. The developer unzips this configuration bundle and saves it in a version control repository for the customer. The developer uses a transformation script to update component versions and configuration properties from the pre-upgrade values to the post-upgrade ones.

Using the VM image, the developer starts an initial subset of the new cluster. They import the post-upgrade configuration bundle using the rhino-console command importdeclarativeconfig <config-bundle.zip> -o output.json and save the resulting output file. Rhino applies the configuration and converges the actual state of the updated components to match the newly imported desired state.

The developer runs the rhino-console command waittilconverged and watches for alarms. Once the actual state has converged, the system administrator redirects traffic to the new cluster and shuts down VMs hosting the old cluster one by one while booting VMs for the new cluster to replace them.

Maintenance

Unlike versions of Rhino prior to 3.0.0, the declarative state management commands do not require disabling and re-enabling symmetric activation state mode when performing operations on a single Rhino node. The default behavior is to have a default desired state for the cluster and use temporary per-node state configuration to override this as needed.

Temporary stop

A system administrator using rhino-console can use the desired state management commands to temporarily deactivate nodes for maintenance.

For example, to deactivate a node while running diagnostics that have the potential to interrupt call flow, the administrator runs the console command setsleedesiredstate <node-ID> stopped. After the task is complete, they then run removepernodesiredstate <node-ID> to return the node to the same state as the rest of the cluster.

Reboot

A system administrator needs to reboot the host VM for OS updates. They run the OS reboot command. The init script for the Rhino service runs the console command shutdown -nodes <node-IDs> -timeout <drain timeout> to shut down the Rhino nodes running on the host. No change is made to the desired state, so when the host restarts, the nodes return to the same state as before the reboot.

Backup

A system administrator makes configuration backups before performing configuration changes. Backups are made before and after importing a partial configuration or after running a series of console commands that change the configuration of components.

The administrator uses the rhino-console command exportdeclarativeconfig <pre-change-bundle.zip> to write the state of the operational cluster to a file. The administrator unzips this configuration bundle and compares it with the latest version in the version control repository. If it is different, the administrator adds it to the repository with a commit message describing the difference.

The administrator makes the planned configuration changes to the system using a management interface. They use the rhino-console command exportdeclarativeconfig <post-change-bundle.zip> to write the state of the operational cluster to a file. They unzip this configuration bundle and save it in the version control repository with a commit message describing the purpose of the change.

Common Problems

A component fails to activate because of a configuration fault

Problem

If a component has a valid but incorrect configuration it may fail to activate. In the case of a service, Rhino will raise an alarm. Resource adaptors are responsible for raising their own alarms.

Solution

Fix the configuration problem and import the new configuration bundle. You can do this by editing the complete configuration previously imported or creating a new partial configuration bundle.

A component fails to start because it is not licensed

Problem

Some components, such as Sentinel VoLTE and the Service Interaction SLEE, require additional license functions to operate. If unlicensed they will fail to activate.

Solution

Install a license that allows the required functions. Rhino will try to activate any components that have a desired state of active and require the newly licensed functions.

Convergence times out

Problem

The state of a component does not converge due to factors inside the SLEE. Typically, this will occur when there are still active calls and SBBs or Activities remain live longer than the convergence timeout period. Rhino raises a convergence-timeout alarm and continues waiting for the component to transition to the next desired state.

2020-08-20 14:37:46.8651200 Minor [rhino.facility.alarm.manager] <ConvergenceExecutor-0> {ns=test} Alarm 101:255170210055681 [SubsystemNotification[subsystem=Convergence],rhino.state.convergence-timeout,ConvergenceTask[, namespace='test', component=ResourceAdaptorEntityPKey[test:activity-test-ra], desiredState=non_existent]] was raised at 2020-08-20 14:37:46.865 to level Minor

        State convergence timed out for "ResourceAdaptorEntityPKey[test:activity-test-ra]".
The component remains in the "stopping" state.
Convergence will be retried periodically until it reaches the desired state.

Solution

Wait for calls to drain or identify the entity that is preventing the component state transition and remove it. The Rhino console commands findactivities, findsbbs and findtimers will list the entities that are preventing a resource adaptor entity or service from stopping. The commands removeactivity, removeallactivities, removesbb, removeallsbbs and canceltimer will remove these entities. You can also use the Rhino Element Manager to manage the live entities in the system.

Rhino Configuration

This section covers procedures for configuring Rhino upon installation, and as needed (for example to tune performance).

This includes configuring:

logging
staging
object pools
licenses
rate limiting (overload control)
security
external databases
cluster membership
the Rhino SNMP subsystem
static replication domaining
data striping
SAS tracing

Logging

Rhino supports logging to multiple different locations and offers granular configuration of loggers and output. The most common configuration used is of log levels and output appenders. Log appenders are used to direct logging output for display and storage, typically to files or the console terminal. Rhino provides management interfaces and commands for configuring the logging framework. It also provides access to the logging framework for deployed components as extensions to the SLEE specification.

This section includes the following topics:

overview of logging
overview of SLEE 1.1 Tracers
Creating a file appender
Create a Socket Appender
Creating a Console Appender
Remove an Appender
Attaching an appender to a logger
Configure a Logger
Mananging a Logger’s Additivity
Managing a Logger’s Log Level
Listing log appenders
Listing log keys
Managing Logging Properties
Define a Plugin Component
Annotating Log files
Rolling-Over All Rolling File Appenders
Logging Plugins
Known Issues

JMX clients can access logging management operations via the Logging Management MBean.

About Logging

The Rhino SLEE uses the Apache Log4j 2 logging framework to provide logging facilities for SLEE components and deployed services.

The Logging Management MBean

Rhino SLEE allows changes to logging configuration at runtime. This is useful for capturing log information to diagnose a problem, without having to restart the SLEE. You configure logging using the Logging Management MBean, through the command console. This MBean lets you query the log configuration, and for most subsystems effect immediate changes (some require a node restart for performance reasons).

Rhino’s logging system includes logger names, log levels, log appenders, filters, and tracers.

Asynchronous Logging

The Log4j 2 logging architecture provides a new approach to asynchronous logging. It uses asynchronous loggers, which submit log events to a work queue for later handling by the appropriate appenders.

More details can be found in the Log4j 2 async loggers documentation

Rhino offers support for mixed synchronous and asynchronous logging through logger configuration commands. Correctly configuring asynchronous logging has some complexity, discussed in how to Configure a Logger.

Mapped Diagnostic Context

Rhino 2.6 introduces access to the Mapped Diagnostic Context (MDC) as a tool to tag and correlate log messages throughout an activity’s life cycle. This tagging can be combined with the new filters to allow very fine grained control of logging and tracing.

A simple SIP example of useful context would be including the P-charging-vector header. As this uniquely identifies a single call, it becomes trivial to identify all log messages related to handling an individual call. Identification (or filtering) remains simple even under load, with multiple calls handled in parallel.

The Logging Context Facility discusses MDC in greater detail.

Logger names

Subsystems within the Rhino SLEE send log messages to specific loggers. For example, the rhino.facility.alarm logger periodically receives messages about which alarms are currently active within the Rhino SLEE.

Examples of logger names include:

root — the root logger, from which all loggers are derived (can be used to change the log level for all loggers at once)
rhino — main Rhino logger
rhino.management — for log messages related to Rhino management systems
trace.<namespace>.<deployable_type>.<notification_source>.<tracer name> — loggers used by deployed SLEE components that use tracers. By default these keys appear abbreviated in console and file logs. Details of tracer abbreviation can be found at Tracer pattern converter.

Log levels

Log levels can be assigned to individual loggers to filter how much information the SLEE produces:

Log level

Information sent

OFF

No messages sent to logs (not recommended).

 FATAL

Error messages for unrecoverable errors only (not recommended).

 ERROR

Error messages (not recommended).

 WARN

Warning messages.

 INFO

Informational messages (especially during node startup or deployment of new resource adaptors or services). The default.

 DEBUG

Detailed log messages. Used for debugging by Metaswitch Rhino SLEE developers.

 TRACE

Finest level. Not currently used.

ALL

All of the above.

Each log level will log all messages for that log level and above. For example, if a logger is set to the INFO level (the default), all of the log messages logged at the INFO, WARN, ERROR, and FATAL levels will be logged as well.

If a logger is not assigned a log level, it inherits its parent’s. For example, if the rhino.management logger has not been assigned a log level, it will have the same effective log level as the rhino logger.

The root logger is a special logger which is considered the parent of all other loggers. By default, the root logger is configured with the INFO log level. In this way, all other logger will output log messages at the INFO log level or above unless explicitly configured otherwise.

Use INFO

A lot of useful or crucial information is output at the INFO log level. Because of this, setting log levels to WARN, ERROR or FATAL is not recommended.

Log appenders

System administrators can use the console to create some simple log appenders. Full appender creation is available through the Rhino Element Manager (REM). These append log messages to destinations such as the console, a log file, socket, or Unix syslog daemon. At runtime, when Rhino logs a message (as permitted by the log level of the associated logger), Rhino sends the message to the log appender for writing. Types of log appenders include:

file appenders — which append messages to files (and may be rolling file appenders)
console appenders — which send messages to the Rhino console
socket appenders — Append to a network socket. Either raw, or syslog formatted.
custom appenders — which can do a wide variety of things. Common "custom" apenders are used to append to various kinds of database.

Rolling file appenders

Typically, to manage disk usage, administrators are interested in sending log messages to a set of rolling files. They do this by setting up rolling file appenders which:

create new log files if the current one gets too large
rename old log files as numbered backups
delete old logs when a certain number of them have been archived.

Log files roll over when they exceed the configured size i.e. size is checked after logging each message, if the log file is larger than the maximum the next message will be written to a new file. Since Rhino 2.6.0 only the SDK rolls over log files on node restart. Production nodes use a size-based policy only.

You can configure the size and number of rolled-over log files and the rollover behaviour. Options include size-based, time-based and on node-restart. All configurations described for Log4j 2 are valid: https://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender

An example logging config containing a complex rollover strategy that increments file numbers, retaining up to 4 historical files younger than 30 days:

        <appender name="RhinoLog" plugin-name="RollingFile">
            <layout name="RhinoLogLayout" plugin-name="PatternLayout">
                <property name="pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%logger] <%threadName> %mdc : %msg%n%throwable"/>
                <property name="header" value="${rhinoVersionHeader}"/>
            </layout>
            <property name="filePattern" value="${logDir}/rhino.log.%i"/>
            <property name="fileName" value="${logDir}/rhino.log"/>
            <component name="RhinoLogPolicy" plugin-name="SizeBasedTriggeringPolicy">
                <property name="size" value="100MB"/>
            </component>
            <component name="RhinoLogStrategy" plugin-name="NotifyingRolloverStrategy">
                <property name="min" value="1"/>
                <property name="max" value="2147483647"/>
                <component name="deleteAction" plugin-name="Delete">
                    <property name="basePath" value="${logDir}"/>
                    <property name="maxDepth" value="1"/>
                    <component name="fileName" plugin-name="IfFileName">
                        <property name="glob" value="rhino.log.*"/>
                    </component>
                    <component name="any" plugin-name="IfAny">
                        <component name="lastmodified" plugin-name="IfLastModified">
                            <property name="age" value="30d"/>
                        </component>
                        <component name="fileCount" plugin-name="IfAccumulatedFileCount">
                            <property name="exceeds" value="4"/>
                        </component>
                    </component>
                </component>
            </component>
        </appender>

Default appenders

By default, the Rhino SLEE comes configured with the following appenders active:

Appender

Where it sends messages

Logger name

Type of appender

 RhinoLog

the Rhino logs directory (work/log/rhino.log)

 root

a rolling file appender

 STDERR

the Rhino console where a node is running (in a standard error stream)

 root

a console appender

 ConfigLog

work/log/config.log

 rhino.config

a rolling file appender

New appenders won’t receive messages until associated with at least one logger

By default, a newly created log appender is not associated with any loggers, so will not receive any log messages.

Appender additivity

If a logger has its additivity flag set to true, all of the output of its log statements goes to all of its appenders and ancestors. If a specific ancestor has its additivity flag set to false, then the output goes to all appenders and ancestors up to and including that specific ancestor — but not to appenders in any of that ancestor’s ancestors. (By default, logger additivity flags are set to true.}

See Apache’s Log4j 2 Architecture page for details on additivity.

Filters

Filters can be applied to both loggers and appenders to restrict the set of log messages that are reported by a logger or through an appender. They provide a more flexible limiting approach than log level alone. Configuring filters can be performed using the Rhino Element Manager or by modifying an export of the logging configuration. A list of the filters available by default and their configuration properties can be found in the Log4j 2 filter documentation

An example filter configuration setting for limiting log levels to Finer in namespace volte and Info in all other namespaces is shown below:

                <component plugin-name="DynamicThresholdFilter">
                    <property name="defaultThreshold" value="Finer"/>
                    <property name="key" value="namespace"/>
                    <component plugin-name="KeyValuePair">
                        <property name="key" value="volte"/>
                        <property name="value" value="Warning"/>
                    </component>
                </component>

If three trace messages are emitted by the service

tracer.warning("TransparentDataCache(MMTEL-Services) (RepositoryDataAccessKey{REPOSITORY_DATA, userId=tel:+34600000002, userIdType=IMPU, serviceIndication=MMTEL-Services}): [DoUDR] failed to send request")
tracer.finer("Cache gave immediate response. Latency: 1 ms")
tracer.finest("Removing service indication: MMTEL-Services from the session state list.Initial items: [MMTEL-Services]")

With the service deployed in namespace volte only the Warning will be logged:

2017-11-14 13:35:38.123+1300 Warning [trace.sh_cache_ra.sh-cache-ra] <jr-4> {ns=volte, txID=101:210487189646097} TransparentDataCache(MMTEL-Services) (RepositoryDataAccessKey{REPOSITORY_DATA, userId=tel:+34600000002, userIdType=IMPU, serviceIndication=MMTEL-Services}): [DoUDR] failed to send request

otherwise both the Finer and Warning messages will be logged:

2017-11-14 13:35:38.123+1300 Warning [trace.sh_cache_ra.sh-cache-ra] <jr-4> {ns=mmtel, txID=101:210487189646097} TransparentDataCache(MMTEL-Services) (RepositoryDataAccessKey{REPOSITORY_DATA, userId=tel:+34600000002, userIdType=IMPU, serviceIndication=MMTEL-Services}): [DoUDR] failed to send request
2017-11-14 13:35:38.137+1300 Finer   [trace.volte_sentinel_sip.2_7_0_copy_1.volte_sentinel_sip.sentinel.sbb] <jr-4> {ns=mmtel, txID=101:210487189646097} Cache gave immediate response. Latency: 1 ms

The default threshold of Finer will cause the Finest message to never be logged.

Logging plugins

Rhino contains several logging plugins to extend the functionality of Log4j 2 to aid SLEE management and provide additional context to logs.

NotifyingRolloverStrategy
NotifyingDirectWriteRolloverStrategy
LogNotificationAppender
PolledMemoryAppender

NotifyingRolloverStrategy

An extended variant of the DefaultRolloverStrategy providing an API for components to receive notification of log file rollover. The RolloverNotificationListener can be registered to receive a callback whenever a log file is rolled over. This strategy should be used instead of the Log4j 2 DefaultRolloverStrategy so Rhino can send notifications to monitoring systems.

NotifyingDirectWriteRolloverStrategy

An extended variant of the DirectWriteRolloverStrategy providing an API for components to receive notification of log file rollover. The RolloverNotificationListener can be registered to receive a callback whenever a log file is rolled over. This strategy should be used instead of the Log4j 2 DirectWriteRolloverStrategy so Rhino can send notifications to monitoring systems.

LogNotificationAppender

A log appender for delivering log messages to a listener inside the application. This is used to send log messages to JMX monitoring clients and as SNMP notifications. It is only necessary to use the LogNotificationAppender if using SNMP to receive log messages.

TraceNotificationAppender

A log appender for delivering log messages to a listener inside the application that extracts tracer messages to send as `TraceNotification`s. This is used to send tracer messages to JMX monitoring clients such as REM. It is necessary to use the TraceNotificationAppender if using JMX to receive tracer messages. Without an instance of this appender in the log configuration REM instances connecting to this Rhino instance will not be able to receive or display tracer messages.

PolledMemoryAppender

A log appender that stores messages in an internal buffer that the REM can poll for live log watching. This implementation is only of use when log output is infrequent enough for human monitoring and has a minor performance cost. It will be removed in a future release of Rhino. We recommend that log files or an external log server be used as the primary log output.

See Logging plugins for instructions on enabling additional appender types.

Other plugins

The Log4j 2 project (https://logging.apache.org/log4j/2.x) provides a number of plugins for extending the functionality of Log4j 2. These plugins provide appenders for sending logs to a number of log servers, files and databases, layouts for configuring the format of log messages, and filters to restrict the logging of messages. System integrators or operators can create plugins to add further functionality or support for other log handling systems.

Rhino log configuration properties

Rhino log configuration variables include a rhino namespace containing options useful for providing additional context in log files. These are:

${rhino:node-id}: The node ID of the Rhino node that wrote the log message parameterised with this variable
${rhino:version}: The version of Rhino running at the time the log message parameterised with this variable was written

Tracer objects

SLEE 1.1 provides tracer objects for logging messages from deployed components.

Rhino logs all messages sent to a Tracer under the trace.<notification source>.<tracer name> logger.

In an extension of the SLEE specification Rhino allows configuration of tracer levels at coarser grain than the component tracer. This extended functionality is accessed through the Rhino logging configuration. For example setloglevel trace Finest will set the default tracer level to Finest. All tracers not explicitly set will log at levels from Finest up. To support this SLEE extension root tracers for individual notification sources inherit their levels from the trace logger. It is also permitted to unset the root tracer level for a given notification source using setTracerLevel. Unsetting the root tracer level reverts to using the inherited level.

A further extension of the SLEE specification allows for full use of logging management commands against Tracers. A SLEE 1.1 Tracer may have appenders and filters added to further customise tracing output, both to JMX notifications, and logging destinations. Any supported appender may be used, so logging destinations are not restricted to file only.

Tracer log levels

Log levels for trace messages are logged at the level they are emitted.

About SLEE 1.1 Tracers

Tracer Interface

In SLEE 1.1, there are more components that may need tracing support. In addition to SBBs, trace messages may also be generated by profile abstract classes and resource adaptors, and potentially any other SLEE subsystem.

All of these components may use the SLEE 1.1 javax.slee.facilities.Tracer interface. The Tracer interface will be familiar to users of other logging APIs. It provides methods for generating traces at different trace levels. Details of the tracing methods available are in the javax.slee.facilities.Tracer javadoc.

Obtaining a Tracer

Components obtain Tracers by calling the getTracer() method on the particular component’s context object. Rhino 2.6 provides 'com.opencloud.rhino.facilities.ExtendedTracer' instances when acquiring a Tracer, If only Rhino 2.6 support is required, the Tracer acquired from a context may be safely cast to ExtendedTracer

Older Rhino versions provide a com.opencloud.rhino.facilities.Tracer. The older Rhino implementation does not offer the extended logging API that the ExtendedTracer does.

For backwards compatibility Rhino 2.6 API library contains a com.opencloud.rhino.facilities.trace.TracerAccessor which handles safely acquiring a Rhino 2.6 ExtendedTracer.

Example Tracer acquisition

Component

Tracer access method

SBB

 ExtendedTracer trace = (ExtendedTracer)SbbContext.getTracer(String)

Profiles

ProfileContext.getTracer(String)

Resource Adaptors

ResourceAdaptorContext.getTracer(String) or TracerAccessor.getExtendedTracer(ResourceAdaptorContext, String)

The string parameter in the above methods is the tracer name. This is a hierarchical name, following Java naming conventions, where the different levels in the hierarchy are delimited by a dot. For example, a tracer named "com.foo" is the parent of "com.foo.bar". Components may create any number of tracers, with different names, for different purposes. Tracers inherit the trace level of their parent in the hierarchy. The tracer named "" (empty string) is the top-level or root tracer. The hierarchical naming is a convention used in most logging APIs, and allows an administrator to easily enable or disable tracing for an entire hierarchy of tracers.

Example

import javax.slee.Sbb;
import javax.slee.SbbContext;
import javax.slee.facilities.Tracer;

public abstract class MySbb implements Sbb {

    private Tracer rootTracer;
    private ExtendedTracer fooTracer;
    private SbbContext context;

    public void setSbbContext(SbbContext context) {
        this.context = context;
        this.rootTracer = context.getTracer("");
        this.fooTracer = (ExtendedTracer)context.getTracer("foo");
    }

    ...

    // Generate an INFO trace on the root tracer
    rootTracer.info("An event has occurred");
    ...

    // Generate a WARNING trace on the fooTracer
    fooTracer.warning("Could not combobulate {}", "discombobulator");

Notification Sources

SLEE 1.1 introduces the javax.slee.management.NotificationSource interface, which the SLEE automatically adds to notifications generated by SLEE tracers. As this is automatically asssociated with the Tracer object, there is no need to manually specify source as in SLEE 1.0. This solves the problem of identifying which SBB in which service generated a trace message. The NotificationSource explicity identifies the component that generated the trace, so a management client can easily see which service and SBB the trace came from, allowing filtering by service or SBB.

Tracer Extensions

To alleviate some limitations of the SLEE 1.1 Tracer system, Rhino offers an extended Tracer API. This extended API offers a larger set of tracing methods, to support tracing without string concatenation to build trace messages. Tracer extensions contains details of the Tracer API extensions, and com.opencloud.rhino.facilities.ExtendedTracer javadoc is available.

Rhino 2.6 Tracer Extensions

In Rhino 2.6, the Tracer subsystem has been substantially reworked. As a result, Tracers are now first class loggers. This means that a Tracer may be manipulated by logging management commands as if it were a logger, with the exception that it will only accept Tracer levels.

Tracers now have very long logger names, as they must be unique to support making Tracers first class loggers. In log files these very long names are inconvenient, as they will frequently cause log entries to run over multiple lines on screen. In order to alleviate this issue, we have included a default tracer name abbreviation system.

Tracer pattern converter

The Tracer abbreviator used by default is based heavily on the logger pattern converter supplied with Log4j 2. See Log4j 2 Pattern Layout for documentation.

The tracer pattern converter shipped with Rhino allows for optionally completely removing a logger/tracer name component. In contrast, the logger pattern converter will always leave a . literal to show where elements have been abbreviated. The tracer pattern converter also does not implement Log4j 2 integer precision abbreviation, only pattern abbreviation.

Tracer name

Pattern

Output

trace.default.resourceadaptorentity.simplera.example

 %logger{\*.0.0.*}

 trace...simplera.example

trace.default.resourceadaptorentity.simplera.example

 %tracer{\*.0.0.*}

 trace.simplera.example

Tracer abbreviation behaviour can be managed through REM or by editing an exported logging.xml.

The default tracer pattern converter shipped with Rhino is shown below

Default tracer pattern converters

<component plugin-name="MarkerPatternSelector" >
  <property name="defaultPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSSZ} %-7level [%logger] &lt;%threadName&gt; %mdc %msg{nolookups}%n%throwable"/>
  <component plugin-name="PatternMatch">
    <property name="key" value="Trace"/>
    <property name="pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSSZ} ${plainLevel} [%tracer{*.0.0.*}] &lt;%threadName&gt; %mdc %msg{nolookups}%n%throwable"/>
  </component>
  <component plugin-name="PatternMatch">
    <property name="key" value="SbbTrace"/>
    <property name="pattern" value="%d{yyyy-MM-dd HH:mm:ss.SSSZ} ${plainLevel} [%tracer{*.0.0.*.0.*.*.0.0.*}] &lt;%threadName&gt; %mdc %msg{nolookups}%n%throwable"/>
  </component>
</component>

Note that there are three patterns in use here.

Marker

Pattern

Usecase

None (defaultPattern)

 %logger

Used for non-tracer log messages

SbbTrace

 %tracer{\*.0.0.*.0.\*.*.0.0.*}

Used for Tracer messages logged from an SBB.

Trace

 %tracer{\*.0.0.*}

Used for Tracer messages logged from anything other than an SBB

Different patterns are required for SBB and non-SBB Tracers, due to the more complex notification source identity of SBB notification sources. An SBB notification source includes both SBB id and Service ID. All other notification sources have no equivalent of Service ID.

Creating a File Appender

To create a file appender, use the following rhino-console command or related MBean operation. Since Rhino 2.6 there are many varieties of file appenders supported.

There are two major classes of file appenders discussed below. Non rolling file appenders do not rollover logfiles ever. Rolling file appenders are able to roll over logfiles, and must be configured with automatic rollover rules.

FileName arguments are paths to files, not just filenames. To create a logfile in the configured logging directory (Default is ${NODE_HOME}/work/log) one can use the property ${logDir} as the leading element of the filename

Non rolling file appenders

These appenders cannot be rolled over with the rolloverlogfiles console command.

Console command: `createfileappender`

Command

createfileappender <appenderName> <fileName> [-append <true|false>] [-bufferedIO
<true|false>] [-bufferSize size] [-createOnDemand <true|false>] [-immediateFlush
<true|false>] [-locking <true|false>] [-ignoreExceptions <true|false>] [-pattern
<pattern>]
  Description
    The FileAppender is an appender that writes to the File named in the <fileName>
    parameter.
  Required Arguments
    appenderName  The name of the Appender.
    fileName  The name of the file to write to. If the file, or any of its parent
    directories, do not exist, they will be created.
  Options
    -append  When true, records will be appended to the end of the file. When set to
    false, the file will be cleared before new records are written. The default is
    true.
    -bufferedIO  When true, records will be written to a buffer and the data will be
    written to disk when the buffer is full or, if immediateFlush is set, when the
    record is written. File locking cannot be used with bufferedIO. The default is
    true.
    -bufferSize  The buffer size. The default is 8192 bytes.
    -createOnDemand  When true, the appender creates the file on-demand. The default
    is false.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -locking  When true, I/O operations will occur only while the file lock is held.
    The default is false.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

To create a logfile in the configured logging directory

$ ./rhino-console createfileappender myappender "${logDir}/myappender.log"
Done.

To create a logfile in an absolute path

$ ./rhino-console createfileappender myappender /var/logs/rhino/myappender.log
Done.

Console command: `createrandomaccessfileappender`

Command

createrandomaccessfileappender <appenderName> <fileName> [-append <true|false>]
[-immediateFlush <true|false>] [-bufferSize size] [-ignoreExceptions
<true|false>] [-pattern <pattern>]
  Description
    The RandomAccessFileAppender is an appender that writes to the File named in the
    <fileName> parameter. It is similar to the standard FileAppender except it is
    always buffered.
  Required Arguments
    appenderName  The name of the Appender.
    fileName  The name of the file to write to. If the file, or any of its parent
    directories, do not exist, they will be created.
  Options
    -append  When true, records will be appended to the end of the file. When set to
    false, the file will be cleared before new records are written. The default is
    true.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -bufferSize  The buffer size. The default is 8192 bytes.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

$ ./rhino-console createrandomaccessfileappender myappender "${logDir}/myappender.log"
Done.

Console command: `creatememorymappedfileappender`

Command

creatememorymappedfileappender <appenderName> <fileName> [-append <true|false>]
[-immediateFlush <true|false>] [-regionLength length] [-ignoreExceptions
<true|false>] [-pattern <pattern>]
  Description
    The MemoryMappedFileAppender maps a part of the specified file into memory and
    writes log events to this memory, relying on the operating system's virtual
    memory manager to synchronize the changes to the storage device
  Required Arguments
    appenderName  The name of the Appender.
    fileName  The name of the file to write to. If the file, or any of its parent
    directories, do not exist, they will be created.
  Options
    -append  When true, records will be appended to the end of the file. When set to
    false, the file will be cleared before new records are written. The default is
    true.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -regionLength  The length of the mapped region, defaults to 32 MB.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

$ ./rhino-console creatememorymappedfileappender myappender "${logDir}/myappender.log"
Done.

Rolling file appenders

Console command: `createrollingfileappender`

Command

createrollingfileappender <appenderName> <fileName> <filePattern> <size>
[-append <true|false>] [-bufferedIO <true|false>] [-bufferSize size]
[-createOnDemand <true|false>] [-immediateFlush <true|false>] [-min <min>] [-max
<max>] [-ignoreExceptions <true|false>] [-pattern <pattern>]
  Description
    The RollingFileAppender is an appender that writes to the File named in the
    <fileName> parameter and rolls the file over according the values set by the
    <size> [-min][-max] options.
  Required Arguments
    appenderName  The name of the Appender.
    fileName  The name of the file to write to. If the file, or any of its parent
    directories, do not exist, they will be created.
    filePattern  The pattern of the file name of the archived log file. Both a
    date/time pattern compatible with SimpleDateFormat and/or a %i which represents
    an integer counter can be used.
    size  The file size required before a roll over will occur.
  Options
    -append  When true, records will be appended to the end of the file. When set to
    false, the file will be cleared before new records are written. The default is
    true.
    -bufferedIO  When true, records will be written to a buffer and the data will be
    written to disk when the buffer is full or, if immediateFlush is set, when the
    record is written. File locking cannot be used with bufferedIO. The default is
    true.
    -bufferSize  The buffer size. The default is 8192 bytes.
    -createOnDemand  When true, the appender creates the file on-demand. The default
    is false.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -min  The minimum value of the roll over counter. The default value is 1.
    -max  The maximum value of the roll over counter. Once this values is reached
    older archives will be deleted on subsequent rollovers.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

$ ./rhino-console createrollingfileappender myappender "${logDir}/myappender.log"
Done.

Console command: `createrollingrandomaccessfileappender`

Command

createrollingrandomaccessfileappender <appenderName> <fileName> <filePattern>
<size> [-append <true|false>] [-bufferSize size] [-immediateFlush <true|false>]
[-min <min>] [-max <max>] [-ignoreExceptions <true|false>] [-pattern <pattern>]
  Description
    The RollingRandomAccessFileAppender is an appender that writes to the File named
    in the <fileName> parameter and rolls the file over according the values set by
    the <size>[-min][-max] options. It is similar to the standard
    RollingFileAppender except it is always buffered.
  Required Arguments
    appenderName  The name of the Appender.
    fileName  The name of the file to write to. If the file, or any of its parent
    directories, do not exist, they will be created.
    filePattern  The pattern of the file name of the archived log file. Both a
    date/time pattern compatible with SimpleDateFormat and/or a %i which represents
    an integer counter can be used.
    size  The file size required before a roll over will occur.
  Options
    -append  When true, records will be appended to the end of the file. When set to
    false, the file will be cleared before new records are written. The default is
    true.
    -bufferSize  The buffer size. The default is 8192 bytes.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -min  The minimum value of the roll over counter. The default value is 1.
    -max  The maximum value of the roll over counter. Once this values is reached
    older archives will be deleted on subsequent rollovers.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

$ ./rhino-console createrollingrandomaccessfileappender myappender "${logDir}/myappender.log"
Done.

MBean operations:

Create a Socket Appender

Rhino 2.6 supports two varieties of socket appenders, configurable format socket appenders, and syslog format. To create either use the following rhino-console commands or related MBean operations

Console command: `createsocketappender`

Command

createsocketappender <appenderName> <host> <port> [-bufferedIO <true|false>]
[-bufferSize size] [-connectTimeoutMillis <timeout(ms)>] [-immediateFail
<true|false>] [-immediateFlush <true|false>] [-protocol <protocol>]
[-reconnectionDelayMillis <delay(ms)>] [-keyStoreLocation <location>]
[-keyStorePassword <password>] [-trustStoreLocation <location>]
[-trustStorePassword <password>] [-ignoreExceptions <true|false>]
  Description
    The SocketAppender is an appender that writes its output to a remote destination
    specified by a host and port. The data can be sent over either TCP or UDP and
    the default format of the data is to send a Serialized LogEvent.
  Required Arguments
    appenderName  The name of the Appender.
    host  The name or address of the system that is listening for log events.
    port  The port on the host that is listening for log events.
  Options
    -bufferedIO  When true, records will be written to a buffer and the data will be
    written to disk when the buffer is full or, if immediateFlush is set, when the
    record is written. File locking cannot be used with bufferedIO. The default is
    true.
    -bufferSize  The buffer size. The default is 8192 bytes.
    -connectTimeoutMillis  The connect timeout in milliseconds. The default is 0
    (infinite timeout).
    -immediateFail  When set to true, log events will not wait to try to reconnect
    and will fail immediately if the socket is not available.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -protocol  'TCP' (default), 'SSL' or 'UDP'.
    -reconnectionDelayMillis  If set to a value greater than 0, after an error there
    will be an attempt to reconnect to the server after waiting the specified number
    of milliseconds.
    -keyStoreLocation  The location of the keystore for SSL connections.
    -keyStorePassword  The password of the keystore for SSL connections.
    -trustStoreLocation  The location of the truststore for SSL connections.
    -trustStorePassword  The password of the truststore for SSL connections.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.

Example

$ ./rhino-console createsocketappender myappender localhost 12000
Done.

Console command: `createsyslogappender`

Command

createsyslogappender <appenderName> <host> <port> <facility> [-advertise
<true|false>] [-appName <name>] [-charset <name>] [-connectTimeoutMillis
<timeout(ms)>] [-enterpriseNumber <number>] [-format <name>] [-id <id>]
[-immediateFail <true|false>] [-immediateFlush <true|false>] [-includeMDC
<true|false>] [-mdcExcludes <key1,key2...>] [-mdcId <id>] [-mdcIncludes
<key1,key2...>] [-mdcRequired <key1,key2...>] [-mdcPrefix <prefix>] [-messageId
<msgid>] [-newLine <true|false>] [-reconnectionDelayMillis <delay(ms)>]
[-keyStoreLocation <location>] [-keyStorePassword <password>]
[-trustStoreLocation <location>] [-trustStorePassword <password>]
[-ignoreExceptions <true|false>] [-protocol <protocol>]
  Description
    The SyslogAppender is a SocketAppender that writes its output to a remote
    destination specified by a host and port in a format that conforms with either
    the BSD Syslog format or the RFC 5424 format.
  Required Arguments
    appenderName  The name of the Appender.
    host  The name or address of the system that is listening for log events.
    port  The port on the host that is listening for log events.
    facility  The facility is used to try to classify the message. The facility
    option must be set to one of 'KERN', 'USER', 'MAIL', 'DAEMON', 'AUTH', 'SYSLOG',
    'LPR', 'NEWS', 'UUCP', 'CRON', 'AUTHPRIV', 'FTP', 'NTP', 'AUDIT', 'ALERT',
    'CLOCK', 'LOCAL0', 'LOCAL1', 'LOCAL2', 'LOCAL3', 'LOCAL4', 'LOCAL5', 'LOCAL6',
    or 'LOCAL7'.
  Options
    -advertise  Indicates whether the appender should be advertised.
    -appName  The value to use as the APP-NAME in the RFC 5424 syslog record.
    -charset  The character set to use when converting the syslog String to a byte
    array. The String must be a valid Charset. If not specified, the default system
    Charset will be used.
    -connectTimeoutMillis  The connect timeout in milliseconds. The default is 0
    (infinite timeout).
    -enterpriseNumber  The IANA enterprise number as described in RFC 5424
    -format  If set to 'RFC5424' the data will be formatted in accordance with RFC
    5424. Otherwise, it will be formatted as a BSD Syslog record.
    -id  The default structured data id to use when formatting according to RFC
    5424. If the LogEvent contains a StructuredDataMessage the id from the Message
    will be used instead of this value.
    -immediateFail  When set to true, log events will not wait to try to reconnect
    and will fail immediately if the socket is not available.
    -immediateFlush  When true, each write will be followed by a flush. This will
    guarantee the data is written to disk but could impact performance. The default
    is true.
    -includeMDC  Indicates whether data from the ThreadContextMap will be included
    in the RFC 5424 Syslog record. Defaults to true.
    -mdcExcludes  A comma separated list of mdc keys that should be excluded from
    the LogEvent.
    -mdcId  The id to use for the MDC Structured Data Element.
    -mdcIncludes  A comma separated list of mdc keys that should be included in the
    FlumeEvent.
    -mdcRequired  A comma separated list of mdc keys that must be present in the
    MDC.
    -mdcPrefix  A string that should be prepended to each MDC key in order to
    distinguish it from event attributes
    -messageId  The default value to be used in the MSGID field of RFC 5424 syslog
    records.
    -newLine  If true, a newline will be appended to the end of the syslog record.
    The default is false.
    -reconnectionDelayMillis  If set to a value greater than 0, after an error there
    will be an attempt to reconnect to the server after waiting the specified number
    of milliseconds.
    -keyStoreLocation  The location of the keystore for SSL connections.
    -keyStorePassword  The password of the keystore for SSL connections.
    -trustStoreLocation  The location of the truststore for SSL connections.
    -trustStorePassword  The password of the truststore for SSL connections.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -protocol  'TCP' (default), 'SSL' or 'UDP'.

Example

$ ./rhino-console createsyslogappender myappender localhost 12000 USER
Done.

MBean operations:

Creating a Console Appender

To create a new Console appender, use the following rhino-console command or related MBean operation.

Console command: `createconsoleappender`

Command

createconsoleappender <appenderName> [-follow <true|false>] [-direct
<true|false>] [-target <SYSTEM_OUT|SYSTEM_ERR>] [-ignoreExceptions <true|false>]
[-pattern <pattern>]
  Description
    Appends log events to System.out or System.err using a layout specified by the
    user.
  Required Arguments
    appenderName  The name of the Appender.
  Options
    -follow  Identifies whether the appender honors reassignments of System.out or
    System.err
    -direct  Write directly to java.io.FileDescriptor and bypass
    java.lang.System.out/.err. Can give up to 10x performance boost when the output
    is redirected to file or other process.
    -target  Either 'SYSTEM_OUT' or 'SYSTEM_ERR'. The default is 'SYSTEM_OUT'.
    -ignoreExceptions  When true, exceptions encountered while appending events will
    be internally logged and then ignored. The default is true.
    -pattern  The pattern to use for logging output.

Example

$ ./rhino-console  createconsoleappender myappender -target SYSTEM_OUT
Done.

MBean operations:

Remove an Appender

To remove a no-longer required appender, use the following rhino-console commands, and related MBean methods.

Console command: `removeappender`

Command

removeappender <appenderName>
  Description
    Remove all references to an appender and remove the appender.
  Required Arguments
    appenderName  The name of the Appender.

Example

$ ./rhino-console removeappender TraceNotification
Removed appender: TraceNotification

MBean operations:

removeAppender(String appenderName)

Attaching an Appender to a Logger

To attach an appender to a logger, use the following rhino-console command or related MBean operation.

Console command: `addappenderref`

Command

addappenderref <logKey> <appenderName>
  Description
    Adds an appender for a log key.
  Required Arguments
    logKey  The log key of the logger.
    appenderName  The name of the Appender.

Example

To configure log keys to output their logger’s messages to a specific file appender:

$ ./rhino-console addappenderref root myappender
Added appender reference of myappender to root.

Console command: `removeappenderref`

Command

removeappenderref <logKey> <appenderName>
  Description
    Removes an appender for a log key.
  Required Arguments
    logKey  The log key of the logger.
    appenderName  The name of the Appender.

Example

$ ./rhino-console removeappenderref rhino.main AlarmsLog
Removed appender reference of AlarmsLog from rhino.main.

MBean operations:

Configure a Logger

To configure/reconfigure a Logger, use the following console commands and related MBean methods. Since 2.6, Rhino has offered fully asynchronous logging through asynchronous loggers. Asynchronous logging is based on the idea of returning control to the processing thread as early as possible, for maximum throughput.

Rhino allows any individual logger to be asynchronous. This requires careful setup, as the way that log messages are logged is not entirely straightfoward.

In order to get the expected behaviour, that messages to logger foo are logged asynchronously, and only once, logger foo must be configured as follows:

asynchronous set to true. Make this logger asynchronous
additivity set to false. This prevents double logging of messages if any parent logger also has a reference to the same appenders.
add relevant appender refs. A non-additive logger must have at least one appender ref to log anything
set level. Asynchronous loggers do not inherit levels from synchronous parents.

As a result of this complexity, there is no rhino-console command to set or get asynchronous alone. Configuring an Asynchronous Logger shows an example.

Possible behaviours with Asynchronous Loggers.

An asynchronous logger may not necessarily behave as expected, with all messages always logged asynchronously. To determine the actual behaviour of an asynchronous logger requires examining the whole path back to the first non-additive parent (or root logger)

Configuration

Behaviour

Logger

name        : rhino.main
level       : INFO
additivity  : false
asynchronous: true
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

Parent

name        : root
level       : INFO
additivity  : true
asynchronous: <not configured - default is false>
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

rhino.main logs asynchronously to STDERR, RhinoLog, LogNotification, and PolledMemoryAppender.

Logger

name        : rhino.main
level       : INFO
additivity  : false
asynchronous: true
appenders   : []

Parent

name        : root
level       : INFO
additivity  : true
asynchronous: <not configured - default is false>
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

rhino.main does not log to parent loggers, as it is not additive. rhino.main logs to nowhere as it has no appenders attached to log to.

Logger

name        : rhino.main
level       : INFO
additivity  : true
asynchronous: true
appenders   : []

Parent

name        : root
level       : INFO
additivity  : true
asynchronous: <not configured - default is false>
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

rhino.main logs nothing directly, and logs synchronously to parent logger. As above, rhino.main has no attached appenders to log to. Calls to the parent logger are always synchronous, regardless of logger asynchrony.

Logger

name        : rhino.main
level       : INFO
additivity  : true
asynchronous: true
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

Parent

name        : root
level       : INFO
additivity  : true
asynchronous: <not configured - default is false>
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

rhino.main logs asynchronously all attached appenders, and synchronously to parent logger. This results in every log message doubling up due to shared appenders.

Logger

name        : rhino.main
level       : INFO
additivity  : true
asynchronous: true
appenders   : [mainAppender]

Parent

name        : root
level       : INFO
additivity  : true
asynchronous: <not configured - default is false>
appenders   : [STDERR, RhinoLog, LogNotification, PolledMemoryAppender]

rhino.main logs asynchronously to mainAppender, and synchronously to parent logger. As no appenders are shared by the loggers, this does not result in messages doubling up. However, calls to rhino.main will synchronously log through appenders attached to root.

Console command: `configurelogger`

Command

configurelogger <logKey> [-level <level>] [-additivity <additivity>]
[-asynchronous <asynchronosity>] [-appender <appender-ref>]* [-plugin
<plugin-name>]*
  Description
    Set the configuration for a logger.  At least one option must be specified.
    Plugins can be defined using the defineplugincomponent command.

Example

$ ./rhino-console configurelogger root -level info -additivity true -appender STDERR -appender RhinoLog -appender LogNotification
Created/updated logger configuration for root

Make rhino.management log asynchronously to rhino.log only

$ ./rhino-console configurelogger rhino.manangement -level info -additivity false -asynchronous true -appender RhinoLog
Created/updated logger configuration for rhino.management

Console command: `getloggerconfig`

Command

getloggerconfig <logKey>
  Description
    Get the configuration for a logger.
  Required Arguments
    logKey  The log key of the logger.

Example

$ ./rhino-console getloggerconfig rhino
Logger rhino is not configured

$ ./rhino-console getloggerconfig rhino.main
name        : rhino.main
level       : INFO
additivity  : <not configured - default is true>
asynchronous: <not configured - default is false>
appenders   : []

Console command: `removeloggerconfig`

Command

removeloggerconfig <logKey>
  Description
    Remove the configuration for a logger.
  Required Arguments
    logKey  The log key of the logger.

Example

$ ./rhino-console removeloggerconfig rhino.main
Configuration for logger rhino.main removed

MBean operations:

Mananging a Logger’s Additivity

To specify whether or not a logger is additive, use the following rhino-console command or related MBean operation.

The meaning of "additivity" is explained in the Appender additivity section of the About Logging page.

Loggers are additive by default.

Console command: `setadditivity`

Command

setadditivity <logKey> <additivity>
  Description
    Sets whether the log key inherits the log filter level of its parent logger.
  Required Arguments
    logKey  The log key of the logger.
    additivity  Set to true for enabled, false for disabled, or - to use the
    platform default

Example

To make a logger additive:

$ ./rhino-console setadditivity rhino.foo true
Done.

Console command: `getadditivity`

Command

getadditivity <logKey>
  Description
    Get the configured additivity for a logger.
  Required Arguments
    logKey  The log key of the logger.

Example

To make a logger additive:

$ ./rhino-console getadditivity rhino
Logger rhino is not configured - the default additivity (true) would apply to this log key

$ ./rhino-console getadditivity root
Additivity for root is true

MBean operations

Managing a Logger’s Log Level

To manage the log level for a log, use the following rhino-console command or related MBean operation.

Console command: `setloglevel`

Command

setloglevel <logKey> <logLevel>
  Description
    Set the log level for a logger.
  Required Arguments
    logKey  The log key of the logger.
    logLevel  The log level.

Example

$ ./rhino-console  setloglevel rhino.main info
Log level for rhino.main set to: INFO

Console command: `getloglevel`

Command

getloglevel <logKey>
  Description
    Get the configured log level for a logger. Displays the effective log level if
    no explicit level is set.
  Required Arguments
    logKey  The log key of the logger.

Examples

$ ./rhino-console getloglevel rhino
Logger rhino does not exist but it has sub-loggers.
Log level for rhino is not set.
Effective (inherited) log level is: INFO

$ ./rhino-console getloglevel rhino.main
Log level for rhino.main is: INFO

MBean operations:

Listing Log Appenders

To list available log appenders, use the following rhino-console command or related MBean operation.

Console command: `listappenders`

Command

listappenders
  Description
    List all currently configured appenders.

Example

[Rhino@localhost (#14)] listAppenders
ConfigLog
STDERR
RhinoLog

MBean operations:

getAppenders()

Listing Log Keys

To list log keys, use the following rhino-console command or related MBean operation.

Console command: `listlogkeys`

Command

listlogkeys [-configured <true|false>] [-prefix <prefix>] [-contains <string>]
  Description
    Returns an array of known log keys. If configured is true, return only
    explicitly configured logkeys, otherwise return all known keys.
  Options
    -configured  If true, list only keys with explicit configuration, otherwise list
    all known keys
    -prefix  Limit results to log keys matching prefix
    -contains  Limit results to log keys containing the specified string

Example

[Rhino@localhost (#3)] listlogkeys
fastserialize
framework
framework.bulk.manager
framework.bulk.ratelimiter
framework.csi
framework.dlv
framework.groupheartbeat
framework.mcp
framework.mcp.preop
framework.mcpclient-mplexer
framework.rmi.network
framework.rmi.result
framework.rmi.server
framework.rmi.skeleton.com.opencloud.rhino.configmanager.runtime.ConfigurationStateImpl
...

MBean operations:

Managing Logging Properties

Rhino 2.6 allows logging configuration to use property subsitutions in almost all logging configuration.

To manage properties available for substitution use the following rhino-console commands and related MBean methods.

Console command: `getloggingproperties`

Command

getloggingproperties [-property <property>]
  Description
    Get the values of Logging properties.
  Options
    -property  The name of the Property

Example

$ ./rhino-console getloggingproperties
name                 value
-------------------  -------------------------------------------------------------------------------------------------------------------------------------------------------------
        colourLevel                                                                                                                        %highlight{%-7level}{${consoleColours}}
     consoleColours  SEVERE=RED BRIGHT, WARNING=YELLOW, INFO=GREEN, CONFIG=CYAN, FINE=BRIGHT_BLACK, FINER=BRIGHT_BLACK, FINEST=BRIGHT_BLACK, CLEAR=GREEN, CRITICAL=RED, MAJOR=RED,
             logDir                                                                                                                                      ${sys:rhino.dir.work}/log
   maxArchivedFiles                                                                                                                                                              4
         plainLevel                                                                                                                                                       %-7level
 rhinoVersionHeader                                                                                               %d{yyyy-MM-dd HH:mm:ss.SSSZ} ${rhino:version} log file started%n
6 rows

Console command: `setloggingproperty`

Command

setloggingproperty <property> <value>
  Description
    Set a Logging property. Overwrites if it already exists
  Required Arguments
    property  The name of the Property
    value  The value of the Property

Example

$ ./rhino-console setloggingproperty maxArchivedFiles 5
Set property maxArchivedFiles to 5

Console command: `removeloggingproperty`

Command

removeloggingproperty <property>
  Description
    Remove a logging property if not in use.
  Required Arguments
    property  The name of the Property

Example

$ ./rhino-console removeloggingproperty consoleColours
An error occurred executing command 'removeloggingproperty':

com.opencloud.rhino.configmanager.exceptions.ConfigurationException: Property consoleColours in use by property colourLevel

$ ./rhino-console removeloggingproperty colourLevel
Removed logging property colourLevel

MBean operations:

Define a Plugin Component

Console command: `defineplugincomponent`

Command

defineplugincomponent <alias-name> <plugin-name> [(<property-name>
<property-value>)]* [(-plugin <name>)]*
  Description
    Define a plugin component that can be used with the configurelogger command or
    other plugin definitions.  Plugin definitions exist only in the client, and will
    be lost when the client terminates

Example

[SLEE Stopped] [admin@localhost (#11)] defineplugincomponent fooPattern PatternLayout pattern "%d{yyyy-MM-dd HH:mm:ss.SSSZ} %-7level [%logger] <%threadName> %mdc %msg{nolookups}%n%throwable"
Defined plugin component with name PatternLayout

MBean Methods

buildGenericComponent(String pluginName, String[] properties, CompositeData[] components)

Annotating Log files

To append a message to a given logger, use the following console commands and related MBean methods

Console command: `annotatelog`

Command

annotatelog <logKey> <logLevel> <message>
  Description
    Logs a message to all nodes in the cluster using Rhino's logging subsystem.
  Required Arguments
    logKey  The log key of the logger.
    logLevel  The log level.
    message  The message to log.

Example

To configure log keys to output their logger’s messages to a specific file appender:

$ ./rhino-console annotatelog root info "a log annotation"
Annotating log.
Done.

rhino.log

...
2017-12-04 11:53:23.010+1300 INFO    [root] <GroupRMI-thread-1> {} a log annotation
...

MBean operations:

annotateLog(String logKey, String logLevel, String message, Throwable throwable)

Rolling-Over All Rolling File Appenders

To backup and truncate all existing rolling file appenders, use the following rhino-console command or related MBean operation.

Overriding default rollover behaviour

The default behaviour for log files is to automatically rollover when they reach 100MB in size. You can instead request rollover at any time, using the rolloverAllLogFiles command. (You can also override the default maximum file size before a log file rolls over and the maximum number of backup files to keep, when creating a file appender).

Console command: `rolloverlogfiles`

Command

rolloverlogfiles
  Description
    Triggers a rollover of all existing rolling appenders.

Example

$ ./rhino-console rolloverLogFiles
Done.

MBean operations:

Logging Plugins

Rhino uses the Log4j 2 plugin architecture to support any Log4j 2 appender and allow addition of custom appenders.

See the Apache Log4j 2 Plugins documentation for more information about plugins.

Many of the appenders provided by Log4j 2 have additional dependencies. These are not packaged with Rhino. Some examples include:

Cassandra Appender
JDBC Appender
Kafka Appender

Installing appender dependencies and custom plugins

If you want to use a custom plugin or a Log4j 2 appender that requires additional dependencies, you must put the required JARs into ${RHINO_HOME}/lib/logging-plugins/. Any jars found in this directory are added to the core logging classloader.

Files in ${RHINO_HOME}/lib/logging-plugins are only scanned at node boot time.

This classloader is not visible to the main application classloaders. Because it is isolated, it can contain versions of libraries that would otherwise conflict with versions of libraries deployed in the SLEE. (It can’t contain multiple versions of the same library if different appenders require different versions.)

Custom plugins may affect the stability of Rhino nodes.

Custom plugins

Log4j 2 provides multiple mechanisms for plugin discovery. The only mechanism supported by Rhino is use of the Log4j 2 annotation processor during the plugin build phase.

The Log4j 2 annotation processor works by scanning for Log4j 2 plugins and generating a metadata file in your processed classes.

The Java compiler will automatically pick up the annotation processor if it is in the classpath. If the annotation processor is disabled during the compilation phase, you must add another compiler pass to your build process that does annotation processing for org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor.

Importing a Rhino export

Dependencies for logging plugins are not included in a Rhino export, even if there is configuration that requires those dependencies. So when using rhino-export and importing into a new Rhino instance, the logging plugins must be copied manually. Copy ${RHINO_HOME}/lib/logging-plugins to the new Rhino location.

Rhino upgrades

The Log4j 2 version may be updated in new Rhino releases, if bug fixes in the core implementation are needed. When this happens, it is possible that plugins may need to updated for the new Log4j version. Changes to the Log4j version will be documented in the Rhino changelog.

Known Issues

Highlighting of alarm and tracer levels does not work in colour console (LOG4J2-1999, LOG4J2-2000, LOG4J2-2005)
Rolling file appenders do not support live reconfiguration between time-based and sequential numbering (LOG4J2-2009)
The createOnDemand configuration option for file appenders, including rolling file appenders does not work if the appender layout has a header (LOG4J2-2027)

Staging

Staging refers to the micro-management of work within the Rhino SLEE.

This work is divided up into items, executed by workers. A system-level thread represents each worker. You can configure the number of threads available to process items on the stage, to minimise latency, and thus increase the performance capacity of the SLEE.

The staging-thread system

Rhino performs event delivery on a pool of threads, called staging threads. The staging-thread system operates a queue of units of work for Rhino to perform, called stage items. Typically, these units of work involve the delivery of SLEE events to SBBs. A stage item enters staging in a processing queue. Then, the first available staging thread removes it, to perform its associated work. How long the thread spends in the staging queue, before a stage worker processes it, contributes to the overall latency in handling the event. Thus, it is important to make sure that the SLEE is using staging threads optimally.

Tunable parameters

To improve performance, you can tune the following staging parameters: maximumSize, threadCount, maximumAge, queueType.

The node must be restarted for any change in maximumSize, maximumAge, or queueType to take effect.

For instructions on tuning staging parameters, see Configuring Staging Parameters. You can observe the effects of configuration changes in the statistics client by simulating heavy concurrency using a load simulator.

`maximumSize`

Description

Maximum size of the staging queue. Determines how many stage items may be queued awaiting processing. When the queue reaches maximum size, the SLEE automatically fails and removes the oldest item, to accomodate new items.

Default

Recommendation

The default works well for most scenarios. Should be high enough that the SLEE can ride out short bursts of peak traffic, but not so large that under extreme overload stage items wait in the queue for too long (to be useful to the protocol generating the event), before being properly failed.

`threadCount`

Description

Number of staging threads in the thread pool.

Of all staging parameters, this has the greatest impact on overall event-processing latency. To achieve optimal performance, give careful attention to tuning the thread count.

Default

Recommendation

The default works well for many applications on a wide range of hardware. However for some applications, or with hardware using four or more CPUs, more staging threads may be useful. In particular, when the SLEE is running services that perform high-latency blocking requests to an external system, more staging threads may often be necessary.

For example, for a credit-check application that only allows a call setup to continue after performing a synchronous call to an external system:

If a credit check takes on average 150ms, the staging thread that processes the call-setup event will be blocked and unable to process other events for 150ms.
With the default configuration of 30 staging threads, such a system would be able to handle an input rate of approximately 200 events/second. Above this rate, the stage worker threads will not be able to service event-processing stage items fast enough, and stage items will begin to back up in staging queues, eventually causing some calls to be dropped.
The problem is easily solved by configuring a higher number of staging threads.

In real-world applications, it is seldom a matter of applying a simple formula to work out the optimal number of staging threads. Instead, performance-monitoring tools would be used to examine the behaviour of staging, alongside such metrics as event-processing time and system-CPU usage, to find a suitable value for this parameter.

`maximumAge`

Description

Maximum possible age of a staging item, in milliseconds. Determines how long an item of work can remain in the staging queue and still be considered valid for processing. Staging threads automatically fail and remove stage items that stay in the staging queue for longer than this maximum age. Tuning this (along with maximumSize), helps determine your application’s behaviour under overload conditions.

Default

`queueType`

Description

Determines ordering of the staging queue. These options are available:

LIFO ("Last In First Out") — the newest item in the queue is processed first
FIFO ("First In First Out") — the oldest item in the queue is processed first
transfer — acts as many FIFO queues, and may perform better under high load on systems with many processors (introduced in Rhino 2.3.1.7).

Default

 LIFO

Recommendation

The default LIFO behaviour works well for most scenarios. In situations where shorts bursts of work exceed capacity then newer work items will see prompt handling at the expense of lengthened delays for those already waiting. In contrast, FIFO behaviour will see delays hit all items in the queue until the queue is cleared.

Configuring Staging Parameters

To configure staging parameters, use the following rhino-console commands or related MBean operations.

`configurestagingqueues` command

Command

configurestagingqueues [maximumAge <age>] [maximumSize <size>] [threadCount
<count>]
  Description
    set some or all of the staging-queues configuration properties

Example

$ ./rhino-console configurestagingqueues maximumAge 11000 maximumSize 4000 threadCount 40
Updated staging-queue config properties:
maximumSize=4000
threadCount=40
maximumAge=11000

`getstagingqueuesconfig` command

Command

getstagingqueuesconfig
  Description
    get the staging-queues configuration properties

Example

$ ./rhino-console getstagingqueuesconfig
Configuration properties for staging-queues:
maximumAge=11000
maximumSize=4000
threadCount=40

MBean operations

Use the following MBean operations to configure staging queue parameters, defined on the Staging Queue Management MBean interface.

Operations

Usage

To get and set the maximum number of items permitted in the staging queue:

public int getMaximumSize()
    throws ConfigurationException;
public void setMaximumSize(int size)
    throws ConfigurationException, ValidationException;

getMaximumAge
setMaximumAge

To get and set the maximum age of items permitted in the staging queue:

public long getMaximumAge()
    throws ConfigurationException;
public void setMaximumAge(long ms)
    throws ConfigurationException, ValidationException;

Queued work items do not immediately expire if their age (measured in milliseconds) exceeds the maximum allowed. Instead, the SLEE discards them when they leave the staging queue (when it’s their turn for processing).

To skip checking the age of queued items (so they can’t get "too old" for processing), you can set this parameter to -1.

getThreadCount
setThreadCount

To get and set the number of threads available for processing items on the staging queue:

public int getThreadCount()
    throws ConfigurationException;
public void setThreadCount(int threads)
    throws ConfigurationException, ValidationException;

Object Pools

The Rhino SLEE uses groups of object pools to manage the Java objects representing SBBs and profile tables.

Throughout the lifecycle of an object, it may move from one pool to another. Although the defaults are generally suitable, each object pool’s maximum size can be configured if needed.

Pools

There are several types of object pools, however Metaswitch recommends that only the initial pooled pool sizes be changed by system administrators. The other pool sizes are best set during performance testing and only after the maximum workload without tuning has been determined. When tuning the pool sizes consideration should be given to the maximum load nodes are expected to process and the memory consumed by the pools.

The JAIN SLEE specification describes the purpose of the object pools with respect to SBBs:

The SLEE creates and manages a pool of SBB objects. At runtime, the SLEE may assign zero or more SBB objects to represent an SBB entity. When an SBB object is assigned to represent an SBB entity, the SBB object is in the Ready state (see Section 6.3). It can receive and fire events, receive synchronous method invocations, and access and update the persistent state of the SBB entity. Another viewpoint is that the SBB object caches a copy of the persistent data of the SBB entity to provide transactional semantics.

Rhino has five types of object pool. Each is managed per-service or per-profile table, if a service or profile table does not have a pool configuration it inherits the default configuration for its type.

SBB object and profile object pools
Pooled pool	Contains SBB objects and profile objects in the `Pooled` state. This means that the object context has been initialised but no SBB entity or profile is associated with the object. If the pool is empty, the SLEE must create and initialise a new object the next time it needs one. This may take time, particularly if the `setSbbContext()` or `setProfileContext()` method on the object performs a lengthy initialisation. To reduce the impact of object initialisation, the pool may be pre-populated with initialised objects, either at service activation/profile table creation or at SLEE startup time. By default, the Rhino SLEE pre-populates this pool with 50 initialised objects for services and zero for profile tables. This initial pool size can be configured with the `initialPooledPoolSize` configuration attribute.
Ready pool	Contains SBB objects and profile objects in the `Ready` state. `Ready` means that the object is associated with the most recent version of an SBB entity or profile. For SBB objects this means that the object is ready to receive and process events. On startup this pool is always empty. It is populated only with objects from the stale pool or pooled pool, or objects created on demand if the pooled pool was empty.
Stale pool	Contains SBB objects and profile objects that are associated with an SBB entity or profile that has been modified in another transaction. This pool exists as a partner for the ready pool to avoid unnecessary calls to `sbbActivate()` and `sbbPassivate`, or `profileActivate()` and `profilePassivate()`. Objects in the stale pool are associated with out-of-date state and must be resynchronised with their persistent state before they can be used. On startup this pool is always empty. It is populated with objects from the ready pool if and when they become stale.

CMP field object pools
Persistent state pool	A persistent state object holds the MemDB representation of CMP and CMR field data for an SBB entity or profile. A new persistent state object is required for every transaction in which CMP or CMR field data is updated. The purpose of the persistent state pool is to reduce the GC impact caused by the cycling of these objects as SBB entities and profiles are created, updated, and removed.
State pool	State objects provide the interface between SBB entities and profiles and the persistent state objects holding their CMP and CMR field data. State objects are associated with an SBB object or profile object when the object is associated with an SBB entity or profile. The state pool should be configured to be at least the size of the ready pool. The maximum amount of state objects in use at any one time, and thus the maximum recommended state pool size, is limited to the sum of: the size of the ready pool; the size of the stale pool; and the number of event processing threads.

Object pool statistics are available in the ObjectPools parameter set.

Configuring Object Pools

The configuration of object pools is structured as follows:

A global defaults object pool configuration contains the set of base defaults.
A service defaults object pool configuration contains the default configuration for services. When a service is deployed, its initial object pool configuration is copied from the service defaults configuration.

If, for some reason, the service defaults configuration is missing when it is required, it will be recreated based on the global defaults configuration.
A profile table defaults object pool configuration contains the default configuration for profile tables. When a profile table is created, its initial object pool configuration is copied from the profile table defaults configuration.

If, for some reason, the profile table defaults configuration is missing when it is required, it will be recreated based on the global defaults configuration.
When a new namespace is created, each of the default object pool configurations for that namespace are initialised as a copy of the corresponding configurations from the default namespace.

Object pools can be configured, for example, with rhino-console using the following commands:

getobjectpoolconfig can be used to view an object pool configuration; and
configureobjectpools can be used to change the sizes of the object pools in a configuration.

Please see the online help in rhino-console for more information on using these commands.

Alternatively, MBeans operations can be used to configure object pools. Please see:

ObjectPoolManagementMBean for obtaining the ObjectName of Object Pool MBeans; and
ObjectPoolMBean for managing an individual object pool configuration.

The useDefaults flag of an object pool configuration is deprecated and no longer has any function.

Licenses

As well as an overview of licenses, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Listing current licenses

 `listlicenses`

License Management → getLicenses

Getting licensed capacity

 `getlicensedcapacity`

License Management → getLicensedCapacity

Installing a license

 `installlicense`

License Management → install

Uninstalling a license

 `uninstalllicense`

License Management → uninstall

About Licenses

To be activated, services and resource adaptors need a valid license loaded into Rhino (at least for core or "default" functions). See the following details on license properties, validity, alarms, statistics and an example.

License properties

Each license has the following properties;

a unique serial identifier
a start date (before which the license is not valid)
an end date (after which the license is not valid)
a set of licenses that are superseded by this license
licensed-product functions — for the Rhino family of products, these are "Rhino" (used by the production Rhino build for its core functions) and "Rhino-SDK" (used by the SDK Rhino build for its core functions)
licensed-product versions
licensed-product capacities
one or more descriptive fields (optional, not actually used for licensing calculations).

Each license can contain one or more sets of (function, version, capacity). For example, a license could be for "Rhino-SDK, version 2.1, 1000" as well as "Rhino, version 2.1, 500".

Highest-capacity licenses display on startup

Licenses display when Rhino starts up — not the complete list, but only those with the highest licensed capacity for each licensed function/version. (If you have a big license and a small license for the same function/version installed, only the largest will display on startup.)

License validity

A license is considered valid if:

The current date is after the license start date, but before the license end date.
The list of license functions in that license contains the required function.
The list of product versions contains the required version.
The license is not superseded by another.

If Rhino finds multiple valid licenses for the same function, it uses the one with the largest licensed capacity.

Upon activating a service or resource adaptor, Rhino checks the list of functions that they require against the list of installed valid licenses. If all required functions are licensed, the service or resource adaptor will activate. (If one or more functions is unlicensed, they will not activate.)

Licensing applies to explicit activation (by way of a management client) and implicit activation (on Rhino restart). There is one exception: if a node joins an existing cluster that has an active service for which there is no valid license, the service does become active on that node.

In the production Rhino, services and resource adaptors that are already active will continue to successfully process events for functions that are no longer licensed, such as when a license has expired. For the SDK Rhino, services and resource adaptors that are already active will stop processing events for the core "Rhino-SDK" function if it becomes unlicensed, typically after a license has expired.

License alarms

Typically, Rhino raises an alarm when:

A license has expired.
A license is due to expire in the next 7 days.
License units are being processed for a currently unlicensed function.
A license function is processing more accounted units than it is licensed for. The audit log shows how long it’s been over limit.

System administrators are responsible for verifying and canceling alarms (through the management console (command console or Rhino Element Manager.

Cancelled capacity alarms are re-generated for licensed functions that continue to run over capacity.

Enforcing license limits

Production Rhino never enforces the "hard limit" on a license. The SDK version of Rhino will enforce the "hard limit" on the core "Rhino-SDK" function, by rejecting incoming work.

Contact your Metaswitch sales representative or Metaswitch support if you require a greater capacity Rhino SDK license during development.

Audit logs

Rhino SLEE generates a license audit log in the same directory where other logs are stored. The Rhino SLEE system administrator can use the log for a self-audit as required. Metaswitch support may also request the audit log to perform a license audit. Audit logs are protected from external tampering using an HMAC (Hash-based Method Authentication Code) checksum on each line output to the log.

Audit logs are subject to "rollover", just like any other rolling log appender log — for a full audit log for a particular period, several logs may need to be concatenated. (Older logs are named audit.log.0, audit.log.1, and so on.)

License statistics

The standard Rhino SLEE statistics interfaces include:

the root License Accounting statistic
statistics for each function, with both accountedUnits and unaccountedUnits values.

Only accountedUnits count towards licensed limits. Rhino records unaccountedUnits for services and resource adaptors with licensed functions configured as accounted="false".

Sample Rhino license

License:       1194a455e0b
  Issued to:   Umbrella Corporation
  Created at:  Mon Apr 14 12:11:10 NZST 2008
  Valid after: Mon Apr 14 12:11:10 NZST 2008
  Valid until: Fri Jun 13 12:11:10 NZST 2008
  Allows functions:
    Rhino version 2.0, 500 capacity
    Rhino-SDK version 2.0, 500 capacity

License Audit Log Format

License audit logs track information over time, about cluster membership, installed licenses, and license function usage.

Each line in the audit logs describes one of these items at a particular time, as detailed below. All lines start with an HMAC (Hash-based Method Authentication Code) checksum for the line followed by a full time stamp which includes a GMT offset.

Every Rhino node writes an audit log; but all audit logs detail cluster-wide information and usage statistics (not per-node information).

Cluster membership

These lines in the audit logs show the current set of node IDs following a cluster membership change.

When logged

Whenever the active node set in the cluster changes.

Format

<checksum>,<timestamp>, CLUSTER_MEMBERS_CHANGED, [<comma>,<separated>,<node>,<list>]

Example

8c7297fec286a6209307920ce2ed6fb7c562099fce760cd8b23721bdb934e81d, 2022-03-21 15:45:57 +0000, CLUSTER_MEMBERS_CHANGED, [101,102]

Installed licenses

These lines in the audit logs list and describe changes to installed licenses.

When logged

Whenever the set of valid licenses changes. For example, when:

a license is installed or removed
an installed license becomes valid
an installed license expires.

Format

<checksum>, <timestamp>,LICENSE,"<license description>"

Example

1a757486f0280bc05f5aa11b61de92a6480401fc06c7c85b757d892204d6720a, 2022-03-21 15:45:46 +0000,LICENSE,"[LicenseInfo serial=116eaaffde9,validFrom=Mon May 10 14:53:49 NZST 2021,...]"

License function usage

These lines in the audit logs show the following information about license function usage:

the start and end timestamps
number of records
minimum, maximum, and average times (each logged period is made up of several smaller periods).

When logged

Every ten minutes.

Format

Each line represents a single license function, in the following format:

<checksum>, <timestamp>, <startTimeMillis>, <endTimeMillis>, <intervalMillis>, <nodeCount>, <function>, <totalAccounted>, <avgAccounted>, <totalUnaccounted>, <avgUnaccounted>, <capacity>

where the fields are as follows:

 <startTimeMillis>

the milliseconds timestamp of the start of this accounting period

 <endTimeMillis

the milliseconds timestamp of the end of this accounting period

 <intervalMillis>

the length in milliseconds of this accounting period (<endTimeMillis> - <startTimeMillis>)

 <nodeCount>

the number of nodes in the cluster at the time

 <function>

the Rhino license function whose usage is described by this record

 <totalAccounted>

total units accounted against <function> for this accounting period

 <avgAccounted>

the average number of accounted units per second against <function> for this accounting period

 <totalUnaccounted>

the total number of unaccounted units against <function> for this period (not counting towards licensed capacity, but presented for informational purposes)

 <avgUnaccounted>

the average number of unaccounted units per second against <function> for this period (not counting towards licensed capacity, but presented for informational purposes)

 <capacity>

current licensed capacity for <function>

Example

a1a9ad29af063e3447db10c8f4432b1843b151d2c5418917f99a4e0458af6887, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino, 17690, 29.48, 0, 0.00, 10000
dafdc99203cbb0ca5a51c16b286450efc0aea73b26b203e3c998032fc92cd352, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS, 6454, 10.76, 0, 0.00, 10000

Sample log file

1a757486f0280bc05f5aa11b61de92a6480401fc06c7c85b757d892204d6720a, 2022-03-21 15:45:46 +0000,LICENSE,"[LicenseInfo serial=179568e2ee7,licensee=Open Cloud Limited (Test license),..."
5d4d60cc73eab2d0b4444473225d8ab25b40583682f17835d9a196bbf08fffce, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-TPS, 0, 0.00, 0, 0.00, -1
9fd1d0d8cd6c253ce95c31b1e559cdcc2df48fb05f52e69697f11374013bfdd7, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-External-Service, 0, 0.00, 0, 0.00, 0
ad454bb83c2e107dc7beb9429d0b10294c9f25b77b5cd553e927e4dbe4ef9633, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-SMS, 0, 0.00, 0, 0.00, 10000
b3e72244b0c7d3f3087ec4de9ddb6e4fd9b11f53220287acdf885426f62a5088, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS, 0, 0.00, 0, 0.00, 10000
72472ba40f76adec16e79dadea8de4bfe9bd3f21aec19a5b413a7b0c8b33299e, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN, 0, 0.00, 0, 0.00, 10000
9c1c7a287b83621345b8713fc8cbdc5e0d7cf00e878f5d1084e7b56d536ebe82, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-Protocol-CAP, 0, 0.00, 0, 0.00, 0
e34dbbf6e2be929dcd378692bbd4b1806df5385684ff438fca34276fe9971ccd, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino, 0, 0.00, 0, 0.00, 10000
704925656fca4281721624c3d8a44932c45a2b93322176f50545e3b5473b2c31, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-Local-Service, 0, 0.00, 0, 0.00, 0
f740e80afea471aa710d65cda4cd9aa9f74a274d562e606a2916f7fd334cde1b, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-Voice, 0, 0.00, 0, 0.00, 10000
105b4c6c8434ec2b2c76806e925084e3a1a68912e0f6ac600092a9fce234a539, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-Protocol-ETSI-INAP-CS1, 0, 0.00, 0, 0.00, 0
bc6c2b572649312f1c70006ee0651606c05d851689c975d25695264ba3556eb5, 2022-03-21 15:55:48 +0000, 1647877548591, 1647878148591, 600000, 2, Rhino-SIS-IN-SPS, 0, 0.00, 0, 0.00, 0
8136300188cb27e3f485795eab499a68d941823bd9a12b7dfb648c4445968dc3, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-TPS, 0, 0.00, 0, 0.00, -1
20f9ca5e038e6a78e28805d3a483411ca0d15a75af415564a989ea46ab1f1a00, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-External-Service, 0, 0.00, 0, 0.00, 0
cacdf27553e8d52d09b10d4825f75e07752dad18b4850451bd407dc384285647, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-SMS, 0, 0.00, 0, 0.00, 10000
dafdc99203cbb0ca5a51c16b286450efc0aea73b26b203e3c998032fc92cd352, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS, 6454, 10.76, 0, 0.00, 10000
5adaa740343637b3284cff421824149a22b6ed2853df6ff14df33e506b9678f5, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN, 6453, 10.76, 0, 0.00, 10000
d35da58ee2b81cf309123a3db203e036dbb3dc23e34477eefd7dc113d9d76024, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-Protocol-CAP, 0, 0.00, 0, 0.00, 0
a1a9ad29af063e3447db10c8f4432b1843b151d2c5418917f99a4e0458af6887, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino, 17690, 29.48, 0, 0.00, 10000
41a2657135c3d2c90ffe8a432c78141e8a6f3b87d72633fbd03b7d6b8d79c0ca, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-Local-Service, 0, 0.00, 0, 0.00, 0
e786da7cac6035af3ebc8113c66ee65818791187c4743f27debeb8d716b1f9ea, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-Voice, 6454, 10.76, 0, 0.00, 10000
c0cf5a8ae5abaaa8c44ba9dcc83b54f6dda60b265cc6c455c2af701c91955a32, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-Protocol-ETSI-INAP-CS1, 0, 0.00, 0, 0.00, 0
555c83c9d1fae7bd57c041cd7f296e6ce382816b28b31cf2a592526b0d26f6f1, 2022-03-21 16:05:48 +0000, 1647878148591, 1647878748590, 599999, 2, Rhino-SIS-IN-SPS, 0, 0.00, 0, 0.00, 0
13ad53e15c6de948fa7c77c7b4318606dd600ddef7ebef1bbe1cbe8ee5cb8795, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-TPS, 0, 0.00, 0, 0.00, -1
055413c02406b7eb4cb9b3303fa3eef50940929a6b1173fc1082bcb0d276fa02, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-External-Service, 0, 0.00, 0, 0.00, 0
8c10fcea238143abdc4cf411b0ab67b5fc89ceb55c9f3d3550dd42ae0e7bf961, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-SMS, 0, 0.00, 0, 0.00, 10000
04bb9793abd5e1e8102a95cb77b146b8a875840eb2e366076e0d44526de993f6, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS, 7474, 12.46, 0, 0.00, 10000
959b430a644d8bac8ce0c70e7b6d5b5accf4d053d19fa7aeabc32f396d54302b, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN, 7475, 12.46, 0, 0.00, 10000
4ece93de1af46e3ceb541ed9e968fb388df8981b26184e3cb2ef92b260abbc48, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-Protocol-CAP, 0, 0.00, 0, 0.00, 0
ecde669ac6a14dedd4b6cd8c73170be5bccc011324663706e1a8afc30914818e, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino, 20456, 34.09, 0, 0.00, 10000
6e29cb20d3b8c28d269e3adeb3d6b224cf9716e0d0b0ead4667adfabca7d24dc, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-Local-Service, 0, 0.00, 0, 0.00, 0
813dbe19c08bddc3042d901626360b5f5856d06165228ec353076c33b7b5d309, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-Voice, 7474, 12.46, 0, 0.00, 10000
095caf0c508ba3804eaa761118c80f2b62121d58404a81100855516696da1915, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-Protocol-ETSI-INAP-CS1, 0, 0.00, 0, 0.00, 0
a23ead3862769c27b6b0d4360a04c5a72eae2852cc06107767d6825d31d9c62c, 2022-03-21 16:15:48 +0000, 1647878748590, 1647879348591, 600001, 2, Rhino-SIS-IN-SPS, 0, 0.00, 0, 0.00, 0

Listing Current Licenses

To list current licenses, use the following rhino-console command or related MBean operation.

Console command: `listlicenses`

Command

listlicenses <brief|verbose>
  Description
    Displays a summary of the currently installed licenses

Example

$ ./rhino-console listLicenses
Installed licenses:
[LicenseInfo serial=107baa31c0e,validFrom=Wed Nov 23 14:00:50 NZDT 2008,
validUntil=Fri Dec 02 14:00:50 NZDT 2008,capacity=400,hardLimited=false,
valid=false,functions=[Rhino],versions=[2.1],supersedes=[]]
[LicenseInfo serial=10749de74b0,validFrom=Tue Nov 01 16:28:34 NZDT 2008,
validUntil=Mon Jan 30 16:28:34 NZDT 2009,capacity=450,hardLimited=false,
valid=true,functions=[Rhino,Rhino-IN-SIS],versions=[2.1,2.1],
supersedes=[]]
Total: 2

In this example, two licenses are installed:

107baa31c0e, enabling the Rhino function
10749de74b0, enabling the Rhino and Rhino-IN-SIS functions.

Both are for Rhino version 2.1.

MBean operation: `getLicenses`

MBean

Rhino operation

public String[] getLicenses()

Getting Licensed Capacity

To get the licensed capacity for a specified product and function (to determine how much throughput the Rhino cluster has), use the following rhino-console command or related MBean operation.

Console command: `getlicensedcapacity`

Command

getlicensedcapacity <function> <version>
  Description
    Gets the currently licensed capacity for the specified function and version

Example

$ ./rhino-console getlicensedcapacity Rhino 2.1
Licensed capacity for function 'Rhino' and version '2.1': 450

MBean operation: `getLicensedCapacity`

MBean

Rhino operation

public long getLicensedCapacity(String function, String version)

Installing a License

To install a license, use the following rhino-console command or related MBean operation.

Console command: `installlicense`

Command

installlicense <license file>
  Description
    Install the license at the specified SLEE-accessible filename or file: URL

Example

$ ./rhino-console installLicense file:/home/user/rhino/rhino.license
Installing license from file:/home/user/rhino/rhino.license

License files must be on the local filesystem of the host where the node is running.

MBean operation: `install`

MBean

Rhino operation

Install a license from a raw byte array

public void install(byte[] filedata)
        throws LicenseException, LicenseAlreadyInstalledException, ConfigurationException

Uninstalling a License

To uninstall a license, use the following rhino-console command or related MBean operation.

Console command: `uninstalllicense`

Command

uninstalllicense <serial id>
  Description
    Uninstalls the specified license

Example

To uninstall the license with serial ID "105563b8895":

$ ./rhino-console uninstalllicense 105563b8895
Uninstalling license with serial ID: 105563b8895

License files must be on the local filesystem of the host where the node is running.

MBean operation: `uninstall`

MBean

SleeManagementMBean.getLimiterManagementMBean()

Rhino operation

public void uninstall(String serial)
        throws UnrecognizedLicenseException, ConfigurationException

Rate Limiting (Overload Control)

Rhino’s carrier-grade overload control manages the rate of work Rhino accepts, processes, and generates for other systems. A Rhino administrator can:

design the entire end-to-end system query, transaction, and session/message rates
configure fine-grained processing rates, so Rhino can participate in the entire end-to-end system without overloading other network equipment.

Rate limiting can be useful in high-load situations, to ensure that once an activity starts it can run to completion without overloading the SLEE or related systems.

Granular control using "rate limiters"

Rhino administrators use rate limiters to set specific "flow rates" for different types of messages. These limit the number of messages per second that the SLEE will process or generate, and throttle-back the flow for certain messages in favor of others. Rhino also lets administrators monitor and refine rate-limiter parameters, to achieve desired throughput.

This section includes the following topics:

About Rate Limiting
Managing Rate Limiting
Monitoring Limiter Statistics
Using Alarms with Limiters

About Rate Limiting

For rate limiting with the Rhino SLEE:

An administrator creates limiters and assembles them into hierarchies.
The administrator connects those limiters to limiter endpoints.
RAs and SBBs determine the number of units needed for a particular piece of work.
RAs, SBBs, and Rhino code use limiter endpoints to determine if a piece of work can be done (for example, if a message can be processed or sent).

Per-node configuration

Some limiter properties can be overridden on a per-node basis (a value set this way is called a per-node value). For example, a rate limiter’s maximum allowed rate could be set differently for different sized machines.

Each node always independently maintains the working state of each limiter (counts of units used and so on).

What are limiters?

A limiter is an object that decides if a piece of work can be done or not. How the decision is made depends on the type of limiter. Limiters are always created and removed "globally". That is, they always exist on all nodes in the cluster.

Limiter names

Each limiter has a name. A limiter’s name must be globally unique within the scope of the Rhino instance.

Name character restriction

The limiter name cannot include the "/" character.

See also Limiter Types for details on limiter properties, and Managing Limiters for procedures to create, remove, set properties, inspect, and list limiters.

Limiter hierarchies

Limiters can optionally be linked to a single parent limiter and/or multiple child limiters. A limiter only allows a piece of work if all of its ancestors (its parent, its parent’s parent, and so on) also allow the work. You configure a hierarchy by setting the parent property on each limiter.

The limiter hierarchy is the same on all nodes — per-node hierarchies are not possible. (Nor is it possible to create a cycle among parent/child limiters.)

Bypassing a limiter

All limiters have a bypassed property. If the flag is true, then the limiter itself takes no part in the decision about allowing work. If it has a parent, it delegates the question to the parent. If is doesn’t have a parent, it always allows all work.

Rhino has no concept of enabling or disabling a limiter. Instead, you use the bypassed property.

Default limiter hierarchy

By default Rhino has two limiters, with the following configuration:

Name

Type

Parent

Bypassed

Configuration

 QueueSaturation

 QUEUE_SATURATION

 <none>

❌

 maxSaturation=85%

 SystemInput

 RATE

 QueueSaturation

✅

 maxRate=0
timeUnit=seconds
depth=1

So by default, limiting only happens when the event staging queue is 85% or more full. Both limiters can be reconfigured as necessary. QueueSaturation can be removed, but SystemInput cannot (although it doesn’t have to be used for anything).

Limiter endpoints

A limiter endpoint is the interface between code that uses rate limiting, and the rate-limiting system itself. Administrators cannot create limiter endpoints — they are created as part of RA entities and SBBs. The only configuration available for a limiter endpoint is whether or not it is connected to a limiter. Limiter endpoints are not the same as SLEE endpoints — they are different concepts.

Endpoints in RAs and SBBs

RAs and SBBs may have any number of limiter endpoints, and there is no restriction on what they can be used for. Documentation with each RA or SBB should list and explain the purpose of its limiter endpoints. Typical uses include throttling output of messages to external resources and throttling input messages before passing them to the SLEE.

RA "Input" endpoints

The SLEE automatically creates a limiter endpoint named RAEntity/<entityname>/Input for every RA entity. These endpoints let the SLEE throttle incoming messages from RA entities. By default each "Input" endpoint is connected to the built-in "SystemInput" limiter, but the administrator can connect or disconnect it to another limiter.

The SLEE will try to use one unit on the "Input" endpoint every time a new activity is started. If the endpoint denies the unit then the SLEE rejects the activity. The SLEE will forcibly use one unit every time the RA passes in an event or ends an activity. This functionality is built into Rhino, and automatically happens for all RA entities, regardless of whether or not they use other limiter endpoints.

See also Managing Limiter Endpoints, for procedures to list limiter endpoints, connect them to and disconnect them from a limiter, and find which limiter is connected to them.

What are units?

Units are an abstract concept representing the cost of doing a piece of work. For example, one unit might represent a normal piece of work, so three units indicate a piece of work that needs three times as much processing.

The RA or SBB determines the number of units used for a particular piece of work. Some might be configurable through configuration properties or deployment descriptors. This information should be documented for each individual RA or SBB.

Using units

Code can ask an endpoint "Can I do x units of work?". If the endpoint is connected to a limiter, the limiter will answer yes or no. If the endpoint is not connected to a limiter, the answer is always yes. If the answer is yes, the units are said to have been used. If the answer is no, the units are said to have been rejected.

Code can also tell the endpoint "I am doing x units of work that cannot be throttled". The endpoint passes this message to a limiter if connected (otherwise, it ignores the message). The units in this case are said to have been forcibly used.

Future limiter decisions do not differentiate between units used and those forcibly used. Rhino just counts both as having been "used".

Example

The following diagram illustrates an example rate limiting configuration, with two limiter hierarchies. Incoming traffic from the MSC is limited by the "FromSwitch" limiter and the limiters further up the chain. The Location and SRI services have defined limiter endpoints, which the administrator has connected (directly or indirectly) to the "To HLR" limiter to limit the total rate of requests to the HLR.

Limiter Types

Rhino comes with the following types of limiters:

rate limiter,
queue-saturation limiter,
absolute stat limiter, and
relative stat limiter.

Rate limiter

Rate limiters limit the rate of work. It is typically used to limit the rate of incoming events, or outgoing requests.

Type Name

RATE
(the type argument specified when creating a limiter)

Rejects work when…

the number of units used (or forced) during a given timeUnit exceeds maxRate. TimeUnit can be one second, one minute, one hour, or one day (24-hour period, not calendar day).

Rhino implements rate limiters with a token bucket algorithm, where the depth property determines the bucket size. The actual bucket size is maxRate * depth.

The default setting for depth is 1.0. So "50/sec" means "allow for 50 per second". When depth is 2, "50/sec" means "allow an initial burst of 100 and then 50 per second." The recommended setting for maxRate is where your CPU is around 85%.

Example

Configured as "maxRate=100.0 timeUnit=seconds depth=2.0", the limiter has a bucket size of 100.0*2.0=200.0. If the bucket is empty when the limiter becomes overloaded, it allows 200+100=300 units in the first second and then 100 units per second after that.

Example

Configured as "maxRate=0.1 timeUnit=seconds depth=10.0", the limiter has a bucket size of 0.1*10.0=1.0. If the bucket is empty when the limiter becomes overloaded, it allows 1.0+0.1=1 units in the first second and then 1 units per 1/maxRate seconds after that.

Properties

Property

Legal values

Default

Settable per node

 bypassed

true false

 false

✅

 parent

name of another limiter or "" for no parent

""

❌

 maxrate

1.0 ⋯ 10000000.0

0.0

✅

 depth

0.01 ⋯ 00.0

1.0

✅

 timeunit

seconds minutes hours days
Note that 'days' means 24 hour periods, not calendar days.

 seconds

❌

Queue-saturation limiter

The queue-saturation limiter rejects work when the event-staging queue (explained in the Staging section) passes a given saturation. It provides some overload protection, by limiting incoming activities in cases where too much work is backlogged, while allowing enough headroom to process existing activities.

For example, the default configuration has the QueueSaturation limiter configured with an allowed capacity of 85%. With the default maximum queue size of 3000, this limiter starts rejecting new activities when 2250 or more items are in the queue (leaving 15% headroom for processing existing activities).

Type Name

QUEUE_SATURATION
(the type argument specified when creating a limiter)

Rejects work when…

the number of items in the staging queue reaches maxSaturation, expressed as a percentage of the queue’s capacity.

Example

Configured as maxSaturation=80.0 and a queue capacity of 200, the limiter will reject work when the queue contains 160 or more items.

Properties

Property

Legal values

Default

Settable per node

 bypassed

true false

 false

✅

 parent

name of another limiter or "" for no parent

""

❌

 maxSaturation

0.0 ⋯ 100.0

✅

Absolute stat limiter

The absolute stat limiter rejects work based on the value of a single gauge-type statistic and supports progressive levels of limiting tiers.

This limiter type should be used in cases where a single parameter set gauge can be compared against one or more fixed values to determine if work load should be limited. On each request for one or more units, the current value of the specified statistic is compared against one or more absolute values. Each absolute value can be configured to limit a different percentage of the total unit requests. The absolute value tiers can be structured in such a way as to progressively limit an increasing percentage of all unit requests as the value of the statistic approaches some known threshold.

Only gauge-type statistics are currently supported. Configuring a limiter with a counter or sample-type statistic will cause the limiter to raise a misconfiguration alarm. Any unit requests made to this limiter while it is misconfigured will be accepted (unless rejected by a parent limiter).

Type Name

ABSOLUTE_STAT
(the type argument specified when creating a limiter)

Rejects work when…

the value of the monitored statistic is equal to or greater than the value of one of the configured limiter tiers that has a non-zero limit percentage.

Example

Configured as

parameterSet=Activities
statistic=active
tiers=[ {10000: 50%} {12000: 75%} {13000: 100%} ]

when the current value of the active gauge in the Activities parameter set is equal to or greater than:

10000 — 50% of all unit requests will be refused;
12000 — 75% of all unit requests will be refused; or
13000 — 100% of all unit requests will be refused.

Properties

Property

Legal values

Default

Settable per node

 bypassed

true false

 false

✅

 parent

name of another limiter or "" for no parent

""

❌

 parameterSet

a registered parameter set

 "unspecified"

❌

 statistic

a statistic in the configured parameter set

 "unspecified"

❌

 tiers

one or more mappings of value to limit percentage

[]

✅

Relative stat limiter

The relative stat limiter rejects work based on the values of two related gauge-type statistics and supports progressive levels of limiting tiers.

This limiter type should be used in cases where two related parameter set gauges can be compared against each other to determine if work load should be limited. On each request for one or more units, the current value of the specified statistic is compared against one or more percentage values of the current value of the relative statistic. Each percentage value can be configured to limit a different percentage of the total unit requests. The relative percentage tiers can be structured in such a way as to progressively limit an increasing percentage of all unit requests as the value of the statistic approaches a known threshold with respect to the value of the related statistic.

Only gauge-type statistics are currently supported. Configuring a limiter with a counter or sample-type statistic will cause the limiter to raise a misconfiguration alarm. Any unit requests made to this limiter while it is misconfigured will be accepted (unless rejected by a parent limiter).

Type Name

RELATIVE_STAT
(the type argument specified when creating a limiter)

Rejects work when…

the value of the monitored statistic is equal to or greater than the relative percentage of the relative statistic as configured in one or more tiers that has a non-zero limit percentage.

Example

Configured as

parameterSet=MemDB-Local
statistic=committedSize
relativeParameterSet=MemDB-Local
relativeStatistic=maxCommittedSize
tiers=[ {75%: 50%} {80%: 75%} {90%: 100%} ]

when the current value of the committedSize gauge in the MemDB-Local parameter set is equal to or greater than:

75% of maxCommittedSize — 50% of all unit requests will be refused;
80% of maxCommittedSize — 75% of all unit requests will be refused; or
90% of maxCommittedSize — 100% of all unit requests will be refused.

Properties

Property

Legal values

Default

Settable per node

 bypassed

true false

 false

✅

 parent

name of another limiter or "" for no parent

""

❌

 parameterSet

a registered parameter set

 "unspecified"

❌

 statistic

a statistic in the configured parameter set

 "unspecified"

❌

 relativeParameterSet

a registered parameter set

 "unspecified"

❌

 relativeStatistic

a statistic in the configured relative parameter set

 "unspecified"

❌

 tiers

one or more mappings of relative percentage to limit percentage

[]

✅

For the stat limiters, when limiting work in a percentage limit tier, the limiter always attempts to keep the percentage of units rejected as close to the specified limit percentage as possible. For example, upon entering a tier configured to limit 50%, if one unit is requested and accepted, then the next unit will be rejected. If a single request for five units is rejected, then around five of the next requests for a single unit will be accepted. When the limiter drops below a certain tier, or the total units requested while limiting surpasses Integer.MAX_VALUE, that tier’s internal running count of total units requested to units rejected is reset.

See how to set limiter properties and per-node configuration.

Managing Rate Limiting

Below are a summary of the MBeans and a listing of the procedures available for managing rate limiting with the Rhino SLEE.

Rate-limiting MBeans

Rate limiting exposes several MBean classes:

MBean What it does Where to get it

LimiterManagementMBean

The main limiting MBean.

RateLimiterMBean, QueueSaturationLimiterMBean, AbsoluteStatLimiterMBean, or RelativeStatLimiterMBean, depending on the type of limiter.

Provides management of a particular limiter, specific to the type of that limiter. All limiter MBeans extend LimiterMBean.

LimiterManagementMBean.getLimiterMBean()

LimiterRampUpMBean

Controls ramp-up of a rate limiter.

LimiterManagementMBean.getLimiterRampUpMBean() LimiterManagementMBean.getLimiterRampUpMBean(java.lang.String)

For convenience, you can get the MBean for the SystemInput limiter through LimiterManagementMBean.getSystemInputLimiterMBean().

Rate-limiting procedures

The rate-limiting MBeans and console commands can be used for the following procedures, detailed in this section:

Managing Limiters
Managing Limiter Endpoints
Managing Rate Limiter Ramp-up

Managing Limiters

This section includes instructions for performing the following Rhino SLEE procedures with explanations, examples, and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Creating a limiter

 createlimiter

LimiterManagementMBean → createLimiter

Removing a limiter

 removelimiter

LimiterManagementMBean → removeLimiter

Setting limiter properties

 configureratelimiter
 configuresaturationlimiter
 configureabsolutestatlimiter
 configurerelativestatlimiter

LimiterMBean
RateLimiterMBean
QueueSaturationLimiterMBean
AbsoluteStatLimiterMBean
RelativeStatLimiterMBean

Inspecting a limiter

 getlimiterinfo

LimiterMBean
RateLimiterMBean
QueueSaturationLimiterMBean
AbsoluteStatLimiterMBean
RelativeStatLimiterMBean

Listing limiters and limiter hierarchies

 listlimiters

LimiterManagementMBean → getLimiters
LimiterManagementMBean → getHierarchySummary

Creating a Limiter

To create a limiter, use the following rhino-console command or related MBean operation.

Name character restriction

The limiter name cannot include the "/" character.

Console command: `createlimiter`

Command

createlimiter <limitername> [-type <limitertype>]
  Description
    Creates a new limiter with the given name, and of the given type if specified.
    If no type is specified, then a RATE limiter is created by default.

Example

To create a queue-saturation type limiter named saturation1:

$ ./rhino-console createlimiter saturation1 -type QUEUE_SATURATION
Successfully created queue_saturation limiter 'saturation1'.

MBean operation: `createLimiter`

MBean

Rhino operation

void createLimiter(String type, String name)
        throws NullPointerException, InvalidArgumentException, ConfigurationException, ManagementException, LimitingManagementException;

See also About Rate Limiting and Limiter Types.

Removing a Limiter

To remove a limiter, use the following rhino-console command or related MBean operation.

A limiter cannot be removed if it is the parent of any limiters or if any limiter endpoints are connected to it.

Also the SystemInput limiter cannot be removed.

Console command: `removelimiter`

Command

removelimiter <limitername>
  Description
    Remove the specified limiter

Example

To remove limiter saturation1:

$ ./rhino-console removelimiter saturation1
The Limiter saturation1 has been successfully removed.

MBean operation: `removeLimiter`

MBean

setMaxSaturationDefault

Rhino operation

void removeLimiter(String name)
        throws NullPointerException, InvalidArgumentException, ConfigurationException, LimitingManagementException;

Setting Limiter Properties

To set limiter properties, use the following rhino-console commands or related MBean operations. For more information on the configuration properties for input limiter types see Limiter Types and About Rate Limiting.

Limiters can only be configured administratively — RAs or services cannot configure limiters.

Console commands

For details of available properties for each limiter type, see Limiter Types.

`configureratelimiter`

Command

configureratelimiter <limitername> [-nodes node1,node2...] <[-property value]
[-property value] ... >
  Description
    Sets the values of the specified configuration properties of the limiter on the
    given node(s). Use a value of '-' to clear existing per-node settings

Example

To set rate limiter properties:

$ ./rhino-console configureratelimiter SystemInput -nodes 101 -bypassed false -maxrate 100
Updated config properties for limiter 'SystemInput':
maxrate=100
bypassed=false

`configuresaturationlimiter`

Command

configuresaturationlimiter <limitername> [-nodes node1,node2...] <[-property
value] [-property value] ... >
  Description
    Sets the values of the specified configuration properties of the limiter on the
    given node(s). Use a value of '-' to clear existing per-node settings

Example

To set saturation limiter properties:

$ ./rhino-console configuresaturationlimiter QueueSaturation -maxsaturation 75
Updated config properties for limiter 'QueueSaturation':
maxsaturation=7

`configureabsolutestatlimiter`

Command

configureabsolutestatlimiter <limitername> [-nodes node1,node2...] <[-property
value] [-property value] ... >
  Description
    Sets the values of the specified configuration properties of the limiter on the
    given node(s). Use a value of '-' to clear existing per-node settings

Example

To configure an absolute stat limiter called ActivitiesActive based on the active statistic in the Activities parameter set with three progressive levels of limiting tiers. When the active gauge reports a value at or above:

10000 — 50% of all unit requests will be refused;
12000 — 75% of all unit requests will be refused; or
13000 — 100% of all unit requests will be refused.

$ ./rhino-console configureabsolutestatlimiter ActivitiesActive \
  -parameterset Activities \
  -statistic active \
  -values 10000,12000,13000 \
  -limitpercentages 50,75,100
Updated config properties for limiter 'ActivitiesActive':
parameterset=Activities
statistic=active
values=10000,12000,13000
limitpercentages=50,75,100

`configurerelativestatlimiter`

Command

configurerelativestatlimiter <limitername> [-nodes node1,node2...] <[-property
value] [-property value] ... >
  Description
    Sets the values of the specified configuration properties of the limiter on the
    given node(s). Use a value of '-' to clear existing per-node settings

Example

To configure a relative stat limiter called MemDBLocalCommitted based on the committedSize and maxCommittedSize statistics in the MemDB-Local parameter set with three progressive levels of limiting tiers. When the committedSize gauge reports a value at or above:

75% of maxCommittedSize — 50% of all unit requests will be refused;
80% of maxCommittedSize — 75% of all unit requests will be refused; or
90% of maxCommittedSize — 100% of all unit requests will be refused.

$ ./rhino-console configurerelativestatlimiter MemDBLocalCommitted \
  -parameterset MemDB-Local \
  -statistic committedSize \
  -relativeparameterset MemDB-Local \
  -relativestatistic maxCommittedSize \
  -relativepercentages 75,80,90 \
  -limitpercentages 50,75,100
Updated config properties for limiter 'MemDBLocalCommitted':
parameterset=MemDB-Local
statistic=committedSize
relativeparameterset=MemDB-Local
relativestatistic=maxCommittedSize
relativepercentages=75,80,90
limitpercentages=50,75,100

If -nodes are specified, these commands set properties for the listed nodes only (the "per-node" properties). Otherwise, these commands update the default properties for the limiter (the properties that apply whenever no per-node property is set).

You cannot change the name or type of a limiter — these are set when a limiter is created.

MBean operations

`Limiter` MBean operations

Operation Usage

setBypassedDefault

void setBypassedDefault(boolean bypassed)
        throws ConfigurationException;

setBypassedForNode

void setBypassedForNode(boolean[] bypassed, int[] nodeIDs)
        throws NullPointerException, ConfigurationException, InvalidArgumentException;

setParent

void setParent(String parentName)
        throws ConfigurationException, ValidationException, NullPointerException, InvalidArgumentException;

`RateLimiter` MBean operations

Operation Usage

setDepthDefault

void setDepthDefault(double depth)
        throws ConfigurationException, ValidationException;

setDepthForNode

void setDepthForNode(double[] depth, int[] nodeIDs)
        throws ConfigurationException,  ValidationException, NullPointerException, InvalidArgumentException;

setMaxRateDefault

void setMaxRateDefault(double depth)
        throws ConfigurationException, ValidationException;

setMaxRateForNode

void setMaxRateForNode(double[] depth, int[] nodeIDs)
        throws ConfigurationException,  ValidationException, NullPointerException, InvalidArgumentException;

setTimeUnit

void setTimeUnit(String timeUnit)
        throws ConfigurationException, ValidationException;

`QueueSaturationLimiter` MBean operations

Operation Usage

void setMaxSaturationDefault(double maxSaturation)
        throws ConfigurationException, ValidationException;

setMaxSaturationForNode

void setMaxSaturationForNode(double[] maxSaturation, int[] nodeIDs)
        throws ConfigurationException, ValidationException,NullPointerException, InvalidArgumentException;

`StatLimiter` MBean operations

Operation Usage

setParameterSetStatistic

void setParameterSetStatistic(String parameterSet, String statistic)
        throws ConfigurationException, ValidationException;

`AbsoluteStatLimiter` MBean operations

Operation Usage

setTier

void setTier(long value, double limitPercentage)
        throws ConfigurationException, ValidationException;

setTierForNode

void setTierForNode(long[] values, double[] limitPercentages, int[] nodeIDs)
        throws ConfigurationException, InvalidArgumentException, ValidationException;

clearTier

void clearTier(long value)
        throws ConfigurationException, ValidationException;

clearTierForNode

void clearTierForNode(long[] values, int[] nodeIDs)
        throws ConfigurationException, InvalidArgumentException, ValidationException;

`RelativeStatLimiter` MBean operations

Operation Usage

setRelativeParameterSetStatistic

void setRelativeParameterSetStatistic(String parameterSet, String statistic)
        throws ConfigurationException, ValidationException;

setTier

void setTier(double relativePercentage, double limitPercentage)
        throws ConfigurationException, ValidationException;

setTierForNode

void setTierForNode(double[] relativePercentages, double[] limitPercentages, int[] nodeIDs)
        throws ConfigurationException, InvalidArgumentException, ValidationException;

clearTier

void clearTier(double relativePercentage)
        throws ConfigurationException, ValidationException;

clearTierForNode

void clearTierForNode(double[] relativePercentages, int[] nodeIDs)
        throws ConfigurationException, InvalidArgumentException, ValidationException;

Inspecting a Limiter

To inspect a limiter, use the following rhino-console command or related MBean operations.

Console command

`getlimiterinfo`

Command

getlimiterinfo <limitername> [-c]
  Description
    Displays the current configuration settings of the specified limiter.  If the -c
    flag is provided, all stored default and per node settings for the limiter are
    listed.  Otherwise the current configuration of all event routing nodes (as
    derived from the stored settings) is listed.

The first variant of this command, using the -c flag (or the LimiterMBean.getConfigSummary method) displays all configuration properties stored for the limiter.

The second variant, without the -c flag (or the LimiterMBean.getInfoSummarymethod) displays the effective configuration for the limiter as derived from the stored configuration properties. This variant of the command lists information for all event-routing nodes, whereas the MBean method lets you specify which nodes you’re interested in. (You can get a list of all event-routing nodes from RhinoHousekeepingMBean.getEventRouterNodes()).

Examples

To view all configuration properties stored for a limiter SystemInput:

$ ./rhino-console getlimiterinfo SystemInput -c
limiter-name   node-id    bypassed   depth   maxrate   parent            time-unit   type
-------------  ---------  ---------  ------  --------  ----------------  ----------  -----
  SystemInput   defaults       true     1.0       0.0   QueueSaturation     SECONDS   RATE
          n/a        101      false       *     100.0               n/a         n/a    n/a
2 rows

'*' means no value set
'n/a' means setting not configurable per node

NOTE: Ramp up of SystemInput limiter is currently disabled

To view the effective configuration for limiter SystemInput, as derived from the stored configuration properties:

$ ./rhino-console getlimiterinfo SystemInput
limiter-name   node-id   bypassed   depth   maxrate   parent            time-unit   type
-------------  --------  ---------  ------  --------  ----------------  ----------  -----
  SystemInput       101      false     1.0     100.0   QueueSaturation     SECONDS   RATE
1 rows

'*' means no value set


NOTE: Ramp up of SystemInput limiter is currently disabled

MBean operations

`Limiter` MBean operations

Operation Usage

getConfigSummary

TabularData getConfigSummary();

getInfoSummary

TabularData getInfoSummary(int[] nodeIDs)
      throws NullPointerException, ConfigurationException, InvalidArgumentException;

getChildLimiters

String[] getChildLimiters()
      throws ConfigurationException;

getConnectedEndPoints

String[] getConnectedEndPoints()
      throws ConfigurationException;

getName

String getName()
      throws ConfigurationException;

getParent

String getParent()
      throws ConfigurationException;

getType

String getType()
      throws ConfigurationException;

isBypassedDefault

boolean isBypassedDefault()
      throws ConfigurationException;

isBypassedForNode

boolean[] isBypassedForNode(int[] nodeIDs)
      throws NullPointerException, ConfigurationException, InvalidArgumentException;

`RateLimiter` MBean operations

Operation Usage

getDepthDefault

double getDepthDefault()
      throws ConfigurationException;

getDepthForNode

double[] getDepthForNode(int[] nodeIDs)
      throws ConfigurationException, NullPointerException, InvalidArgumentException;

getMaxRateDefault

double getMaxRateDefault()
      throws ConfigurationException;

getMaxRateForNode

double[] getMaxRateForNode(int[] nodeIDs)
      throws ConfigurationException, NullPointerException, InvalidArgumentException;

getTimeUnit

double getTimeUnit()
      throws ConfigurationException;

`SaturationLimiter` MBean operations

Operation Usage

getMaxSaturationDefault

double getMaxSaturationDefault()
      throws ConfigurationException;

getMaxSaturationForNode

double[] getMaxSaturationForNode(int[] nodeIDs)
      throws ConfigurationException, NullPointerException, InvalidArgumentException;

`StatLimiter` MBean operations

Operation Usage

getParameterSet

String getParameterSet()
      throws ConfigurationException;

getStatistic

String getStatistic()
      throws ConfigurationException;

getLimitPercentages

double[] getLimitPercentages()
      throws ConfigurationException;

getLimitPercentagesForNodes

String double[][] getLimitPercentagesForNodes(int[] nodeIDs)
      throws ConfigurationException, InvalidArgumentException;

`AbsoluteStatLimiter` MBean operations

Operation Usage

getTierValues

long[] getTierValues()
        throws ConfigurationException;

getTierValuesForNodes

String long[][] getTierValuesForNodes(int[] nodeIDs)
      throws ConfigurationException, InvalidArgumentException;

`RelativeStatLimiter` MBean operations

Operation Usage

getRelativeParameterSet

String getRelativeParameterSet()
      throws ConfigurationException;

getRelativeStatistic

String getRelativeStatistic()
      throws ConfigurationException;

getTierPercentages

double[] getTierPercentages()
        throws ConfigurationException;

getTierPercentagesForNodes

String double[][] getTierPercentagesForNodes(int[] nodeIDs)
      throws ConfigurationException, InvalidArgumentException;

Listing Limiters and Limiter Hierarchies

To list all limiters and limiter hierarchies, use the following rhino-console command or related MBean operations.

Console command: `listlimiters`

Command

listlimiters [-v]
  Description
    Lists all limiters. If the '-v' flag is provided, all limiter hierarchies and
    connected endpoints are displayed.

Example

To list all limiters:

$ ./rhino-console listlimiters
QueueSaturation
rate1
SystemInput

To display all limiter hierarchies and connected endpoints:

$ ./rhino-console listlimiters -v
QueueSaturation
+- SystemInput
   +- Endpoint:RAEntity/entity1/Input
   +- Endpoint:RAEntity/entity2/Input
rate1
   (Has no children or endpoints)

MBean operation: `getLimiters`

MBean

Connecting a limiter endpoint to a limiter

Rhino operation

String[] getLimiters() throws ConfigurationException;

MBean operation: `getHierarchySummary`

MBean

LimiterManagement

Rhino operation

String getHierarchySummary()
          throws ConfigurationException, ManagementException;

Managing Limiter Endpoints

This section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

 connectlimiterendpoint

LimiterManagementMBean → connectLimiterEndpoint

Disconnecting a limiter endpoint from a limiter

 disconnectlimiterendpoint

LimiterManagementMBean → disconnectLimiterEndpoint

Listing limiter endpoints

 listlimiterendpoints

LimiterManagementMBean → getLimiterEndpoints

Finding which limiter is connected to a limiter endpoint

 getlimiterforlimiterendpoint

LimiterManagementMBean → getLimiterForEndpoint

Connecting a Limiter Endpoint to a Limiter

To connect a limiter endpoint to a limiter, use the following rhino-console command or related MBean operation.

Console command: `connectlimiterendpoint`

Command

connectlimiterendpoint <limiterendpoint> <limiter>
  Description
    Sets the limiter endpoint to use the specified limiter

Example

To connect limiter endpoint RAEntity/entity1/Input to limiter rate1:

$ ./rhino-console connectlimiterendpoint RAEntity/entity1/Input rate1
Connected limiter endpoint 'RAEntity/entity1/Input' to limiter 'rate1'

MBean operation: `connectLimiterEndpoint`

MBean

Rhino operation

void connectLimiterEndpoint(String limiterEndpointID, String limiterName)
        throws NullPointerException, InvalidArgumentException, ConfigurationException, ManagementException, LimitingManagementException;

Disconnecting a Limiter Endpoint from a Limiter

To disconnect a limiter endpoint from a limiter, use the following rhino-console command or related MBean operation.

Console command: `disconnectlimiterendpoint`

Command

disconnectlimiterendpoint <limiterendpoint>
  Description
    Removes the limiter for a limiter endpoint

Example

To disconnect limiter endpoint RAEntity/entity1/Input:

$ ./rhino-console disconnectlimiterendpoint RAEntity/entity1/Input
Disconnected limiter endpoint 'RAEntity/entity1/Input'

MBean operation: `disconnectLimiterEndpoint`

MBean

Rhino operation

void disconnectLimiterEndpoint(String limiterEndpointID)
        throws NullPointerException, InvalidArgumentException, ConfigurationException,  LimitingManagementException, ManagementException;

Listing Limiter Endpoints

To list all limiter endpoints, use the following rhino-console command or related MBean operation.

Console command: `listlimiterendpoints`

Command

listlimiterendpoints [-v]
  Description
    Lists all available limiter endpoints. If the '-v' flag is provided, the limiter
    endpoint's current used limiter is also provided.

Example

$ ./rhino-console listlimiterendpoints
RAEntity/entity1/Input
RAEntity/entity1/inbound

MBean operation: `getLimiterEndpoints`

MBean

Rhino operation

String[] getLimiterEndpoints()
        throws ConfigurationException, ManagementException;

Finding which Limiter is Connected to a Limiter Endpoint

To find which limiter is connected to a limiter endpoint, use the following rhino-console command or related MBean operation.

Console command: `getlimiterforlimiterendpoint`

Command

getlimiterforlimiterendpoint <limiterendpoint>
  Description
    Returns the name of the limiter that the limiter endpoint is using

Example

To find which limiter is connected to limiter endpoint RAEntity/entity1/Input:

$ ./rhino-console getlimiterforlimiterendpoint RAEntity/entity1/Input
LimiterEndpoint 'RAEntity/entity1/Input' is using the limiter 'rate1'

MBean operation: `getLimiterForEndpoint`

MBean

Enabling rate limiter ramp-up

Rhino operation

String getLimiterForEndpoint(String limiterEndpointID)
        throws NullPointerException, InvalidArgumentException, ConfigurationException, ManagementException;

Managing Rate Limiter Ramp-up

As well as an overview of ramp-up of the rate limiters, this section includes instructions for performing the following Rhino SLEE procedures, with explanations, examples, and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

 enablerampup

LimiterRampUpMBean → enableRampUp

Disabling rate limiter ramp-up

 disablerampup

LimiterRampUpMBean → disableRampUp

Inspecting a rate limiter’s ramp-up configuration

 getrampupconfiguration

LimiterRampUpMBean → isEnabled
LimiterRampUpMBean → getStartRate
LimiterRampUpMBean → getEventsPerIncrement
LimiterRampUpMBean → getRateIncrement

About Rate Limiter Ramp-up

Ramp-up is an optional procedure that gradually increases the rate that a rate limiter allows — from a small value at the beginning of the ramp, typically when a node starts, up to the configured maximum.

This allows time for events such as Just-In-Time compilation and cache-loading, before the maximum work rate applies to the node.

Ramp-up configuration units

SystemInput rate limiter

The ramp-up configuration of the SystemInput rate limiter is expressed in terms of a raw number of SLEE events. That is, the startRate and rateIncrement specify an exact number of events.

For example, if the startRate is 50 then ramp-up begins with an allowance of 50 events per time unit.

The rateIncrement is added to the allowed rate every time Rhino processes eventsPerIncrement events with no rejected events. Rhino counts all events processed, regardless of whether or not they go through the SystemInput limiter.

Other rate limiters

The ramp-up configuration of all other rate limiters is expressed in terms of a percentage of the maximum rate of the limiter.

For example, if maxRate = 500 and startRate = 25.0 then ramp-up begins with an allowance of 500 x 25.0% = 125 units of work per time unit.

Following on with this example, if rateIncrement = 10.0 then the allowed rate increases by 500 x 10.0% = 50 units of work per time unit every time that eventsPerIncrement units of work are used in the limiter.

Enabling or disabling ramp-up

Below is a summary of what happens depending on whether ramp-up is enabled or disabled for the SystemInput rate limiter.

Enabled Disabled

First, Rhino sets the SystemInput limiter’s maximum allowed rate to startRate.
Then, every time Rhino processes eventsPerIncrement events, with no rejected events, it adds rateIncrement to the rate.
Ramp-up finishes when the current rate reaches the maxRate value set for the SystemInput limiter.

Nothing special happens when the node starts — the maximum rate the SystemInput limiter allows is simply maxRate.

Below is a summary of what happens when ramp-up is enabled or disabled for any other rate limiter.

Enabled Disabled

First, Rhino sets the rate limiter’s maximum allowed rate to maxRate x startRate%.
Then, every time eventsPerIncrement units of work are used in the limiter, it adds maxRate x rateIncrement% to the rate.
Ramp-up finishes when the current rate reaches the maxRate value set for the rate limiter.

Nothing special happens when the node starts — the maximum rate the rate limiter allows is simply maxRate.

Ramp-up has no effect if the rate limiter’s bypassed flag is true.

Ramp-up restarts from startRate if any property of the ramp-up’s configuration is modified, if the limiter’s maxRate is changed, or if the limiter changes from being bypassed to not being bypassed.

You configure ramp-up globally, but each node ramps up independently. So if a node restarts, it ramps up again — without affecting other already running nodes.

Enabling Rate Limiter Ramp-up

To enable rate limiter ramp-up, use the following rhino-console command or related MBean operation.

Console command: `enablerampup`

Command

enablerampup [-limiter <limitername>] <startrate> <rateincrement>
<eventsperincrement>
  Description
    Enables rampup of a limiter's rate with the provided startrate, rateincrement,
    and eventsperincrement.  If no limiter name is given then the SystemInput
    limiter is updated.

Example

To enable SystemInput limiter ramp-up:

$ ./rhino-console enablerampup 10 10 1000
Enabled rampup of the SystemInput limiter rate with config properties:
startrate=10.0 events per time unit
rateincrement=10.0 events per time unit
eventsperincrement=1000
Tip: the ramp up will only be effective when the SystemInput limiter is not bypassed.

To enable ramp-up on a rate limiter named "From MSC":

$ ./rhino-console -limiter "From MSC" enablerampup 5 5 800
Enabled rampup of the From MSC limiter rate with config properties:
startrate=5.0% of maximum rate
rateincrement=5.0% of maximum rate
eventsperincrement=800
Tip: the ramp up will only be effective when the From MSC limiter is not bypassed.

MBean operation: `enableRampUp`

MBean

LimiterRampUpMBean

Rhino operation

void enableRampUp(double startRate, double rateIncrement, int eventsPerIncrement)
        throws ConfigurationException;

The rate limiter’s bypassed flag must be false for ramp-up to have any effect.

Disabling Rate Limiter Ramp-up

To disable rate limiter ramp-up, use the following rhino-console command or related MBean operation.

Console command: `disablerampup`

Command

disablerampup [limitername]
  Description
    Disables rampup of the specified limiter's rate, or the rate of the SystemInput
    limiter if no limiter name is given.

Example

To disable SystemInput limiter ramp-up:

$ ./rhino-console disablerampup
Disabled rampup of the SystemInput limiter rate

To disable ramp-up on a rate limiter named "From MSC":

$ ./rhino-console disablerampup "From MSC"
Disabled rampup of the From MSC limiter rate

MBean operation: `disableRampUp`

MBean

LimiterRampUp

Rhino operation

void disableRampUp()
        throws ConfigurationException;

Inspecting Rate Limiter Ramp-up Configuration

To inspect a rate limiter’s ramp-up configuration, use the following rhino-console command or related MBean operation.

Console command: `getrampupconfiguration`

Command

getrampupconfiguration [limitername]
  Description
    Retrieves the limiter rampup configuration settings, if it is enabled. If no
    limiter is specified the settings of the SystemInput limiter are shown.

Example

To inspect the SystemInput limiter’s ramp-up configuration:

$ ./rhino-console getrampupconfiguration
Rampup of the SystemInput limiter is active with the following config properties:
startrate=10.0 events per time unit
rateincrement=10.0 events per time unit
eventsperincrement=100

To inspect the ramp-up configuration of the "From MSC" rate limiter:

$ ./rhino-console getrampupconfiguration "From MSC"
Rampup of the From MSC limiter is active with the following config properties:
startrate=5.0% of maximum rate
rateincrement=5.0% of maximum rate
eventsperincrement=800

`LimiterRampUp` MBean operations

Operation

Usage

isEnabled

boolean isEnabled()
        throws ConfigurationException;

getStartRate

double getStartRate()
        throws ConfigurationException;

getRateIncrement

double getRateIncrement()
        throws ConfigurationException;

getEventsPerIncrement

int getEventsPerIncrement()
        throws ConfigurationException;

Monitoring Limiter Statistics

You can monitor limiter statistics using the rhino-stats tool or the StatsManagement MBean.

The root parameter set is called Limiters, and it has one child parameter set for each limiter; for example, Limiters.SystemInput or Limiters.QueueSaturation.

Limiter parameters recorded

Rhino records the following limiter parameters:

Records the total number of.. Increments.. Decrements..

Total units successfully used or forced — unitsUsed

…units allowed to be used and units forced to be used.

…whenever units are allowed to or forced to be used.

…never.

Total units denied by a limiter’s parent — unitsRejectedByParent

…units not allowed to be used, because the parent of the limiter denied their use. This includes units denied because of any ancestor (such as the parent of the parent).

The unitsRejected statistic also includes units rejected by a parent.

…whenever the parent of a limiter denies unit use.

…never.

Total units denied by a limiter — unitsRejected

…units not allowed to be used, because a limiter or its parent denied their use. This includes units denied because of any ancestor (such as the parent of the parent).

…whenever a limiter denies unit use.

…never.

Example

The following excerpt shows the number of units a limiter allows and rejects, second by second.

$ ./rhino-stats -m Limiters.SystemInput
2009-03-11 06:57:43.903  INFO    [rhinostat]  Connecting to localhost:1199
2009-03-11 06:57:44.539  INFO    [dispatcher]  Establish direct session DirectSession[host=server1 port=17400 id=56928181032135173]
2009-03-11 06:57:44.542  INFO    [dispatcher]  Connecting to localhost/127.0.0.1:17400

                                 Limiters.SystemInput
time                       rejected  rejectedByParent  used
-----------------------   ---------------------------------
2009-03-11 06:57:46.604           -                 -     -
2009-03-11 06:57:47.604          14                 0   103
2009-03-11 06:57:48.604          14                 0   102
2009-03-11 06:57:49.604          11                 0   101
2009-03-11 06:57:50.604          12                 0    99
2009-03-11 06:57:51.604          13                 0   102
2009-03-11 06:57:52.604          14                 0   101
2009-03-11 06:57:53.604           8                 0    96

(In this example, rejectedByParent is 0, as SystemInput has no parent.)

Using Alarms with Limiters

Threshold Alarms can be configured for a limiter based on any limiter statistics.

See the Configuring Rules section for general instructions on installing threshold alarm rules, and the configuration example on this page.

Pre-existing alarms

By default Rhino has two threshold alarms pre-configured to indicate when one of the two pre-configured limiters rejects work: the SystemInput Rejecting Work alarm for the SystemInput limiter, and the QueueSaturation Rejecting Work alarm for the QueueSaturation limiter. Each rate limiter may also generate a Negative capacity alarm if it reaches a limit to the amount of forced work it can keep track of.

SystemInput rejecting work

Alarm Message

SystemInput rate limiter is rejecting work

Type

 LIMITING

Instance ID

system-input-limiter-rejecting-work

Level

 MAJOR

Raised if…

…the SystemInput limiter is rejecting work for more than one second.

Cleared if…

…the SystemInput limiter has not rejected any work for five seconds.

Example output

2009-03-02 17:13:43.893  Major   [rhino.facility.alarm.manager]   <Timer-2> Alarm 101:136455512705:8
    [SubsystemNotification[subsystem=ThresholdAlarms],LIMITING,system-input-limiter-rejecting-work]
    was raised at 2009-03-02 17:13:43.893 to level Major
        SystemInput rate limiter is rejecting work

QueueSaturation Rejecting Work

Alarm Message

QueueSaturation limiter is rejecting work

Type

 LIMITING

Instance ID

queue-saturation-limiter-rejecting-work

Level

 MAJOR

Raised if…

…the QueueSaturation limiter is rejecting work for more than one second.

Cleared if…

…the QueueSaturation limiter has not rejected any work for five seconds.

Example output

2009-03-02 17:16:37.697  Major   [rhino.facility.alarm.manager]   <Timer-1> Alarm 101:136455512705:10
    [SubsystemNotification[subsystem=ThresholdAlarms],LIMITING,queue-saturation-limiter-rejecting-work]
    was raised at 2009-03-02 17:16:34.592 to level Major
          QueueSaturation limiter is rejecting work

Negative capacity alarm

Alarm Message

Token count in rate limiter "<LIMITER_NAME>" capped at negative saturation point on node <NODE_ID>. Too much work has been forced. Alarm will clear once token count >= 0.

Type

 ratelimiter.below_negative_capacity

Instance ID

 nodeID=<NODE_ID>,limiter=<LIMITER_NAME>

Level

 WARNING

Raised if…

…a very large number of units have been forcibly used and the internal token counter has reached the biggest possible negative number (-2,147,483,648).

Cleared if…

…token counter >= 0

Example output

2009-03-05 01:14:59.597  Warning [rhino.facility.alarm.manager]   <Receiver for switchID 1236168893> Alarm 101:136654511648:16

[SubsystemNotification[subsystem=LimiterManager],limiting.ratelimiter.below_negative_capacity,nodeID=101,limiter=SystemInput]
  was raised at 2009-03-05 01:14:59.596 to level Warning
        Token count in rate limiter "SystemInput" capped at negative saturation point on node 101.
        Too much work has been forced. Alarm will clear once token count >= 0.

Threshold alarm example

The following configuration example defines the pre-existing system-input-limiter-rejecting-work alarm.

<threshold-rules active="true" name="system-input-limiter-rejecting-work">
    <trigger-conditions name="Trigger conditions" operator="OR" period="1000">
        <simple-threshold operator="&gt;" value="0.0">
            <select-statistic calculate-delta="true" parameter-set="Limiters.SystemInput" statistic="unitsRejected"/>
        </simple-threshold>
    </trigger-conditions>
    <reset-conditions name="Reset conditions" operator="OR" period="5000">
        <simple-threshold operator="==" value="0.0">
            <select-statistic calculate-delta="true" parameter-set="Limiters.SystemInput" statistic="unitsRejected"/>
        </simple-threshold>
    </reset-conditions>
    <trigger-actions>
        <raise-alarm-action level="Major" message="SystemInput rate limiter is rejecting work" type="LIMITING"/>
    </trigger-actions>
    <reset-actions>
        <clear-raised-alarm-action/>
    </reset-actions>
</threshold-rules>

The default threshold alarms can be modified or removed as needed.

Security

Security is an essential feature of the JAIN SLEE standard and Rhino.

It provides access control for: m-lets (management applets), JAIN SLEE components (including resource adaptors, services and libraries) and Rhino node and cluster administration. Rhino’s security subsystem implements a pessimistic security model — to prevent untrusted resource adaptors, m-lets, services or human users from performing restricted functions.

Transport-layer security and the general security of the remote host and server are important considerations when interconnection with third-party servers.

Any security planning can be foiled by an incumbent with a key!

The Rhino security model is based on: the standard Java security model, the Java Authentication and Authorisation Service (JAAS), and the SLEE specification default permission sets for components.

Key features of Rhino security include:

Configuring Java Security of Rhino
Secure Access for OA&M Staff
Configuring the SLEE Component Java Sandbox

Configuring Java Security of Rhino

The following standard Java security policy file defines the Rhino codebase security configuration.

As Rhino starts, it:

pre-processes configuration files (including rhino.policy
substitutes configuration variables (such as @RHINO_HOME@)
creates working configuration files (in the node-XXX/work/config subdirectory).

Disabling or debugging security

There may be times when you want to disable security (for example, during development), or enable fine-grained security tracing in Rhino (for example, to track down security-related issues in Rhino).

Disabling security completely

You can disable security two ways:

Insert a rule into the policy file that grants AllPermission to all code:
```
grant {
permission java.security.AllPermission;
};
```
Disable the use of a security manager — edit $RHINO_HOME/node-XXX/read-config-variables, commenting out the following line:
```
#OPTIONS="$OPTIONS -Djava.security.manager"
```

Enable security when running Rhino

Metaswitch recommends you always run Rhino with security enabled.

Debugging security

You can debug Rhino’s security configuration by enabling security tracing (so that the security manager produces trace logs) — edit $RHINO_NODE_HOME/read-config-variables, adding the following line:

OPTIONS="$OPTIONS -Djava.security.debug=access,failure"

Capture console output

This option will produce a lot of console output. To capture it, redirect the standard out and standard error streams from Rhino to a file. For example:

$ start-rhino.sh > out 2>&1

Excerpt of rhino.policy

Below is an excerpt of $RHINO_HOME/node-XXX/config/rhino.policy:

grant {
  permission java.io.FilePermission "${java.home}${/}lib${/}jaxp.properties","read";

  // Needed by default logging configuration.
  permission java.io.FilePermission "$${rhino.dir.work}$${/}log", "read";
  permission java.io.FilePermission "$${rhino.dir.work}$${/}log$${/}-","read,write,delete";

  // Needed by netty specifically, but it's a sensible top-level permission to grant
  permission java.io.FilePermission "/etc/os-release", "read";
  permission java.io.FilePermission "/usr/lib/os-release", "read";

  // Java "standard" properties that can be read by anyone
  permission java.util.PropertyPermission "java.version", "read";
  permission java.util.PropertyPermission "java.vendor", "read";
  permission java.util.PropertyPermission "java.vendor.url", "read";
  permission java.util.PropertyPermission "java.class.version", "read";
  permission java.util.PropertyPermission "os.name", "read";
  permission java.util.PropertyPermission "os.version", "read";
  permission java.util.PropertyPermission "os.arch", "read";
  permission java.util.PropertyPermission "file.encoding", "read";
  permission java.util.PropertyPermission "file.separator", "read";
  permission java.util.PropertyPermission "path.separator", "read";
  permission java.util.PropertyPermission "line.separator", "read";

  permission java.util.PropertyPermission "java.specification.version", "read";
  permission java.util.PropertyPermission "java.specification.vendor", "read";
  permission java.util.PropertyPermission "java.specification.name", "read";

  permission java.util.PropertyPermission "java.vm.specification.version", "read";
  permission java.util.PropertyPermission "java.vm.specification.vendor", "read";
  permission java.util.PropertyPermission "java.vm.specification.name", "read";
  permission java.util.PropertyPermission "java.vm.version", "read";
  permission java.util.PropertyPermission "java.vm.vendor", "read";
  permission java.util.PropertyPermission "java.vm.name", "read";
};

// Standard java and jdk modules we use get all permissions by default.
// Actual access will be limited by the caller's security context.
grant codeBase "jrt:/java.security.jgss" {
  permission java.security.AllPermission;
};

// ...

Java Security Properties

A per node configuration file $RHINO_NODE_HOME/config/rhino.java.security has been added to allow overriding of JVM security settings. This file includes default values for the following networking security properties:

networkaddress.cache.ttl=30
networkaddress.negative.cache.ttl=10

The value of these properties control how long Resource Adaptors and Rhino based applications cache network addresses after successful and unsuccessful DNS queries. These values override the ones specified in the JVMs java.security file. See Oracle’s Networking Properties documentation for more details. The JVM default for networkaddress.cache.ttl is to cache forever. (-1) Therefore the introduction of this file to Rhino’s per-node configuration will alter an applications caching behavior on upgrade to a newer Rhino version.

Use of a different java.security configuration file can be achieved by modifying the following line in $RHINO_NODE_HOME/read-config-variables:

OPTIONS="$OPTIONS -Djava.security.properties=${SCRIPT_WORK_DIR}/config/rhino.java.security"

Secure Access for OA&M Staff

Rhino provides a set of management tools for OA&M staff, including the Rhino Element Manager and various command-line tools.

The following topics explain how you can:

manage user authentication
encrypt the interconnections between Rhino and all command-line tools
configure Rhino to allow management access from a remote trusted host.

Authentication

The Java Authentication and Authorization Service (JAAS) allows integration with enterprise systems, identity servers, — databases and password files.

JAAS configuration

The file rhino.jaas defines the JAAS modules Rhino uses for authentication:

/** Login Configuration for OpenCloud Rhino **/

jmxr-adaptor {
    com.opencloud.rhino.security.auth.FileLoginModule REQUIRED
    file="$${rhino.dir.base}/rhino.passwd"
    hash="md5";
};

See the Javadoc for the JAAS Configuration class for details about flags such as REQUIRED.

The system property java.security.auth.login.config defines the location of rhino.jaas (in read-config-variables for a production Rhino instance and jvm_args for the Rhino SDK.)

Rhino contains JAAS login modules based on files, LDAP, and JAIN SLEE profiles.

The FileLoginModule reads login credentials and roles from a file. It is the default login module for a new Rhino installation.

The parameters to the FileLoginModule are:

file - specifies location of password file.
hash - password hashing algorithm. Use none for clear text passwords, or a valid java.security.MessageDigest algorithm name (e.g. md5 or sha-1). If not specified, clear text passwords are used.

Password File Format

<username>:<password>:<role,role...>

username - user’s name
password - user’s password (or hashed password). May be prefixed by the hash method in {}.
roles - comma-separated list of role names that the user belongs to, eg. rhino,view.

Using flags and hashed passwords

By default, Rhino stores passwords in cleartext, in the password file. For increased security, store a secure one-way hash of the password instead:

Configure the file login module, changing the hash="none" option to hash="md5".
Use the client/bin/rhino-passwd utility to generate hashed passwords.
Copy those hashed passwords into the password file.

The LdapLoginModule reads login credentials and roles from an LDAP directory server.

To use this module, edit the JAAS configuration file ${RHINO_HOME}/config/rhino.jaas, and add an entry to the jmxr-adaptor declaration:

jmxr-adaptor {
    com.opencloud.rhino.security.auth.LdapLoginModule SUFFICIENT
        properties="config/ldapauth.properties";

    /* a "backup" login module would typically go here */
};

Configuration Properties

The properties file contains LDAP connection parameters. The properties that this module uses are documented in the example ldapauth.properties file, along with default values and examples

The file config/ldapauth.properties defines the LDAP-connection configuration:

### Properties for JAAS LDAP login module (LdapLoginModule)
#
# The commented values are the default values that will be used if the given property is not specified.
# The ldap.url property has no default and must be specified.
#
# This properties file should be supplied to the LdapLoginModule using the "properties" property, e.g.
#
# jmxr-adaptor {
#     com.opencloud.rhino.security.auth.LdapLoginModule SUFFICIENT
#     properties="config/ldapauth.properties";
#  };
#

### Connection properties

# An LDAP URL of the form ldap://[host[:port]]/basedn or ldaps://host[:port]/basedn
# Some examples:
# Connect to local directory server
#ldap.url=ldap:///dc=example,dc=com
# Connect to remote directory server
#ldap.url=ldap://remoteserver/dc=example,dc=com
# Connect to remote directory server using SSL
#ldap.url=ldaps://remoteserver/dc=example,dc=com
ldap.url=

# Use TLS.  When set to true, the LdapLoginModule attempts a "Start TLS" request when it connects to the
# directory server.  This should NOT be set to true when using an ldaps:// (SSL) URL.
#ldap.usetls=true

# To use TLS or SSL, you must have your directory server's X509 certificate installed in Rhino's trust
# store, located at $RHINO_BASE/rhino-server.keystore.

### Authentication properties

## Direct mode
# In "direct mode", the login module attempts to bind using a DN calculated from the pattern property.
# Direct mode is used if the ldap.userdnpattern property is specified.

# A DN pattern that can be used to directly login users to LDAP.  This pattern is used for creating a DN string for
# 'direct' user authentication, where the pattern is relative to the base DN in the LDAP URL.
# {0} will be replaced with the submitted username.
# A typical value for this property might be "uid={0},ou=People"

#ldap.userdnpattern=

## Search mode
# In "search mode", the login module binds using the given manager credentials and searches for the user.
# Authentication to LDAP will be done from the DN found if successful.
# Search mode is used if the ldap.userdnpattern property is not specified.

# Bind credentials to search for the user.  May be blank if the directory server allows anonymous connections, or if
# using direct mode.
#ldap.managerdn=
#ldap.managerpw=

# A filter expression used to search for the user DN that will be used in LDAP authentication.
# {0} will be replaced by the submitted username.
#ldap.searchfilter=(uid={0})

# Context name to search in, relative to the base DN in the LDAP URL.
#ldap.searchbase=

### Role resolution properties

# A search is performed using the search base (ldap.role.searchbase), and filter (ldap.role.filter).  The results of
# the search define the Rhino roles.  The role name is in the specified attribute (ldap.roles.nameattr) and must match
# role definitions in Rhino configuration.  The members of each role are determined by examining the values of the
# member attribute (ldap.role.memberattr) and must contain the DN of the authenticated user.

# Attribute on the group entry which denotes the group name.
#ldap.rolenameattr=cn

# A multi-value attribute on the group entry which contains user DNs or ids of the group members (e.g. uniqueMember,member)
#ldap.rolememberattr=uniqueMember

# The LDAP filter used to search for group entries.
#ldap.rolefilter=(objectclass=groupOfUniqueNames)

# A search base for group entry DNs, relative to the DN that already exists on the LDAP server's URL.
#ldap.rolesearchbase=ou=Groups

# Do case-sensitive search by default. Allowed values are true and false.
#ldap.casesensitive=true

TLS setup for ldaps:// or starttls

For security reasons, always use TLS for LDAP authentication, either via an ldaps:// URL or via ldap.usetls=true. Rhino does not use the JDK’s default CA certificates list, so you must add a TLS certificate that Rhino should trust to Rhino’s rhino-server.keystore. This must be done whether you are using a TLS certificate from a public or private CA, or a self-signed certificate, for your LDAP server, otherwise Rhino will refuse to trust the LDAP server. You may use any of:

The CA’s root certificate, which will be the most durable choice as it should continue to work across LDAP server certificate rotations for a number of years.
One of the CA’s intermediate certificates, which will be less durable than using the root CA certificate.
The LDAP server’s certificate, which is the least durable choice because you will need to replace it in each Rhino keystore file whenever the LDAP server gets a new certificate.

To add an LDAP TLS certificate to rhino-server.keystore, run the following, with $PATH_TO_YOUR_CA_CERT replaced with the correct path to your certificate:

keytool -importcert -noprompt -alias ldap-server-ca-cert -file $PATH_TO_YOUR_CA_CERT -keystore rhino-server.keystore

The ProfileLoginModule reads login credentials and roles from a SLEE profile table.

To use this module, edit the JAAS configuration file ${RHINO_HOME}/config/rhino.jaas, and add an entry to the jmxr-adaptor declaration:

jmxr-adaptor {
    com.opencloud.rhino.security.auth.ProfileLoginModule SUFFICIENT
        profiletable="UserLoginProfileTable"
        passwordattribute="HashedPassword"
        rolesattribute="Roles"
        hash="md5";

    /* a "backup" login module would typically go here */
};

ProfileLoginModule supports the following options:

Option Description Default

 profiletable

name of the profile table to use

 UserLoginProfileTable

 passwordattribute

profile attribute to compare the password against
profile attribute type must be java.lang.String

 HashedPassword

 rolesattribute

profile attribute to load the roles from
profile attribute type must be array of java.lang.String

 Roles

 hash

hashing algorithm to use for the password
use none for clear text passwords, or a valid java.security.MessageDigest algorithm name (e.g. md5 or sha-1).

If anything other than md5 is specified, then the environment variable on the UserLoginProfile profile spec must be set to match the algorithm used.

md5

The profile login module:

finds the profile in a specified table with a name matching the supplied username
compares the supplied password with the password stored in the profile; if authentication succeeds, retrieves the roles for that user from the profile.

Rhino comes with a profile specification that you can use to create a profile table for the profile login module (in $RHINO_HOME/lib/user-login-profile-du.jar). It contains a profile specification called UserLoginProfileSpec. You can install it using the rhino-console:

[Rhino@localhost (#3)] installlocaldu ../../lib/user-login-profile-du.jar
installed: DeployableUnitID[url=file:/tmp/rhino/lib/user-login-profile-du.jar]
[Rhino@localhost (#4)] listprofilespecs
ProfileSpecificationIDname=AddressProfileSpec,vendor=javax.slee,version=1.0
ProfileSpecificationIDname=AddressProfileSpec,vendor=javax.slee,version=1.1
ProfileSpecificationIDname=ResourceInfoProfileSpec,vendor=javax.slee,version=1.0
ProfileSpecificationID[name=UserLoginProfileSpec,vendor=Open Cloud,version=1.0]

A profile table named UserLoginProfileTable created using this specification will work with the default configuration values listed above.

Creating a profile table fallback

OpenCloud recommends configuring a file login module as a fallback mechanism, in case the profile table is accidentally deleted or renamed, or the admin user profile is deleted or changed.

Without a fallback you would not be able to fix the profile table problem, since no user would be able to login using a management client!

To create a profile table fallback, give ProfileLoginModule a SUFFICIENT flag and the FileLoginModule a REQUIRED flag:

jmxr-adaptor {
    com.opencloud.rhino.security.auth.ProfileLoginModule SUFFICIENT
        profiletable="UserLoginProfileTable"
        passwordattribute="HashedPassword"
        rolesattribute="Roles"
        hash="md5";

    com.opencloud.rhino.security.auth.FileLoginModule REQUIRED
        file="$${rhino.dir.base}/rhino.passwd"
        hash="md5";
};

Encrypted Communication with SSL

By default, the interconnection between Rhino and a management client uses the Secure Sockets Layer (SSL) protocol.

(You can disable SSL by editing the JMX Remote Adaptor m-let configuration.)

How does SSL work?

An SSL connection for sending data protects it by using encryption, which prevents eavesdropping and tampering. SSL uses a cryptographic system that doubly encrypts the data, with both a public key known to everyone and a private (or "secret") key known only to the recipient of the message.

For more about SSL, please see SSL Certificates HOWTO from the Linux Documentation Project, and Java SE Security Documentation from Oracle.

Below are descriptions of Rhino SSL keystores and using the keytool utility to manage them.

SSL in Rhino

Several keystores store the keys Rhino uses during user authentication. For example, a Rhino SDK installation includes:

Keystore Used by… To…

 $RHINO_HOME/rhino-public.keystore

clients

identify themselves, and confirm the server’s identity

 $RHINO_HOME/rhino-private.keystore

Rhino

identify itself, confirm a client’s identity

 $RHINO_HOME/client/rhino-public.keystore

Rhino OA&M clients (like command line console)

duplicate $RHINO_HOME/rhino-public.keystore (this is a copy of that), when copying the client directory to another location

The installation process generates keystores, keys, and certificates for Rhino.

Using `keytool` to manage keystores

You can use keytool to manage keystores. For example:

$ keytool -list -keystore rhino-public.keystore
Enter keystore password:  <password>

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

jmxr-ssl-client, Apr 25, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA-256): B4:5A:4E:E3:B8:73:22:C4:94:1C:C7:B7:B5:B0:BF:7E:06:B2:68:D3:D3:21:A4:98:63:2A:12:9B:53:FB:9F:C3
jmxr-ssl-server, Apr 29, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): BE:B8:00:AD:8B:5E:B3:0D:D5:5A:4B:61:AE:7B:36:F9:CD:DE:8D:8F:98:5A:13:3E:F7:27:C4:D9:D9:89:BA:F7

Change the default passphrase

Rhino keystores and keys have a default passphrase of changeit. As the name suggests, Metaswitch recommends changing it, for example with keytool:

keytool -storepasswd -keystore rhino-public.keystore

Enabling Remote Access

By default, only Rhino’s management tools (such as the command-line console or stats console) can run on the same host as Rhino. You can, however, securely manage Rhino from a remote host.

As discussed in the preceding topic, Rhino uses SSL to secure its interconnect with management clients. To configure Rhino to support remote management clients:

Copy the client directory to the remote machine.
Allow the remote host to connect to the JMX remote adaptor.

Set up the client directory on the remote machine

The client directory (and subdirectories) contain all the scripts, configuration files and other dependencies needed for Rhino management clients. To setup the client directory on the remote machine:

Copy the entire directory structure to the remote host: $ scp -r client <user>@<host>:<destination>/

Edit client/etc/client.properties and change rhino.remote.host:

# RMI properties, file names are relative to client home directory
rhino.remote.host=<rhino host>
rhino.remote.port=1199
# ...

Allow the remote host to connect to the JMX remote adaptor

All management tools connect to Rhino using the JMX Remote Adaptor m-let. By default this component only permits access from the same host that Rhino is running on.

The security-spec section of the node-XXX/config/permachine-mlet.conf and node-XXX/config/pernode-mlet.conf files defines the security environment of an m-let. To allow a remote host to connect to the JMX remote adaptor, edit the security-permission-spec sections of the node-XXX/config/permachine-mlet.conf file, to enable remote access with appropriate java.net.SocketPermission:

<mlet enabled="true">
<classpath>
<jar-url>@FILE_URL@@RHINO_BASE@/lib/jmxr-adaptor.jar</jar-url>
<jar-url>@FILE_URL@@RHINO_BASE@/lib/jmxr-adaptor-gpl2.jar</jar-url>
<security-permission-spec>
grant {
...
permission java.net.SocketPermission "<REMOTE_HOST>","accept";
...
};
...
</mlet>

If you would like to connect to Rhino SDK, the file that defines the m-let configuration is $RHINO_SDK/config/mlet.conf.

Configuring the SLEE Component Java Sandbox

All JAIN SLEE components run within a "sandbox" defined by a set of Java security permissions.

The JAIN SLEE 1.1 specification:

defines a default set of security permissions for each SLEE component jar (such as SBB jars, resource adaptor jars and library jars).
explains how you can grant additional security permissions for SBB, profile specification, resource adaptor and library components (over and above the default set).

This section draws heavily from material in the JAIN SLEE 1.1 specification.

Default Security Permissions for SLEE Components

The following table defines the Java platform security permissions that Rhino grants to the instances of SLEE component classes at runtime.

The term "grant" means that Rhino grants the permission, the term "deny" means that Rhino denies the permission.

Permission name SLEE policy

 java.security.AllPermission

deny

 java.awt.AWTPermission

deny

 java.io.FilePermission

deny

 java.net.NetPermission

deny

 java.util.PropertyPermission

grant read, *
deny all other

 java.lang.reflect.ReflectPermission

deny

 java.lang.RuntimePermission

deny

 java.lang.SecurityPermission

deny

 java.io.SerializablePermission

deny

 java.net.SocketPermission

deny

This permission set is defined by the JAIN SLEE 1.1 specification section 12.1.1.1-slee-component-security-permissions. This section also explains how SBB, profile specification, resource adaptor and library components can be granted additional security permissions over and above the default set.

Adding Security Permissions to SLEE Components

SBB, profile specification, resource adaptor and library components can be granted additional security permissions over and above the default set of security permissions granted by the SLEE — by using the security-permissions element in their respective deployment descriptor.

Each security-permissions element contains the following sub-elements:

description — an optional informational element
security-permission-spec — an element that identifies the security permission policies used by component jar file classes. (For the security-permission-spec element syntax definition, please see the J2SE security documentation).

If the codeBase argument:

is not specified for a grant entry — the SLEE assumes the codebase to be the component jar file, and grants security permissions to all classes loaded from the component jar file (that is, to all SLEE components defined in the component jar file). The SLEE does not, however, grant the security permissions to classes loaded from any other dependent component jar required by the components defined in the deployment descriptor.
is specified for a grant entry — the SLEE assumes the specified path is relative to the root directory of the component jar within the deployable unit jar (but its use is otherwise undefined by the SLEE specification).

Below are a sample component jar deployment descriptor with added security permissions, and a table of security requirements that apply to methods invoked on classes loaded from different types of component jars with added permissions.

Sample component jar deployment descriptor with added security permissions

Below is an example of a resource adaptor component jar with added security permissions:

<resource-adaptor-jar>
<resource-adaptor>
<description> ... </description>
<resource-adaptor-name> Foo JCC </resource-adaptor-name>
<resource-adaptor-vendor> com.foo </resource-adaptor-vendor>
<resource-adaptor-version> 10/10/20 </resource-adaptor-version>
...
</resource-adaptor>

<security-permissions>
<description>
Allow the resource adaptor to modify thread groups and connect to remotehost on port 1234
</description>
<security-permission-spec>
grant {
permission java.lang.RuntimePermission "modifyThreadGroup";
permission java.net.SocketPermission "remotehost:1234", "connect";
};
</security-permission-spec>
</security-permissions>
</resource-adaptor-jar>

Security requirements for methods invoked on classes loaded from component jars

The following table describes the security requirements that apply to methods invoked on classes loaded from different types of component jars:

Component jar type Security requirements

SBB

Event-handler and initial event-selector methods run with the default set of security permissions granted by the SLEE, plus any additional security permissions specified in the SBB’s component jar deployment descriptor.
The isolate-security-permissions attribute of the sbb-local-interface element in the SBB’s deployment descriptor controls whether or not security permissions of other protection domains in the call stack are propagated to the SBB when a business method on the SBB Local interface is invoked:
- If False — then the method in the SBB abstract class invoked as a result of a business method invoked on the SBB Local interface runs with an access control context that includes the protection domain(s) of the SBB as well as the protection domains of any other classes in the call stack as prescribed by the Java security model, such as the SBB that invoked the SBB Local interface method.
- If True — the SLEE automatically wraps the method invoked on the SBB abstract class in response to the SBB Local interface method invocation in an AccessController.doPrivileged block in order to isolate the security permissions of the invoked SBB. That is, the security permissions of other protection domains in the call stack do not affect the invoked SBB.
All methods defined in the javax.slee.Sbb interface can be invoked on an SBB object from an unpredictable call path. If any of these methods need to execute privileged code requiring security permissions over and above the standard set of permissions granted by the SLEE, the additional security permissions must be declared in the SBB component jar’s deployment descriptor and the relevant methods must wrap the privileged code in an AccessController.doPrivileged block to ensure that potentially more restrictive security permissions of other protection domains in the call stack do not prohibit the privileged code from being executed.

Profile spec

All management methods invoked on the Profile Management interface run with the default set of security permissions granted by the SLEE, plus any additional security permissions specified in the Profile Specification’s component jar deployment descriptor.
The isolate-security-permissions attribute of the profile-local-interface element in the Profile Specification’s deployment descriptor controls whether or not security permissions of other protection domains in the call stack are propagated to the Profile when a business method on the Profile Local interface is invoked.
- If False — the method in the Profile abstract class invoked as a result of a business method invoked on the Profile Local interface runs with an access control context that includes the protection domain(s) of the Profile Specification as well as the protection domains of any other classes in the call stack as prescribed by the Java security model, such as the SLEE Component that invoked the Profile Local interface method.
- If True — the SLEE automatically wraps the method invoked on the Profile abstract class in response to the Profile Local interface method invocation in an AccessController.doPrivileged block in order to isolate the security permissions of the invoked Profile, i.e. the security permissions of other protection domains in the call stack do not affect the invoked Profile.
The setProfileContext, unsetProfileContext, profilePostCreate, profileActivate, profilePassivate, profileLoad, profileStore, and profileRemove methods defined in the javax.slee.profile.Profile interface can be invoked on a Profile object from an unpredictable call path. If any of these methods need to execute privileged code requiring security permissions over and above the standard set of permissions granted by the SLEE, the additional security permissions must be declared in the Profile Specification component jar’s deployment descriptor and the relevant methods must wrap the privileged code in an AccessController.doPrivileged block to ensure that potentially more restrictive security permissions of other protection domains in the call stack do not prohibit the privileged code from being executed.
The profileInitialize and profileVerify methods defined in the javax.slee.profile.Profile interface are invoked as a result of management operations and therefore run with the default set of security permissions granted by the SLEE, plus any additional security permissions specified in the Profile Specification’s component jar deployment descriptor.

Resource adaptor

All methods invoked by the SLEE on the javax.slee.resource.ResourceAdaptor and javax.slee.resource.Marshaler interfaces run with the default set of security permissions granted by the SLEE, plus any additional security permissions specified in the Resource Adaptor’s component jar deployment descriptor.
All methods that may be invoked by other SLEE Components such as SBBs run with the set of security permissions that is the intersection of the permissions of all protection domains traversed by the current execution thread (up until any AccessController.doPrivileged invocation in the call stack). If any of these methods need to execute privileged code requiring security permissions over and above the standard set of permissions granted by the SLEE, the additional security permissions must be declared in the Resource Adaptor component jar’s deployment descriptor and the relevant methods must wrap the privileged code in an AccessController.doPrivileged block to ensure that potentially more restrictive security permissions of other protection domains in the call stack do not prohibit the privileged code from being executed.

Library

All methods run with the set of permissions that is the intersection of the permissions of all protection domains traversed by the current execution thread (up until any AccessController.doPrivileged invocation in the call stack). If a library component method needs to execute privileged code requiring security permissions over and above the standard set of permissions granted by the SLEE, the additional security permissions must be declared in the library component jar’s deployment descriptor and the method must wrap the privileged code in an AccessController.doPrivileged block to ensure that potentially more restrictive security permissions of other protection domains in the call stack do not prohibit the privileged code from being executed.

External Databases

The Rhino SLEE requires the use of an external database for persistence of management and profile data. Rhino can also provide SLEE applications with access to an external database for persistence of their own data.

Rhino can connect to any external database which has support for JDBC 2.0 and JDBC 2.0’s standard extensions. The JDBC API is the industry standard for database-independent connectivity between the Java programming language and a wide range of databases. The JDBC API provides a call-level API for SQL-based database access. JDBC technology lets you use the Java programming language to exploit "Write Once, Run Anywhere" capabilities for applications that require access to enterprise data. For more information, please see https://docs.oracle.com/javase/tutorial/jdbc.

External database integration is managed in Rhino using the following configurable entities:

Configurable entity

What it does

Persistence instance

Defines the parameters Rhino needs to be able to connect to an external database using the database vendor’s database driver code.

Persistence resource

Links a Rhino in-memory database with one or more persistence instances.

JDBC resource

Provides a SLEE application with access to a persistence instance.

This section includes instructions and details on:

adding the JDBC driver for a database
managing persistence instances
managing persistence resources
managing JDBC resources
the external persistence configuration file format

Adding the JDBC Driver

The JDBC driver for an external database needs to be added to Rhino’s runtime environment before Rhino can connect to it. You’ll need the JDBC 2.0 driver from the database vendor. (You’ll only need to do this once per Rhino installation and database vendor.)

To install the driver, you need to add it to Rhino’s runtime environment and grant permissions to the classes in the JDBC driver. Rhino needs to be restarted after making these changes for them to take effect.

Add the library

To add the library to Rhino’s runtime environment, copy the JDBC driver jar file to $RHINO_BASE/lib, then add the jar to Rhino’s classpath. The method for adding classpath entries differs between the Rhino SDK and Rhino production versions.

Rhino SDK
For the Rhino SDK, add an entry for the JDBC driver jar file into the rhino.runtime.classpath system property in $RHINO_HOME/config/jvm_args.

Below is an example that includes the PostgreSQL and Oracle JDBC drivers.

# Required classpath
-Drhino.runtime.classpath=${RHINO_BASE}/lib/postgresql.jar;${RHINO_BASE}/lib/derby.jar;${RHINO_BASE}/lib/ojdbc6.jar

Rhino Production
For the production version of Rhino, add an entry for the JDBC driver jar file into the RUNTIME_CLASSPATH environment variable in $RHINO_BASE/defaults/read-config-variables, and the $RHINO_HOME/read-config-variables file in any node directory that has already been created.

Below is an example for adding the Oracle JDBC driver:

# Set classpath
LIB=$RHINO_BASE/lib
CLASSPATH="${CLASSPATH:+${CLASSPATH}:}$LIB/RhinoBoot.jar"
RUNTIME_CLASSPATH="$LIB/postgresql.jar"
# Add Oracle JDBC driver to classpath
RUNTIME_CLASSPATH="$RUNTIME_CLASSPATH:$LIB/ojdbc6.jar"

Grant permissions to the JDBC driver

To grant permissions to the classes in the JDBC driver, edit the Rhino security policy file, adding an entry for the JDBC driver jar file.

In the Rhino SDK, the policy file is $RHINO_HOME/config/rhino.policy. In the production version, the policy files are $RHINO_BASE/defaults/config/rhino.policy, and $RHINO_HOME/config/rhino.policy in any node directory that has already been created.

Below is an example for the Oracle JDBC driver:

// Add permissions to Oracle JDBC driver
grant codeBase "file:$${rhino.dir.base}/lib/ojdbc6.jar" {
  permission java.net.SocketPermission "*", "connect,resolve";
  permission java.lang.RuntimePermission "getClassLoader";
  permission java.util.PropertyPermission "oracle.*", "read";
  permission java.util.PropertyPermission "javax.net.ssl.*", "read";
  permission java.util.PropertyPermission "user.name", "read";
  permission javax.management.MBeanPermission "oracle.jdbc.driver.OracleDiagnosabilityMBean", "registerMBean";
};

Persistence Instances

As well as an overview of persistence instances, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Creating a persistence instance

 createpersistenceinstance

Persistence Management → createPersistenceInstance

Displaying persistence instances

 listpersistenceinstances
 dumppersistenceinstance

Persistence Management → getPersistenceInstances Persistence Management → getPersistenceInstance

Updating a persistence instance’s configuration

 updatepersistenceinstance

Persistence Management → updatePersistenceInstance

Removing a persistence instance

 removepersistenceinstance

Persistence Management → removePersistenceInstance

About Persistence Instances

A persistence instance defines how Rhino connects to an external database endpoint.

A persistence instance requires the following configuration properties:

A unique name that identifies the persistence instance in the SLEE.
The fully-qualified name of the Java class from the database driver that implements the javax.sql.DataSource interface or the javax.sql.ConnectionPoolDataSource interface. For more information on the distinction between these interfaces and their implications for application-level JDBC connection pooling in Rhino, please see Managing database connections.
Configuration properties for the datasource. Each datasource has a number of JavaBean properties (as defined by the JDBC specification). For each configured property, its name, expected Java type, and value must be specified.
Variables may be used in the construct of JavaBean property values. Variables are indicated using the ${...} syntax, where the value between the braces is the variable name. Rhino attempts to resolve the variable name by looking in the following places in this order:
- The content of the $RHINO_HOME/config/config_variables file
- Java system properties
- User environment variables

At a minimum, configuration properties that inform the JDBC driver where to connect to the database server must be specified.

Creating Persistence Instances

To create a persistence instance, use the following rhino-console command or related MBean operation.

Console command: `createpersistenceinstance`

Command

createpersistenceinstance <name> <type> [-ds <datasource-class-name>]
[-set-property (<property-name> <property-type> <property-value)*]
  Description
    Create a persistence instance configuration.  The type may be 'jdbc' or
    'cassandra'.  A datasource class name must be specified for 'jdbc'
    configurations.

Example

This example creates a new persistence instance with the following configuration properties:

Name: oracle
Type: jdbc
Datasource class name: oracle.jdbc.pool.OracleDataSource
JavaBean properties:
- name: URL
  type: java.lang.String
  value: jdbc:oracle:thin:@oracle_host:1521:db
- name: user
  type: java.lang.String
  value: ${MANAGEMENT_DATABASE_USER}
- name: password
  type: java.lang.String
  value: ${MANAGEMENT_DATABASE_PASSWORD}
- name: loginTimeout
  type: java.lang.Integer
  value: 30

$ ./rhino-console createpersistenceinstance oracle jdbc \
    -ds oracle.jdbc.pool.OracleDataSource \
    -set-property URL java.lang.String jdbc:oracle:thin:@oracle_host:1521:db \
    -set-property user java.lang.String '${MANAGEMENT_DATABASE_USER}' \
    -set-property password java.lang.String '${MANAGEMENT_DATABASE_PASSWORD}' \
    -set-property loginTimeout java.lang.Integer 30
Created persistence instance oracle

Configuration properties

JDBC persistence instances

A JDBC persistence instance has configuration properties defined by the JavaBean properties of the target datasource class. Reference must be made to the datasource documentation for the available properties.

Cassandra persistence instances

A Cassandra persistence instance can be configured using any configuration property names recognised by the DataStax Java Driver, for example basic.contact-points, advanced.reconnection-policy.class, etc. The reference configuration (used in Rhino as the base configuration) provides a comprehensive list of the configuration properties recognised by the driver. Driver execution profiles are supported for application use using appropriately flattened configuration property names such as profiles.myprofile.basic.request.timeout. Driver metrics can be enabled by configuring the advanced.metrics.session.enabled or advanced.metrics.node.enabled configuration properties as described in the reference configuration. Metrics are exposed over JMX from the Rhino MBean server with the object name domain com.datastax.oss.driver.

If the basic.contact-points configuration property is specified in a persistence instance configuration, the driver requires that the basic.load-balancing-policy.local-datacenter configuration property is also specified to provide the name of the datacentre. If the datacentre name is not specified, the driver may still successfully connect to the contact points but will refuse to utilise any of the database servers due to datacentre name mismatch.

The Java type of all Cassandra persistence instance configuration properties for the DataStax Java Driver must be java.lang.String. This is to accommodate things like duration properties being specified with values containing unit qualifiers such as 100ms.

A Cassandra persistence instance that is used in a persistence resource referenced by a memdb configuration in rhino-config.xml, such as the ManagementDatabase or ProfileDatabase, must specify the basic.session-keyspace configuration property with the name of the Cassandra keyspace where persistent state will be stored. In any other case, specifying the keyspace in configuration is optional. An application may, for example, specify the desired keyspace at runtime using CQL queries instead.

Finally, all Cassandra persistence instances must define the Rhino-specific configuration property rhino.ddl-statement-timeout of type java.lang.String. This property defines the timeout duration that Rhino will use, if configured to use the persistence instance for internal functions such as the key/value store or session ownership store, when executing schema-altering statements such as CREATE TABLE and DROP TABLE.

An example Cassandra persistence instance configuration (as it would appear in the $RHINO_HOME/config/persistence.xml file) is illustrated below:

<persistence-instance name="cassandra" type="cassandra">
  <parameter name="rhino.ddl-statement-timeout" type="java.lang.String" value="10s"/>
  <parameter name="basic.contact-points" type="java.lang.String" value="${CASSANDRA_CONTACT_POINTS}"/>
  <parameter name="basic.load-balancing-policy.local-datacenter" type="java.lang.String" value="${CASSANDRA_DATACENTRE}"/>
  <parameter name="advanced.reconnection-policy.class" type="java.lang.String" value="ConstantReconnectionPolicy"/>
  <parameter name="advanced.reconnection-policy.base-delay" type="java.lang.String" value="5000ms"/>
  <parameter name="basic.request.consistency" type="java.lang.String" value="LOCAL_QUORUM"/>
</persistence-instance>

MBean operation: `createPersistenceInstance`

MBean

Rhino operation

public void createPersistenceInstance(String name, PersistenceInstanceType type, String dsClassName, ConfigProperty[] configProperties)
    throws NullPointerException, InvalidArgumentException,
      DuplicateNameException, ConfigurationException;

Displaying Persistence Instances

To list current persistence instances or display the configuration parameters of a persistence instance, use the following rhino-console commands or related MBean operations.

listpersistenceinstances

Command

listpersistenceinstances
  Description
    List all currently configured persistence instances.

Example

$ ./rhino-console listpersistenceinstances
oracle
postgres
postgres-jdbc

dumppersistenceinstance

Command

dumppersistenceinstance <name> [-expand]
  Description
    Dump the current configuration for the named persistence instance. The -expand
    option will cause any property values containing variables to be expanded with
    their resolved value (if resolvable)

Example

$ ./rhino-console dumppersistenceinstance postgres
datasource-class-name : org.postgresql.ds.PGSimpleDataSource
name                  : postgres
type                  : jdbc
config-properties     :
       > name               type                value
       > -----------------  ------------------  --------------------------------
       >        serverName    java.lang.String       ${MANAGEMENT_DATABASE_HOST}
       >        portNumber   java.lang.Integer       ${MANAGEMENT_DATABASE_PORT}
       >      databaseName    java.lang.String       ${MANAGEMENT_DATABASE_NAME}
       >              user    java.lang.String       ${MANAGEMENT_DATABASE_USER}
       >          password    java.lang.String   ${MANAGEMENT_DATABASE_PASSWORD}
       >      loginTimeout   java.lang.Integer                                30
       >     socketTimeout   java.lang.Integer                                15
       >  prepareThreshold   java.lang.Integer                                 1
       > 8 rows

MBean operations

`getPersistenceInstances`

MBean

Rhino operation

public String[] getPersistenceInstances()
    throws ConfigurationException;

This operation returns an array containing the names of the persistence instances.

`getPersistenceInstance`

MBean

Rhino operation

public CompositeData getPersistenceInstance(String name)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

This operation returns a JMX CompositeData object that contains the current configuration parameters for the specified persistence instance. The javadoc for this operation describes the format of this data.

Updating Persistence Instances

The configuration properties of an existing persistence instance can be updated at runtime. If the persistence instance is in use at the time of a reconfiguration, then new connections will be established with the new configuration properties, and any existing connections to the database will be closed when they become idle.

To update an existing persistence instance, use the following rhino-console command or related MBean operation.

Console command: `updatepersistenceinstance`

Command

updatepersistenceinstance <name> [-type type] [-ds <datasource-class-name>]
[-set-property <property-name> <property-type> <property-value)]*
[-remove-property <property-name>]*
  Description
    Update a persistence instance configuration.

Example

$ ./rhino-console updatepersistenceinstance oracle \
    -set-property URL java.lang.String jdbc:oracle:thin:@oracle_backup:1521:db \
    -set-property user java.lang.String '${MANAGEMENT_DATABASE_USER}' \
    -set-property password java.lang.String '${MANAGEMENT_DATABASE_PASSWORD}'
Updated persistence instance oracle

MBean operation: `updatePersistenceInstance`

MBean

Rhino operation

public void updatePersistenceInstance(String name, PersistenceInstanceType type, String dsClassName, ConfigProperty[] configProperties)
    throws NullPointerException, InvalidArgumentException,
      NameNotFoundException, ConfigurationException;

Removing Persistence Instances

To remove an existing persistence instance, use the following rhino-console command or related MBean operation.

A persistence instance cannot be removed while it is referenced by a persistence resource or JDBC resource.

Console command: `removepersistenceinstance`

Command

removepersistenceinstance <name>
  Description
    Remove an existing persistence instance configuration.

Example

$ ./rhino-console removepersistenceinstance oracle
Removed persistence instance oracle

MBean operation: `removePersistenceInstance`

MBean

Creating a persistence resource

Rhino operation

public void removePersistenceInstance(String name)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

Persistence Resources

As well as an overview of persistence resources, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Managing persistence resources

 createdatabaseresource

Persistence Management → createPersistenceResource

Displaying persistence resources

 listpersistenceresources

Persistence Management → getPersistenceResources

Removing a persistence resource

 removedatabaseresource

Persistence Management → removePersistenceResource

Managing persistence instance references

Adding a persistence instance to a persistence resource

 addpersistenceinstanceref

Persistence Management → addPersistenceResourcePersistenceInstanceRef

Displaying a persistence resource’s persistence instance references

 listpersistenceinstancerefs

Persistence Management → getPersistenceResourcePersistenceInstanceRefs

Removing persistence instances from a persistence resource

 removepersistenceinstanceref

Persistence Management → removePersistenceResourcePersistenceInstanceRef

About Persistence Resources

A persistence resource links a Rhino in-memory database with one or more persistence instances. State stored in the in-memory database is replicated to the external databases for persistence, so that if the Rhino SLEE cluster is shut down then management and provisioned data can be restored.

The persistence resources that Rhino requires are defined in the config/rhino-config.xml file. An in-memory database, identified by a <memdb> element in this file, that persists its state externally contains a reference to a persistence resource using a <persistence-resource-ref> element. In the default configuration, Rhino requires the persistence resources named below:

Persistence Resource

What it’s used for

 management

Persistence of installed deployable units, component desired states, configuration information, and so on.

 profiles

Persistence of all provisioned data in profile tables.

While it is possible to add and remove persistence resources from Rhino, there is typically never a need to do so. Rhino only utilises the persistence resources named in config/rhino-config.xml, and all must exist for Rhino to function correctly.

Active session state is stored in an in-memory database that is not backed by a persistence resource.

Persistence resources and persistence instances

A persistence resource can be associated with zero or more persistence instances. By associating a persistence resource with a persistence instance, in-memory database state corresponding with that resource will be persisted to the external database endpoint identified by that persistence instance. Any given persistence instance may be used concurrently by multiple persistence resources. Each persistence resource uses a unique set of tables such that overlap in a single database will not occur.

Upon a successful connection, Rhino will keep each persistence instance synchronised with the state of the persistence resource. Naturally, at least one persistence instance reference must be configured for persistence to occur.

When the first node of a cluster boots, Rhino will attempt to connect to all persistence instances used by a persistence resource, and will initialise corresponding in-memory database state from a connected persistence instance that contains the most recent data. The node will fail to boot if it cannot successfully connect to at least one persistence instance for each required persistence resource.

If Rhino connects to any persistence instance that contain out-of-date data, it will be resynchronised with the latest data. Native database replication should not be used between the persistence instances that Rhino connects to — Rhino will handle the synchronisation itself.

A persistence resource should never be associated with two persistence instances that connect to the same physical database. Due to table locking this causes a deadlock when the first Rhino cluster node boots, and it can also cause corruption to database state.

Using multiple persistence instances for a persistence resource

While only a single PostgreSQL or Oracle database is required for the entire Rhino SLEE cluster, the Rhino SLEE supports communications with multiple database servers.

Multiple servers add an extra level of fault tolerance for the runtime configuration and the working state of the Rhino SLEE. Rhino’s in-memory databases will be constantly synchronized to each persistence instance so if the cluster is restarted it will be able to restore state if any of the databases are operational. If a persistence instance database fails or is no longer network-reachable, Rhino will continue to persist updates to the other instances associated with the persistence resource. Updates will be queued for unreachable instances and stored when the instances come back online.

Configuring multiple instances

Prepare the database servers

Before adding a database to a persistence resource you must prepare the database, by executing $RHINO_NODE_HOME/init-management-db.sh for each server.

$ init-management-db.sh -h dbhost-1 -p dbport -u dbuser -d database postgres
$ init-management-db.sh -h dbhost-2 -p dbport -u dbuser -d database postgres

You will be prompted for a password on the command line

Create persistence instances for the databases

Once the databases are initialised on each database server, configure new persistence instances in Rhino and attach them to the persistence resources. To create persistence instances, follow the instructions at Creating persistence instances.

Add the new persistence instances to the configured persistence resources

When persistence instances have been created for each database, add them to the persistence resources. Instructions to do so are at Adding persistence instances to a persistence resource. An example of the procedure is shown below:

$ ./rhino-console createpersistenceinstance oracle \
    oracle.jdbc.pool.OracleDataSource \
    URL java.lang.String jdbc:oracle:thin:@oracle_host:1521:db \
    user java.lang.String ${MANAGEMENT_DATABASE_USER} \
    password java.lang.String ${MANAGEMENT_DATABASE_PASSWORD} \
    loginTimeout java.lang.Integer 30
Created persistence instance oracle
$ ./rhino-console addpersistenceinstanceref persistence management oracle
Added persistence instance reference 'oracle' to persistence resource management
$ ./rhino-console addpersistenceinstanceref persistence profiles oracle
Added persistence instance reference 'oracle' to persistence resource profiles

It is also possible to configure the persistence instances before starting Rhino by editing the persistence.xml configuration file. This is useful for initial setup of the cluster but should not be used to change a running configuration as changes to the file cannot be reloaded without restarting. An example persistence.xml is shown below:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE rhino-persistence-config PUBLIC "-//Open Cloud Ltd.//DTD Rhino Persistence Config 2.3//EN" "rhino-persistence-config-2.3.dtd">
<rhino-persistence-config config-version="2.3" rhino-version="Rhino (version='2.5', release='0-TRUNK.0-SNAPSHOT.1-DEV13-pburrowes', build='201610251631', revision='6c862fc (dirty)')" timestamp="1477629656508">
    <!--Generated Rhino configuration file: 2016-10-28 17:40:56.507-->
    <persistence>
        <jdbc-resource jndi-name="jdbc">
            <persistence-instance-ref name="postgres-jdbc"/>
            <connection-pool connection-pool-timeout="5000" idle-check-interval="30" max-connections="15" max-idle-connections="15" max-idle-time="600" min-connections="0"/>
        </jdbc-resource>
        <persistence-instances>
            <persistence-instance datasource-class-name="org.postgresql.ds.PGSimpleDataSource" name="postgres-1">
                <parameter name="serverName" type="java.lang.String" value="${MANAGEMENT_DATABASE_HOST}"/>
                <parameter name="portNumber" type="java.lang.Integer" value="${MANAGEMENT_DATABASE_PORT}"/>
                <parameter name="databaseName" type="java.lang.String" value="${MANAGEMENT_DATABASE_NAME}"/>
                <parameter name="user" type="java.lang.String" value="${MANAGEMENT_DATABASE_USER}"/>
                <parameter name="password" type="java.lang.String" value="${MANAGEMENT_DATABASE_PASSWORD}"/>
                <parameter name="loginTimeout" type="java.lang.Integer" value="30"/>
                <parameter name="socketTimeout" type="java.lang.Integer" value="15"/>
                <parameter name="prepareThreshold" type="java.lang.Integer" value="1"/>
            </persistence-instance>
            <persistence-instance datasource-class-name="org.postgresql.ds.PGSimpleDataSource" name="postgres-2">
                <parameter name="serverName" type="java.lang.String" value="${MANAGEMENT_DATABASE_HOST2}"/>
                <parameter name="portNumber" type="java.lang.Integer" value="${MANAGEMENT_DATABASE_PORT}"/>
                <parameter name="databaseName" type="java.lang.String" value="${MANAGEMENT_DATABASE_NAME}"/>
                <parameter name="user" type="java.lang.String" value="${MANAGEMENT_DATABASE_USER}"/>
                <parameter name="password" type="java.lang.String" value="${MANAGEMENT_DATABASE_PASSWORD}"/>
                <parameter name="loginTimeout" type="java.lang.Integer" value="30"/>
                <parameter name="socketTimeout" type="java.lang.Integer" value="15"/>
                <parameter name="prepareThreshold" type="java.lang.Integer" value="1"/>
            </persistence-instance>
        </persistence-instances>
        <persistence-resource name="management">
            <persistence-instance-ref name="postgres-1"/>
            <persistence-instance-ref name="postgres-2"/>
        </persistence-resource>
        <persistence-resource name="profiles">
            <persistence-instance-ref name="postgres-1"/>
            <persistence-instance-ref name="postgres-2"/>
        </persistence-resource>
    </persistence>
</rhino-persistence-config>

Creating Persistence Resources

To create a persistence resource, use the following rhino-console command or related MBean operation.

Console command: `createdatabaseresource`

Command

createdatabaseresource <resource-type> <name>
  Description
    Create a database resource. The resource-type parameter must be either
    'persistence' or 'jdbc'. Note that when creating JDBC resources the supplied
    name will automatically be prefixed with 'jdbc/` when determining the internal
    JNDI name for the resource, so this prefix should not normally be included when
    specifying the resource name.

Example

$ ./rhino-console createdatabaseresource persistence myresource
Created persistence resource myresource

MBean operation: `createPersistenceResource`

MBean

Rhino operation

public void createPersistenceResource(String name)
    throws NullPointerException, InvalidArgumentException,
      DuplicateNameException, ConfigurationException;

Displaying Persistence Resources

To list the current persistence resources use the following rhino-console command or related MBean operation.

Console command: `listdatabaseresources`

Command

listdatabaseresources <resource-type>
  Description
    List all currently configured database resources. The resource-type parameter
    must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console listdatabaseresources persistence
management
profiles

MBean operation: `getPersistenceResources`

MBean

Rhino operation

public String[] getPersistenceResources()
    throws ConfigurationException;

This operation returns an array containing the names of the persistence resources that have been created.

Removing Persistence Resources

To remove an existing persistence resource, use the following rhino-console command or related MBean operation.

Console command: `removedatabaseresource`

Command

removedatabaseresource <resource-type> <name>
  Description
    Remove an existing database resource. The resource-type parameter must be either
    'persistence' or 'jdbc'.

Example

$ ./rhino-console removedatabaseresource persistence myresource
Removed persistence resource myresource

MBean operation: `removePersistenceResource`

MBean

Rhino operation

public void removePersistenceResource(String name)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

Adding Persistence Instances to a Persistence Resource

To add a persistence instance to a persistence resource, use the following rhino-console command or related MBean operation.

Console command: `addpersistenceinstanceref`

Command

addpersistenceinstanceref <resource-type> <resource-name>
<persistence-instance-name>
  Description
    Add a persistence instance reference to a database resource. The resource-type
    parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console addpersistenceinstanceref persistence management oracle
Added persistence instance reference 'oracle' to persistence resource management

MBean operation: `addPersistenceResourcePersistenceInstanceRef`

MBean

Rhino operation

public void addPersistenceResourcePersistenceInstanceRef(String persistenceResourceName, String persistenceInstanceName)
    throws NullPointerException, NameNotFoundException,
      DuplicateNameException, ConfigurationException;

Displaying a Persistence Resource’s Persistence Instances

To display the persistence instances that have been added to a persistence resource, use the following rhino-console command or related MBean operation.

Console command: `listpersistenceinstancerefs`

Command

listpersistenceinstancerefs <resource-type> <resource-name>
  Description
    List the persistence instance references for a database resource. The
    resource-type parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console listpersistenceinstancerefs persistence management
postgres

MBean operation: `getPersistenceResourcePersistenceInstanceRefs`

MBean

Rhino operation

public String[] getPersistenceResourcePersistenceInstanceRefs(String persistenceResourceName)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

This operation returns an array containing the names of the persistence instances used by the persistence resource.

Removing Persistence Instances from a Persistence Resource

To remove a persistence instance from a persistence resource, use the following rhino-console command or related MBean operation.

Console command: `removepersistenceinstanceref`

Command

removepersistenceinstanceref <resource-type> <resource-name>
<persistence-instance-name>
  Description
    Remove a persistence instance reference from a database resource. The
    resource-type parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console removepersistenceinstanceref persistence management oracle
Removed persistence instance reference 'oracle' from persistence resource management

MBean operation: `removePersistenceResourcePersistenceInstanceRef`

MBean

Displaying JDBC resources

Rhino operation

public void removePersistenceResourcePersistenceInstanceRef(String persistenceResourceName, String persistenceInstanceName)
    throws NullPointerException, NameNotFoundException,
        ConfigurationException;

JDBC Resources

JDBC resources are used by application components such as service building blocks (SBBs) to execute SQL statements against an external database. A systems administrator can configure new external database resources for applications to use.

As well as an overview on how SBBs can use JDBC to execute SQL and an overview on managing physical database connections, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean(s) → Operation

Managing JDBC resources

Creating a JDBC resource

 createdatabaseresource

Persistence Management → createJdbcResource

 listdatabaseresources

Persistence Management → getJdbcResources

Removing a JDBC resource

 removedatabaseresource

Persistence Management → removeJdbcResource

Managing persistence instance references

Adding a persistence instance to a JDBC resource

 addpersistenceinstanceref

Persistence Management → setJdbcResourcePersistenceInstanceRef

Displaying a JDBC resource’s persistence instance reference

 listpersistenceinstancerefs

Persistence Management → getJdbcResourcePersistenceInstanceRef

Removing a persistence instance from a JDBC resource

 removepersistenceinstanceref

Persistence Management → removePersistenceResourcePersistenceInstanceRef

Managing database connections

Adding a connection pool configuration to a JDBC resource

 createjdbcresourceconnectionpoolconfig

Persistence Management → createJdbcResourceConnectionPoolConfig

Displaying a JDBC resource’s connection pool configuration

 dumpjdbcresourceconnectionpoolconfig

Persistence Management → getJdbcResourceConnectionPoolConfigMBean

JDBC Resource Connection Pool Management → get…

Updating a JDBC resource’s connection pool configuration

 setjdbcresourceconnectionpoolconfig

JDBC Resource Connection Pool Management → set…

Removing the connection pool configuration from a JDBC resource

 removejdbcresourceconnectionpoolconfig

Persistence Management → removeJdbcResourceConnectionPoolConfig

How SBBs use JDBC to execute SQL

An SBB can use JDBC to execute SQL statements. It must declare this intent in an extension deployment descriptor: the oc-sbb-jar.xml file (contained in the SBB jar file in the META-INF directory). The <resource-ref> element (which must be inside the <sbb> element of oc-sbb-jar.xml) defines the JDBC datasource it will use.

Sample `<resource-ref>`

Below is a sample <resource-ref> element defining a JDBC datasource:

<resource-ref>
    <!-- Name under the SBB's java:comp/env tree where this datasource will be bound -->
    <res-ref-name>foo/datasource</res-ref-name>
    <!-- Resource type - must be javax.sql.DataSource -->
    <res-type>javax.sql.DataSource</res-type>
    <!-- Only Container auth supported -->
    <res-auth>Container</res-auth>
    <!-- Only Shareable scope supported -->
    <res-sharing-scope>Shareable</res-sharing-scope>
    <!-- JNDI name of target JDBC resource, relative to Rhino's java:resource tree. -->
    <res-jndi-name>jdbc/myresource</res-jndi-name>
</resource-ref>

In the above example, the <res-jndi-name> element has the value jdbc/myresource, which maps to the JDBC resource created in the example.

How an SBB obtains a JDBC connection

An SBB can get a reference to an object that implements the datasource interface using a JNDI lookup. Using that object, the SBB can then obtain a connection to the database. The SBB uses that connection to execute SQL queries and updates.

For example:

import javax.naming.*;
import javax.slee.*;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
...

public abstract class SimpleSbb implements Sbb {
    public void setSbbContext(SbbContext context) {
        try {
            Context myEnv = (Context)new InitialContext().lookup("java:comp/env");
            ds = (DataSource)myEnv.lookup("foo/datasource");
        }
        catch (NamingException e) {
            // JNDI lookup failed
        }
    }

    public void onSimpleEvent(SimpleEvent event, ActivityContextInterface context) {
        Connection conn;
        try {
            conn = ds.getConnection();
        }
        catch (SQLException e) {
            // could not get database connection
        }
        ...
    }
    ...

    private DataSource ds;
    ...
}

SQL programming

When an SBB executes in a transaction and invokes SQL statements, the SLEE controls transaction management of the JDBC connection. This lets the SLEE perform last-resource-commit optimisation.

Invoking JDBC methods which affect transaction management have no effect or undefined semantics when called from an application component isolated by a SLEE transaction. The methods (including any overridden form) that affect transaction management on the java.sql.Connection interface are listed below. These methods should not be invoked by SLEE components:

close
commit
rollback
setAutoCommit
setIsolationLevel
setSavePoint
releaseSavePoint

Creating JDBC Resources

JDBC resources are identified by a unique name that identifies where in the JNDI tree the JDBC resource will be bound. This name is relative to the java:resource/jdbc namespace, for example the JNDI name oracle/db1 will result in the JDBC resource being bound to the name java:resource/jdbc/oracle/db1.

The JNDI location is not accessible to SBBs directly. Each SBB links to the JNDI name in the SBB deployment descriptor. (For more on SBB deployment descriptor entries please see how SBBs use JDBC to execute SQL.)

All JDBC resources required by the SBBs in a service must exist before that service can be activated. A JDBC resource must also have a persistence instance associated with it in order for it to be able to provide database connections to SBBs that request them.

To create a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `createdatabaseresource`

Command

createdatabaseresource <resource-type> <name>
  Description
    Create a database resource. The resource-type parameter must be either
    'persistence' or 'jdbc'. Note that when creating JDBC resources the supplied
    name will automatically be prefixed with 'jdbc/` when determining the internal
    JNDI name for the resource, so this prefix should not normally be included when
    specifying the resource name.

Example

$ ./rhino-console createdatabaseresource jdbc myresource
Created JDBC resource myresource

MBean operation: `createJdbcResource`

MBean

Rhino operation

public void createJdbcResource(String jndiName)
    throws NullPointerException, InvalidArgumentException,
      DuplicateNameException, ConfigurationException;

Displaying JDBC Resources

To list current JDBC resources, use the following rhino-console command or related MBean operation.

Console command: `listdatabaseresources`

Command

listdatabaseresources <resource-type>
  Description
    List all currently configured database resources. The resource-type parameter
    must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console listdatabaseresources jdbc
jdbc
myresource

MBean operation: `getJdbcResources`

MBean

Rhino operation

public String[] getJdbcResources()
    throws ConfigurationException;

This operation returns an array containing the names of the JDBC resources that have been created.

Removing JDBC Resources

To remove an existing JDBC resource, use the following rhino-console command or related MBean operation.

A JDBC resource cannot be removed while it is referenced by an SBB in an activated service.

Console command: `removedatabaseresource`

Command

removedatabaseresource <resource-type> <name>
  Description
    Remove an existing database resource. The resource-type parameter must be either
    'persistence' or 'jdbc'.

Example

$ ./rhino-console removedatabaseresource jdbc myresource
Removed JDBC resource myresource

MBean operation: `removeJdbcResource`

MBean

Rhino operation

public void removeJdbcResource(String jndiName)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

Adding A Persistence Instance to a JDBC Resource

A JDBC resource can be associated with at most one persistence instance.

Rhino SLEE treats different JDBC resources as different database managers, even if they use the same persistence instance. Therefore, even if two JDBC resources use the same persistence instance, and a single transaction uses both JDBC resources, Rhino treats them as multiple resource managers.

To add a persistence instance to a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `addpersistenceinstanceref`

Command

addpersistenceinstanceref <resource-type> <resource-name>
<persistence-instance-name>
  Description
    Add a persistence instance reference to a database resource. The resource-type
    parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console addpersistenceinstanceref jdbc myresource oracle
Added persistence instance reference 'oracle' to JDBC resource myresource

MBean operation: `setJdbcResourcePersistenceInstanceRef`

MBean

Rhino operation

public void setJdbcResourcePersistenceInstanceRef(String jdbcResourceJndiName, String persistenceInstanceName)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

Displaying a JDBC Resource’s Persistence Instance

To display the persistence instance that has been added to a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `listpersistenceinstancerefs`

Command

listpersistenceinstancerefs <resource-type> <resource-name>
  Description
    List the persistence instance references for a database resource. The
    resource-type parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console listpersistenceinstancerefs jdbc myresource
oracle

MBean operation: `getJdbcResourcePersistenceInstanceRef`

MBean

Rhino operation

public String getJdbcResourcePersistenceInstanceRef(String jndiName)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

This operation returns the name of any persistence instance that has been associated with the JDBC resource.

Removing the Persistence Instance from a JDBC Resource

To remove the persistence instance from a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `removepersistenceinstanceref`

Command

removepersistenceinstanceref <resource-type> <resource-name>
<persistence-instance-name>
  Description
    Remove a persistence instance reference from a database resource. The
    resource-type parameter must be either 'persistence' or 'jdbc'.

Example

$ ./rhino-console removepersistenceinstanceref jdbc myresource oracle
Removed persistence instance reference 'oracle' from JDBC resource myresource

MBean operation: `setJdbcResourcePersistenceInstanceRef`

MBean

Rhino operation

public void setJdbcResourcePersistenceInstanceRef(String jdbcResourceJndiName, String persistenceInstanceName)
    throws NullPointerException, NameNotFoundException,
      ConfigurationException;

To remove an existing persistence instance reference, pass in null for the persistenceInstanceName parameter.

Managing Database Connections

JDBC 2.0 with standard extensions provides two mechanisms for connecting to the database:

The javax.sql.DataSource interface provides unmanaged physical connections.
The javax.sql.ConnectionPoolDataSource interface provides managed physical connections. To connect to a connection pooling data source, you need a managed ConnectionPoolDataSource connection.

Using a connection pool with a JDBC resource

By default, a JDBC resource does not use connection pooling. A connection pool may, however, be attached to a JDBC resource to improve efficiency. When a JDBC resource uses connection pooling, the way Rhino manages connections depends on what interface the datasource class of the persistence instance used by the JDBC resource is an implementation of, as follows:

Interface How Rhino manages connections

 javax.sql.DataSource

uses an internal implementation of ConnectionPoolDataSource to create managed connections

 javax.sql.ConnectionPoolDataSource

uses managed connections from the ConnectionPoolDataSource provided by the persistent instance’s datasource class.

Connection pool configurable parameters

A connection pool has the following configurable parameters:

Parameter What it specifies

 max-connections

Maximum number of active connections a Rhino process can use at any one time.

 max-idle-connections

Maximum number of inactive connections that should be maintained in the connection pool.

This value must be less than or equal to max-connections.

 min-connections

Minimum number of connections that should be maintained in the connection pool.

 max-idle-time

Time in seconds after which an inactive connection may become eligible for discard. An idle connection will not be discarded if doing so would reduce the number of idle connections below the min-connections setting.

If this parameter has the value 0, idle connections will never be discarded.

 idle-check-interval

Time in seconds between idle connection discard checks.

 connection-pool-timeout

Maximum time in milliseconds an SBB will wait for a free connection before a timeout error occurs.

Adding a Connection Pool Configuration to a JDBC Resource

To add a connection pool configuration to a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `createjdbcresourceconnectionpoolconfig`

Command

createjdbcresourceconnectionpoolconfig <name>
  Description
    Create a connection pool configuration for a JDBC resource.

Example

$ ./rhino-console createjdbcresourceconnectionpoolconfig myresource
Connection pool configuration created

MBean operation: `createJdbcResourceConnectionPoolConfig`

MBean

Rhino operation

public ObjectName createJdbcResourceConnectionPoolConfig(String jndiName)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

This method returns the JMX ObjectName of a JDBC Resource Connection Pool Management MBean, which can be used to manage the connection pool configuration parameters.

Displaying a JDBC Resource’s Connection Pool Configuration

To display the connection pool configuration for a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `dumpjdbcresourceconnectionpoolconfig`

Command

dumpjdbcresourceconnectionpoolconfig <name>.
  Description
    Dump the connection pool configuration of a JDBC resource.

Example

$ ./rhino-console dumpjdbcresourceconnectionpoolconfig myresource
connection-pool-timeout : 5000
idle-check-interval     : 30
max-connections         : 2147483647
max-idle-connections    : 2147483647
max-idle-time           : 0
min-connections         : 0

MBean operations:

`getJdbcResourceConnectionPoolConfigMBean`

MBean

JDBC Resource Connection Pool Management

Rhino operation

public ObjectName getJdbcResourceConnectionPoolConfigMBean(String jndiName)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

This method returns the JMX ObjectName of a JDBC Resource Connection Pool Management MBean, which can be used to manage the connection pool configuration parameters.

`JDBC Resource Connection Pool Management`

MBean

Rhino operations

public int getMaxConnections()
    throws ConfigurationException;

public int getMinConnections()
    throws ConfigurationException;

public int getMaxIdleConnections()
    throws ConfigurationException;

public int getMaxIdleTime()
    throws ConfigurationException;

public int getIdleCheckInterval()
    throws ConfigurationException;

public long getConnectionPoolTimeout()
    throws ConfigurationException;

These methods return the current value of the corresponding connection pool configuration parameter.

public CompositeData getConfiguration()
    throws ConfigurationException;

This operation returns a JMX CompositeData object that contains the current configuration parameters for the connection pool configuration. The javadoc for this operation describes the format of this data.

Updating a JDBC Resource’s Connection Pool Configuration

To update the connection pool configuration for a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `setjdbcresourceconnectionpoolconfig`

Command

setjdbcresourceconnectionpoolconfig <name> [-max-connections max-size]
[-min-connections size] [-max-idle-connections size] [-max-idle-time time]
[-idle-check-interval time] [-connection-pool-timeout time]
  Description
    Update the connection pool configuration of a JDBC resource. Size parameters
    must be integer values. The max-idle-time and idle-check-interval parameters are
    measured in seconds and must be integer values. The connection-pool-timeout
    parameter is measured in milliseconds and must be a long value.

Example

In the example below, the maximum idle connections is set to 20, the maximum number of connections is set to 30, and the maximum time an idle connection remains in the connection pool is set to 60s:

$ ./rhino-console setjdbcresourceconnectionpoolconfig myresource \
        -max-idle-connections 20 -max-connections 30 -max-idle-time 60
Connection pool configuration updated for JDBC resource myresource
$ ./rhino-console dumpjdbcresourceconnectionpoolconfig myresource
connection-pool-timeout : 5000
idle-check-interval     : 30
max-connections         : 30
max-idle-connections    : 20
max-idle-time           : 60
min-connections         : 0

MBean operations:

`getJdbcResourceConnectionPoolConfigMBean`

MBean

JDBC Resource Connection Pool Management

Rhino operation

public ObjectName getJdbcResourceConnectionPoolConfigMBean(String jndiName)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

This method returns the JMX ObjectName of a JDBC Resource Connection Pool Management MBean, which can be used to manage the connection pool configuration parameters.

`JDBC Resource Connection Pool Management`

MBean

Rhino operations

public void setMaxConnections(int maxConnections)
    throws ConfigurationException;

public void setMinConnections(int minConnections)
    throws ConfigurationException;

public void setMaxIdleConnections(int maxIdleConnections)
    throws ConfigurationException;

public void setMaxIdleTime(int maxIdleTime)
    throws ConfigurationException;

public void setIdleCheckInterval(int idleCheckInterval)
    throws ConfigurationException;

public void setConnectionPoolTimeout(long timeout)
    throws ConfigurationException;

These methods set a new value for the corresponding connection pool configuration parameter.

Removing the Connection Pool Configuration from a JDBC Resource

To remove the connection pool configuration from a JDBC resource, use the following rhino-console command or related MBean operation.

Console command: `removejdbcresourceconnectionpoolconfig`

Command

removejdbcresourceconnectionpoolconfig <name>.
  Description
    Remove the connection pool configuration from a JDBC resource.

Example

$ ./rhino-console removejdbcresourceconnectionpoolconfig myresource
Connection pool configuration removed

MBean operation: `removeJdbcResourceConnectionPoolConfig`

MBean

Rhino operation

public void removeJdbcResourceConnectionPoolConfig(String jndiName)
    throws NullPointerException, NameNotFoundException,
      InvalidStateException, ConfigurationException;

Persistence Configuration File Format

In most circumstances, it will never be necessary to manually edit the external persistence configuration file. However, as the default Rhino install is configured to connect to a PostgreSQL database, one of the most likely reasons for needing to manually edit the file is if some other vendor’s database needs to be used, for example Oracle, as Rhino will not start if it doesn’t have an external database that it can connect to.

This section describes the format of the persistence configuration file.

Persistence configuration file location

The persistence configuration file can be found in ${RHINO_HOME}/config/persistence.xml. However this file only exists if the Rhino node has been started at least once. If the node has yet to be started, or if the persistence.xml file is deleted, then the persistence configuration is obtained from ${RHINO_HOME}/config/defaults.xml

Every node in a Rhino cluster has the same persistence configuration. A Rhino node that boots and joins an existing cluster will obtain its persistence configuration from the other nodes in the cluster. The cluster configuration will be saved into the node’s ${RHINO_HOME}/config/persistence.xml file, potentially overwriting any local changes that may have been made to it.

XML Format of a Persistence Configuration

The persistence configuration is contained within the <persistence> element in the configuration file. The <persistence> element may contain any number of the following elements:

Element

Description

 <persistence-instance>

Contains the configuration information for a single persistence instance

 <presistence-resource>

Contains the configuration information for a single persistence resource

 <jdbc-resource>

Contains the configuration information for a single jdbc resource

Persistence instance configuration

A persistence instance configuration is contained in a <persistence-instance> element. This element must have the following attributes:

Attribute Description

 name

The name of the persistence instance. This name must be unique between all persistence instance configurations.

 datasource-class-name

The fully-qualified name of the Java class from the database driver that implements the javax.sql.DataSource interface or the javax.sql.ConnectionPoolDataSource interface.

A <persistence-instance> element may also include zero or more <parameter> elements. Each <parameter> element identifies the name, Java type, and value of a configuration property of the datasource class using the following element attributes:

Attribute Description

 name

The name of a JavaBean property defined by the datasource class.

 type

The fully-qualified Java class name of the JavaBean property’s type.

 value

The value that should be assigned to the configuration property.
Variables may be used in the construct of JavaBean property values. Variables are indicated using the ${...} syntax, where the value between the braces is the variable name. Rhino attempts to resolve the variable name by looking in the following places in this order:

The content of the $RHINO_HOME/config/config_variables file
Java system properties
User environment variables

Example

Below is an example of the default configuration that connects to a PostgreSQL database:

<persistence-instance datasource-class-name="org.postgresql.ds.PGSimpleDataSource" name="postgres">
    <parameter name="serverName" type="java.lang.String" value="${MANAGEMENT_DATABASE_HOST}"/>
    <parameter name="portNumber" type="java.lang.Integer" value="${MANAGEMENT_DATABASE_PORT}"/>
    <parameter name="databaseName" type="java.lang.String" value="${MANAGEMENT_DATABASE_NAME}"/>
    <parameter name="user" type="java.lang.String" value="${MANAGEMENT_DATABASE_USER}"/>
    <parameter name="password" type="java.lang.String" value="${MANAGEMENT_DATABASE_PASSWORD}"/>
    <parameter name="loginTimeout" type="java.lang.Integer" value="30"/>
    <parameter name="socketTimeout" type="java.lang.Integer" value="15"/>
    <parameter name="prepareThreshold" type="java.lang.Integer" value="1"/>
</persistence-instance>

Persistence resource configuration

A persistence resource configuration is contained in a <persistence-resource> element. This element must have a name attribute, which specifies the name of the persistence resource. The name must be unique between all persistence resource configurations.

A <persistence-resource> element may also include zero or more <persistence-instance-ref> elements. Each <persistence-instance-ref> element must have a name attribute, which must be the name of a persistence instance defined elsewhere in the configuration file. The persistence resource will store relevant in-memory database state into each referenced persistence instance.

Example

Below is an example of a persistence resource that stores state into two PostgreSQL databases:

<persistence-resource name="management">
    <persistence-instance-ref name="postgres-db1"/>
    <persistence-instance-ref name="postgres-db2"/>
</persistence-resource>

JDBC resource configuration

A JDBC resource configuration is contained in a <jdbc-resource> element. This element must have a jndi-name attribute, which specifies the JNDI name relative to the java:resource/jdbc namespace where the resource will be bound in the JNDI tree. The JNDI name must be unique between all JDBC resource configurations.

A <jdbc-resource> element may also optionally include a <persistence-instance-ref> element and a <connection-pool> element.

The <persistence-instance-ref> element must have a name attribute, which must be the name of a persistence instance defined elsewhere in the configuration file. The JDBC resource will use the database identified by the referenced persistence instance to execute SQL queries.

The presence of a <connection-pool> element indicates to Rhino that a connection pool should be used to manage the physical connections used by the JDBC resource. The element may define attributes with the names of the connection pool configurable parameters. If a given parameter is absent in the element’s attribute list then the default value for that parameter is assumed.

Example

Below is an example of a JDBC resource:

<jdbc-resource jndi-name="jdbc">
    <persistence-instance-ref name="postgres-jdbc"/>
    <connection-pool
        connection-pool-timeout="5000"
        idle-check-interval="30"
        max-connections="15"
        max-idle-connections="15"
        max-idle-time="600"
        min-connections="0"/>
</jdbc-resource>

Cluster Membership

Rhino maintains a single system image by preventing inconsistent nodes from forming a cluster. It determines cluster membership based on the set of cluster nodes reachable within a time-out period.

This page explains the strategies available for managing and configuring cluster membership.

Cluster membership is not a concern when using the Rhino SDK Getting Started Guide, where the cluster membership is always just the single SDK node.

Below are descriptions of:

how Rhino uses component selectors to determine which node is in the primary component
how cluster membership changes in Rhino
configuring what sort of component selector you are using
the differences between DLV and 2-node selectors
using quorum nodes to distinguish node failure from network segmentation.

How nodes "go primary"

What is primary component selection?

A cluster node runs a primary component selection algorithm to determine whether the component it belongs to is primary or non-primary — without a priori global knowledge of the cluster.

The primary component is the authoritative set of nodes in the cluster. A node can only perform work when in the primary component. When a node enters the primary component, we say it "goes primary". Likewise, when a node leaves the primary component, we say it "goes non-primary".

The component selector manages which nodes are in the primary component. Rhino provides a choice of two component selectors: DLV or 2-node. The component selector needs to maintain a consistent view of the primary component in several scenarios, to maintain the single system image provided by Rhino.

Segmentation and split-brain

Nodes can become isolated from each other if some networking failure causes a network segmentation. This carries the risk of a "split brain" scenario, where nodes on both sides of the segment consider themselves primary. Rhino, which is managed as a single system image, does not allow split brain scenarios. The DLV and 2-node selectors use different strategies for avoiding split-brain scenarios.

Starting and stopping nodes

Nodes may stop and start the following ways:

node failure — Individual cluster nodes may fail, for example due to a hardware failure. From the point of view of the remaining nodes, node failures are indistinguishable from network segmentation. Behaviour of the surviving members is determined by the component selector.
automatic shutdown with restart — There are cases described in this guide where the component selector "shuts down" a node, for example to prevent split-brain scenarios. It does this by shifting the node from primary to non-primary. Whenever a node goes from primary to non-primary, it self-terminates. The node will still restart if the “-k” flag was passed to the start-rhino.sh script. The node will become primary again as soon as the component selector determines it’s safe to do so.
node start or restart — When a booting node enters a cluster which is primary, the node will also go primary, and will receive state from existing nodes.
remerge — A remerge happens after a network segmentation, when connectivity between network segments is restored. When a network segment of non-primary nodes merges with a segment of primary nodes, the non-primary nodes will also go primary, and receive state from the other nodes. In the unlikely case that two primary segments try to merge, Rhino will shut down the nodes in one of the segments, to maintain the single system image. This should only happen if two sides of a network segment are manually activated using the -p flag when using DLV (an administrative error), or after a network failure when using the 2-node selector.

Specifying the component selector

The main configuration choice related to cluster membership is the choice of component selector. If no component selector is specified, Rhino uses DLV as the default.

To specify the component selector, set the system property com.opencloud.rhino.component_selection_strategy to 2node or dlv on each node. Add this line near the end of read-config-variables file under the node directory to use the 2-node strategy:

OPTIONS="$OPTIONS -Dcom.opencloud.rhino.component_selection_strategy=2node"

This property must be set consistently on every node in the cluster. Rhino will shut down a node trying to enter a cluster using a different component selector.

The DLV component selector

What is DLV?

The DLV component selector is inspired by the dynamic-linear voting (DLV) algorithm described by Jajodia and Mutchler in their research paper Dynamic voting algorithms for maintaining the consistency of a replicated database.

DLV is the default primary component strategy. It is suitable for most deployments, and recommended when using three or more Rhino nodes, or two nodes plus a quorum node.

The DLV component selector uses a voting algorithm where the membership of previous primary components plays a role in the selection of the next primary component. Each node persists its knowledge of the last known primary component. When the cluster membership changes, each node exchanges a voting message that contains its own knowledge of previous primary components. Once voting completes, each node, independently, uses these votes to make the same decision on whether to be primary or non-primary. A component can be primary if there are enough members present from the last known configuration to form a quorum.

The DLV component selector guarantees that in the case of a network segmentation (where sets of nodes are isolated from each other), that at most one of the segments will remain primary, to avoid a 'split-brain' scenario where two segments consider themselves primary. This is achieved by considering any component smaller than cluster_size/2 to be non-primary. In the case of an exactly even split (4 node cluster with 2 nodes failed), the component with the lowest nodeID survives.

Manually activating DLV

Upon first starting a cluster using DLV, the primary component must be activated. You do this by passing the -p flag to start-rhino.sh when booting the first node. DLV persists the primary/non-primary state to disk, so specifying the -p flag is not required after the first time.

Using quorum nodes to distinguish node failure from network segmentation

What is a quorum node?

A quorum node is a lightweight node added to distinguish between network segmentation and node failure (as described above). It does not process events or run SLEE services (nodes that are not quorum nodes are sometimes called "event-router nodes"). Quorum nodes have much lighter hardware requirements than event-router nodes.

To start a quorum node, you pass the -q flag to the start-rhino.sh script. You should always set a quorum node’s node ID to be higher than the other nodes, so that the loss of a quorum node never causes an event-router node to shut down. (The node with the lowest ID is the distinguished node.)

A quorum node is useful to help distinguish between node failure and network segmentation when using just two event-router nodes. Given a cluster of nodes {1,2}, there are two node-failure cases:

If node {2} fails, the remaining node {1} will stay primary because it is the distinguished node (having the lowest node ID).
If node {1} fails, the remaining node {2} will go non-primary and shut down. DLV can’t distinguish this from network segmentation, so it shuts down node {2} to prevent the possibility of a split-brain scenario. This usually isn’t desirable, and there are two approaches for solving this case: use a 2-node component selector or add a single quorum node to the cluster.

The 2-node component selector

The 2-node selector is designed exclusively for configurations with exactly two Rhino nodes, with a redundant network connection between them. It differs from DLV in how it handles node failures. When one node fails, the other node stays primary, regardless of which of the nodes failed. Conceptually, the responsibility of avoiding a split-brain scenario shifts to the redundant network connection. For this reason, this strategy should only be used when a redundant connection is available. If network segmentation happens, and two primary components remerge, one side of the segment will be shut down.

Quorum nodes cannot be used with a 2-node selector. If you choose a 2-node selector, Rhino will prevent quorum nodes from booting.

Activating 2-node selectors automatically

2-node selectors automatically go primary when booting. (The -p flag is not necessary when using the 2-node selector, and is ignored.) When both nodes are visible, they become primary without delay. When a single node boots, it waits for a short time (defaulting to five seconds) before going primary. This prevents a different split-brain case when introducing new nodes.

Communications mode

Cluster membership may be run on exactly one of two communication methods. The communication mode must be chosen at cluster creation time and cannot be live reconfigured.

Multicast

This communication mode uses UDP multicast for communication between nodes. This requires that UDP multicast be available and correctly working on all hosts in the cluster.

Scattercast

This communication mode uses UDP unicast in a mesh topology for communication between nodes. This mode is intended for use where multicast support is not available, such as in the cloud. Scattercast requires significantly more complex configuration, and incurs some network overhead. Thus we do not recommend scattercast where multicast is available.

About Multicast

Multicast is the default communications mode used in Rhino.

This mode allows for automated cluster membership discovery by using the properties of multicast behaviour. When the network supports multicast this is the preferred communication mode, as it is much easier to configure in Rhino.

Nodes communicate by sending messages to well-known multicast groups. These are received by all nodes within the same network.

Configuring multicast

Configuration of multicast is very simple. A multicast address range must be specified; addresses from this range are used for different internal groups. Configuring this is handled by the rhino-install.sh script.

Troubleshooting multicast

A troubleshooting guide for multicast clusters can be found in Clustering.

About Scattercast

Scattercast is implemented as a replacement for UDP multicast clustering in environments that do not support multicast.

Cluster-wide communication mode

Choosing a cluster communication mode is a cluster-wide decision. It should be made before installation begins. The cluster cannot correctly form when the cluster communication mode is inconsistent; two independent, primary clusters will form.

How does it work?

Normally Savanna will send UDP datagrams to a well-known multicast group address / port combination to maintain cluster membership. Message transfer happens on separate multicast group address / port combinations that are allocated at runtime from a pool.

Scattercast replaces each multicast UDP datagram with multiple unicast UDP datagrams, one to each involved node. Each node has a unique unicast address / port combination ("scattercast endpoint") used for cluster membership. A separate unicast address is used for each message group. Another separate unicast address is used for co-ordinating state transfer between members. This is created from membership IP address, membership port + state_distribution_port_offset (default 100. Configured in {$NODE_HOME}/config/savanna/cluster.properties).

All nodes must a priori know the endpoint addresses of all other nodes in the same cluster. To achieve this, a configuration file scattercast.endpoints is stored on each node. This file is created during install and is subsequently managed using the Scattercast Management commands.

Separate endpoints for message transfer are allocated at runtime based on the membership address, and a port chosen from the range first_port to last_port (defaults to 46700, 46800). This is configured in {$NODE_HOME}/config/savanna/cluster.properties).

Scattercast uses separate groups for message transfer and membership. Ports used for the membership group in scattercast endpoints must not overlap with the port range used for message groups.

UDP broadcast addresses are not supported in scattercast endpoints. These will not be rejected by the installer, scattercast commands, or the recovery tool, but must be avoided by users.

Scattercast endpoint configuration is versioned, and hashed to ensure consistency. The cluster will prefer the newest version if multiple versions are detected when a node tries to join the cluster. Nodes that detect an out-of-date local version will shut down immediately. Nodes that detect a hash mismatch will also shut down immediately, as this indicates corrupt or manually modified contents.

All clustering configuration is stored per node, and must be updated on all nodes to remain in sync. It is expected that this should not be changed often, if at all.

What’s the downside?

Scattercast requires sending additional traffic. For an N-node cluster, scattercast will generate about (N-1) times as many datagrams as the equivalent multicast cluster. That is, there is no penalty for a 2-node cluster; a 3-node cluster will generate about 2x traffic; a 4-node cluster will generate 3x traffic; and so on. At high loads you may run out of network bandwidth sooner; also, there is some CPU overhead involved in sending the extra datagrams.

Scattercast cannot automatically discover nodes; you must explicitly provide endpoint information for all nodes in the cluster. To add nodes, remove nodes, or update nodes at runtime, online management commands should be used.

Manual editing of the configuration file scattercast.endpoints is not supported. Manual editing will cause edited nodes to fail to boot.

Initial setup

A cluster must be seeded with an initial scattercast endpoints file containing valid mappings for all initial nodes. Without a valid scattercast endpoints file a node is unable to boot in scattercast comms mode. This initial endpoints set may be generated by the Rhino installer. When choosing to install in scattercast mode, the installer script must be provided with an initial endpoints set. Details can be found in Unpack and Gather Information.

If the initial cluster size is known at installation time, providing the full endpoint set here is recommended, as there is no manual step required when this is done.

Troubleshooting guide

A troubleshooting guide for scattercast can be found in Scattercast Clustering.

Scattercast Management

Below is an overview of procedures for managing and repairing scattercast endpoints.

Online management

Once a cluster has been established following the procedures in initial setup, online management of the scattercast endpoints becomes possible. There are four basic management commands, to get, add, delete, or update scattercast endpoints.

Each command applies the result to all currently executing nodes. If there is a node not currently executing that requires the new endpoints set, this must be done manually. To provide an up-to-date endpoints set to an offline node, copy {$NODE_HOME}/config/savanna/scattercast.endpoints from any up-to-date node to the matching path for the new node. All currently running nodes should have an up-to-date copy of this file. This can be verified by using the getscattercastendpoints command.

scattercast.endpoints cannot be manually edited. Nodes will not boot with a manually edited scattercast.endpoints.

Multicast, localhost, and wildcard addresses are not permitted in scattercast endpoints. As an endpoint address is used to both send and recieve, the localhost address confines the cluster to a single host. Wildcard addresses are only usable to listen on all interfaces. These addresses cannot be sent from, and thus are invalid in scattercast endpoints.

Inconsistent states

When manually copying scattercast endpoint sets to all cluster members, the cluster will reject all write-management commands until it is rebooted. This occurs because persistent and in-memory state are not identical across all nodes.

This may also occur in other ways, and can be resolved with the following steps

If the disk state is correct on all nodes, reboot the cluster.
If the disk state is not correct or not the same on all nodes:
- If the disk state incorrect, use the recover-scattercast-endpoints.sh tool in repairing to create a new, correct file; and copy to all nodes before reboot.
- If disk state is correct on some but not all nodes, copy the file from correct nodes to all other nodes.

Repair

In most cases where the scattercast configuration is inconsistent, the faulty nodes can be restored by copying the scattercast.endpoints file from an operational node. If the node has been deleted from the current configuration, it should first be re-added using the addscattercastendpoints rhino-console command. The configuration file can be found at $RHINO_HOME/$NODE/config/savanna/scattercast.endpoints.

If no nodes are operational, such as after a major change to network addressing, the tool recover-scattercast-endpoints.sh can be used to rebuild the configuration from scratch.

After running recover-scattercast-endpoints.sh, you must copy the generated file to $RHINO_HOME/$NODE/config/savanna/scattercast.endpoints for each node.

Recovering scattercast endpoints

The recover-scattercast-endpoints.sh script is used to rebuild the scattercast config file after a major network change or configuration data loss. It can be run interactively, prompting for node,IP,port tuples, or using the new configuration on the command line. Options must be provided before the list of endpoints. If you want to use automatic port assignment, you must provide the baseport and offset options to allow calculation of a valid port set.

Usage:

$ cd rhino
$ ./recover-scattercast-endpoints.sh -?

 Usage: ./recover-scattercast-endpoints.sh [options] [node,ip-address[,port]]*
       Creates a seed scattercast endpoints file. The generated file needs to be copied to {$NODE_HOME}/config/savanna/scattercast.endpoints in all cluster nodes.
       If no endpoints are provided, enters interactive mode.
       arguments:
        -f, --file           Relative path to output file.
        -b, --baseport       The scattercast base port, used to derive a port when no port is specified in endpoints.
        -o, --offset         The scattercast port offset, used to derive a port when no port is specified in endpoints.
        -?, --help           Displays this message.

Example:

$RHINO_HOME/recover-scattercast-endpoints.sh -b 19000 -o 100 101,192.168.1.1,19000 102,192.168.1.2 103,192.168.1.2,19003

If baseport and offset are provided, they are used only for the recovery tool. Nodes added or updated with management commands will continue to use values in cluster.properties.

Add Scattercast Endpoint(s)

addscattercastendpoints adds one or more new endpoints to the scattercast endpoints set.

This must be done before the new node is booted because a node cannot boot if it is not in the scattercast endpoints set. After running the add command successfully, the scattercast endpoints file must be copied from an existing node to the new node. This cannot be done with rhino-management commands.

If an endpoint is added with the wrong ip/port, this can be resolved by deleting and re-adding the endpoint.

Command

addscattercastendpoints <node,ip-address[,port]>*
  Description
    Add scattercast endpoints for new cluster members. If port is omitted, one will
    be assigned automatically.

Examples

Add endpoints for nodes 102, 103:

[Rhino@localhost (#1)] addscattercastendpoints 102,192.168.0.127 103,192.168.0.127
Endpoints added successfully. Displaying new scattercast endpoints mappings:
NodeID   Address
-------  --------------------
    101   192.168.0.127:12000
    102   192.168.0.127:12001
    103   192.168.0.127:12002
3 rows

Attempt to add an invalid address:

[Rhino@localhost (#4)] addscattercastendpoints 104,224.0.101.1
Multicast addresses are not permitted in scattercast endpoints: 224.0.101.1
Invalid usage for command 'addscattercastendpoints'.  Usage:
  addscattercastendpoints <node,ip-address[,port]>*
     Add scattercast endpoints for new cluster members. If port is omitted, one will
     be assigned automatically.

Add a node while node 102 has changed disk state:

[Rhino@localhost (#7)] addscattercastendpoints 104,192.168.0.127
Failed to add endpoints:
Node 102 reports: Disk state does not match memory state. No write commands available.

Delete Scattercast Endpoint(s)

deletescattercastendpoints removes endpoints for shut-down nodes.

A node’s endpoint cannot be deleted while in use. This means that the node must be shut down and have left the cluster before a delete can be issued.

A node that has been deleted cannot rejoin the cluster unless it is re-added, and the new scattercast endpoints file copied over. Copying an older scattercast endpoints file will not work, as the cluster uses versioning to protect against out-of-sync endpoints files.

Command

deletescattercastendpoints <-nodes node1,node2,...>
  Description
    Delete scattercast endpoints for cluster members being removed.

Examples

Delete a shut-down node:

[Rhino@localhost (#3)] deletescattercastendpoints -nodes 104
Endpoints deleted successfully, removed nodes shut down.
New scattercast endpoints mappings:
NodeID   Address
-------  --------------------
    101   192.168.0.127:12000
    102   192.168.0.127:12001
    103   192.168.0.127:12002
3 rows

Delete a running node:

[Rhino@localhost (#9)] deletescattercastendpoints -nodes 101
Failed to delete scattercast endpoints due to: Node: 101 currently running. Please shutdown nodes before deleting.

Get Scattercast Endpoints

getscattercastendpoints reports the set of scattercast endpoints known to all currently running cluster members.

This command may be issued at any time. If cluster membership changes during the read command, this causes an immediate failure, reporting that the cluster membership changed. This is for consistency with write commands.

Command

getscattercastendpoints
  Description
    Get the scattercast endpoints for the cluster.

Examples

Single node read with consistent endpoints:

[Rhino@localhost (#1)] getscattercastendpoints
[Consensus] Disk Mapping   : Coherent
[Consensus] Memory Mapping :
      [101] Address  : 192.168.0.127:12000
      [102] Address  : 192.168.0.127:12001

Two nodes read, where node 102 has different in-memory and disk mappings:

[Rhino@localhost (#2)] getscattercastendpoints
[101] Disk Mapping   : Coherent
[101] Memory Mapping :
      [101] Address  : 192.168.0.127:12000
      [102] Address  : 192.168.0.127:12001

[102] Disk Mapping   :
      [101] Address  : 192.168.0.127:12000
      [102] Address  : 192.168.0.127:12001
      [103] Address  : 192.168.0.127:18000

[102] Memory Mapping :
      [101] Address  : 192.168.0.127:12000
      [102] Address  : 192.168.0.127:12001

Read failed due to a cluster-membership change:

[Rhino@localhost (#3)] getscattercastendpoints
[Consensus] Disk Mapping   : Cluster membership change detected, command aborting
[Consensus] Memory Mapping : Cluster membership change detected, command aborting

Update Scattercast Endpoint(s)

updatescattercastendpoints updates the endpoints for currently running nodes.

In order to update scattercast endpoints, the SLEE must be stopped clusterwide. If this is successful, it triggers an immediate cluster restart to reload scattercast state.

Update commands make a best-efforts attempt to validate that the updated value will be usable after cluster reboot. This is done by attempting to bind the new address.

Updates cannot be applied to non-running cluster nodes. To update a node that is out of the cluster, simply delete the node and add with the new address.

Command

updatescattercastendpoints <node,ip-address[,port]>*
  Description
    Update scattercast endpoints for existing cluster members. If port is omitted,
    one will be assigned automatically. WARNING: This command will cause a cluster
    restart.

Examples

Update with the whole cluster in a stopped state:

[Rhino@localhost (#0)] updatescattercastendpoints 101,192.168.0.127,18000
Update executed successfully, cluster shutting down now.

Update while the SLEE is in a running state:

[Rhino@localhost (#3)] updatescattercastendpoints 101,192.168.0.127,12000
Failed to update scattercast endpoints due to: Cannot update scattercast endpoints while SLEE is running.

Update a non-running node:

[Rhino@localhost (#5)] updatescattercastendpoints 102,192.168.0.127,12000
Failed to update scattercast endpoints due to: 102 is not currently alive. Updates can only be done against currently live nodes

Errors

If the update command fails part way through execution, it is likely that the update will not have been applied. Under some rare circumstances, such as multiple disk errors or filesystem-access problems, the update will have only been applied to some nodes. Check the current configuration by running the getscattercastendpoints command to verify that the on-disk config is either coherent with the in-memory config (the command rolled back cleanly) or consistent across all nodes (the command failed after writing all changes). Identify and fix the fault that caused the command to fail, then reboot the cluster. If the update failed before writing the new configuration, rerun the update after fixing the fault that caused the initial attempt to fail.

Alarms

If the updatescattercastendpoints command is unable to reboot the cluster automatically, for example due to a timeout writing state to the persistent database, it raises a CRITICAL alarm of type rhino.scattercast.update-reboot-required and a message:

Scattercast endpoints have been updated. A cluster reboot is required
to apply the update as soon as possible otherwise a later partial reboot
e.g. due to network segmentation could result in a split-brain cluster.

Static Replication Domaining

This section covers what resources are domainable in Rhino 2.3 and late, instructions for configuring basic and advanced features of static replication domaining, and how to display the current domaining configuration.

What is static replication domaining?

Static replication domaining means partitioning Rhino’s replication mechanisms to perform replication only between selected subsets of nodes.

A subset of nodes is called a "domain". This provides better scaling for larger clusters, while still providing a level of replication to ensure fault tolerance. Prior to Rhino 2.3.0, a cluster could be considered as having one and only one domain, and every node could be considered a member of that domain.

Domain configuration consists of a set of domain definitions, each associated with one or more domainable resources and one or more cluster nodes.

Domainable resources

Rhino includes two types of domainable resources:

persistence resources — instances of MemDB (Rhino’s in-memory database) that act as storage for SBB, RA, or profile replication
activity handler resources — the existence and the state of Activity Context Interfaces, Activity Contexts, and associated attributes.

Activity Handlers, SBB Persistence, and RA Persistence replicated resources are domainable in Rhino 2.3.0 and later

The Null Activity Factory and Activity Context Naming are not domainable. This means that these resources are replicated cluster wide.

Configuring Static Replication Domaining

Below are instructions for configuring static replication domaining.

Configuring basic domaining settings

To configure domaining, you edit the config/rhino-config.xml in each Rhino node directory. The domain definitions in those files look like this:

<domain name="domain-name" nodes="101,102,...,n">
  ... resources associated with the domain ...
</domain>

Domainable resources

Inside each domain configuration block, each resource is defined using the following format and resource names:

Persistence resources Activity Handler resources

Format

Inside a memdb-resource section:

<memdb-resource>
  ...memory database name...
</memdb-resource>

Inside an ah-resource section:

<ah-resource>
  ...activity handler name...
</ah-resource>

Name

Same as the jndi-name used in its declaration in rhino-config.xml:

<memdb>
  <jndi-name>DomainedMemoryDatabase</jndi-name>
  <message-id>10005</message-id>
  <group-name>rhino-db</group-name>
  <committed-size>100M</committed-size>
  <resync-rate>100000</resync-rate>
</memdb>

Same as its group-name in rhino-config.xml:

<activity-handler>
  <group-name>rhino-ah</group-name>
  <message-id>10000</message-id>
  <resync-rate>100000</resync-rate>
</activity-handler>

It is extremely important that the domaining configuration section of rhino-config.xml is identical between all cluster nodes and does not change for the lifetime of the cluster. Any changes to the domaining configuration must be made while the cluster is offline.

Some persistence resources are not domainable as they contain data which either makes no sense to domain, or which must be global to the entire cluster. The current undomainable persistence resources are ReplicatedMemoryDatabase, LocalMemoryDatabase, and ManagementDatabase.

Example configuration

rhino-config.xml includes the following sample domaining configuration, commented out by default. It configures an 8-node cluster into 4 domains, with each domain containing 2 nodes — specifying that replication of SBB and RA shared state only happens between each pair of nodes.

<!--
  Example replication domain configuration.

  This example splits the cluster into several 2-node domain pairs for the purposes of
  service state replication. This example does not cover replication domaining for writeable
  profiles.
-->
<domain name="domain-1" nodes="101,102">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-2" nodes="201,202">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-3" nodes="301,302">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-4" nodes="401,402">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>

This example contains node IDs which start with the same number as their corresponding domain. While it’s not required, Metaswitch recommends this naming scheme as it clarifies which nodes are associated with a particular domain.

Default domain

The default domain (named domain-0) is not configurable and contains all replicated resources which are not explicitly domained as part of the configuration in rhino-config.xml. If a node is booted into the cluster, and does not have an associated domain configuration associated with it, it will use the default domain for all persistence resources. If no domains are configured at all, all resources will belong to the default domain.

Advanced configuration

It is possible, though less usual, to configure overlapping domains with different resources. The only constraint on the domaining configuration is that for each domainable resource, it may only occur in a single domain for any given node. For example, the following configuration is valid, despite multiple nodes containing the same NodeIDs.

This example builds on the basic example, adding two more domains (domain-profiles-1 and domain-profiles-2). These additional domains allow replication of writeable profiles (backed by MyWriteableProfileDatabase) across a larger set of nodes than the domains used for service replication.

<domain name="domain-profiles-1" nodes="101,102,201,202">
  <memdb-resource>MyWriteableProfileDatabase</memdb-resource>
</domain>
<domain name="domain-profiles-2" nodes="301,302,401,402">
  <memdb-resource>MyWriteableProfileDatabase</memdb-resource>
</domain>

<domain name="domain-services-1" nodes="101,102">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-services-2" nodes="201,202">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-services-3" nodes="301,302">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>
<domain name="domain-services-4" nodes="401,402">
  <memdb-resource>DomainedMemoryDatabase</memdb-resource>
  <ah-resource>rhino-ah</ah-resource>
</domain>

The configuration and setup of the memory database for use with writeable profiles is beyond the scope of this documentation.

Displaying the Current Domaining Configuration

To display the current domaining configuration, use the following rhino-console command or MBean operation.

Console command: `getdomainstate`

Command

getdomainstate
  Description
    Display the current state of all configured domains

Output

Display the current state of all configured domains.

Example

$ ./rhino-console getDomainState
domain-1:
DomainedMemoryDatabase, rhino-ah
  101                              Running
  102                              Running

domain-2:
DomainedMemoryDatabase, rhino-ah
  201                              Running
  202                              Running

domain-3:
DomainedMemoryDatabase, rhino-ah
  301                              Stopped
  302                                    -

domain-4:
DomainedMemoryDatabase, rhino-ah
  401                                    -
  402                                    -

Nodes which are configured with domain information but are not current part of the cluster are represented by a -.

MBean operation: `getDomainConfig`

MBean

com.opencloud.savanna2.framework.GroupHeartbeat.interval

Rhino extension

public TabularData getDomainConfig()
    throws ManagementException;

(See the javadoc for the structure of the TabularData returned by this operation.)

Data Striping

This section covers which MemDB instances support data striping, instructions for configuring basic and advanced features of MemDB data striping, how to display the current striping configuration, and striping-related statistics.

What is MemDB data striping?

MemDB data striping means dividing a MemDB instance into partially independent "stripes". This can remove bottlenecks in MemDB, letting Rhino better use more available cores (in machines with many CPU cores). In other words, the primary purpose of data striping is to increase vertical scalability. MemDB data striping was introduced in Rhino 2.3.1.

Data striping should not be used for replicated MemDB instances. Under some conditions it can corrupt the management database.

MemDB instances

Rhino includes two types of MemDB (Rhino’s in-memory database):

local MemDB — contains state local to the Rhino node, used by non-replicated applications running in "high-availability mode".
replicated MemDB — contains state replicated across the cluster, domain, or sub-cluster, used by replicated applications running in "fault-tolerant mode".

MemDB instances backed by disk storage — including the profile database and management database — do not support striping.

Configuring Data Striping

Below are instructions for configuring data striping.

Configuring basic striping settings

The number of stripes can be configured for each instance of MemDB.

How does the stripe count work?

To scale well on increasingly multi-core systems, it’s important to understand how the stripe count works:

Each MemDB stripe has a single commit order. While concurrent transactions (different events processed in parallel) execute concurrently against a single MemDB stripe, the commit protocol enforces an ordering, which means that no more than one commit at a time can occur against a single stripe.
While a single CPU can process thousands of commits per second, eventually a single commit order becomes a latency and scalability bottleneck.
Therefore, configuring a stripe count greater than 1 means that there is now more than 1 commit order — which means there is more scalability. So a stripe count of 8 means that up to 8 transactions can commit concurrently.

In summary, stripe count is the measure of commit concurrency.

Below are details on the default settings for stripe counts, and how to choose and set the stripe count for your MemDB instances.

Default settings

By default, Rhino disables data striping for all MemDB instances.

Choosing a stripe count

The stripe count must be 1 or greater, and must be a power of two (1, 2, 4, 8, 16, …). The stripe count should be proportional to the number of CPU cores in a server. A good rule of thumb is that the stripe count should be about 1/2 the number of CPU cores.

To disable striping, use a stripe count of 1.

In some cases when nodes are regularly leaving and joining a cluster, there is a chance of all cluster nodes being restarted as a result of striping being enabled.

We recommend that you consult with Metaswitch before enabling striping to ensure it is configured correctly in a stable and consistent network.

Setting the stripe count

Each MemDB instance has its own stripe count. To configure the stripe count for a particular MemDB instance, you edit the MemDB configuration for that instance, in the config/rhino-config.xml file in each Rhino node directory.

The stripe count for a MemDB instance must be the same on all nodes in the cluster. A new node will not start if it contains a stripe count which is inconsistent with other nodes in the cluster. Therefore, the stripe count cannot be changed while a cluster is running.

The striping configuration for a local MemDB instance looks like this:

<memdb-local>
  ...
  <stripe-count>8</stripe-count>
</memdb-local>

Data striping is not a supported configuration for replicated MemDB instance.

Displaying the Current Striping Configuration

To display the current striping configuration, use the following rhino-console command or MBean operations.

Console command: `getstripingstate`

Command

getstripingstate
  Description
    Display the striping configuration of all MemDB instances

Output

Display the striping configuration of all MemDB instances.

Example

$ ./rhino-console getstripingstate
Striping configuration for replicated MemDB instances:

memdb-resource             stripe-count   stripe-offset
-------------------------  -------------  --------------
       ManagementDatabase              1               0
          ProfileDatabase              1               0
 ReplicatedMemoryDatabase              1               0
3 rows

Striping configuration for local MemDB instances on node 101:

memdb-resource        stripe-count
--------------------  -------------
 LocalMemoryDatabase              8
1 rows

MBean operation: `getReplicatedMemDBStripingConfig`

MBean

RhinoHousekeepingMBean

Rhino extension

public TabularData getReplicatedMemDBStripingConfig()
  throws ManagementException;

(See the javadoc for the structure of the TabularData returned by this operation.)

MBean operation: `getLocalMemDBStripingNodeConfig`

MBean

NodeHousekeepingMBean

Rhino extension

public TabularData getLocalMemDBStripingNodeConfig()
  throws ManagementException;

(See the javadoc for the structure of the TabularData returned by this operation.)

MemDB and striping statistics

There are two sets of statistics related to MemDB data striping: MemDB statistics and striping-statistics.

MemDB statistics and striping

MemDB collects statistics under the MemDB-Replicated and MemDB-Local parameter sets, within each data stripe. They can be monitored on a per-stripe basis, or viewed as an aggregate across all stripes.

The parameter set names of the per-stripe statistics end with suffix of the form .stripe-N. For example, the stats for the first stripe will have suffix .stripe-0.

Striping statistics

MemDB maintains atomicity, consistency and isolation of data across stripes. This involves managing the versions of data exposed to various client transactions. The MemDB-Timestamp parameter set contains the relevant statistics.

This is a listing of the statistics available for a particular MemDB instance, within the MemDB-Timestamp parameter set:

Counter type statistics:
Id: Name:             Label:     Description:
0   waitingThreads    waiting    The number of threads waiting for a timestamp to become safe
1   unexposedCommits  unexposed  The number of commits which are not yet safe to expose

A database transaction containing at least one write is considered "safe to expose to client transactions" when (as shown by these statistics) all its changes — as well as all the write transactions that precede them — are available across all stripes.

These statistics are expected to have low values even under load (often with value zero), and should stay at zero when Rhino is not under load.

Service Assurance Server (SAS) Tracing

Metaswitch Service Assurance Server (SAS) is a platform that records traces of network flows and service logic.

Rhino TAS provides a SAS facility for components to send events to SAS.

This section describes the commands for managing the SAS facility and resource bundles.

SAS Configuration
Enabling and Disabling SAS Tracing
SAS Bundle Mappings
SAS Bundle Generation

For information about developing Rhino applications with the SAS tracing functionality, refer to the Rhino SAS API Development Guide.

SAS Configuration

This page describes the commands used to configure and enable the SAS facility. SAS configuration is namespace-aware: all these commands apply to the current namespace for the client (selected with setactivenamespace).

Resource Identifiers

Events sent to SAS are associated with a resource identifier. All components within a Rhino namespace use the same resource identifier. The resource identifier can be set with the setsasresourceid command.

The resource identifier will be in the generated resource bundle that will be imported into SAS.

SAS Servers

Rhino supports connecting to some or all SAS server instances in a federation. This is maintained as an internal list of servers and ports. Servers may be added with addsasserver and removed with removesasserver. By default SAS listens on port 6761. If the port is omitted from the add command, then the default port will be used.

Changing SAS configuration

Note that all commands that change SAS configuration on this page require the SLEE be in the Stopped state on all cluster nodes and/or SAS tracing to be disabled.

If you want to disable SAS tracing without stopping the SLEE, you can do so using setsasenabled false -force true. Then make changes and re-enable SAS using setsasenabled true.

The getsasconfiguration command can be run at any time.

Console command: `addsasserver`

Command

addsasserver <servers>
  Description
    Add one or more servers to the set of configured SAS servers
  Required Arguments
    servers  Comma delimited list of host:port pairs for SAS servers

Example

[Rhino@localhost (#1)] addsasserver localhost:12000
Added server(s) to SAS configuration properties:
servers=localhost:12000

[Rhino@localhost (#2)] addsasserver 127.0.0.1:12001,127.0.0.2
Added server(s) to SAS configuration properties:
servers=127.0.0.1:12001,127.0.0.2

Console command: `removesasserver`

Command

removesasserver <servers>
  Description
    Remove one or more servers from the set of configured SAS servers
  Required Arguments
    servers  Comma delimited list of host:port pairs for SAS servers

Example

[Rhino@localhost (#1)] removesasserver localhost:12000
Removed server(s) from SAS configuration properties:
servers=localhost:12000

[Rhino@localhost (#2)] removesasserver 127.0.0.1:12001,127.0.0.2
Removed server(s) from SAS configuration properties:
servers=127.0.0.1:12001,127.0.0.2

Console command `setsassystemname`

Command

setsassystemname <systemName> [-appendID <appendID>]
  Description
    Configure the SAS system name.
  Required Arguments
    systemName  The unique system name to use. Cluster wide
  Options
    -appendID  If true, append node ID to system name

Example

$ ./rhino-console setsassystemname mmtel
Set SAS system name:
systemName=mmtel

$ ./rhino-console setsassystemname mmtel -appendID true
Set SAS system name:
systemName=mmtel
appendID=true

Console command `setsassystemtype`

Command

setsassystemtype <systemType>
  Description
    Configure the SAS system type.
  Required Arguments
    systemType  The system type to use. Cluster wide

For systems running Sentinel products on the Rhino platform, Metaswitch recommends the following system type strings:

VoLTE TAS for Sentinel VoLTE
IPSMGW for Sentinel IPSMGW
BSF for Sentinel GAA
ShCM for Sh Cache Microservice

Example

$ ./rhino-console setsassystemtype BSF
Set SAS system type:
systemType=BSF

Console command `setsassystemversion`

Command

setsassystemversion <systemVersion>
  Description
    Configure the SAS system version.
  Required Arguments
    systemVersion  The system version to use. Cluster wide

For systems running Sentinel products on the Rhino platform, Metaswitch recommends using the three-digit version number of the installed product as the system version string, for example 3.0.0.

Example

$ ./rhino-console setsassystemversion 3.0.0
Set SAS system version:
systemVersion=3.0.0

Console command `setsasresourceid`

Command

setsasresourceid <resourceIdentifier>
  Description
    Configure the SAS resource identifier.
  Required Arguments
    resourceIdentifier  The resource identifier to use.

Example

$ ./rhino-console setsasresourceid com.metaswitch.rhino
Set SAS resource identifier:
resourceIdentifier=com.metaswitch.rhino

Console command `setsasqueuesize`

Command

setsasqueuesize <queueSize>
  Description
    Configure the per server SAS message queue limit.
  Required Arguments
    queueSize  The maximum number of messages to queue for sending to the SAS
    server.

Example

$ ./rhino-console setsasqueuesize 100000
Set SAS queue size:
queueSize=100000

Console command: `getsasconfiguration`

Command

getsasconfiguration
  Description
    Display SAS tracing configuration

Example

$ ./rhino-console getsasconfiguration
SAS tracing is currently disabled.

Configuration properties for SAS:
servers=[sas-server]
systemName=mmtel
appendNodeIdToSystemName=true
resourceIdentifier=com.metaswitch.rhino
queueSize=10000 per server

MBean methods

Enabling and Disabling SAS Tracing

SAS tracing can be enabled and disabled using the setsasenabled command. The Rhino SAS facility must be configured with both a resource identifier and server list before being enabled. SAS tracing state is namespace-aware: this command applies to the current namespace for the client (selected with setactivenamespace).

Disabling SAS tracing on a running SLEE requires use of the -force option in order to shut down SAS tracing. When the SLEE is running, there may be activities actively tracing to SAS. Live reconfiguration of the SAS facility will result in breaking all trails started before the reconfiguration. If this is acceptable, then the -force parameter will allow a clean shutdown of SAS tracing for reconfiguration.

Console command: `setsasenabled`

Command

setsasenabled <enable> [-force <force>]
  Description
    Enable or disable SAS tracing. Configure SAS before enabling.
  Required Arguments
    enable  True to enable SAS tracing, false to disable.
  Options
    -force  True to override the SLEE state check when disabling SAS tracing state.
    SAS tracing state cannot normally be disabled when the SLEE is not in the
    Stopped state, because this may cause incomplete trails to be created in SAS for
    sessions that are in progress.

Example

To enable SAS tracing:

$ ./rhino-console setsasenabled true
SAS tracing enabled

SAS Bundle Mappings

Rhino TAS uses a prefix per mini-bundle to generate full event IDs included in the exported SAS resource bundle.

In general, you need to manually define the prefixes and map them to the mini-bundles. This section describes the Rhino management console commands that you can use to manage the mappings.

For more information about defining bundle mappings, see Define bundle mappings in the Rhino SAS API Development Guide.

Console command: `listsasbundlemappings`

Command

listsasbundlemappings [-sortBy <sortBy>]
  Description
    Lists all the SAS bundle mappings.
  Options
    -sortBy  The column to sort the bundle mappings by for display. Either 'name' or
    'prefix'

Example

[Rhino@localhost (#1)] listsasbundlemappings
name                                      prefix
----------------------------------------  -------
 com.opencloud.slee.services.example.sas   0x0001
1 rows

Console command: `setsasbundlemapping`

Command

setsasbundlemapping <name> <prefix>
  Description
    Sets a SAS bundle mapping.
  Required Arguments
    name  The fully qualified name of the bundle.
    prefix  The prefix for the bundle mapping, as a decimal, hex, or octal string.

Example

[Rhino@localhost (#1)] setsasbundlemapping com.opencloud.slee.services.example.sas 0x0001
Added a SAS bundle mapping from com.opencloud.slee.services.example.sas to 0x0001.

Console command: `removesasbundlemapping`

Command

removesasbundlemapping <name>
  Description
    Removes a SAS bundle mapping.
  Required Arguments
    name  The fully qualified name of the bundle.

Example

[Rhino@localhost (#1)] removesasbundlemapping com.opencloud.slee.services.example.sas
Prefix for com.opencloud.slee.services.example.sas removed.

Console command: `listunmappedsasbundles`

Command

listunmappedsasbundles
  Description
    Display unmapped SAS bundles

Example

[Rhino@localhost (#1)] listunmappedsasbundles
Unmapped SAS bundles found:
com.opencloud.slee.resources.http

MBean methods

SAS Bundle Generation

SAS requires at least one resource bundle file containing definitions of all events that will be sent to the server. These definitions show SAS how to display and interpret data sent to the server.

Rhino verifies that a SAS enabled deployable unit includes a resource bundle containing definitions for all events it uses. These per DU resource bundles are called mini-bundles.

Rhino provides console commands to export a resource bundle suitable for use by SAS, containing mini-bundles from all installed deployable units.

Console command: `exportsasbundle`

Command

exportsasbundle <bundleFileName>
  Description
    Export SAS bundle.
  Required Arguments
    bundleFileName  The bundle file name.

Example

[Rhino@localhost (#1)] exportsasbundle my-bundle.yaml
Wrote combined bundle to: ~/my-bundle.yaml

Exported bundle

info:
  identifier: my-rhino
  minimum_sas_version: '9.1'
  version: '1522714397579'
events:
  0x000100:
    summary: Test Event
    level: 100
enums: {
  }

MBean methods

getSasBundle()

System Properties

Below is a list of system properties that can be used to modify Rhino behaviour.

Name

Description

Interval between sending per-group heartbeat messages

com.opencloud.savanna2.framework.GroupHeartbeat.loss_threshold

Number of unreceived pending heartbeats before the group heartbeat watchdog condition triggers

eventrouter.transaction_timeout

Transaction timeout for eventrouter transactions

logging.status.level

Log level to set on the Log4j status logger

notifications.max_pending_notifications

Maximum number of pending JMX notifications before notifications are dropped

notifications.notification_threads

Number of JMX notification threads to run

rhino.ah.gcthreads

Set the max number of AH GC threads

rhino.ah.pessimistic.maxlocktimeout

Set the maximum lock timeout for local activity handler

rhino.ah.replicating.maxlocktimeout

Set the maximum lock timeout for replicated activities

rhino.ah.replicating.migrationtimeout

Default ah migration timeout

rhino.audit.log_format

Format of CSV management audit log

rhino.bootup.locktimeout

Distributed lock timeout used during node bootup

rhino.config.yaml_code_point_limit

Set the maximum number of code points supported in a declarative YAML bundle for importing

rhino.er.ra_notifier_threads

Number of resource adaptor entity notifier (callback) threads

rhino.group_rmi.max_threads

Maximum number of threads available to handle group RMI operations.

rhino.halttimeout

The timeout in milliseconds before a JVM shutdown is forcibly terminated with a Runtime.halt().

rhino.lenient_config_property_validation

Relax resource adaptor entity config property validation.

rhino.license.default

Location of default Rhino license file

rhino.lock.lockmanager.defaultlocktimeout

Set the default timeout for lock acquisitions

rhino.mbean.boot.timeout

Timeout used for write operations during exceptionally long SLEE starts (during Rhino boot)

rhino.misc.queue_size

Queue size for Distributed Resource Manager’s misc. runnable stage

rhino.misc.thread_count

Number of threads used in Distributed Resource Manager’s misc. runnable stage

rhino.monitoring.clocksync.check_interval

Interval in milliseconds between cluster node clock sync checks

rhino.monitoring.clocksync.max_fail_count

Number of times a clock sync check must fail for a node before an alarm is raised

rhino.monitoring.clocksync.threshold

Threshold in milliseconds over which a cluster node will be reported as being out of clock synchronisation

rhino.rem.enabled

Enable the embedded Rhino Element Manager (SDK only).

rhino.skip_lifecycle_checks

Whether to skip check that prevents a resource adaptor from creating an activity in the STOPPING state.

rhino.slee_restricted_lifecycle

Restrict SLEE lifecycle changes depending on current node actual states.

rhino.state.convergence.interval_seconds

Interval between state convergence checks

rhino.state.convergence.retry_interval_millis

Interval between state convergence task retries

rhino.state.convergence.timeout_seconds

Time before a convergence task is considered to have timed out

rhino.tm.resources.queue_size

Queue size for Transaction Manager’s executor for blocking resource callbacks

rhino.tm.resources.thread_count

Number of threads used in Transaction Manager’s executor for blocking resource callbacks

rhino.tm.synchronization.queue_size

Queue size for Transaction Manager’s executor for synchronization callbacks

rhino.tm.synchronization.thread_count

Number of threads used in Transaction Manager’s executor for synchronization callbacks

rhino.verification.traceexceptions

Print exception stack traces for DU verification errors

sas.queue_full_interval

Interval in milliseconds to wait before clearing the queue-full alarm

savanna.receive_watchdog_interval

Maximum time a Savanna receive-thread may remain busy before it is considered stuck

snmp.filename.max-length

The maximum allowed filename length, used when generating MIBs during a MIB export

snmp.locktimeout

Default lock acquisition timeout for SNMP config update thread

snmp.max_config_update_retries

Maximum number of times the SNMP config update thread will attempt to obtain an exclusive lock in order to apply a given configuration update work item

snmp.name.max-length

The maximum number of characters allowed in object identifiers included in generated MIBs

snmp.name.uppercase

Determines if MIB identifiers and filenames will be forced to uppercase, used when generating MIBs during a MIB export

snmp.suppress_service_oid_conflict_alarms

Suppress the raising of alarms for duplicate parameter set type OID mappings when multiple services use the same base OID

staging.lifo_scan_interval

Interval between scans of the LIFO queue’s tail to check for expired items

staging.live_threads_fraction

Minimum percentage of staging threads that must remain alive to prevent a watchdog restart

timer.remote.http_timeout

HTTP response timeout in milliseconds for remote timer server requests

transaction.default_timeout

Default transaction age in milliseconds before a long-running transaction is aborted

transaction.timeout_check_interval

Interval in milliseconds between checks for transactions that need timing out

transaction.timeout_warn_percent

Transaction age (as a percentage of transaction timeout) to warn about long-running transactions at

watchdog.check_interval

Interval in milliseconds between watchdog checks

watchdog.max_pause_margin

Maximum delay in watchdog scheduling before a warning is displayed

watchdog.no_exit

Override the default behaviour of the watchdog to disable terminating the JVM.Do not use in a production deployment. An alarm will be raised when this mode is active.

watchdog.reverse_timewarp_margin

Maximum watchdog 'early wakeup' in milliseconds before a reverse-timewarp warning is displayed

watchdog.warn_interval

Minimum interval in milliseconds between displaying timewarp warnings

com.opencloud.savanna2.framework.GroupHeartbeat.interval
Description	Interval between sending per-group heartbeat messages
Valid values	time in milliseconds
Default value	5000

com.opencloud.savanna2.framework.GroupHeartbeat.loss_threshold
Description	Number of unreceived pending heartbeats before the group heartbeat watchdog condition triggers
Valid values	positive integer
Default value	10

eventrouter.transaction_timeout
Description	Transaction timeout for eventrouter transactions
Valid values	milliseconds
Default value	30000

logging.status.level
Description	Log level to set on the Log4j status logger
Valid values	ERROR,WARN,INFO,DEBUG,TRACE
Default value	ERROR

notifications.max_pending_notifications
Description	Maximum number of pending JMX notifications before notifications are dropped
Valid values	number of pending notifications, >= 0
Default value	500

notifications.notification_threads
Description	Number of JMX notification threads to run
Valid values	number of threads; ⇐0 implies same-thread delivery
Default value	1

rhino.ah.gcthreads
Description	Set the max number of AH GC threads
Valid values	>2
Default value	2

rhino.ah.pessimistic.maxlocktimeout
Description	Set the maximum lock timeout for local activity handler
Valid values	time in milliseconds
Default value	15000

rhino.ah.replicating.maxlocktimeout
Description	Set the maximum lock timeout for replicated activities
Valid values	time in milliseconds
Default value	15000

rhino.ah.replicating.migrationtimeout
Description	Default ah migration timeout
Valid values	time in milliseconds
Default value	60000

rhino.audit.log_format
Description	Format of CSV management audit log
Valid values	2.4 (old format) or 2.5 (includes an extra namespace field)
Default value	2.5

rhino.bootup.locktimeout
Description	Distributed lock timeout used during node bootup
Valid values	Positive integer (seconds)
Default value	120

rhino.config.yaml_code_point_limit
Description	Set the maximum number of code points supported in a declarative YAML bundle for importing
Valid values	positive integers
Default value	10485760

rhino.er.ra_notifier_threads
Description	Number of resource adaptor entity notifier (callback) threads
Valid values	Positive integer
Default value	1

rhino.group_rmi.max_threads
Description	Maximum number of threads available to handle group RMI operations.
Valid values	Positive integer
Default value	10

rhino.halttimeout
Description	The timeout in milliseconds before a JVM shutdown is forcibly terminated with a Runtime.halt().
Valid values	null
Default value	60000

rhino.lenient_config_property_validation
Description	Relax resource adaptor entity config property validation.
Extended Description	When set to `false` (the default), Rhino will reject an attempt to create a resource adaptor entity or update the config properties of an existing resource adaptor entity if the config properties provided by the client contains unrecognised config property names, ie. names not defined by the resource adaptor or any known vendor-specific properties. If this system property is set to `true`, this validation check is skipped and any config property name will be accepted.
Valid values	true,false
Default value	false

rhino.license.default
Description	Location of default Rhino license file
Valid values	absolute or relative file path
Default value	../rhino.license (rhino-sdk.license for Rhino SDK)

rhino.lock.lockmanager.defaultlocktimeout
Description	Set the default timeout for lock acquisitions
Valid values	time in milliseconds
Default value	60000

rhino.mbean.boot.timeout
Description	Timeout used for write operations during exceptionally long SLEE starts (during Rhino boot)
Valid values	Positive integer (seconds)
Default value	120

rhino.misc.queue_size
Description	Queue size for Distributed Resource Manager’s misc. runnable stage
Valid values	Positive integer
Default value	100

rhino.misc.thread_count
Description	Number of threads used in Distributed Resource Manager’s misc. runnable stage
Valid values	Positive integer
Default value	3

rhino.monitoring.clocksync.check_interval
Description	Interval in milliseconds between cluster node clock sync checks
Valid values	time in milliseconds
Default value	10000

rhino.monitoring.clocksync.max_fail_count
Description	Number of times a clock sync check must fail for a node before an alarm is raised
Valid values	Positive integer
Default value	5

rhino.monitoring.clocksync.threshold
Description	Threshold in milliseconds over which a cluster node will be reported as being out of clock synchronisation
Valid values	time in milliseconds
Default value	2000

rhino.rem.enabled
Description	Enable the embedded Rhino Element Manager (SDK only).
Valid values	true,false
Default value	true

rhino.skip_lifecycle_checks
Description	Whether to skip check that prevents a resource adaptor from creating an activity in the STOPPING state.
Extended Description	This property should be set to `false` (the default) to enforce the restriction on creating activities in the `STOPPING` state. When set to `true`, resource adaptors should check the state before creating an activity, to avoid a situation where a resource adaptor entity never deactivates because new activities are being created. See the documentation reference for more details.
Valid values	true,false
Default value	false
Reference	Resource adaptor entity lifecycle states

rhino.slee_restricted_lifecycle
Description	Restrict SLEE lifecycle changes depending on current node actual states.
Extended Description	Normally the start and stop SLEE lifecycle operations only consider the current node desired state(s) when determining if the operation is valid. When this property is set to `true`, Rhino will revert to legacy behaviour and also consider current node actual state(s) as well. The legacy behaviour, for example, doesn’t allow a node to be started if it is currently in the `STOPPING` state.
Valid values	true,false
Default value	false

rhino.state.convergence.interval_seconds
Description	Interval between state convergence checks
Valid values	an integer specifying the delay from the end of one scheduled convergence check to the start of the next
Default value	30

rhino.state.convergence.retry_interval_millis
Description	Interval between state convergence task retries
Valid values	an integer specifying the delay from the completion of a batch of state convergence operations after which to retry ones that did not meet the required preconditions
Default value	1000

rhino.state.convergence.timeout_seconds
Description	Time before a convergence task is considered to have timed out
Valid values	An integer specifying the interval in seconds from the creation of a convergence task to when it can be considered to have timed out and an alarm raised
Default value	300

rhino.tm.resources.queue_size
Description	Queue size for Transaction Manager’s executor for blocking resource callbacks
Valid values	Positive integer
Default value	100

rhino.tm.resources.thread_count
Description	Number of threads used in Transaction Manager’s executor for blocking resource callbacks
Valid values	Positive integer
Default value	2

rhino.tm.synchronization.queue_size
Description	Queue size for Transaction Manager’s executor for synchronization callbacks
Valid values	Positive integer
Default value	500

rhino.tm.synchronization.thread_count
Description	Number of threads used in Transaction Manager’s executor for synchronization callbacks
Valid values	Positive integer
Default value	2

rhino.verification.traceexceptions
Description	Print exception stack traces for DU verification errors
Valid values	true, false
Default value	false

sas.queue_full_interval
Description	Interval in milliseconds to wait before clearing the queue-full alarm
Valid values	positive integer
Default value	5000

savanna.receive_watchdog_interval
Description	Maximum time a Savanna receive-thread may remain busy before it is considered stuck
Valid values	time in milliseconds
Default value	5000

snmp.filename.max-length
Description	The maximum allowed filename length, used when generating MIBs during a MIB export
Valid values	Any positive integer. Any value less than or equal to zero will disable the length limit.
Default value	255

snmp.locktimeout
Description	Default lock acquisition timeout for SNMP config update thread
Valid values	time in milliseconds
Default value	30000

snmp.max_config_update_retries
Description	Maximum number of times the SNMP config update thread will attempt to obtain an exclusive lock in order to apply a given configuration update work item
Valid values	Positive integer
Default value	10

snmp.name.max-length
Description	The maximum number of characters allowed in object identifiers included in generated MIBs
Valid values	Any positive integer. Any value less than or equal to zero will disable the length limit.
Default value	127

snmp.name.uppercase
Description	Determines if MIB identifiers and filenames will be forced to uppercase, used when generating MIBs during a MIB export
Valid values	true, false
Default value	false

snmp.suppress_service_oid_conflict_alarms
Description	Suppress the raising of alarms for duplicate parameter set type OID mappings when multiple services use the same base OID
Valid values	true,false
Default value	false

staging.lifo_scan_interval
Description	Interval between scans of the LIFO queue’s tail to check for expired items
Valid values	time in milliseconds
Default value	1000

staging.live_threads_fraction
Description	Minimum percentage of staging threads that must remain alive to prevent a watchdog restart
Valid values	0 - 100
Default value	25

timer.remote.http_timeout
Description	HTTP response timeout in milliseconds for remote timer server requests
Valid values
Default value	2000

transaction.default_timeout
Description	Default transaction age in milliseconds before a long-running transaction is aborted
Valid values
Default value	180000

transaction.timeout_check_interval
Description	Interval in milliseconds between checks for transactions that need timing out
Valid values	time in milliseconds
Default value	10000

transaction.timeout_warn_percent
Description	Transaction age (as a percentage of transaction timeout) to warn about long-running transactions at
Valid values	0 - 100
Default value	75

watchdog.check_interval
Description	Interval in milliseconds between watchdog checks
Valid values	positive integer
Default value	1000

watchdog.max_pause_margin
Description	Maximum delay in watchdog scheduling before a warning is displayed
Valid values
Default value	1000

watchdog.no_exit
Description	Override the default behaviour of the watchdog to disable terminating the JVM.Do not use in a production deployment. An alarm will be raised when this mode is active.
Valid values	true,false
Default value	false

watchdog.reverse_timewarp_margin
Description	Maximum watchdog 'early wakeup' in milliseconds before a reverse-timewarp warning is displayed
Valid values
Default value	500

watchdog.warn_interval
Description	Minimum interval in milliseconds between displaying timewarp warnings
Valid values
Default value	15000

Changes in Rhino 2.6

Rhino 2.6 has removed some system properties previously available for extended configuration. These include:

Property

Replaced by

 rhino.tracer.defaultlevel

 setloglevel trace <trace_level>

Application-State Maintenance

As well as an overview of application-state maintenance, this section includes instructions for performing the following Rhino SLEE procedures with explanations, examples and links to related javadocs:

Procedure rhino-console command(s) MBean → Operation(s)

Finding Housekeeping MBeans

Rhino Housekeeping → getClusterHousekeeping
Rhino Housekeeping → getNodeHousekeeping

Finding activities in the SLEE

 findactivities

Housekeeping → getActivities

Getting detailed information about an activity

 getactivityinfo

Housekeeping → getActivityInfo

Forcefully removing an activity

 removeactivity

Housekeeping → removeActivity

Marking all activities of a resource adaptor entity for removal

 removeallactivities

Rhino Housekeeping → markAllActivitiesForRemoval

Finding SBB entities in the SLEE

 findsbbs

Housekeeping → getSbbs

Getting detailed information about an SBB entity

 getsbbinfo

Housekeeping → getSbbInfo

Forcefully removing an SBB entity

 removesbb

Housekeeping → removeSbb

Removing all SBB entities of a service

 removeallsbbs

Rhino Housekeeping → removeAllSbbs

Finding timers in the SLEE

 findtimers

Housekeeping → getTimers

Finding remote timers in the SLEE

 findremotetimers

Housekeeping → getRemoteTimers

Getting detailed information about a timer

 gettimerinfo

Housekeeping → getTimerInfo

Getting detailed information about a remote timer

 getremotetimerinfo

Housekeeping → getRemoteTimerInfo

Administratively cancelling a timer

 canceltimer

Housekeeping → cancelTimer

Administratively cancelling a remote timer

 cancelremotetimer

Housekeeping → cancelRemoteTimer

Finding activity context name bindings in the SLEE

 findactivitybindings

Housekeeping → getBoundActivities

Removing activity context name bindings in the SLEE

 removeactivitybinding

Housekeeping → removeActivityBinding

Inspecting environment entries

 getenventries

Deployment → getEnvEntries

Setting environment entries

 setenventries

Deployment → setEnvEntries

Inspecting security permissions

 getsecuritypolicy

Deployment → getSecurityPolicy

Modifying security permissions

 setsecuritypolicy

Deployment → setSecurityPolicy

Activity state garbage collection

 initiateactivitycleanup

Housekeeping → initiatecleanup

Global state garbage collection

 initiateglobalcleanup

Housekeeping → initiateglobalcleanup

About Application-State Maintenance

During normal operation, Rhino removes SBB entities when they are no longer needed to process events on the activities they are attached to — usually when all those activities have ended.

Sometimes, however, the normal SBB lifecycle is interrupted and obsolete entities remain. For example:

An SBB might be attached to an activity that didn’t end correctly, due to a problem in the resource adaptor entity that created it.
The sbbRemove method might throw an exception.

Unexpected problems such as these, with deployed resource adaptors or services, may cause resource leaks. Rhino provides an administration interface, the Node Housekeeping MBean, which lets you find and remove stale or problematic:

activities
SBB entities
activity context name bindings
timers.

The following topics include procedures for:

finding Housekeeping MBeans
finding, inspecting and removing activities, SBB entities and activity context name bindings
configuring runtime components.

Finding Housekeeping MBeans

To find Node or Cluster Housekeeping MBeans, if using MBean operations directly, use the Rhino Housekeeping MBean, as follows.

Cluster vs Node Housekeeping

Rhino includes two types of Housekeeping MBean, which provide the same set of functions for either an entire cluster or a single node:

Cluster Housekeeping MBeans operate on all event-router nodes in the primary component.
Node Housekeeping MBeans operate on a single cluster node only.

Many of the housekeeping commands available in rhino-console accept a -node parameter, which lets you execute the housekeeping command for a specified node, using the relevant Node Housekeeping MBean. Without this parameter, the command executes for the cluster, using the Cluster Housekeeping MBean.

MBean operation: `getClusterHousekeeping`

MBean

Rhino operation

public ObjectName getClusterHousekeeping()
  throws ManagementException;

This operation returns the JMX Object Name of a Cluster Housekeeping MBean.

MBean operation: `getNodeHousekeeping`

MBean

Rhino operation

public ObjectName getNodeHousekeeping(int)
  throws InvalidArgumentExcepiton, ManagementException;

This operation returns the JMX Object Name of a Node Housekeeping MBean for the given node.

Both the Cluster Housekeeping MBean and Node Housekeeping MBean expose the NodeHousekeepingMBean interface.

Activities

Rhino includes procedures for:

Finding Activities
Inspecting Activities
Removing Activities
Removing All Activities

Finding Activities

To find activities in the SLEE, use the following rhino-console command or related MBean operations.

Console command: `findactivities`

Command

findactivities [-maxpernode maxrows] [-node nodeid] [-removed|-all] [-ra
<ra-entity>] [-created-after date|time|offset] [-created-before
date|time|offset] [-updated-after date|time|offset] [-updated-before
date|time|offset]
  Description
    Find activities.  Use -removed to list only activities removed but not garbage
    collected.  Use -all to list all active and removed activities combined.

Options

Option

Abbreviation

Description

 -maxpernode <maxrows>

Retrieve at most this many activities from each event-router node (default is 100). Can be used to limit Rhino’s load when processing the request.

 -node <node-id>

Only display activities owned by the given node.

 -removed

Display removed but not yet garbage collected activities instead of active activities.

 -all

Display both active and removed but not yet garbage collected activities.

 -ra <ra-entity>

Only display activities created by the named resource adaptor entity.

 -created-after <time>

-ca

Only display activities created after the given time.

 -created-before <time>

-cb

Only display activities created before the given time.

 -updated-after <time>

-ua

Only display activities updated after the given time.

 -updated-before <time>

-ub

Only display activities updated before the given time.

Times for the above options may be entered in either absolute or relative format:

Type Format Description Examples

Absolute

 [[finding-activitiesyyyy/]MM/dd] [HH:mm[:ss]]

yyyy = the year
MM = the month (1-12)
dd = the date of the month (1-31)
HH = the hour (0-23)
mm = the minute (0-59)
ss = the second (0-59)

 2008/04/15
04/15
10:57
10:57:35
2008/04/15 10:57:35

Relative

 (<nn><d|h|m|s>)*

nn = a number
d = days
h = hours
m = minutes
s = seconds

 1d
1h
1m
30s
1m
7m30s
1d12h5m30s

Rhino assumes relative time format is in the past. For example, 1h30m means 1 hour and 30 minutes ago.

Examples

To display all activities in the SLEE:

$ ./rhino-console findactivities
pkey                      attach-count   handle                                 namespace   node   ra-entity    ref-count   replication-mode   submission-time     update-time
------------------------  -------------  -------------------------------------  ----------  -----  -----------  ----------  -----------------  ------------------  ------------------
1.101:219852641476607.1               0  ServiceActivity[ServiceID[name=Simple                101   Rhino Inte           0            SAVANNA   20180613 18:35:32   20180613 18:35:32
C.101:219852641476608.0               1    SAH[switchID=1528911304,connectionI                101       simple           3    KEY_VALUE_STORE   20180613 18:35:47   20180613 18:35:47
C.101:219852641476609.0               1    SAH[switchID=1528911304,connectionI                101       simple           3    KEY_VALUE_STORE   20180613 18:35:48   20180613 18:35:48
C.102:219852644015615.0               1    SAH[switchID=1528911304,connectionI                102       simple           3    KEY_VALUE_STORE   20180613 18:35:48   20180613 18:35:48
C.102:219852644015616.0               1    SAH[switchID=1528911304,connectionI                102       simple           3    KEY_VALUE_STORE   20180613 18:35:49   20180613 18:35:49
C.103:219852646067199.0               1    SAH[switchID=1528911304,connectionI                103       simple           3    KEY_VALUE_STORE   20180613 18:35:48   20180613 18:35:48
C.103:219852646067200.0               1    SAH[switchID=1528911304,connectionI                103       simple           3    KEY_VALUE_STORE   20180613 18:35:49   20180613 18:35:49
7 rows

Finding stale activities

A common search would be for stale activities. Rhino performs a periodic activity-liveness scan, checking all active activities and ending those detected as stale. Sometimes, however, a failure in the network or inside a resource adaptor might prevent the liveness scan from detecting and ending some activities. In this case, the Administrator would have to locate and end the stale activities manually.

To narrow the search:

To search for activities belonging to node 101 (replicated or non-replicated activities owned by 101) that are more than one hour old, you would use the arguments -node 101 and -cb 1h:

$ ./rhino-console findactivities -node 101 -cb 1h
pkey                      attach-count   handle                                 namespace   node   ra-entity    ref-count   replication-mode   submission-time     update-time
------------------------  -------------  -------------------------------------  ----------  -----  -----------  ----------  -----------------  ------------------  ------------------
C.101:219852641476608.0               1    SAH[switchID=1528911304,connectionI                101       simple           3    KEY_VALUE_STORE   20180613 18:35:47   20180613 18:35:47
C.101:219852641476609.0               1    SAH[switchID=1528911304,connectionI                101       simple           3    KEY_VALUE_STORE   20180613 18:35:48   20180613 18:35:48
2 rows

(This example returned two activities.)

MBean operation: `getActivities`

MBean

Rhino operations

Get summary information for all activities

public TabularData getActivities(int maxPerNode, boolean includeRemoved)
    throws ManagementException;

This operation returns tabular data summarising all activities.

Get summary information for activities belonging to a resource adaptor entity

public TabularData getActivities(int maxPerNode, String entityName, boolean includeRemoved)
    throws UnrecognizedResourceAdaptorEntityException, ManagementException;

This operation returns tabular data summarising the activities owned by the given resource adaptor entity.

Get summary information for activities using time-based criteria

public TabularData getActivities(int maxPerNode, String entityName, long createdAfter, long createdBefore, long updatedAfter, long updatedBefore, boolean includeRemoved)
    throws UnrecognizedResourceAdaptorEntityException, ManagementException;

This operation returns tabular data summarising the activities owned by the given resource adaptor entity using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular parameter).

Get summary information only for removed but not yet garbage collected activities using time-based criteria

public TabularData getRemovedActivities(int maxPerNode, String entityName, long createdAfter, long createdBefore, long updatedAfter, long updatedBefore)
    throws ManagementException, UnrecognizedResourceAdaptorEntityException;

This operation returns tabular data summarising the removed activities owned by the given resource adaptor entity using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular parameter).

Results depend on the Housekeeping MBean that invokes the operation:

Cluster Housekeeping MBean — returns results from all event-router nodes in the primary component
Node Housekeeping MBean — returns results from that node only.

For a description of the format of the tabular data that these operations return, see the javadoc.

Inspecting Activities

To get detailed information about an activity, use the following rhino-console command or related MBean operation.

Console command: `getactivityinfo`

Command

getactivityinfo [-non-resident] [-v] <activity pkey>*
  Description
    Get activity information [-v = verbose].  Use -non-resident to get activity
    information on activities not currently owned by any cluster node.

Example

To display activity information for activity C.101:219852641476611.0:

$ ./rhino-console getactivityinfo C.101:219852641476611.0
pkey             : C.101:219852641476611.0
activity         : SAH[switchID=1528911304,connectionID=7,address=1]
creating-gen     : 25
ending           : false
events-submitted : 2
flags            : 0x0
handle           : SAH[switchID=1528911304,connectionID=7,address=1]
head-event       :
last-event-time  : 20180614 14:40:55
namespace        :
node             : 101
ra-entity        : simple
replication-mode : KEY_VALUE_STORE
submission-time  : 20180614 14:40:25
submitting-node  : 101
update-time      : 20180614 14:40:55
event-queue      : no rows
generations      :
      [27] refcount      : 0
      [27] removed       : false
      [27] attached-sbbs : no rows
      [27] timers        : no rows

This command returns a snapshot of the activity’s state at the time you execute it. Some values (such as fields pkey, activity, creating-gen, flags, handle, namespace, ra-entity, replication-mode, submission-time and submitting-node) are fixed for the lifetime of the activity. Others change as events on the activity are processed.

See Activity Information Fields for a description of the fields getactivityinfo returns.

MBean operation: `getActivityInfo`

MBean

Rhino operation

public CompositeData getActivityInfo(String activityPKey, boolean showAllGenerations, boolean nonResident)
    throws InvalidPKeyException, UnknownActivityException, ManagementException;

This operation returns tabular data with detailed information on the given activity.

For a description of the format of the tabular data that this operation returns, see the javadoc.

Activity Information Fields

The getactivityinfo console command displays information about:

an activity
events in the activity’s event queue
how the activity was generated
SBB entities attached to the activity
active timers on the activity.

Activity information

The getactivityinfo console command displays the following values about an activity:

Field Description

 pkey

The activity’s primary key. Uniquely identifies this activity within Rhino.

 activity

The activity object, in string form. Its exact content is resource adaptor dependent (and may or may not contain useful human-readable information).

 creating-gen

The database generation in which the activity was created.

 ending

Boolean flag indicating if the activity is ending.

 events-submitted

The number of events that have been submitted for processing on the activity.

 flags

Hexadecimal value of the flags the activity was created with (if any).

 handle

The activity handle assigned by the activity’s resource adaptor entity, in string form. The exact content is resource adaptor dependent (and may or may not contain useful human-readable information).

 head-event

The event at the head of the activity’s event queue (the next event to be processed on the activity).

 last-event-time

When the most recent event was submitted on the activity.

 namespace

The namespace that the activity resides in.

 node

The Rhino cluster node that currently owns the activity. If this value is different to the submission-node field, the activity must be a replicated activity that was reassigned to this node.

If this value is negative, then the activity is currently non-resident. This means that the state for the activity was replicated to an external key/value store, the node that the activity was previously assigned to has failed, and the activity has not yet been adopted by any remaining cluster node. The absolute value of the node ID represents the node that the activity was last assigned to.

 ra-entity

The resource adaptor entity that created this activity.

 replication-mode

The method of activity replication. This field will have one of the following values:

NONE — the activity is not replicated
SAVANNA — the activity is replicated using the traditional savanna framework
KEY_VALUE_STORE — the activity is replicated using an external key/value store.

 submission-time

When the activity was created.

 submission-node

The Rhino cluster node that created the activity.

 update-time

When the activity was last updated (when the most recent database generation record was created). Useful in some situations for evaluating whether an activity is still live.

event-queue

A list of events queued for processing on the activity.

generations

A list of generational information stored in the database for the activity. If getactivityinfo includes the -v option, all generations display (otherwise just the most recent displays).

Event-queue information

The getactivityinfo console command displays the following values for each event in an activity’s event queue:

Field

Description

 position

The position of the event in the queue.

 event-type

The event-type component identifier of the event.

 event

The event object, in string form. Its exact content is resource adaptor dependent (and may or may not contain useful human-readable information).

 flags

Hexadecimal value of the flags the event was fired with (if any).

Generational information

The getactivityinfo console command displays values for the following fields, in an activity’s generational information:

Field Description

 generation

Not displayed as a field but included in square brackets before the rest of the generational information, for example: [76343].

 refcount

The number of references made to the activity by the Timer Facility and the Activity Context Naming Facility.

 removed

Boolean flag indicating if the activity no longer exists in the SLEE. Only true if the activity has ended but has not yet been garbage collected.

attached-sbbs

A list of SBBs attached to the activity.

timers

A list of Timer Facility timers set on the activity.

Attached-SBB information

The getactivityinfo console command displays values for the following fields, for each SBB entity attached to an activity:

Field

Description

 pkey

The primary key of the SBB entity.

 namespace

The namespace the attached SBB entity resides in. This will always be equal to the namespace in which the activity resides.

 sbb-component-id

The component identifier of the SBB for the SBB entity.

 service-component-id

The component identifier of the service the SBB belongs to.

Activity-timer information

The getactivityinfo console command displays values for the following fields, for each timer active on an activity:

Field Description

 pkey

The primary key of the timer.

 namespace

The namespace that the timer exists in. This will always be equal to the namespace in which the activity resides.

 activity-pkey

The primary key of the activity the timer is set on.

 submission-time

The time the timer was initially set.

 period

The timer period (for periodic timers).

 repetitions

The number of repetitions the timer will fire before it expires.

 preserve-missed

Boolean flag indicating if missed timers should still fire an event into the SLEE.

 replicated

Boolean flag indicating whether or not the timer is replicated. This flag will only be set to true if the activity the timer was set on has a replication mode of SAVANNA.

Removing Activities

To forcefully remove an activity, use the following rhino-console command or related MBean operation.

Consult the spec before ending an activity

The JAIN SLEE 1.1 specification provides detailed rules for ending no-longer-required activities.

Console command: `removeactivity`

Command

removeactivity [-non-resident] <activity pkey>*
  Description
    Remove activities.  Use -non-resident to remove activities not currently owned
    by any cluster node.

Example

To remove the activities with the primary keys C.101:219852641476611.0 and C.101:219852641476612.0

$ ./rhino-console removeactivity C.101:219852641476611.0 C.101:219852641476612.0
2 activities removed

MBean operation: `removeActivity`

MBean

Rhino operation

public void removeActivity(String activityPKey, boolean nonResident)
    throws InvalidPKeyException, UnknownActivityException,
            ManagementException;

This operation removes the activity with the given primary key. The nonResident argument must be true in order to remove a non-resident activity.

Removing All Activities

To mark all activities of a resource adaptor entity for removal, use the following rhino-console command or related MBean operation.

Use extreme care when removing forcibly

Occasionally an administrator will want to remove all activities belonging to a resource adaptor entity. Typically, this would be to deactivate a resource adaptor when upgrading or reconfiguring. Under normal conditions, these actions would be performed automatically, by allowing existing activities to drain over time. Rhino provides the following housekeeping commands to forcibly speed up the draining process, although these should be used with extreme care on production systems — they will interrupt service for any existing network activities belonging to the resource adaptor entity.

Console command: `removeallactivities`

Command

removeallactivities <ra-entity> [-nodes node1,node2,...]
  Description
    Remove all activities belonging to a resource adaptor entity in the Stopping
    state (on the specified nodes)

Example

To remove all activities owned by the resource adaptor entity called sipra on nodes 101 and 102:

$ ./rhino-console removeallactivities sipra -nodes 101,102
Activities marked for removal on node(s) [101,102]

MBean operation: `markAllActivitiesForRemoval`

MBean

RhinoHousekeepingMBean

Rhino operation

public void markAllActivitiesForRemoval(String entityName, int[] nodeIDs)
    throws NullPointerException, UnrecognizedResourceAdaptorEntityException,
          InvalidStateException, ManagementException;

This operation marks all the activities owned by the given resource adaptor entity on the given nodes for removal.

Resource adaptor entity (or SLEE) must be STOPPING

As a safeguard, this command (or MBean operation) cannot be run unless the specified resource adaptor entity, or the SLEE, is in the STOPPING state on the specified nodes. (It may also be run against nodes where the resource adaptor entity is in the INACTIVE state (or the SLEE is in the STOPPED state) for convenience in asymmetric cluster configurations, but has no effect against such nodes since no activities exist for the resource adaptor entity on nodes where it is INACTIVE (or the SLEE is STOPPED).)

Why "mark" (instead of just ending)?

This command does not remove all activities immediately, because that might overload the system (from processing too many activity-end events at once). Instead, removeallactivities marks targeted activities as ENDING. Marked activities end over the course of the next pass of the activity query-liveness scan (typically, with the default settings, each scan lasts three-to-five minutes).

Timers

Rhino includes procedures for:

Finding Timers
Finding Remote Timers
Inspecting Timers
Inspecting Remote Timers
Cancelling Timers
Cancelling Remote Timers

Finding Timers

To find timers in the SLEE, use the following rhino-console command or related MBean operations.

Console command: `findtimers`

Command

findtimers [-maxpernode maxrows] [-node nodeid] [-created-after
date|time|offset] [-created-before date|time|offset]
  Description
    Find timers

Options

Option

Abbreviation

Description

 -maxpernode <maxrows>

Retrieve at most this many timers from each event-router node (default is 100). Can be used to limit Rhino’s load when processing the request.

 -node <node-id>

Only display timers owned by the given node.

 -ra <ra-entity>

Only display timers set on activities belonging to the named resource adaptor entity.

 -created-after <time>

-ca

Only display timers created after the given time.

 -created-before <time>

-cb

Only display timers created before the given time.

Times for the above options may be entered in either absolute or relative format:

Type Format Description Examples

Absolute

 [[finding-timersyyyy/]MM/dd] [HH:mm[:ss]]

yyyy = the year
MM = the month (1-12)
dd = the date of the month (1-31)
HH = the hour (0-23)
mm = the minute (0-59)
ss = the second (0-59)

 2008/04/15
04/15
10:57
10:57:35
2008/04/15 10:57:35

Relative

 (<nn><d|h|m|s>)*

nn = a number
d = days
h = hours
m = minutes
s = seconds

 1d
1h
1m
30s
1m
7m30s
1d12h5m30s

Rhino assumes relative time format is in the past. For example, 1h30m means 1 hour and 30 minutes ago.

Examples

To display all timers in the SLEE:

$ ./rhino-console findtimers
pkey                          activity-pkey             namespace   period                preserve-missed   remote-timer-pkey         repetitions   replicated   submission-time
----------------------------  ------------------------  ----------  --------------------  ----------------  ------------------------  ------------  -----------  ------------------
3.103:244326037547631.0.420    3.103:244326037547631.0               9223372036854775807              Last   102/102:244326037604633             1        false   20191218 23:29:40
3.103:244326037547664.0.43c    3.103:244326037547664.0                            290000              Last   102/102:244326037604669           307        false   20191218 23:29:40
3.103:244326037547664.0.43d    3.103:244326037547664.0               9223372036854775807              Last   102/102:244326037604668             1        false   20191218 23:29:40
3.103:244326037547664.0.43e    3.103:244326037547664.0               9223372036854775807              Last   102/102:244326037604667             1        false   20191218 23:29:40
3.103:244326037547668.0.443    3.103:244326037547668.0                            290000              Last   101/101:244326026950453           307        false   20191218 23:29:40
3.103:244326037547668.0.444    3.103:244326037547668.0               9223372036854775807              Last   101/101:244326026950454             1        false   20191218 23:29:40
3.103:244326037547668.0.445    3.103:244326037547668.0               9223372036854775807              Last   102/102:244326037604673             1        false   20191218 23:29:40
7 rows

MBean operation: `getTimers`

MBean

Rhino operations

Get summary information for all timers

public TabularData getTimers(int maxPerNode)
    throws ManagementException;

This operation returns tabular data summarising all timers.

Get summary information for timers set on activities belonging to a resource adaptor entity using time-based criteria

public TabularData getTimers(int maxPerNode, String raEntity, long createdAfter, long createdBefore)
    throws ManagementException;

This operation returns tabular data summarising the timers set on activities belonging to the given resource adaptor entity using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular time-based parameter).

Results depend on the Housekeeping MBean that invokes the operation:

Cluster Housekeeping MBean — returns results from all event-router nodes in the primary component
Node Housekeeping MBean — returns results from that node only.

For a description of the format of the tabular data that these operations return, see the javadoc.

Finding Remote Timers

To find remotely-armed timers in the SLEE, use the following rhino-console command or related MBean operations.

Console command: `findremotetimers`

Command

findremotetimers [-maxpernode maxrows] [-node nodeid] [-created-after
date|time|offset] [-created-before date|time|offset]
  Description
    Find remote timers

Options

Option

Abbreviation

Description

 -maxpernode <maxrows>

Retrieve at most this many remote timers from each event-router node (default is 100). Can be used to limit Rhino’s load when processing the request.

 -node <node-id>

Only display remote timers armed on the given node.

 -ra <ra-entity>

Only display remote timers set on activities belonging to the named resource adaptor entity.

 -created-after <time>

-ca

Only display remote timers created after the given time.

 -created-before <time>

-cb

Only display remote timers created before the given time.

Times for the above options may be entered in either absolute or relative format:

Type Format Description Examples

Absolute

 [[finding-remote-timersyyyy/]MM/dd] [HH:mm[:ss]]

yyyy = the year
MM = the month (1-12)
dd = the date of the month (1-31)
HH = the hour (0-23)
mm = the minute (0-59)
ss = the second (0-59)

 2008/04/15
04/15
10:57
10:57:35
2008/04/15 10:57:35

Relative

 (<nn><d|h|m|s>)*

nn = a number
d = days
h = hours
m = minutes
s = seconds

 1d
1h
1m
30s
1m
7m30s
1d12h5m30s

Rhino assumes relative time format is in the past. For example, 1h30m means 1 hour and 30 minutes ago.

Examples

To display all remote timers in the SLEE:

$ ./rhino-console findremotetimers
pkey                      activity-pkey             callback-node   expiry-time         interval   namespace   period   submission-time
------------------------  ------------------------  --------------  ------------------  ---------  ----------  -------  ------------------
101/101:244326026949631    F.103:244326037546522.0             103   20191218 23:30:23      65000                    0   20191218 23:29:18
101/101:244326026949632    F.103:244326037546502.0             103   20191218 23:39:23     605000                    0   20191218 23:29:18
101/101:244326026949633    F.103:244326037546538.0             103   20191218 23:39:23     605000                    0   20191218 23:29:18
101/101:244326026949634    F.103:244326037546538.0             103   20191218 23:30:23      65000                    0   20191218 23:29:18
101/101:244326026949635    F.103:244326037546517.0             103   20191218 23:30:23      65000                    0   20191218 23:29:18
101/101:244326026949636    F.103:244326037546503.0             103   20191218 23:30:23      65000                    0   20191218 23:29:18
101/101:244326026949637    F.103:244326037546506.0             103   20191218 23:30:23      65000                    0   20191218 23:29:18
7 rows

MBean operation: `getRemoteTimers`

MBean

Rhino operations

Get summary information for all remote timers

public TabularData getRemoteTimers(int maxPerNode)
    throws ManagementException;

This operation returns tabular data summarising all timers.

Get summary information for remote timers set on activities belonging to a resource adaptor entity using time-based criteria

public TabularData getRemoteTimers(int maxPerNode, String raEntity, long createdAfter, long createdBefore)
    throws ManagementException;

This operation returns tabular data summarising the remote timers set on activities belonging to the given resource adaptor entity using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular time-based parameter).

Results depend on the Housekeeping MBean that invokes the operation:

Cluster Housekeeping MBean — returns results from all event-router nodes in the primary component
Node Housekeeping MBean — returns results from that node only.

For a description of the format of the tabular data that these operations return, see the javadoc.

Inspecting Timers

To get detailed information about a timer, use the following rhino-console command or related MBean operation.

Console command: `gettimerinfo`

Command

gettimerinfo <timer id>*
  Description
    Get timer info

Example

To display information for timer 10.103:244325031551168.0.227d7:

$ ./rhino-console gettimerinfo 10.103:244325031551168.0.227d7
activity-pkey     : 10.103:244325031551168.0
pkey              : 10.103:244325031551168.0.227d7
remote-timer-pkey :
namespace         :
next-firing-time  : 20191218 23:01:32
node              : 103
period            : 9223372036854775807
preserve-missed   : Last
ra-entity         : sip-sis-ra
remaining         : 1
repetitions       : 1
replicated        : false
submission-time   : 20191218 23:00:27

This command returns a snapshot of the timer’s state at the time you execute it. Some values (such as fields pkey, activity-pkey, remote-timer-pkey, period, namespace, ra-entity, replicated, submission-time, and preserve-missed) are fixed for the lifetime of the timer. Others values change as the timer fires.

See Timer Information Fields for a description of the fields gettimerinfo returns.

MBean operation: `getTimerInfo`

MBean

Rhino operation

public CompositeData getTimerInfo(String timerID)
    throws InvalidPKeyException, UnknownTimerException, ManagementException;

This operation returns composite data with detailed information on the given timer.

For a description of the format of the composite data that this operation returns, see the javadoc.

Timer Information Fields

The gettimerinfo console command displays the following values about a timer:

Field

Description

 pkey

The primary key of the timer. This key uniquely identifies the timer within the SLEE.

 namespace

The namespace that the timer exists in.

 activity-pkey

The primary key of the SLEE activity this timer is attached to.

 ra-entity

The name of the resource adaptor entity which created the activity on which this timer is set.

 submission-time

The date the timer was created.

 next-firing-time

The date the timer will next fire.

 period

The timers period.

 repetitions

Number of repetitions of the timer.

 remaining

Number of remaining repetitions of the timer.

 preserve-missed

Behaviour when a timer is missed. One of:

NONE - do not preserve missed,
ALL - preserve all missed events, or
LAST - Preserve last missed event.

 node

The node currently responsible for scheduling the timer.

 replicated

Replicated flag, true for replicated timers entities, false for non-replicated timers. A timer is replicated if the activity it is attached to is replicated.

 remote-timer-pkey

The primary key of the remote timer created for this timer, if any.

Inspecting Remote Timers

To get detailed information about a remote timer, use the following rhino-console command or related MBean operation.

Console command: `getremotetimerinfo`

Command

getremotetimerinfo <remote timer id>*
  Description
    Get remote timer info

Example

To display remote timer information for 103/103:244326037626200:

$ ./rhino-console getremotetimerinfo 103/103:244326037626200
activity-pkey    : 3.101:244326026885993.0
pkey             : 103/103:244326037626200
callback-node    : 101
expiry-time      : 20191218 23:35:12
interval         : 300000
namespace        :
next-firing-time : 20191218 23:35:12
period           : 0
pops             : 0
ra-entity        : Rhino Null Activities
submission-time  : 20191218 23:30:12

This command returns a snapshot of the remote timer’s state at the time you execute it. Some values (such as fields pkey, namespace, activity-pkey, ra-entity, submission-time, interval, period, and expiry-time) are fixed for the lifetime of the remote timer. Other values change as the timer fires or cluster membership changes.

See Remote Timer Information Fields for a description of the fields getremotetimerinfo returns.

MBean operation: `getRemoteTimerInfo`

MBean

Rhino operation

public CompositeData getRemoteTimerInfo(String remoteTimerID)
    throws InvalidPKeyException, UnknownTimerException, ManagementException;

This operation returns composite data with detailed information on the given remote timer.

For a description of the format of the composite data that this operation returns, see the javadoc.

Remote Timer Information Fields

The getremotetimerinfo console command displays the following values about a remote timer:

Field

Description

 pkey

The primary key of the remote timer. This key uniquely identifies the remote timer within the SLEE.

 namespace

The namespace that the timer exists in.

 activity-pkey

The primary key of the SLEE activity this timer is attached to.

 ra-entity

The name of the resource adaptor entity which created the activity on which this timer is set.

 submission-time

The date the timer was created.

 next-firing-time

The date the timer will next fire.

 interval

The initial time period after the timer was created before the first firing.

 period

The timers period, if a repeating timer.

 expiry-time

The date that the timer will expire.

 callback-node

The node that will receive a callback with the timer event when the timer next fires. May be null if the assigned callback address for the timer is not currently reachable.

 pops

The number of times that the timer has already fired.

Cancelling Timers

To administratively cancel a timer, use the following rhino-console command or related MBean operation.

Console command: `canceltimer`

Command

canceltimer <timer id>
  Description
    Cancel timer

Example

To cancel the timer with the primary key 10.101:244325023613578.0.7661

$ ./rhino-console canceltimer 10.101:244325023613578.0.7661
Timer removed

MBean operation: `cancelTimer`

MBean

Rhino operation

public void cancelTimer(String timerID)
    throws InvalidPKeyException, UnknownTimerException,
           ManagementException;

This operation cancels the timer with the given primary key.

Cancelling Remote Timers

To administratively cancel a remotely-armed timer, use the following rhino-console command or related MBean operation.

Console command: `cancelremotetimer`

Command

cancelremotetimer <remote timer id>
  Description
    Cancel remote timer

Example

To cancel the remote timer with the primary key 101/101:244326026949631

$ ./rhino-console cancelremotetimer 101/101:244326026949631
Remote timer removed

MBean operation: `cancelRemoteTimer`

MBean

Rhino operation

public void cancelRemoteTimer(String remoteTimerID)
    throws InvalidPKeyException, UnknownTimerException,
           ManagementException;

This operation cancels the remote timer with the given primary key.

SBB Entities

Rhino includes procedures for:

Finding SBB Entities
Inspecting SBB Entities
Diagnosing SBB Entities
Removing SBB Entities
Removing All SBB Entities

Finding SBB Entities

To find SBB entities in the SLEE, use the following rhino-console command or related MBean operation.

Console command: findsbbs

Command

findsbbs [-maxpernode maxrows] [-node nodeid] <-service service> [-sbb sbb]
[-created-after date|time|offset] [-created-before date|time|offset]
  Description
    Find SBBs.

Options

Option

Abbreviation

Description

 -maxpernode <maxrows>

Retrieve at most this many SBB entities from each event-router node (default is 100). Can be used to limit Rhino’s load when processing the request.

 -node <node-id>

Only display activities owned by the given node.

 -service <service>

Look for SBB entities owned by the named service (must be specified).

 -sbb <sbb>

Only display SBB entities of the named SBB.

 -created-after <time>

-ca

Only display SBB entities created after the given time.

 -created-before <time>

-cb

Only display SBB entities created before the given time.

Times for the above options may be entered in either absolute or relative format:

Type Format Description Examples

Absolute

 [[finding-sbb-entitiesyyyy/]MM/dd] [HH:mm[:ss]]

yyyy = the year
MM = the month (1-12)
dd = the date of the month (1-31)
HH = the hour (0-23)
mm = the minute (0-59)
ss = the second (0-59)

 2008/04/15
04/15
10:57
10:57:35
2008/04/15 10:57:35

Relative

 (<nn><d|h|m|s>)*

nn = a number
d = days
h = hours
m = minutes
s = seconds

 1d
1h
1m
30s
1h30m
7m30s
1d12h5m30s

Rhino assumes relative time format is in the past. For example, 1h30m means 1 hour and 30 minutes ago.

Examples

To display all SBB entities owned by the SimpleService service in the SLEE:

$ ./rhino-console findsbbs -service name=SimpleService,vendor=OpenCloud,version=1.1
pkey                             creation-time       namespace   node  parent-pkey   replicated   sbb-component-id                                   service-component-id
-------------------------------  ------------------  ----------  ----- ------------  -----------  -------------------------------------------------  -------------------------------------------------
101:219902028358655/965121714     20180614 21:23:16                101                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
101:219902028358656/996141521     20180614 21:23:18                101                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
101:219902028358657/1027161328    20180614 21:23:19                101                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
101:219902028358658/1058181135    20180614 21:23:21                101                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482559/499417899     20180614 21:23:16                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482560/561457513     20180614 21:23:17                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482561/623497127     20180614 21:23:18                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482562/685536741     20180614 21:23:21                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
103:219902030044671/-392412623    20180614 21:23:17                103                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
103:219902030044672/-361392816    20180614 21:23:18                103                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
103:219902030044673/-330373009    20180614 21:23:19                103                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
11 rows

To narrow the search:

To search for SBB entities belonging to node 102 (replicated or non-replicated SBB entities owned by 102) that are more than one hour old, you would use the parameters -node 102 and -cb 1h:

$ ./rhino-console findsbbs -service name=SimpleService,vendor=OpenCloud,version=1.1 -node 102 -cb 1h
pkey                             creation-time       namespace   node  parent-pkey   replicated   sbb-component-id                                   service-component-id
-------------------------------  ------------------  ----------  ----- ------------  -----------  -------------------------------------------------  -------------------------------------------------
102:219902028482559/499417899     20180614 21:23:16                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482560/561457513     20180614 21:23:17                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482561/623497127     20180614 21:23:18                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
102:219902028482562/685536741     20180614 21:23:21                102                     false  SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1  ServiceID[name=SimpleService,vendor=OpenCloud,ver
4 rows

This example returned four SBB entities.

MBean operations: `getSbbs`

MBean

Rhino operations

Get summary information for all SBB entities owned by a service

public TabularData getSbbs(int maxPerNode, ServiceID serviceID)
    throws UnrecognizedServiceException, ManagementException;

This operation returns tabular data summarising all SBB entities in the given service.

Get summary information for all SBB entities owned by a service using time-based criteria

public TabularData getSbbs(int maxPerNode, ServiceID serviceID, long createdAfter, long createdBefore)
    throws UnrecognizedServiceException, ManagementException;

This operation returns tabular data summarising the SBB entities owned by the given service using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular parameter).

Get summary SBB entity information for a particular SBB in a service using time-based criteria

public TabularData getSbbs(int maxPerNode, ServiceID serviceID, SbbID sbbType, long createdAfter, long createdBefore)
    throws UnrecognizedServiceException, UnrecognizedSbbException, ManagementException

This operation returns tabular data summarising only SBB entities of the given SBB in the given service using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular parameter).

Results depend on the Housekeeping MBean that invokes the operation:

Cluster Housekeeping MBean — returns results from all event-router nodes in the primary component
Node Housekeeping MBean — returns results from that node only.

For a description of the format of the tabular data that these operations return, see the javadoc.

Inspecting SBB Entities

To get detailed information about an SBB entity, use the following rhino-console command or related MBean operation.

Console command: `getsbbinfo`

Command

getsbbinfo <serviceid> <sbbid> [-non-resident] <sbb pkeys>*
  Description
    Get SBB information.  Use -non-resident to get information on SBB entities not
    currently owned by any cluster node.

Example

To display information for SBB entity 102:219902028482559/499417899 of the SimpleSbb SBB in the SimpleService service:

$ ./rhino-console getsbbinfo name=SimpleService,vendor=OpenCloud,version=1.1 name=SimpleSbb,vendor=OpenCloud,version=1.1 102:219902028482559/499417899
parent-pkey          :
pkey                 : 102:219902028482559/499417899
convergence-name     : APK[ns=,ah=10,id=102:219902022645761,replication=NONE]:::::-1
creating-node-id     : 102
creation-time        : 20180614 21:23:16
namespace            :
owning-node-id       : 102
priority             : 10
replicated           : false
sbb-component-id     : SbbID[name=SimpleSbb,vendor=OpenCloud,version=1.1]
service-component-id : ServiceID[name=SimpleService,vendor=OpenCloud,version=1.1]
attached-activities  :
       > pkey                      handle                                              namespace   ra-entity   replicated
       > ------------------------  --------------------------------------------------  ----------  ----------  -----------
       > A.102:219902022645761.0    SAH[switchID=1528911304,connectionID=10,address=]                  simple        false
       > 1 rows

This command returns a snapshot of the SBB entity’s state at the time you execute it. Some values (such as pkey, parent-pkey, convergence-name, creating-node-id, creation-time, namespace, replicated, sbb-component-id and service-component-id) are fixed for the lifetime of the SBB entity. Others change as the SBB entity processes events.

The default mode of this command only retrieves information for SBB entities currently owned by at least one active cluster node. If an SBB entity is non-resident, then the -non-resident option must be used to obtain information for that SBB entity’s state.

See SBB Entity Information Fields for a description of the fields getsbbinfo returns.

MBean operation: `getSbbInfo`

MBean

Rhino operation

public CompositeData getSbbInfo(ServiceID serviceID, SbbID sbbID, String sbbPKey, boolean nonResident)
    throws UnrecognizedServiceException, UnrecognizedSbbException, InvalidPKeyException,
           UnknownSbbEntityException, ManagementException;

This operation returns tabular data with detailed information on the given SBB entity.

For a description of the format of the tabular data that this operation returns, see the javadoc.

SBB Entity Information Fields

The getsbbinfo console command displays information about an SBB entity and each activity it is attached to.

SBB entity information

The getsbbinfo console command displays the following values about an SBB entity:

Field Description

 pkey

The primary key of the SBB entity. Unique to the SBB within the service (SBB entities of other SBBs in the same or another service may have the same primary key).

 parent-pkey

The primary key of the SBB entity’s parent SBB entity. (For a root SBB entity, with no parent, this field is empty.)

 convergence-name

The convergence name, for a root SBB entity. (If not a root SBB entity, this field is empty.)

 creating-node-id

The Rhino node that created the SBB entity.

 creation-time

The date and time the SBB entity was created.

 namespace

The namespace that the SBB entity resides in.

 owning-node-id

The Rhino node that is currently responsible for the SBB entity.

If this value is negative, then the SBB entity is currently non-resident. This means that the state for the SBB entity tree was replicated to an external key/value store, the node that the SBB entity tree was previously assigned to has failed, and the SBB entity tree has not yet been adopted by any remaining cluster node. The absolute value of the node ID represents the node that the SBB entity was last assigned to.

 priority

The SBB entity’s current event-delivery priority.

 replicated

Boolean flag indicating if the SBB entity’s state is replicated.

 sbb-component-id

The SBB-component identifier, in string form. Identifies the SBB component that the SBB entity was created from.

 service-component-id

The service-component identifier, in string form. Identifies the service that the SBB entity is providing a function for.

attached-activities

A list of the activities the SBB entity is attached to.

Attached-activity information

The getsbbinfo console command displays the following values about each activity the SBB entity is attached to:

 Field

Description

 pkey

The primary key of the activity. Uniquely identifies this activity within Rhino.

 handle

The activity handle assigned by the activity’s resource adaptor entity, in string form. Its exact content is resource adaptor dependent (and may or may not contain useful human-readable information).

 namespace

The namespace the activtiy resides in. This will always be equal to the namespace in which the SBB entity resides.

 ra-entity

The resource adaptor entity that created this activity.

 replicated

Boolean flag indicating if the activity is a replicated activity. This flag will only be set to true if the activity has a replication mode of SAVANNA.

Diagnosing SBB Entities

SBB diagnostics lets you pull service-defined diagnostic information from SBB objects at runtime.

You can use the getsbbdiagnostics console command (or associated MBean operation) to access diagnostic information (in string form) from particular methods implemented in the SBB abstract class.

For an SBB to provide this diagnostics information, it must implement one of these methods:

public String ocGetSbbDiagnostics() — When queried with the getsbbdiagnostics console command, the return result from this method will be included in the SBB diagnostics output.
public void ocGetSbbDiagnostics(StringBuilder sb) — When queried with the getsbbdiagnostics console command, this method will be invoked on the SBB object with a SLEE-provided StringBuilder. The toString() output of the StringBuilder will be included in the SBB diagnostics output. This method is intended for chaining diagnostics output between extending classes and child SBBs.

Both of these methods are invoked with a valid transaction context on an SBB entity in the ready state. The methods may read any SBB CMP fields as necessary to produce diagnostic information.

See the example below.

A deployment error will occur if more than one of these methods is implemented.
The implementation of an ocGetSbbDiagnostics method should not modify SBB entity state.

To get detailed diagnostics information about an SBB entity, use the following rhino-console command or related MBean operation.

Console command: `getsbbdiagnostics`

Command

getsbbdiagnostics <serviceid> <sbbid> [-non-resident] <sbb pkeys>*
  Description
    Get SBB info and diagnostics (if supported by sbb implementation).  Use
    -non-resident to get information on SBB entities not currently owned by any
    cluster node.

Example

To display information for SBB entity 103:219902030054321/213428623 of the Sentinel SIP service:

$ ./rhino-console getsbbdiagnostics name=sentinel.sip,vendor=OpenCloud,version=1.0 name=sentinel.sip,vendor=OpenCloud,version=1.0 103:219902030054321/213428623
parent-pkey          :
pkey                 : 103:219902030054321/213428623
convergence-name     : APK[ns=,ah=10,id=103:219902023292928,replication=NONE]:::::-1
creating-node-id     : 103
creation-time        : 20180614 21:10:11
namespace            :
owning-node-id       : 103
priority             : 10
replicated           : false
sbb-component-id     : SbbID[name=sentinel.sip,vendor=OpenCloud,version=1.0]
service-component-id : ServiceID[name=sentinel.sip,vendor=OpenCloud,version=1.0]
attached-activities  :
    > pkey                       handle         ra-entity    replicated
    > -------------------------  -------------  -----------  -----------
    >  A.101:219902021728256.0    SessionAH[3]   sip-sis-ra        false
    > 1 rows


SBB diagnostics:
SentinelSipFeatureAndOcsSbbSupportImpl Child SBBs
=================================================

SubscriberDataLookup SBB:
No diagnostics available for this feature sbb.

SipAdhocConference SBB:
No child SBB currently exists for SipAdhocConference.

DiameterRoOcs SBB:
DiameterRoOcsMultiFsmSbb Service FSM States
===========================================

DiameterIECFSM [current state = NotCharging, InputRegister[scheduled=[], execution=[]], Endpoints[Endpoint[local],Endpoint[DiameterMediation],Endpoint[DiameterToOcs]]]

DiameterSCURFSM [previous state = WaitForInitialCreditCheckAnswer, current state = WaitForNextCreditControlRequest, InputRegister[scheduled=[local_errorsEndSession], execution=[]], Endpoints[Endpoint[local],Endpoint[DiameterMediation],Endpoint[DiameterToOcs,aci=[set,sbb-attached]]]]

SentinelSipSessionStateAggregate Session State
==============================================

Account: 6325
ActivityTestHasFailed: false
AllowPresentationOfDivertedToUriToOriginatingUser: false
AllowPresentationOfServedUserUriToDivertedToUser: false
AllowPresentationOfServedUserUriToOriginatingUser: false
AnnouncementCallIds: null
AnnouncementID: 0
AnytimeFreeDataPromotionActive: false
CFNRTimerDuration: 0
CallHasBeenDiverted: false
CallType: MobileOriginating
CalledPartyAddress: tel:34600000001
CalledPartyCallId: null
CallingPartyAddress: tel:34600000002
CallingPartyCallId: null
ChargingResult: 2001
ClientChargingType: sessionCharging
ClientEventChargingMethod: null
ClosedUserGroupCall: null
ClosedUserGroupDropIfNoGroupMatch: null
ClosedUserGroupEnabled: true
ClosedUserGroupIncomingAccessAllowed: null
ClosedUserGroupList: [CUG1Profile]
ClosedUserGroupOutgoingAccessAllowed: null

...
etc.

This command returns a snapshot of the SBB entity’s state and SBB-defined diagnostics information at the time you execute it. Some values (such as pkey, parent-pkey, convergence-name, creating-node-id, creation-time, namespace, replicated, sbb-component-id, and service-component-id) are fixed for the lifetime of the SBB entity. Others change as the SBB entity processes events.

The diagnostics output (from the "SBB diagnostics:" line onwards) is free-form and SBB defined. The above output is only representative of a single service-defined diagnostics method.

The default mode of this command only retrieves information for SBB entities currently owned by at least one active cluster node. If an SBB entity is non-resident, then the -non-resident option must be used to obtain information for that SBB entity’s state.

See SBB Entity Information Fields for a description of the fields getsbbdiagnostics returns.

MBean operation: `getSbbDiagnostics`

MBean

Rhino operation

public CompositeData getSbbDiagnostics(ServiceID serviceID, SbbID sbbID, String sbbPKey, boolean nonResident)
    throws UnrecognizedServiceException, UnrecognizedSbbException, InvalidPKeyException,
           UnknownSbbEntityException, ManagementException;

This operation returns tabular data with detailed information on the given SBB entity, including SBB-defined diagnostics information.

For a description of the format of the tabular data that this operation returns, see the javadoc.

Example usage

The following is a basic example showing an auto-generated ocGetSbbDiagnostics(StringBuilder sb) method. In this case, the method was generated based on CMP fields declared by the SBB, and demonstrates diagnostics information being obtained from both the current class and the super class:

public abstract class ExampleSessionStateImpl extends com.opencloud.sentinel.ant.BaseSbb implements ExampleSessionState {
    public void ocGetSbbDiagnostics(StringBuilder sb) {
        String header = "ExampleSessionState Session State";
        sb.append(header).append("\n");
        for (int i=0; i<header.length(); i++) sb.append("=");
        sb.append("\n\n");
        // diagnostics: ClashingType (from com.opencloud.sentinel.ant.ExampleSessionStateInterface)
        if (getClashingType() == null) {
            sb.append("ClashingType: null\n");
        }
        else {
          sb.append("ClashingType: ").append(getClashingType()).append("\n");
        }
        // diagnostics: ExampleField (from com.opencloud.sentinel.ant.ExampleSessionStateInterface)
        if (getExampleField() == null) {
            sb.append("ExampleField: null\n");
        }
        else {
            sb.append("ExampleField: ").append(getExampleField()).append("\n");
        }
        sb.append("\n");
        super.ocGetSbbDiagnostics(sb);
    }
    ...

Removing SBB Entities

To forcibly remove an SBB entity, use the following rhino-console) command or related MBean operation.

Only forcibly remove SBB entities if necessary

SBB entities should only be forcibly removed if they do not remove themselves due to some unforeseen error during event processing.

Console command: `removesbb`

Command

removesbb <serviceid> <sbbid> [-non-resident] <sbb pkey>*
  Description
    Remove SBBs.  Use -non-resident to remove SBB entities not currently owned by
    any cluster node.

Example

To remove the SBB entity of the SimpleSbb SBB in the SimpleService service with the primary key 101:219902028358655/965121714:

$ ./rhino-console removesbb name=SimpleService,vendor=OpenCloud,version=1.1 \
    name=SimpleSbb,vendor=OpenCloud,version=1.1 101:219902028358655/965121714
SBB entity 101:219902028358655/965121714 removed

The default mode of this command only removes SBB entities currently owned by at least one active cluster node. If an SBB entity is non-resident, then the -non-resident option must be used to remove that SBB entity.

MBean operation: `removeSbb`

MBean

Rhino operation

public void removeSbb(ServiceID serviceID, SbbID sbbID, String sbbPKey, boolean nonResident)
    throws UnrecognizedServiceException, UnrecognizedSbbException, InvalidPKeyException,
          UnknownSbbEntityException, ManagementException;

This operation removes the SBB entity with the given primary key from the given service.

Removing All SBB Entities

To remove all SBB entities of a service, use the following rhino-console command or related MBean operation.

Use extreme care when removing forcibly

Occasionally an administrator will want to remove all SBB entities in a particular service. Typically, this would be to deactivate the service when upgrading or reconfiguring. Under normal conditions, these actions would be performed automatically, by allowing existing SBB entities to drain over time. Rhino provides the following housekeeping commands to forcibly speed up the draining process, although these should be used with extreme care on production systems — they will interrupt service for any existing network activities belonging to the service.

Service (or SLEE) must be STOPPING

As a safeguard, this command (or MBean operation) cannot be run unless the specified service, or the SLEE, is in the STOPPING state on the specified nodes. (It may also be run against nodes where the service is in the INACTIVE state (or the SLEE is in the STOPPED state) for convenience in asymmetric cluster configurations, but has no affect against such nodes since no SBB entities exist for the service on nodes where the service is INACTIVE (or the SLEE is STOPPED).)

Console command: `removeallsbbs`

Command

removeallsbbs <serviceid> [-nodes node1,node2,...]
  Description
    Remove all SBBs from a service in the Stopping state (on the specified nodes)

Example

To remove all SBB entities for the SimpleService service on nodes 101 and 102:

$ ./rhino-console removeallsbbs name=SimpleService,vendor=OpenCloud,version=1.1 \
    -nodes 101,102
SBB entities removed from node(s) [101,102]

MBean operation: `removeAllSbbs`

MBean

RhinoHousekeepingMBean

Rhino operation

public void removeAllSbbs(ServiceID serviceID, int[] nodeIDs)
    throws NullPointerException, UnrecognizedServiceException, InvalidStateException, ManagementException;

This operation removes all SBB entities for the given service on the given nodes.

Activity Context Name Bindings

Rhino includes procedures for:

Finding Activity Context Name Bindings
Removing Activity Context Name Bindings

Finding Activity Context Name Bindings

To find activity context name bindings in the SLEE, use the following rhino-console command or related MBean operations.

Console command: `findactivitybindings`

Command

findactivitybindings [-maxpernode maxrows] [-node nodeid] [-created-after
date|time|offset] [-created-before date|time|offset]
  Description
    Find activity context naming facility bindings.

Options

Option

Abbreviation

Description

 -maxpernode <maxrows>

Retrieve at most this many activity context name bindings from each event-router node (default is 100). Can be used to limit Rhino’s load when processing the request.

 -node <node-id>

Only display activity context name bindings owned by the given node.

 -created-after <time>

-ca

Only display activity context name bindings created after the given time.

 -created-before <time>

-cb

Only display activity context name bindings created before the given time.

Times for the above options may be entered in either absolute or relative format:

Type Format Description Examples

Absolute

 [[finding-activity-context-name-bindingsyyyy/]MM/dd] [HH:mm[:ss]]

yyyy = the year
MM = the month (1-12)
dd = the date of the month (1-31)
HH = the hour (0-23)
mm = the minute (0-59)
ss = the second (0-59)

 2008/04/15
04/15
10:57
10:57:35
2008/04/15 10:57:35

Relative

 (<nn><d|h|m|s>)*

nn = a number
d = days
h = hours
m = minutes
s = seconds

 1d
1h
1m
30s
1m
7m30s
1d12h5m30s

Rhino assumes relative time format is in the past. For example, 1h30m means 1 hour and 30 minutes ago.

Examples

To display all activity context name bindings in the SLEE:

$ ./rhino-console findactivitybindings
name                                pkey                      namespace   replicated   submission-time
----------------------------------  ------------------------  ----------  -----------  ------------------
connection[1542896514/0]/non-repl    2.101:227012595280421.0                    false   20181122 14:22:05
connection[1542896514/0]/repl        3.101:227012595280422.0                     true   20181122 14:22:05
connection[1542896514/1]/non-repl    2.102:227012872179724.0                    false   20181122 14:22:05
connection[1542896514/1]/repl        3.102:227012872179725.0                     true   20181122 14:22:05
connection[1542896514/2]/non-repl    2.103:227012872965639.0                    false   20181122 14:22:05
connection[1542896514/2]/repl        3.103:227012872965640.0                     true   20181122 14:22:05
connection[1542896514/3]/non-repl    2.101:227012595280424.0                    false   20181122 14:23:15
connection[1542896514/3]/repl        3.101:227012595280425.0                     true   20181122 14:23:15
connection[1542896514/4]/non-repl    2.102:227012872179727.0                    false   20181122 14:23:16
connection[1542896514/4]/repl        3.102:227012872179728.0                     true   20181122 14:23:16
10 rows

Narrowing a name bindings search

To search for activities belonging to node 101 (replicated or non-replicated activities owned by 101) that are more than one hour old, you would use the arguments -node 101 and -cb 1h:

$ ./rhino-console findactivitybindings -node 101 -cb 1h
name                                pkey                      namespace   replicated   submission-time
----------------------------------  ------------------------  ----------  -----------  ------------------
connection[1542896514/0]/non-repl    2.101:227012595280421.0                    false   20181122 14:22:05
connection[1542896514/0]/repl        3.101:227012595280422.0                     true   20181122 14:22:05
2 rows

(This example returned two name bindings.)

MBean operation: `getBoundActivities`

MBean

Rhino operations

Get summary information for all activity context name bindings

public TabularData getBoundActivities(int maxPerNode)
    throws ManagementException;

This operation returns tabular data summarising all activity context name bindings.

Get summary information for activity context name bindings using time-based criteria

public TabularData getBoundActivities(int maxPerNode, long createdAfter, long createdBefore)
    throws ManagementException;

This operation returns tabular data summarising the activity context name bindings using the time-based criteria specified (in milliseconds, as used by java.util.Date, or the value 0 to ignore a particular parameter).

Results depend on the Housekeeping MBean that invokes the operation:

Cluster Housekeeping MBean — returns results from all event-router nodes in the primary component
Node Housekeeping MBean — returns results from that node only.

For a description of the format of the tabular data that these operations return, see the javadoc.

Removing Activity Context Name Bindings

To forcefully remove an activity context name binding, use the following rhino-console command or related MBean operation.

Console command: `removeactivitybinding`

Command

removeactivitybinding <activity pkey> <name> [-non-resident]
  Description
    Remove an activity context naming facility binding. Use -non-resident to remove
    bindings not currently owned by any cluster node.

Example

To remove the activity context name binding with the name connection[1542896514/0]/repl from the activity with the primary key 3.101:227012595280422.0

$ ./rhino-console removeactivitybinding 3.101:227012595280422.0 connection[1542896514/0]/repl
Activity binding removed

MBean operation: `removeActivityBinding`

MBean

Rhino operation

public void removeActivityBinding(String activityPKey, String name, boolean nonResident)
    throws ManagementException, InvalidPKeyException, UnknownActivityException,
           NameNotBoundException;

This operation removes the activity context name binding with the given name from the activity with the given primary key. The nonResident argument must be true in order to remove a name binding from a non-resident activity.

Runtime Component Configuration

As of Rhino 2.3, Rhino supports modifying environment entry configuration and security permissions for deployed components.

To configure runtime components, see:

Inspecting Environment Entries
Setting Environment Entries
Inspecting Security Permissions
Modifying Security Permissions

Inspecting Environment Entries

To inspect a component’s environment entries, use the following rhino-console command or related MBean operation.

Console command: `getenventries`

Command

getenventries <ComponentID> [<true|false>]
  Description
    Returns the env entries for the specified SbbID or ProfileSpecificationID. The
    original env entries will be returned if the final argument is 'true'.

Example

To list all environment entries for the SIP Registrar SBB:

./rhino-console getenventries SbbID[name=RegistrarSbb,vendor=OpenCloud,version=1.8]
Getting env entries for component: SbbID[name=RegistrarSbb,vendor=OpenCloud,version=1.8]
  sipACIFactoryName:slee/resources/ocsip/acifactory
  sipProviderName:slee/resources/ocsip/provider

MBean operation: `getEnvEntries`

MBean

Rhino operation

public Map<String, String> getEnvEntries(ComponentID id, boolean original)
    throws NullPointerException, UnrecognizedComponentException,
    ManagementException, IllegalArgumentException;

This operation returns the environment entries associated with a component as a map of strings.

Setting Environment Entries

To modify a component’s environment entries, use the following rhino-console command or related MBean operation.

Console command: `setenventries`

Command

setenventries <ComponentID> <name1:value1> <name2:value2> ...
  Description
    Sets the env entries associated with the specified SbbID or
    ProfileSpecificationID.

Example

To modify the environment entries for the SIP Registrar SBB:

./rhino-console setenventries SbbID[name=RegistrarSbb,vendor=OpenCloud,version=1.8] sipACIFactoryName:slee/resources/ocsip/mycustomacifactory,sipProviderName:slee/resources/ocsip/mycustomprovider
Setting env entries for component: SbbID[name=RegistrarSbb,vendor=OpenCloud,version=1.8]

MBean operation: `getEnvEntries`

MBean

Rhino operation

public void setEnvEntries(ComponentID id, Map<String, String> entries)
    throws NullPointerException, UnrecognizedComponentException,
           ManagementException, IllegalArgumentException;

This operation sets the environment entries associated with a component.

Inspecting Security Permissions

To inspect a component’s security permissions, use the following rhino-console command or related MBean operation.

The security permissions for a component may be shared with multiple other components. For example, SBBs in the same jar may share their permissions.

Console command: `getsecuritypolicy`

Command

getsecuritypolicy (<ComponentID> | <LibraryID> [jarname]) [true|false]
  Description
    Returns the security policy associated with the specified ComponentID. The
    optional 'jarname' argument can be used to specify a nested library jar for
    LibraryIDs. The original policy will be returned if the final argument is
    'true'.

Example

To list the security permissions for the SIP resource adaptor:

./rhino-console getsecuritypolicy ResourceAdaptorID[name=OCSIP,vendor=OpenCloud,version=2.3.1]
grant {
  permission java.util.PropertyPermission "opencloud.sip.*", "read";
  permission java.io.FilePermission "/etc/resolv.conf", "read";
  permission java.net.SocketPermission "*", "resolve";
  permission java.net.SocketPermission "*:1024-", "listen,resolve";
  permission java.net.SocketPermission "*:1024-", "accept,connect";
  permission java.lang.RuntimePermission "modifyThread";
  permission java.io.FilePermission "sip-ra-ssl.truststore", "read";
  permission java.util.PropertyPermission "javax.sip.*", "read";
  permission java.io.FilePermission "sip-ra-ssl.keystore", "read";
  permission java.net.SocketPermission "*:53", "connect,accept";
};

MBean operation: `getSecurityPolicy`

MBean

Rhino operation

public String getSecurityPolicy(ComponentID id, String subId, boolean original)
    throws NullPointerException, UnrecognizedComponentException,
           IllegalArgumentException, ManagementException;

This operation returns the security permissions associated with a component.

Modifying Security Permissions

To modify a component’s security permissions, use the following rhino-console command or related MBean operation.

The security permissions for a component may be shared with multiple other components. For example, SBBs in the same jar may share their permissions.

Console command: `setsecuritypolicy`

Command

setsecuritypolicy (<ComponentID> | <LibraryID> [jarname]) <SecurityPolicy>
  Description
    Sets the current security policy associated with the specified ComponentID.

Example

To set the security permissions for the SIP resource adaptor:

./rhino-console setsecuritypolicy 'ResourceAdaptorID[name=OCSIP,vendor=OpenCloud,version=2.3.1]' 'grant { permission java.net.SocketPermission "*:53", "connect,accept"; };'
Setting security policy for component: ResourceAdaptorID[name=OCSIP,vendor=OpenCloud,version=2.3.1]

The command-line console only supports a single line as an argument. To easily modify multi-line security policies, use the Rhino Element Manager instead.

MBean operation: `getSecurityPolicy`

MBean

Rhino operation

public void setSecurityPolicy(ComponentID id, String subId, String policyText)
    throws NullPointerException, UnrecognizedComponentException,
           IllegalArgumentException, ManagementException;

This operation sets the security permissions associated with a component.

Garbage Collection

During normal operation, Rhino periodically cleans up removed application state by running GC (garbage collection) algorithms over state that has been updated.

Application state GC is, however, triggered after a certain amount of churn has occurred within a given in-memory database. Sometimes, for example when looking at Rhino heap dumps, it’s useful to disregard state that’s eligible for GC and not have it pollute the heap dump unnecessarily. However, in a cluster that has returned to an idle state, GC won’t normally run to perform these cleanups as the trigger conditions only occur with load.

To help with these situations, Rhino offers two housekeeping functions that force GC to occur. These operations can be used to remove unnecessary state from the heap before taking a heap dump, etc. These procedures are:

Activity State Cleanup
Global State Cleanup

Activity State Cleanup

To clean up state for activities that have been removed but not yet garbage collection, use the following rhino-console command or related MBean operation.

Console command: `initiateactivitycleanup`

Command

initiateactivitycleanup [ra entity name]
  Description
    Initiate activity GC

Example

To initiate garbage collection for the sipra resource adaptor entity

$ ./rhino-console initiateactivitycleanup sipra
Initiated activity GC for insis-ptc-1a

MBean operations: `initiateCleanup`

MBean

Rhino operations

Initiate activity cleanup

public void initiateCleanup(String raEntityName)
    throws UnrecognizedResourceAdaptorEntityException, ManagementException;

This operation initiates the garbage collection process on the activity state of the specified resource adaptor entity in the current namespace. If no resource adaptor entity name is specified, ie. the raEntityName argument is null, then garbage collection is initiated on the activity state for all resource adaptor entities in the current namespace.

Activity state GC runs asynchronously after the operation has been initiated, but typically completes within a few seconds.

Global State Cleanup

Rhino offers functionality to initiate garbage collection of all internal in-memory databases in a single operation.

This affects all removed but not yet garbage collected application and activity state in all namespaces.

To clean up all application and activity state for SBBs and activities that have been removed but not yet garbage collected, use the following rhino-console command or related MBean operation.

Console command: `initiateglobalcleanup`

Command

initiateglobalcleanup
  Description
    Initiate global GC

Example

To initiate garbage collection for all in-memory databases

$ ./rhino-console initiateglobalcleanup
Initiated global GC

MBean operations: `initiateGlobalCleanup`

MBean

Rhino operations

Initiate global cleanup

public void initiateGlobalCleanup()
    throws ManagementException;

This operation initiates the garbage collection process for all application and activity in-memory databases on all Rhino cluster nodes.

The GC process runs asynchronously after the operation has been initiated, but typically completes within a few seconds.

State Flushing

During normal operation, Rhino’s in-memory state can sometimes be slightly ahead of its persisted management and key/value store state. This is caused by asynchronous writes, query batching, and similar implementation details designed to minimise unnecessary or blocking database interactions. Any unpersisted state is usually only seconds ahead of the persisted state, excepting exceptional error cases such as database connection failures where unpersisted state may accumulate until connection restoration.

A graceful termination of a Rhino node (e.g. via the shutdown command) will implicitly perform appropriate persistence operations for any in-memory state which has yet to be persisted, but it can be useful in some cases to explicitly trigger persistence:

where a Rhino node needs to be forcefully terminated and guarantees regarding persisted state are required; or
where external automated tooling needs certainty regarding configuration changes which have been applied via management APIs.

Rhino offers two housekeeping functions that assist with these cases by manually initiating persistence and blocking until completion.

These procedures are:

Key/Value Store Persistence
Configuration Persistence

Key/Value Store Persistence

To force persistence of in-memory key/value store state, use the following rhino-console command or related MBean operation.

Console command: `flushkeyvaluestores`

Command

flushkeyvaluestores [-nodes node1,node2,...]
  Description
    Flush any state currently stored locally in key/value stores (on the specified
    nodes) to the backing database.  This operation blocks until the flush has
    completed.

Example

To force key/value stores to be flushed for node 101

$ ./rhino-console flushkeyvaluestores -nodes 101
Flushing key/value stores on nodes [101]...
Flush complete

MBean operations: `fluskKeyValueStores`

MBean

Rhino operations

Flush KeyValueStores

public void flushKeyValueStores()
    throws ManagementException;

public void flushKeyValueStores(int[] nodeIDs)
    throws ManagementException;

Request that all application state waiting for persistence in all key/value stores in all event router nodes in the cluster be persisted to the backing database as soon as possible.

Any unwritten application state buffered in the key/value stores at the time this operation is invoked will be flushed. Any further application state committed after the flush operation is initiated will be persisted at some future time in accordance with normal key/value store behaviour.

This method blocks until the flush operation has completed.

This operation requests that all application state waiting for persistence in all key/value stores in all event router nodes in the cluster be persisted to the backing database as soon as possible.

Configuration Persistence

To force persistence of in-memory configuration state to the management database, use the following rhino-console command or related MBean operation.

Console command: `flushconfiguration`

Command

flushconfiguration
  Description
    Flush any outstanding configuration changes which have yet to be persisted to
    the management database. This operation blocks until the flush has completed.

Example

To force persistence of configuration to the management database

$ ./rhino-console flushconfiguration
Flushing configuration changes to database...
Flush complete.

MBean operations: `flushConfiguration`

MBean