In console mode, you can run the rhino-stats client with options to:
List root parameter sets
To list the different types of statistics that can be monitored in Rhino, run rhino-stats with the -l parameter.
For example:
$ ./rhino-stats -l The following root parameter sets are available for monitoring: Activities, ActivityHandler, EventRouter, Events, JVM, LicenseAccounting, LockManagers, MemDB-Local, MemDB-Replicated, ObjectPools, SLEE-Usage, Savanna-Protocol, Services, StagingThreads, SystemInfo, Transactions For parameter set type descriptions and a list of available parameter sets use the -l <root parameter set name> option
Display parameter set descriptions
|
The output below illustrates the root parameter set (Events) with many different child parameter sets.
You can use this information to select the level of granularity at which you want statistics reported.
(See the instructions for monitoring parameters in real time.)
|
To query the available child parameter sets within a particular root parameter set, use -l <root parameter set name>.
For example, for the root parameter set Events:
$ ./rhino-stats -l Events
Parameter Set: Events
Parameter Set Type: Events
Description: Event Statistics
Counter type statistics:
Id: Name: Label: Description:
0 accepted acc Accepted events
1 rejected rej Events rejected due to overload
2 failed fail Events that failed in event processing
3 successful succ Event processed successfully
Sample type statistics:
Id: Name: Label: Description:
4 eventRouterSetupTime ERT Event router setup time
5 sbbProcessingTime SBBT SBB processing time
6 eventProcessingTime EPT Total event processing time
Found 164 parameter sets under 'Events':
-> "Events"
-> "Events.Rhino Internal"
-> "Events.Rhino Internal.[javax.slee.ActivityEndEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.facilities.TimerEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.profile.ProfileAddedEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.profile.ProfileRemovedEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.profile.ProfileUpdatedEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.serviceactivity.ServiceStartedEvent javax.slee, 1.0]"
-> "Events.Rhino Internal.[javax.slee.serviceactivity.ServiceStartedEvent javax.slee, 1.1]"
-> "Events.insis-cap1a"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.CloseInd OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.DelimiterInd OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.NoticeInd OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.OpenConf OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.OpenInd OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.ProviderAbortInd OpenCloud, 2.0]"
-> "Events.insis-cap1a.[com.opencloud.slee.resources.in.dialog.UserAbortInd OpenCloud, 2.0]"
...
Parameter set types — required for monitoring
|
A parameter set can only be monitored by a statistics client such as rhino-stats if it has a parameter set type.
|
A parameter set’s type is listed in its description.
Most parameter sets have a type, such as the Events parameter set, which has the type "Events".
However, the SLEE-Usage root parameter set, for example, does not have a type, as shown below:
$ ./rhino-stats -l SLEE-Usage
Parameter Set: SLEE-Usage
(no parameter set type defined for this parameter set)
Found 16 parameter sets under 'SLEE-Usage':
-> "SLEE-Usage"
-> "SLEE-Usage.ProfileTables"
-> "SLEE-Usage.RAEntities"
-> "SLEE-Usage.Services"
-> "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2]"
-> "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]"
-> "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2].(default)"
-> "SLEE-Usage.Services.ServiceID[name=Call Duration Logging Service,vendor=OpenCloud,version=0.2]"
-> "SLEE-Usage.Services.ServiceID[name=Call Duration Logging Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Duration Logging SBB,vendor=OpenCloud,version=0.2]"
-> "SLEE-Usage.Services.ServiceID[name=Call Duration Logging Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Duration Logging SBB,vendor=OpenCloud,version=0.2].(default)"
Neither the SLEE-Usage parameter set, nor its immediate child parameter sets (SLEE-Usage.ProfileTables, SLEE-Usage.RAEntities, and SLEE-Usage.Services), have a parameter set type — as usage parameters are defined by SLEE components.
The parameter set representing usage for an SBB within a particular service does however have a parameter set type and can be monitored:
$ ./rhino-stats -l "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]"
Parameter Set: SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]
Parameter Set Type: Usage.Services.SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]
Description: Usage stats for SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]
Counter type statistics:
Id: Name: Label: Description:
0 missingParameters n/a missingParameters
1 tCallAttempts n/a tCallAttempts
2 unknownSubscribers n/a unknownSubscribers
3 oCallAttempts n/a oCallAttempts
4 callsBarred n/a callsBarred
5 callsAllowed n/a callsAllowed
Sample type statistics: (none defined)
Found 2 parameter sets under 'SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]':
-> "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2]"
-> "SLEE-Usage.Services.ServiceID[name=Call Barring Service,vendor=OpenCloud,version=0.2].SbbID[name=Call Barring SBB,vendor=OpenCloud,version=0.2].(default)"
|
|
Rhino guarantees that if a particular parameter set has a non-null parameter set type, then all its child parameter sets will also have a non-null parameter set type and can therefore also be individually monitored. |
Monitor parameters of a cluster in real time
|
|
Once started, rhino-stats will continue to extract and print the latest statistics at regular intervals.
How often the statistics are printed can be changed using the -i option when using the default (direct statistics download) mode or by using the -s option when using the deprecated JMX remote mode (-j option).
|
To monitor a parameter set of a cluster in real time using the console interface, run rhino-stats with the -m command-line argument followed by the parameter set name.
For example:
$ ./rhino-stats -m "Events.insis-cap1a.[com.opencloud.slee.resources.incc.operation.InitialDPInd OpenCloud, 3.0]"
2008-05-01 17:37:20.687 INFO [rhinostat] Connecting to localhost:1199
2008-05-01 17:37:21.755 INFO [dispatcher] Establish direct session DirectSession[host=x.x.x.x port=17400 id=56914320286194693]
2008-05-01 17:37:21.759 INFO [dispatcher] Connecting to localhost/127.0.0.1:17400
Events.insis-cap1a.[com.opencloud.slee.resources.incc.operation.InitialDPI
time acc fail rej succ EPT ERT SBBT
50% 90% 95% 50% 90% 95% 50% 90% 95%
----------------------- --------------------------------------------------------------------------
2008-05-01 17:37:25.987 69 0 0 69 0.7 1.2 1.4 0.2 0.4 0.4 0.5 0.8 1.0
2008-05-01 17:37:26.989 69 0 0 69 0.9 1.2 1.4 0.2 0.4 0.4 0.5 0.8 1.0
2008-05-01 17:37:27.991 61 0 0 61 0.9 1.3 1.6 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:28.993 67 0 0 67 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:29.996 69 0 0 69 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:30.996 63 0 0 63 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:31.999 71 0 0 71 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:33.001 64 0 0 64 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:34.002 68 0 0 68 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:35.004 60 0 0 60 0.9 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:36.006 64 0 0 64 1.0 1.3 1.4 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:37.008 67 0 0 66 1.0 1.3 1.5 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:38.010 61 0 0 62 1.0 1.4 1.5 0.3 0.4 0.4 0.6 0.9 1.0
2008-05-01 17:37:39.012 61 0 0 61 1.0 1.4 1.5 0.3 0.4 0.4 0.6 0.9 1.0
|
|
The "50% 90% 95%" headers indicate percentile buckets for sample type statistics. |
The above invocation of rhino-stats works against both Savanna and pool clusters.
Monitor parameters of an adhoc set of nodes
To monitor a parameter set of an adhoc set of nodes (excluding other members of their clusters), using the console interface, run rhino-stats using the -a command-line argument followed by comma separated addresses of those nodes' JMX MBean servers.
Nodes monitored can be a part of separate clusters, a subset of a single cluster or Rhino SDK nodes.
The nodes monitored can be in any clustering mode.
The only requirement is that rhino-stats is able to connect to their JMX MBean servers.
For example, events fired by a resource adaptor on three Rhino SDK nodes, whose JMX MBean servers are listening on localhost:1199, localhost:2199 and localhost:3199, can be monitored as:
$ ./rhino-stats -a localhost:1199,localhost:2199,localhost:3199 -S -r -m Events.simplera
Connecting to adhoc addresses: [localhost:1199, localhost:2199, localhost:3199]
Established direct session DirectSession[host=127.0.0.1 port=17400 id=28438765674092033]
Connecting to localhost/127.0.0.1:17400
Established direct session DirectSession[host=127.0.0.1 port=17402 id=28438765666482689]
Connecting to localhost/127.0.0.1:17402
Established direct session DirectSession[host=127.0.0.1 port=17404 id=28438765661958657]
Connecting to localhost/127.0.0.1:17404
/Events.simplera
time node acc fail rej succ EPT
50% 90% 95%
----------------------- ------------------------ ----------------------------------------
2022-11-14 19:43:43.379 localhost:1199:node-101 33 0 0 34 0.5 0.7 0.7
localhost:2199:node-101 20 0 0 18 0.5 0.7 0.9
localhost:3199:node-101 24 0 0 24 0.5 21.8 22.1
cluster 77 0 0 76 0.5 20.4 21.4
2022-11-14 19:43:44.379 localhost:1199:node-101 31 0 0 30 0.5 0.7 0.7
localhost:2199:node-101 31 0 0 30 0.5 0.7 0.9
localhost:3199:node-101 30 0 0 31 0.5 21.7 22.1
cluster 92 0 0 91 0.5 20.4 21.3
2022-11-14 19:43:45.379 localhost:1199:node-101 25 0 0 26 0.5 0.7 0.7
localhost:2199:node-101 31 0 0 30 0.5 0.7 0.9
localhost:3199:node-101 23 0 0 23 0.5 21.7 22.1
cluster 79 0 0 79 0.5 20.4 21.3
|
|
The stats in the above lines whose node column value is cluster are values aggregated across given adhoc nodes, as though they are part of the same Rhino cluster.
|
|
|
JMX MBean server addresses of adhoc nodes can be specified as value of rhino.adhoc.addresses property in client.properties file also.
For example, having rhino.adhoc.addresses=localhost:1199,localhost:2199,localhost:3199 in client.properties file and running ./rhino-stats -S -r -m Events.simplera is equivalent to using the -a option as shown above.
|
Configure console output
The default console output is not particularly useful when you want to do automated processing of the logged statistics.
To make post-processing of the statistics easier, rhino-stats supports a number of command-line arguments to modify the format of statistics output:
-
-R— outputs raw (single number) timestamps -
-C— outputs comma-separated statistics -
-d— display actual value in addition to deltas for counter stats -
-S— no per second conversion of counter deltas -
-r— output one row per node -
-q— suppresses printing non-statistics information
For example, to output a comma-separated log of event statistics:
$ ./rhino-stats -m "Events.insis-cap1a.[com.opencloud.slee.resources.incc.operation.InitialDPInd OpenCloud, 3.0]" -R -C -q time,acc,fail,rej,succ,EPT,ERT,SBBT 1209620311166,64,0,0,64,0.9 1.2 1.3,0.3 0.4 0.4,0.6 0.8 0.9 1209620312168,63,0,0,63,0.9 1.3 1.3,0.3 0.4 0.4,0.6 0.9 0.9 1209620313169,67,0,0,67,0.9 1.3 1.3,0.3 0.4 0.4,0.6 0.9 0.9 1209620314171,66,0,0,66,0.9 1.3 1.3,0.3 0.4 0.4,0.6 0.9 0.9 1209620315174,65,0,0,65,0.9 1.3 1.3,0.3 0.4 0.4,0.6 0.9 0.9 1209620316176,65,0,0,65,0.9 1.3 1.5,0.3 0.4 0.4,0.6 0.9 1.0 1209620317177,62,0,0,62,0.9 1.3 1.4,0.3 0.4 0.4,0.6 0.9 0.9 1209620318179,66,0,0,66,1.0 1.3 1.5,0.3 0.4 0.4,0.6 0.9 1.0 1209620319181,58,0,0,58,1.0 1.3 1.6,0.3 0.4 0.4,0.6 0.9 1.1 1209620320181,69,0,0,69,1.0 1.3 1.6,0.3 0.4 0.4,0.6 0.9 1.0 1209620321182,68,0,0,68,1.0 1.3 1.6,0.3 0.4 0.4,0.6 0.9 1.0 1209620322183,65,0,0,65,1.0 1.3 1.5,0.3 0.4 0.4,0.6 0.9 1.0 1209620323184,67,0,0,67,1.0 1.3 1.5,0.3 0.4 0.4,0.6 0.9 1.0 ...
Write output to file
To write the statistical output to rolling CSV files, the -o command-line argument is used. The files will be written to a subdirectory named output.
There is a limit of ten 10MB files that will be compressed once they reach their max size. These values are configurable through client/etc/rhino-console-log4j2.xml.
The output rows are comma separated by default (same effect as -C command-line argument).
All other console output modifiers will still work:
-
-R— outputs raw (single number) timestamps -
-d— display actual value in addition to deltas for counter stats -
-S— no per second conversion of counter deltas -
-r— output one row per node -
-q— suppresses printing non-statistics information
