- Sentinel IP-SM-GW or OCSS7 not running after installation
- Rhino Alarms
- OCSS7 SGC
Sentinel IP-SM-GW can be disabled in smo-vmpool-config.yaml. If Sentinel IP-SM-GW has been disabled, Rhino will not be running.
Check that bootstrap and configuration were successful:
[sentinel@smo1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log 2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete [sentinel@smo1 ~]$
bootstrap.log does not contain that string, examine the log for any exceptions or errors.
[sentinel@smo1 ~]$ report-initconf status status=vm_converged [sentinel@smo1 ~]$
If the status is different, examine the output from
report-initconf for any problems.
If that is not sufficient, examine the
~/initconf/initconf.log file for any exceptions or errors.
If bootstrap and configuration were successful, check the Rhino and OCSS7 journalctl logs.
[sentinel@smo1 ~]$ journalctl -u rhino -l [sentinel@smo1 ~]$ journalctl -u ocss7 -l
Further information can be found from the SMO logs in
In particular, the Rhino logs are found in a subdirectory of
/var/log/tas with the same
name as the Rhino directory has in the home directory, e.g.
Similarly, the OCSS7 logs are found in
Rhino alarms indicate issues that should be reserved promptly. Rhino alarms can be monitored using MetaView Server or REM on the MAG node. Some common Rhino alarms are described below.
Node: 101 Level: Critical Type: CassandraCQLRA.ConnectToCluster Message: Not connected to Cassandra. Attempting to connect each 10s
Check that the Cassandra server is active on the TSN nodes.
Check the network connectivity to the TSN nodes.
As TSN nodes are discovered automatically, no further configuration should be necessary. Ensure this node has been provided (as part of its configuration bundle) with the correct SDF for the TSN nodes, as the IP addresses to connect to are derived from this SDF.
Node: 101 Level: Major Type: noconnection Message: Lost connection to SGC localhost:11002
Check that SGC on this node is active.
Node: 101 Level: Warning Type: diameter.peer.connectiondown Message: Connection to [host]:[port] is down
Check the Diameter Ro peers are configured correctly.
Check the network connectivity to the Diameter Ro peer host and port.
Node: 101 Level: Major Type: rhino.sas.connection.lost Message: Connection to SAS server at [host]:[port] is down
Check that SAS is active.
Check the network connectivity to the SAS server host and port.
Node: 101 Level: Critical Type: ShCMRA.ShCMConnectFailed Message: Not connected to any instances of the configured Sh Cache Microservice host.
Check that ShCM is active on the ShCM nodes.
Check the network connectivity to the ShCM nodes.
Check that the DNS SRV records for ShCM are set up correctly. The ShCM domain that is configured in
common-config.yamlshould have DNS SRV records set up for every ShCM node.
systemctl status ocss7to determine if the
ocss7service is enabled and running.
jpsto see if an
SGCprocess is running.
Check the most recent
/var/log/tas/ocss7/for information relating to any failed startup.
The OCSS7 SGC CLI may be used to query the SGC for its active alarms.
The SGC CLI executable is located at
display-active-alarm command in the SGC CLI to show the active alarms.
See the OCSS7 Installation and Administration Guide for a full description of the alarms that can be raised by the OCSS7 SGC.