Sentinel IP-SM-GW or OCSS7 not running after installation
Sentinel IP-SM-GW can be disabled in smo-vmpool-config.yaml. If Sentinel IP-SM-GW has been disabled, Rhino will not be running. |
Check that bootstrap and configuration were successful:
[sentinel@smo1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log 2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete [sentinel@smo1 ~]$
If the bootstrap.log
does not contain that string, examine the log for any exceptions or errors.
[sentinel@smo1 ~]$ report-initconf status status=vm_converged [sentinel@smo1 ~]$
If the status is different, examine the output from report-initconf
for any problems.
If that is not sufficient, examine the ~/initconf/initconf.log
file for any exceptions or errors.
If bootstrap and configuration were successful, check the Rhino and OCSS7 journalctl logs.
[sentinel@smo1 ~]$ journalctl -u rhino -l [sentinel@smo1 ~]$ journalctl -u ocss7 -l
Further information can be found from the SMO logs in /var/log/tas
.
In particular, the Rhino logs are found in a subdirectory of /var/log/tas
with the same
name as the Rhino directory has in the home directory, e.g. gaa-4.0.0.0-cluster-110
.
Similarly, the OCSS7 logs are found in /var/log/tas/ocss7
.
Rhino Alarms
Rhino alarms indicate issues that should be reserved promptly. Rhino alarms can be monitored using MetaView Server or REM on the MAG node. Some common Rhino alarms are described below.
Not Connected to Cassandra
Node: 101 Level: Critical Type: CassandraCQLRA.ConnectToCluster Message: Not connected to Cassandra. Attempting to connect each 10s
-
Check that the Cassandra server is active on the TSN nodes.
-
Check the network connectivity to the TSN nodes.
-
As TSN nodes are discovered automatically, no further configuration should be necessary. Ensure this node has been provided (as part of its configuration bundle) with the correct SDF for the TSN nodes, as the IP addresses to connect to are derived from this SDF.
Lost connection to SGC
Node: 101 Level: Major Type: noconnection Message: Lost connection to SGC localhost:11002
-
Check that SGC on this node is active.
Connection to Diameter Ro peer is down
Node: 101 Level: Warning Type: diameter.peer.connectiondown Message: Connection to [host]:[port] is down
-
Check the Diameter Ro peers are configured correctly.
-
Check the network connectivity to the Diameter Ro peer host and port.
Connection to SAS server is down
Node: 101 Level: Major Type: rhino.sas.connection.lost Message: Connection to SAS server at [host]:[port] is down
-
Check that SAS is active.
-
Check the network connectivity to the SAS server host and port.
Not connected to any instances of the configured Sh Cache Microservice host.
Node: 101 Level: Critical Type: ShCMRA.ShCMConnectFailed Message: Not connected to any instances of the configured Sh Cache Microservice host.
-
Check that ShCM is active on the ShCM nodes.
-
Check the network connectivity to the ShCM nodes.
-
Check that the DNS SRV records for ShCM are set up correctly. The ShCM domain that is configured in
common-config.yaml
should have DNS SRV records set up for every ShCM node.
OCSS7 SGC
The OCSS7 SGC is not running
-
Use
systemctl status ocss7
to determine if theocss7
service is enabled and running. -
Check using
jps
to see if anSGC
process is running. -
Check the most recent
startup.log
andss7.log
in/var/log/tas/ocss7/
for information relating to any failed startup.
OCSS7 SGC Alarms
The OCSS7 SGC CLI may be used to query the SGC for its active alarms.
The SGC CLI executable is located at ~/ocss7/<deployment_id>/<node_id>/current/cli/bin/sgc-cli.sh
.
Use the display-active-alarm
command in the SGC CLI to show the active alarms.
See the OCSS7 Installation and Administration Guide for a full description of the alarms that can be raised by the OCSS7 SGC.