REM, XCAP or Sentinel AGW not running after installation
Check that bootstrap and configuration were successful:
[sentinel@mag1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log 2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete [sentinel@mag1 ~]$
If the bootstrap.log does not contain that string, examine the log for any exceptions or errors.
[sentinel@mag1 ~]$ report-initconf status status=vm_converged [sentinel@mag1 ~]$
If the status is different, examine the output from report-initconf for any problems.
If that is not sufficient, examine the ~/initconf/initconf.log file for any exceptions or errors.
If bootstrap and configuration were successful, check the Rhino journalctl logs.
[sentinel@mag1 ~]$ journalctl -u rhino -l
Further information can also be found from the MAG logs in /var/log/tas and its subdirectories.
-
Rhino logs are found in
/var/log/tas/rhino. -
REM logs are found in
/var/log/tas/apache-tomcat.
Cannot connect to REM
Connect to REM using a web browser. The connection should be over HTTPS to port 8443 of the
management interface, and to the /rem/ page. For example: https://192.168.10.10:8443/rem/
If you connect using a hostname rather than the IP address, be sure that the hostname refers only to a single server in DNS.
If connections to REM fail despite use of the correct hostname/IP and port, try the following:
-
Check the REM service status on the node you are trying to connect to with
sudo systemctl status rhino-element-manager. It should be listed asactive (running). -
Check that
jpslists aBootstrapprocess (this is the Apache Tomcat process). -
Check that
netstat -ant6shows two listening sockets, one on the loopback address127.0.0.1, port 8005, and the other on the management address, port 8443:tcp6 0 0 127.0.0.1:8005 :::* LISTEN tcp6 0 0 192.168.10.10:8443 :::* LISTEN
If any of the above checks fail, try restarting REM with sudo systemctl restart rhino-element-manager.
You can also check for errors in the log files in the /var/log/tas/apache-tomcat directory.
Cannot log in to REM
When connecting to REM, you should use one of the accounts set up in the mag-vmpool-config.yaml file. The default username/password documented in the REM product documentation is not available on the REM node.
When trying to connect to Rhino, REM asks for credentials
When trying to connect to a Rhino instance, you need to enter the credentials REM can use to connect to Rhino. The Rhino username and password are configured in the VM pool YAML file for the Rhino nodes being monitored.
The mapping from REM users to Rhino users is deployment-specific (for example, you may wish to allocate a separate Rhino user to each REM user, so it is clear in Rhino audit logs which user made a certain change to Rhino configuration). As such, the VM software is unable to set up these credentials automatically.
It is recommended to use the "Save credentials" option so that you only need to specify the Rhino credentials once (per user, per instance).
Known REM product issues
For known REM issues, refer to the Known issues in REM section in the REM documentation.
Cannot connect to the XCAP server, NAF authentication filter, or BSF
The XCAP server, NAF authentication filter, and BSF run as services in Rhino.
First, check that you are connecting over the access interface, on port 80 (HTTP) or 443 (HTTPS).
If connections to the XCAP server, NAF authentication filter, or BSF fail despite use of the correct IP, try the following:
-
Check the NGINX service status on the node you are trying to connect to with
sudo systemctl status nginx. It should be listed asactive (running). -
Check the NGINX container is running on the node you are trying to connect to with
docker ps --filter name=nginx. It should be listed asUp. -
Check that
netstat -antshows the following listening sockets:tcp 0 0 [access ip]:8080 0.0.0.0:* LISTEN tcp 0 0 [access ip]:80 0.0.0.0:* LISTEN tcp 0 0 [access ip]:8443 0.0.0.0:* LISTEN tcp 0 0 [access ip]:443 0.0.0.0:* LISTEN tcp6 0 0 127.0.0.1:8080 :::* LISTEN tcp6 0 0 [signaling ip]:8443 :::* LISTEN tcp6 0 0 [signaling ip]:8001 :::* LISTEN -
(If a dual-stack access network is configured) Check that
netstat -antalso shows the following listening sockets:tcp6 0 0 [access ipv6]:8080 :::* LISTEN tcp6 0 0 [access ipv6]:80 :::* LISTEN tcp6 0 0 [access ipv6]:8443 :::* LISTEN tcp6 0 0 [access ipv6]:443 :::* LISTEN
If any of the above checks fail, try restarting nginx with sudo systemctl restart nginx.
You can also check for errors in the log files in the /var/log/tas/nginx directory.
Rhino Alarms
Rhino alarms indicate issues that should be reserved promptly. Rhino alarms can be monitored using MetaView Server or REM on the MAG node. Some common Rhino alarms are described below.
Not Connected to Cassandra
Node: 101 Level: Critical Type: CassandraCQLRA.ConnectToCluster Message: Not connected to Cassandra. Attempting to connect each 10s
-
Check that the Cassandra server is active on the TSN nodes.
-
Check the network connectivity to the TSN nodes.
-
As TSN nodes are discovered automatically, no further configuration should be necessary. Ensure this node has been provided (as part of its configuration bundle) with the correct SDF for the TSN nodes, as the IP addresses to connect to are derived from this SDF.
