REM, XCAP or Sentinel AGW not running after installation

Check that bootstrap and configuration were successful:

[sentinel@mag1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log
2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete
[sentinel@mag1 ~]$

If the bootstrap.log does not contain that string, examine the log for any exceptions or errors.

[sentinel@mag1 ~]$ report-initconf status
status=vm_converged
[sentinel@mag1 ~]$

If the status is different, examine the output from report-initconf for any problems. If that is not sufficient, examine the ~/initconf/initconf.log file for any exceptions or errors.

If bootstrap and configuration were successful, check the Rhino journalctl logs.

[sentinel@mag1 ~]$ journalctl -u rhino -l

Further information can also be found from the MAG logs in /var/log/tas and its subdirectories.

  • Rhino logs are found in /var/log/tas/rhino.

  • REM logs are found in /var/log/tas/apache-tomcat.

Cannot connect to REM

Connect to REM using a web browser. The connection should be over HTTPS to port 8443 of the management interface, and to the /rem/ page. For example: https://192.168.10.10:8443/rem/

If you connect using a hostname rather than the IP address, be sure that the hostname refers only to a single server in DNS.

If connections to REM fail despite use of the correct hostname/IP and port, try the following:

  • Check the REM service status on the node you are trying to connect to with sudo systemctl status rhino-element-manager. It should be listed as active (running).

  • Check that jps lists a Bootstrap process (this is the Apache Tomcat process).

  • Check that netstat -ant6 shows two listening sockets, one on the loopback address 127.0.0.1, port 8005, and the other on the management address, port 8443:

    tcp6       0      0 127.0.0.1:8005          :::*        LISTEN
    tcp6       0      0 192.168.10.10:8443      :::*        LISTEN

If any of the above checks fail, try restarting REM with sudo systemctl restart rhino-element-manager. You can also check for errors in the log files in the /var/log/tas/apache-tomcat directory.

Cannot log in to REM

When connecting to REM, you should use one of the accounts set up in the mag-vmpool-config.yaml file. The default username/password documented in the REM product documentation is not available on the REM node.

When trying to connect to Rhino, REM asks for credentials

When trying to connect to a Rhino instance, you need to enter the credentials REM can use to connect to Rhino. The Rhino username and password are configured in the VM pool YAML file for the Rhino nodes being monitored.

The mapping from REM users to Rhino users is deployment-specific (for example, you may wish to allocate a separate Rhino user to each REM user, so it is clear in Rhino audit logs which user made a certain change to Rhino configuration). As such, the VM software is unable to set up these credentials automatically.

It is recommended to use the "Save credentials" option so that you only need to specify the Rhino credentials once (per user, per instance).

Known REM product issues

For known REM issues, refer to the Known issues in REM section in the REM documentation.

Cannot connect to the XCAP server, NAF authentication filter, or BSF

The XCAP server, NAF authentication filter, and BSF run as services in Rhino.

First, check that you are connecting over the access interface, on port 80 (HTTP) or 443 (HTTPS).

If connections to the XCAP server, NAF authentication filter, or BSF fail despite use of the correct IP, try the following:

  • Check the NGINX service status on the node you are trying to connect to with sudo systemctl status nginx. It should be listed as active (running).

  • Check the NGINX container is running on the node you are trying to connect to with docker ps --filter name=nginx. It should be listed as Up.

  • Check that netstat -ant shows the following listening sockets:

    tcp        0      0 [access ip]:8080         0.0.0.0:*               LISTEN
    tcp        0      0 [access ip]:80           0.0.0.0:*               LISTEN
    tcp        0      0 [access ip]:8443         0.0.0.0:*               LISTEN
    tcp        0      0 [access ip]:443          0.0.0.0:*               LISTEN
    tcp6       0      0 127.0.0.1:8080           :::*                    LISTEN
    tcp6       0      0 [signaling ip]:8443      :::*                    LISTEN
    tcp6       0      0 [signaling ip]:8001      :::*                    LISTEN
  • (If a dual-stack access network is configured) Check that netstat -ant also shows the following listening sockets:

    tcp6       0      0 [access ipv6]:8080       :::*                    LISTEN
    tcp6       0      0 [access ipv6]:80         :::*                    LISTEN
    tcp6       0      0 [access ipv6]:8443       :::*                    LISTEN
    tcp6       0      0 [access ipv6]:443        :::*                    LISTEN

If any of the above checks fail, try restarting nginx with sudo systemctl restart nginx. You can also check for errors in the log files in the /var/log/tas/nginx directory.

Rhino Alarms

Rhino alarms indicate issues that should be reserved promptly. Rhino alarms can be monitored using MetaView Server or REM on the MAG node. Some common Rhino alarms are described below.

Not Connected to Cassandra

Node: 101
Level: Critical
Type: CassandraCQLRA.ConnectToCluster
Message: Not connected to Cassandra. Attempting to connect each 10s
  • Check that the Cassandra server is active on the TSN nodes.

  • Check the network connectivity to the TSN nodes.

  • As TSN nodes are discovered automatically, no further configuration should be necessary. Ensure this node has been provided (as part of its configuration bundle) with the correct SDF for the TSN nodes, as the IP addresses to connect to are derived from this SDF.

Connection to [host]:[port] is down

Node: 101
Level: Warning
Type: diameter.peer.connectiondown
Message: Connection to [host]:[port] is down
  • Check the Diameter Zh peers are configured correctly.

  • Check the network connectivity to the Diameter Zh peer host and port.

Connection to SAS server is down

Node: 101
Level: Major
Type: rhino.sas.connection.lost
Message: Connection to SAS server at [host]:[port] is down
  • Check that SAS is active.

  • Check the network connectivity to the SAS server host and port.

Previous page Next page
Rhino VoLTE TAS VMs Version 4.1