Cassandra not running after installation
Check that bootstrap and configuration were successful:
[sentinel@tsn1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log 2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete [sentinel@tsn1 ~]$
If the bootstrap.log does not contain that string, examine the log for any exceptions or errors.
[sentinel@tsn1 ~]$ report-initconf status status=vm_converged [sentinel@tsn1 ~]$
If the status is different, examine the output from report-initconf for any problems.
If that is not sufficient, examine the ~/initconf/initconf.log file for any exceptions or errors.
If bootstrap and configuration were successful, check that the docker containers are present and up:
[sentinel@tsn1 ~]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6999eacf6868 art-docker.metaswitch.com/rhino/cassandra:3.11.4-4 "docker-entrypoint..." 8 minutes ago Up 8 minutes cassandra-ramdisk 77520b74d274 art-docker.metaswitch.com/rhino/cassandra:3.11.4-4 "docker-entrypoint..." 8 minutes ago Up 8 minutes cassandra [sentinel@tsn1 ~]$
If the containers are present and Cassandra is not running, use journalctl and systemctl to check system logs for any errors or exceptions.
For the on-disk Cassandra:
$ journalctl -u cassandra -l $ systemctl status cassandra -l
For the ramdisk Cassandra:
$ journalctl -u cassandra-ramdisk -l $ systemctl status cassandra-ramdisk -l
Confirm that the two Cassandra processes are running and listening on ports 9042 and 19042:
[sentinel@tsn1 ~]$ sudo netstat -plant | grep 9042 tcp 0 0 0.0.0.0:19042 0.0.0.0:* LISTEN 1856/java tcp 0 0 0.0.0.0:9042 0.0.0.0:* LISTEN 1889/java [sentinel@tsn1 ~]$
Check that the Cassandra cluster has formed and each node is UN (Up and Normal).
For the on-disk Cassandra:
[sentinel@tsn1 ~]$ docker exec cassandra nodetool status Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 172.31.58.207 678.58 KiB 256 ? f81bc71d-4ba3-4400-bed5-77f317105cce rack1 UN 172.31.53.62 935.66 KiB 256 ? aa134a07-ef93-4e09-8631-0e438a341e57 rack1 UN 172.31.55.24 958.34 KiB 256 ? 8ce540ea-8b52-433f-9464-1581d32a99bc rack1 Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless [sentinel@tsn1 ~]$
For the ramdisk Cassandra:
[sentinel@tsn1 ~]$ docker exec cassandra-ramdisk nodetool status Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.58.207 204.68 KiB 256 69.0% 1df3c9c5-3159-42af-91bd-0869d0cecf44 rack1 UN 172.31.53.62 343.98 KiB 256 67.1% 77d05776-14bd-49e9-8bcd-9834670c2907 rack1 UN 172.31.55.24 291.58 KiB 256 63.9% 7a0e9deb-4903-483a-8702-4508ca17c42c rack1 [sentinel@tsn1 ~]$
Bootstrap and/or initconf failures are often caused by networking issues.
-
Check that each TSN node can ping all of the other TSN signaling IPs.
-
Check that each TSN node is configured to use its signaling interface for Cassandra.
[sentinel@tsn1 ~]$ docker exec cassandra grep "seeds:" /basedir/config/cassandra.yaml
- seeds: "172.31.58.207,172.31.53.62,172.31.55.24"
[sentinel@tsn1 ~]$
[sentinel@tsn1 ~]$ docker exec cassandra grep "listen_address:" /basedir/config/cassandra.yaml
listen_address: 172.31.58.207
[sentinel@tsn1 ~]$
Cassandra resource exhaustion
To check the resource usage of the docker containers:
[sentinel@tsn1 ~]$ docker stats CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 6999eacf6868 0.45% 2.374 GiB / 14.95 GiB 15.88% 0 B / 0 B 57 MB / 856 kB 73 77520b74d274 0.76% 3.217 GiB / 14.95 GiB 21.52% 0 B / 0 B 38.1 MB / 1.7 MB 81
To check diskspace usage:
[sentinel@tsn1 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p3 7.9G 2.5G 5.1G 33% / devtmpfs 7.5G 0 7.5G 0% /dev tmpfs 7.5G 0 7.5G 0% /dev/shm tmpfs 7.5G 716K 7.5G 1% /run tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup tmpfs 7.5G 0 7.5G 0% /tmp /home/sentinel/cassandra-ramdisk/data 8.0G 0 8.0G 0% /home/sentinel/cassandra-ramdisk/data /dev/nvme0n1p2 6.7G 799M 5.6G 13% /var/log /dev/nvme0n1p1 93M 44M 45M 50% /boot tmpfs 1.5G 0 1.5G 0% /run/user/5101 tmpfs 1.5G 0 1.5G 0% /run/user/0 [sentinel@tsn1 ~]$
-
The on-disk Cassandra runs in the root partition.
-
The ramdisk Cassandra runs in
/home/sentinel/cassandra-ramdisk/data -
Cassandra logs are stored in
/var/log/tas/cassandraand/var/log/tas/cassandra-ramdisk
Cassandra keyspaces missing
The on-disk Cassandra contains keyspaces for:
-
MMT PSAP Callback
-
MMT ECT
-
MMT LBD
-
MMT Third Party Reg
-
MAG.
The ramdisk Cassandra contains keyspaces for Rhino Session Ownership and possibly Rhino Key/Value Stores.
Both the on-disk and ramdisk Cassandra contain keyspaces for CDS and system functionality.
To check if an expected Cassandra keyspace is present:
[sentinel@tsn1 ~]$ docker exec cassandra cqlsh <signaling ip> 9042 -e 'describe keyspaces'; opencloud_location_based_dialling system opencloud_psap_callback opencloud_mmtel_ect opencloud_sh_cache_microservice system_distributed system_schema system_traces system_auth metaswitch_tas_deployment_info opencloud_thirdparty_reg opencloud_gaa_bootstrap_info opencloud_ipsmgw_routing_info opencloud_ipsmgw_registration_ownership [sentinel@tsn1 ~]$
[sentinel@tsn1 ~]$ docker exec cassandra-ramdisk <signaling ip> 19042 cqlsh -e 'describe keyspaces'; system system_distributed system_schema system_traces system_auth metaswitch_tas_deployment_info rhino_session_ownership_0_default rhino_kv_0_default [sentinel@tsn1 ~]$
Cannot run cqlsh command when using ssh
The cqlsh command is set up as a Bash alias.
It can be run as-is from an interactive ssh session.
If running the cqlsh command directly from an ssh command, e.g. as ssh tsn1 cqlsh,
these aliases are not loaded.
Instead, run the command as ssh -t tsn1 bash -ci cqlsh.
Cassandra troubleshooting
Refer to Cassandra documentation for detailed troubleshooting of Cassandra itself:
