Cassandra not running after installation

Check that bootstrap and configuration were successful:

[sentinel@tsn1 ~]$ grep 'Bootstrap complete' ~/bootstrap/bootstrap.log
2019-10-28 13:53:54,226 INFO bootstrap.main Bootstrap complete
[sentinel@tsn1 ~]$

If the bootstrap.log does not contain that string, examine the log for any exceptions or errors.

[sentinel@tsn1 ~]$ report-initconf status
status=vm_converged
[sentinel@tsn1 ~]$

If the status is different, examine the output from report-initconf for any problems. If that is not sufficient, examine the ~/initconf/initconf.log file for any exceptions or errors.

If bootstrap and configuration were successful, check that the docker containers are present and up:

[sentinel@tsn1 ~]$ docker ps

CONTAINER ID        IMAGE                                                COMMAND                  CREATED             STATUS              PORTS               NAMES

6999eacf6868        art-docker.metaswitch.com/rhino/cassandra:3.11.4-4   "docker-entrypoint..."   8 minutes ago       Up 8 minutes                            cassandra-ramdisk

77520b74d274        art-docker.metaswitch.com/rhino/cassandra:3.11.4-4   "docker-entrypoint..."   8 minutes ago       Up 8 minutes                            cassandra
[sentinel@tsn1 ~]$

If the containers are present and Cassandra is not running, use journalctl and systemctl to check system logs for any errors or exceptions.

For the on-disk Cassandra:

$ journalctl -u cassandra -l
$ systemctl status cassandra -l

For the ramdisk Cassandra:

$ journalctl -u cassandra-ramdisk -l
$ systemctl status cassandra-ramdisk -l

Confirm that the two Cassandra processes are running and listening on ports 9042 and 19042:

[sentinel@tsn1 ~]$ sudo netstat -plant | grep 9042

tcp        0      0 0.0.0.0:19042           0.0.0.0:*               LISTEN      1856/java

tcp        0      0 0.0.0.0:9042            0.0.0.0:*               LISTEN      1889/java

[sentinel@tsn1 ~]$

Check that the Cassandra cluster has formed and each node is UN (Up and Normal).

For the on-disk Cassandra:

[sentinel@tsn1 ~]$ docker exec cassandra nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  172.31.58.207  678.58 KiB  256          ?       f81bc71d-4ba3-4400-bed5-77f317105cce  rack1
UN  172.31.53.62   935.66 KiB  256          ?       aa134a07-ef93-4e09-8631-0e438a341e57  rack1
UN  172.31.55.24   958.34 KiB  256          ?       8ce540ea-8b52-433f-9464-1581d32a99bc  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
[sentinel@tsn1 ~]$

For the ramdisk Cassandra:

[sentinel@tsn1 ~]$ docker exec cassandra-ramdisk nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.31.58.207  204.68 KiB  256          69.0%             1df3c9c5-3159-42af-91bd-0869d0cecf44  rack1
UN  172.31.53.62   343.98 KiB  256          67.1%             77d05776-14bd-49e9-8bcd-9834670c2907  rack1
UN  172.31.55.24   291.58 KiB  256          63.9%             7a0e9deb-4903-483a-8702-4508ca17c42c  rack1

[sentinel@tsn1 ~]$

Bootstrap and/or initconf failures are often caused by networking issues.

  • Check that each TSN node can ping all of the other TSN signaling IPs.

  • Check that each TSN node is configured to use its signaling interface for Cassandra.

[sentinel@tsn1 ~]$ docker exec cassandra grep "seeds:" /basedir/config/cassandra.yaml
      - seeds: "172.31.58.207,172.31.53.62,172.31.55.24"
[sentinel@tsn1 ~]$
[sentinel@tsn1 ~]$ docker exec cassandra grep "listen_address:" /basedir/config/cassandra.yaml
listen_address: 172.31.58.207
[sentinel@tsn1 ~]$

Cassandra resource exhaustion

To check the resource usage of the docker containers:

[sentinel@tsn1 ~]$ docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS

6999eacf6868        0.45%               2.374 GiB / 14.95 GiB   15.88%              0 B / 0 B           57 MB / 856 kB      73

77520b74d274        0.76%               3.217 GiB / 14.95 GiB   21.52%              0 B / 0 B           38.1 MB / 1.7 MB    81

To check diskspace usage:

[sentinel@tsn1 ~]$ df -h

Filesystem                             Size  Used Avail Use% Mounted on
/dev/nvme0n1p3                         7.9G  2.5G  5.1G  33% /
devtmpfs                               7.5G     0  7.5G   0% /dev
tmpfs                                  7.5G     0  7.5G   0% /dev/shm
tmpfs                                  7.5G  716K  7.5G   1% /run
tmpfs                                  7.5G     0  7.5G   0% /sys/fs/cgroup
tmpfs                                  7.5G     0  7.5G   0% /tmp
/home/sentinel/cassandra-ramdisk/data  8.0G     0  8.0G   0% /home/sentinel/cassandra-ramdisk/data
/dev/nvme0n1p2                         6.7G  799M  5.6G  13% /var/log
/dev/nvme0n1p1                          93M   44M   45M  50% /boot
tmpfs                                  1.5G     0  1.5G   0% /run/user/5101
tmpfs                                  1.5G     0  1.5G   0% /run/user/0

[sentinel@tsn1 ~]$
  • The on-disk Cassandra runs in the root partition.

  • The ramdisk Cassandra runs in /home/sentinel/cassandra-ramdisk/data

  • Cassandra logs are stored in /var/log/tas/cassandra and /var/log/tas/cassandra-ramdisk

Cassandra keyspaces missing

The on-disk Cassandra contains keyspaces for:

The ramdisk Cassandra contains keyspaces for Rhino Session Ownership and possibly Rhino Key/Value Stores.

Both the on-disk and ramdisk Cassandra contain keyspaces for CDS and system functionality.

To check if an expected Cassandra keyspace is present:

[sentinel@tsn1 ~]$ docker exec cassandra cqlsh <signaling ip> 9042 -e 'describe keyspaces';

opencloud_location_based_dialling  system
opencloud_psap_callback            opencloud_mmtel_ect
opencloud_sh_cache_microservice    system_distributed
system_schema                      system_traces
system_auth                        metaswitch_tas_deployment_info
opencloud_thirdparty_reg           opencloud_gaa_bootstrap_info
opencloud_ipsmgw_routing_info      opencloud_ipsmgw_registration_ownership

[sentinel@tsn1 ~]$
[sentinel@tsn1 ~]$ docker exec cassandra-ramdisk <signaling ip> 19042 cqlsh -e 'describe keyspaces';

system                             system_distributed
system_schema                      system_traces
system_auth                        metaswitch_tas_deployment_info
rhino_session_ownership_0_default  rhino_kv_0_default

[sentinel@tsn1 ~]$

Cannot run cqlsh command when using ssh

The cqlsh command is set up as a Bash alias. It can be run as-is from an interactive ssh session. If running the cqlsh command directly from an ssh command, e.g. as ssh tsn1 cqlsh, these aliases are not loaded. Instead, run the command as ssh -t tsn1 bash -ci cqlsh.

Cassandra troubleshooting

Refer to Cassandra documentation for detailed troubleshooting of Cassandra itself:

Previous page Next page
Rhino VoLTE TAS VMs Version 4.0.0