Scheduled tasks on Rhino VoLTE TAS VMs
The Rhino VoLTE TAS VMs run scheduled tasks to perform housekeeping and maintain stability. The following table shows all scheduled tasks present on the Rhino VoLTE TAS VMs:
| Scheduled task | Description | Configurable? |
|---|---|---|
Restart Rhino |
Runs on all Rhino nodes. Restarts Rhino to avoid issues caused by memory leaks and heap fragmentation in a long-running process. |
Yes (can be disabled), through the |
Cassandra repair |
Runs on the TSN nodes. Repairs Cassandra to ensure data integrity and that the cluster contains sufficient replicas of the Cassandra data. |
Yes (but cannot be disabled), through the |
Cleanup of long-running SBBs and activities |
Runs on all Rhino nodes. Removes long-running SBBs and activities from Rhino - these are assumed to be stuck, for example as a result of an error. |
Yes (but cannot be disabled), through the |
Collect stats |
Runs on the MAG leader node. Connects to all Rhino nodes in the site to collect stats. These stats can then be exported to allow near-realtime monitoring. |
No |
Configuring scheduled tasks
You can configure the scheduled tasks for any VM
by adding appropriate configuration options
to the relevant <node type>-vmpool-config.yaml file.
The VM must be of a node type that supports that particular task,
and it must be marked as configurable. Refer to the table above for details.
To disable Rhino restarts, omit the scheduled-rhino-restarts option from the configuration file.
Changes to task schedules take effect immediately. If a task is already in progress at the time of pushing a configuration change, it will complete its current run, and then run according to the new schedule.
For VMs in a group (that is, all VMs of a particular node type), we recommend the following:
-
If a scheduled task is configured on one VM, it is configured on all VMs in the group.
-
The frequency (daily, weekly or monthly) of the schedules is the same for all VMs in the group.
If you upload configuration where the enabled/disabled state and/or frequency varies between VMs in a group,
the configuration is still applied, but rvtconfig will issue warnings
and the VMs will raise a corresponding configuration warning alarm.
Restrictions
You cannot schedule two Rhino restarts on any one VM within 30 minutes of each other, nor two SBB/activity cleanups on any one VM within 10 minutes of each other. (Such configuration would be excessive anyway; outside of exceptional circumstances, you only need to run these tasks at most once per day per VM.)
Additionally, two nodes in a group cannot restart Rhino within 30 minutes of each other. This is to prevent having a period where there are too few Rhino nodes to handle incoming traffic. While Rhino will normally restart in much less than 30 minutes, all traffic does need to drain from the node first, which can take some time.
There are additional restrictions regarding scheduled Rhino restarts and SBB/activity cleanups:
-
An SBB/activity cleanup on a particular VM cannot be scheduled until 30 minutes after any Rhino restart on that VM.
-
A Rhino restart on a particular VM cannot be scheduled until 10 minutes after any SBB/activity cleanup on that VM.
All the above restrictions are checked by rvtconfig: configuration that doesn’t satisfy these requirements will not be accepted.
We recommend that you avoid scheduling any Rhino restart and Cassandra repair within one hour of each other
to avoid any potential overloading of the Rhino and/or TSN node(s) concerned.
This particular schedule clash is not policed in rvtconfig.
Example schedules for Rhino restarts
Scheduled Rhino restarts are applied per Rhino VM node, so they are defined under each virtual-machine element.
For clarity, the examples below omit various fields that would normally be required.
Daily
For a daily schedule, specify only the time-of-day field.
The format of this field is a 24-hour clock time, which must include any leading zeroes.
virtual-machines:
- vm-id: mag-1
scheduled-rhino-restarts:
time-of-day: 02:00
- vm-id: mag-2
scheduled-rhino-restarts:
time-of-day: 02:30Weekly
For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day.
The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.
virtual-machines:
- vm-id: shcm-1
scheduled-rhino-restarts:
- day-of-week: Monday
time-of-day: 02:00
- day-of-week: Thursday
time-of-day: 03:00
- vm-id: shcm-2
scheduled-rhino-restarts:
- day-of-week: Tuesday
time-of-day: 02:00
- day-of-week: Friday
time-of-day: 03:00Monthly
For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day.
The day-of-month field takes a number between 1 and 28
(29 to 31 are not included to avoid the task unexpectedly not running in certain months).
virtual-machines:
- vm-id: smo-1
scheduled-rhino-restarts:
- day-of-month: 1
time-of-day: 02:00
- day-of-month: 11
time-of-day: 03:00
- day-of-month: 21
time-of-day: 04:00
- vm-id: smo-2
scheduled-rhino-restarts:
- day-of-month: 6
time-of-day: 02:00
- day-of-month: 16
time-of-day: 03:00
- day-of-month: 26
time-of-day: 04:00Example schedules for Cassandra repairs
Scheduled Cassandra repairs are executed on the whole TSN cluster, so they are set globally for all the virtual-machines element.
For clarity, the examples below omit various fields that would normally be required.
Daily
For a daily schedule, specify only the time-of-day field.
The format of this field is a 24-hour clock time, which must include any leading zeroes.
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
time-of-day: "16:30"Weekly
For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day.
The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
- day-of-week: Monday
time-of-day: 02:00
- day-of-week: Thursday
time-of-day: 03:00Monthly
For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day.
The day-of-month field takes a number between 1 and 28
(29 to 31 are not included to avoid the task unexpectedly not running in certain months).
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
- day-of-month: 1
time-of-day: 02:00
- day-of-month: 11
time-of-day: 03:00
- day-of-month: 21
time-of-day: 04:00Maintenance window support
When performing maintenance activities that involve reconfiguring, restarting or replacing VMs,
notably patching or upgrades,
use the rvtconfig enter-maintenance-window command
to temporarily disable all scheduled tasks on all VMs in a site.
You can disable the scheduled tasks for a given number of hours (1 to 24).
Once the maintenance window is finished, run the rvtconfig leave-maintenance-window command.
Scheduled tasks will then resume running as per the VMs' configuration.
|
While a maintenance window is active, you can still make configuration changes as normal. Uploading configuration that includes (changes to) schedules won’t reactivate the scheduled tasks. Once the maintenance window ends, the tasks will run according to the most recent configuration. The stats gathering task cannot be disabled and isn’t affected by maintenance windows. However, there is no impact as this task only performs read-only operations and gracefully handles any attempt to connect to a node that is temporarily out of service. |
|
Scheduled tasks that are already running at the time you run |
For more details on the enter-maintenance-window and leave-maintenance-window commands,
see the rvtconfig page.
