Scheduled tasks on Rhino VM Automation VMs
The Rhino VM Automation VMs run scheduled tasks to perform housekeeping and maintain stability. The following table shows all scheduled tasks present on the Rhino VM Automation VMs:
| Scheduled task | Description | Configurable? |
|---|---|---|
Restart Rhino |
Runs on all Rhino nodes. Restarts Rhino to avoid issues caused by memory leaks and heap fragmentation in a long-running process. |
Yes (can be disabled), through the |
Configuring scheduled tasks
You can configure the scheduled tasks for any VM
by adding appropriate configuration options
to the relevant <node type>-vmpool-config.yaml file.
The VM must be of a node type that supports that particular task,
and it must be marked as configurable. Refer to the table above for details.
To disable Rhino restarts, omit the scheduled-rhino-restarts option from the configuration file.
Changes to task schedules take effect immediately. If a task is already in progress at the time of pushing a configuration change, it will complete its current run, and then run according to the new schedule.
For VMs in a group (that is, all VMs of a particular node type), we recommend the following:
-
If a scheduled task is configured on one VM, it is configured on all VMs in the group.
-
The frequency (daily, weekly or monthly) of the schedules is the same for all VMs in the group.
If you upload configuration where the enabled/disabled state and/or frequency varies between VMs in a group,
the configuration is still applied, but rvtconfig will issue warnings
and the VMs will raise a corresponding configuration warning alarm.
Restrictions
You cannot schedule two Rhino restarts on any one VM within 30 minutes of each other. (Such configuration would be excessive anyway; outside of exceptional circumstances, you only need to run these tasks at most once per day per VM.)
Additionally, two nodes in a group cannot restart Rhino within 30 minutes of each other. This is to prevent having a period where there are too few Rhino nodes to handle incoming traffic. While Rhino will normally restart in much less than 30 minutes, all traffic does need to drain from the node first, which can take some time.
All the above restrictions are checked by rvtconfig: configuration that doesn’t satisfy these requirements will not be accepted.
Example schedules for Rhino restarts
Scheduled Rhino restarts are applied per Rhino VM node, so they are defined under each virtual-machine element.
For clarity, the examples below omit various fields that would normally be required.
Daily
For a daily schedule, specify only the time-of-day field.
The format of this field is a 24-hour clock time, which must include any leading zeroes.
virtual-machines:
- vm-id: mag-1
scheduled-rhino-restarts:
time-of-day: 02:00
- vm-id: mag-2
scheduled-rhino-restarts:
time-of-day: 02:30Weekly
For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day.
The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.
virtual-machines:
- vm-id: shcm-1
scheduled-rhino-restarts:
- day-of-week: Monday
time-of-day: 02:00
- day-of-week: Thursday
time-of-day: 03:00
- vm-id: shcm-2
scheduled-rhino-restarts:
- day-of-week: Tuesday
time-of-day: 02:00
- day-of-week: Friday
time-of-day: 03:00Monthly
For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day.
The day-of-month field takes a number between 1 and 28
(29 to 31 are not included to avoid the task unexpectedly not running in certain months).
virtual-machines:
- vm-id: smo-1
scheduled-rhino-restarts:
- day-of-month: 1
time-of-day: 02:00
- day-of-month: 11
time-of-day: 03:00
- day-of-month: 21
time-of-day: 04:00
- vm-id: smo-2
scheduled-rhino-restarts:
- day-of-month: 6
time-of-day: 02:00
- day-of-month: 16
time-of-day: 03:00
- day-of-month: 26
time-of-day: 04:00Example schedules for Cassandra repairs
Scheduled Cassandra repairs are executed on the whole TSN cluster, so they are set globally for all the virtual-machines element.
For clarity, the examples below omit various fields that would normally be required.
Daily
For a daily schedule, specify only the time-of-day field.
The format of this field is a 24-hour clock time, which must include any leading zeroes.
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
time-of-day: "16:30"Weekly
For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day.
The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
- day-of-week: Monday
time-of-day: 02:00
- day-of-week: Thursday
time-of-day: 03:00Monthly
For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day.
The day-of-month field takes a number between 1 and 28
(29 to 31 are not included to avoid the task unexpectedly not running in certain months).
virtual-machines:
- vm-id: tsn-1
- vm-id: tsn-2
- vm-id: tsn-3
scheduled-cassandra-repairs:
- day-of-month: 1
time-of-day: 02:00
- day-of-month: 11
time-of-day: 03:00
- day-of-month: 21
time-of-day: 04:00Maintenance window support
When performing maintenance activities that involve reconfiguring, restarting or replacing VMs,
notably patching or upgrades,
use the rvtconfig enter-maintenance-window command
to temporarily disable all scheduled tasks on all VMs in a site.
You can disable the scheduled tasks for a given number of hours (1 to 24).
Once the maintenance window is finished, run the rvtconfig leave-maintenance-window command.
Scheduled tasks will then resume running as per the VMs' configuration.
|
While a maintenance window is active, you can still make configuration changes as normal. Uploading configuration that includes (changes to) schedules won’t reactivate the scheduled tasks. Once the maintenance window ends, the tasks will run according to the most recent configuration. |
|
Scheduled tasks that are already running at the time you run |
For more details on the enter-maintenance-window and leave-maintenance-window commands,
see the rvtconfig page.
