Scheduled tasks on Rhino VM Automation VMs

The Rhino VM Automation VMs run scheduled tasks to perform housekeeping and maintain stability. The following table shows all scheduled tasks present on the Rhino VM Automation VMs:

Scheduled task Description Configurable?

Restart Rhino

Runs on all Rhino nodes. Restarts Rhino to avoid issues caused by memory leaks and heap fragmentation in a long-running process.

Yes (can be disabled), through the scheduled-rhino-restarts option in *-vmpool-config.yaml

Configuring scheduled tasks

You can configure the scheduled tasks for any VM by adding appropriate configuration options to the relevant <node type>-vmpool-config.yaml file. The VM must be of a node type that supports that particular task, and it must be marked as configurable. Refer to the table above for details.

To disable Rhino restarts, omit the scheduled-rhino-restarts option from the configuration file.

Changes to task schedules take effect immediately. If a task is already in progress at the time of pushing a configuration change, it will complete its current run, and then run according to the new schedule.

For VMs in a group (that is, all VMs of a particular node type), we recommend the following:

  • If a scheduled task is configured on one VM, it is configured on all VMs in the group.

  • The frequency (daily, weekly or monthly) of the schedules is the same for all VMs in the group.

If you upload configuration where the enabled/disabled state and/or frequency varies between VMs in a group, the configuration is still applied, but rvtconfig will issue warnings and the VMs will raise a corresponding configuration warning alarm.

Restrictions

You cannot schedule two Rhino restarts on any one VM within 30 minutes of each other. (Such configuration would be excessive anyway; outside of exceptional circumstances, you only need to run these tasks at most once per day per VM.)

Additionally, two nodes in a group cannot restart Rhino within 30 minutes of each other. This is to prevent having a period where there are too few Rhino nodes to handle incoming traffic. While Rhino will normally restart in much less than 30 minutes, all traffic does need to drain from the node first, which can take some time.

All the above restrictions are checked by rvtconfig: configuration that doesn’t satisfy these requirements will not be accepted.

Example schedules for Rhino restarts

Scheduled Rhino restarts are applied per Rhino VM node, so they are defined under each virtual-machine element. For clarity, the examples below omit various fields that would normally be required.

Daily

For a daily schedule, specify only the time-of-day field. The format of this field is a 24-hour clock time, which must include any leading zeroes.

virtual-machines:
  - vm-id: mag-1
    scheduled-rhino-restarts:
      time-of-day: 02:00
  - vm-id: mag-2
    scheduled-rhino-restarts:
      time-of-day: 02:30

Weekly

For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day. The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.

virtual-machines:
  - vm-id: shcm-1
    scheduled-rhino-restarts:
      - day-of-week: Monday
        time-of-day: 02:00
      - day-of-week: Thursday
        time-of-day: 03:00

  - vm-id: shcm-2
    scheduled-rhino-restarts:
      - day-of-week: Tuesday
        time-of-day: 02:00
      - day-of-week: Friday
        time-of-day: 03:00

Monthly

For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day. The day-of-month field takes a number between 1 and 28 (29 to 31 are not included to avoid the task unexpectedly not running in certain months).

virtual-machines:
  - vm-id: smo-1
    scheduled-rhino-restarts:
      - day-of-month: 1
        time-of-day: 02:00
      - day-of-month: 11
        time-of-day: 03:00
      - day-of-month: 21
        time-of-day: 04:00
  - vm-id: smo-2
    scheduled-rhino-restarts:
      - day-of-month: 6
        time-of-day: 02:00
      - day-of-month: 16
        time-of-day: 03:00
      - day-of-month: 26
        time-of-day: 04:00

Example schedules for Cassandra repairs

Scheduled Cassandra repairs are executed on the whole TSN cluster, so they are set globally for all the virtual-machines element. For clarity, the examples below omit various fields that would normally be required.

Daily

For a daily schedule, specify only the time-of-day field. The format of this field is a 24-hour clock time, which must include any leading zeroes.

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  time-of-day: "16:30"

Weekly

For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day. The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  - day-of-week: Monday
    time-of-day: 02:00
  - day-of-week: Thursday
    time-of-day: 03:00

Monthly

For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day. The day-of-month field takes a number between 1 and 28 (29 to 31 are not included to avoid the task unexpectedly not running in certain months).

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  - day-of-month: 1
    time-of-day: 02:00
  - day-of-month: 11
    time-of-day: 03:00
  - day-of-month: 21
    time-of-day: 04:00

Maintenance window support

When performing maintenance activities that involve reconfiguring, restarting or replacing VMs, notably patching or upgrades, use the rvtconfig enter-maintenance-window command to temporarily disable all scheduled tasks on all VMs in a site. You can disable the scheduled tasks for a given number of hours (1 to 24).

Once the maintenance window is finished, run the rvtconfig leave-maintenance-window command. Scheduled tasks will then resume running as per the VMs' configuration.

Tip

While a maintenance window is active, you can still make configuration changes as normal. Uploading configuration that includes (changes to) schedules won’t reactivate the scheduled tasks. Once the maintenance window ends, the tasks will run according to the most recent configuration.

Important

Scheduled tasks that are already running at the time you run rvtconfig enter-maintenance-window are not canceled; they will complete their current run. As such, it is best to run the enter-maintenance-window command at a time when no tasks are scheduled, and/or perform a manual check that tasks aren’t running on the VMs concerned before starting any maintenance activity.

For more details on the enter-maintenance-window and leave-maintenance-window commands, see the rvtconfig page.

Previous page Next page
VM Build Container Version 3.2