Scheduled tasks on Rhino VoLTE TAS VMs

The Rhino VoLTE TAS VMs run scheduled tasks to perform housekeeping and maintain stability. The following table shows all scheduled tasks present on the Rhino VoLTE TAS VMs:

Scheduled task Description Configurable?

Restart Rhino

Runs on all Rhino nodes. Restarts Rhino to avoid issues caused by memory leaks and heap fragmentation in a long-running process.

Yes (can be disabled), through the scheduled-rhino-restarts option in *-vmpool-config.yaml

Cassandra repair

Runs on the TSN nodes. Repairs Cassandra to ensure data integrity and that the cluster contains sufficient replicas of the Cassandra data.

Yes (but cannot be disabled), through the scheduled-cassandra-repairs option in tsn-vmpool-config.yaml

Cleanup of long-running SBBs and activities

Runs on all Rhino nodes. Removes long-running SBBs and activities from Rhino - these are assumed to be stuck, for example as a result of an error.

Yes (but cannot be disabled), through the scheduled-sbb-cleanups option in *-vmpool-config.yaml

Collect stats

Runs on the MAG leader node. Connects to all Rhino nodes in the site to collect stats. These stats can then be exported to allow near-realtime monitoring.

No

Configuring scheduled tasks

You can configure the scheduled tasks for any VM by adding appropriate configuration options to the relevant <node type>-vmpool-config.yaml file. The VM must be of a node type that supports that particular task, and it must be marked as configurable. Refer to the table above for details.

To disable Rhino restarts, omit the scheduled-rhino-restarts option from the configuration file.

Changes to task schedules take effect immediately. If a task is already in progress at the time of pushing a configuration change, it will complete its current run, and then run according to the new schedule.

For VMs in a group (that is, all VMs of a particular node type), we recommend the following:

  • If a scheduled task is configured on one VM, it is configured on all VMs in the group.

  • The frequency (daily, weekly or monthly) of the schedules is the same for all VMs in the group.

If you upload configuration where the enabled/disabled state and/or frequency varies between VMs in a group, the configuration is still applied, but rvtconfig will issue warnings and the VMs will raise a corresponding configuration warning alarm.

Restrictions

You cannot schedule two Rhino restarts on any one VM within 30 minutes of each other, nor two SBB/activity cleanups on any one VM within 10 minutes of each other. (Such configuration would be excessive anyway; outside of exceptional circumstances, you only need to run these tasks at most once per day per VM.)

Additionally, two nodes in a group cannot restart Rhino within 30 minutes of each other. This is to prevent having a period where there are too few Rhino nodes to handle incoming traffic. While Rhino will normally restart in much less than 30 minutes, all traffic does need to drain from the node first, which can take some time.

There are additional restrictions regarding scheduled Rhino restarts and SBB/activity cleanups:

  • An SBB/activity cleanup on a particular VM cannot be scheduled until 30 minutes after any Rhino restart on that VM.

  • A Rhino restart on a particular VM cannot be scheduled until 10 minutes after any SBB/activity cleanup on that VM.

All the above restrictions are checked by rvtconfig: configuration that doesn’t satisfy these requirements will not be accepted.

We recommend that you avoid scheduling any Rhino restart and Cassandra repair within one hour of each other to avoid any potential overloading of the Rhino and/or TSN node(s) concerned. This particular schedule clash is not policed in rvtconfig.

Example schedules for Rhino restarts

Scheduled Rhino restarts are applied per Rhino VM node, so they are defined under each virtual-machine element. For clarity, the examples below omit various fields that would normally be required.

Daily

For a daily schedule, specify only the time-of-day field. The format of this field is a 24-hour clock time, which must include any leading zeroes.

The following example will schedule a Rhino restart:

  • every day at 02:00 on the mag-1 VM.

  • every day at 02:30 on the mag-2 VM.

virtual-machines:
  - vm-id: mag-1
    scheduled-rhino-restarts:
      time-of-day: 02:00
  - vm-id: mag-2
    scheduled-rhino-restarts:
      time-of-day: 02:30

Weekly

For a weekly schedule, specify a list of field pairs under the container weekly, each pair being day-of-week and time-of-day. The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.

The following example will schedule a Rhino restart:

  • every Monday at 02:00 on the shcm-1 VM.

  • every Thursday at 03:00 on the shcm-1 VM.

  • every Tuesday at 02:00 on the shcm-2 VM.

  • every Friday at 03:00 on the shcm-2 VM.

virtual-machines:
  - vm-id: shcm-1
    scheduled-rhino-restarts:
      weekly:
        - day-of-week: Monday
          time-of-day: 02:00
        - day-of-week: Thursday
          time-of-day: 03:00

  - vm-id: shcm-2
    scheduled-rhino-restarts:
      weekly:
        - day-of-week: Tuesday
          time-of-day: 02:00
        - day-of-week: Friday
          time-of-day: 03:00

Monthly

For a monthly schedule, specify a list of field pairs under the container monthly, each pair being day-of-month and time-of-day. The day-of-month field takes a number between 1 and 28 (29 to 31 are not included to avoid the task unexpectedly not running in certain months).

The following example will schedule a Rhino restart:

  • on the 1st of every month at 02:00 on the smo-1 VM.

  • on the 11th of every month at 03:00 on the smo-1 VM.

  • on the 21st of every month at 04:00 on the smo-1 VM.

  • on the 6th of every month at 02:00 on the smo-2 VM.

  • on the 16th of every month at 03:00 on the smo-2 VM.

  • on the 26th of every month at 04:00 on the smo-2 VM.

virtual-machines:
  - vm-id: smo-1
    scheduled-rhino-restarts:
      monthly:
        - day-of-month: 1
          time-of-day: 02:00
        - day-of-month: 11
          time-of-day: 03:00
        - day-of-month: 21
          time-of-day: 04:00
  - vm-id: smo-2
    scheduled-rhino-restarts:
      monthly:
        - day-of-month: 6
          time-of-day: 02:00
        - day-of-month: 16
          time-of-day: 03:00
        - day-of-month: 26
          time-of-day: 04:00

Combining Frequencies

You can combine the frequency of schedules together for the same VM.

The following example will schedule a Rhino restart:

  • every Wednesday at 02:00 on the shcm-1 VM.

  • on the 15th of every month at 03:00 on the shcm-1 VM.

virtual-machines:
  - vm-id: shcm-1
    scheduled-rhino-restarts:
      weekly:
        - day-of-week: Wednesday
          time-of-day: 02:00
      monthly:
        - day-of-month: 15
          time-of-day: 03:00

Example schedules for Cassandra repairs

Scheduled Cassandra repairs are executed on the whole TSN cluster, so they are set globally for all the virtual-machines element. For clarity, the examples below omit various fields that would normally be required.

Daily

For a daily schedule, specify only the time-of-day field. The format of this field is a 24-hour clock time, which must include any leading zeroes.

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  time-of-day: "16:30"

Weekly

For a weekly schedule, specify a list of pairs of fields, each pair being day-of-week and time-of-day. The day-of-week field takes an English day of the week name with leading capital letter, for example Monday.

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  - day-of-week: Monday
    time-of-day: 02:00
  - day-of-week: Thursday
    time-of-day: 03:00

Monthly

For a monthly schedule, specify a list of pairs of fields, each pair being day-of-month and time-of-day. The day-of-month field takes a number between 1 and 28 (29 to 31 are not included to avoid the task unexpectedly not running in certain months).

virtual-machines:
  - vm-id: tsn-1
  - vm-id: tsn-2
  - vm-id: tsn-3
scheduled-cassandra-repairs:
  - day-of-month: 1
    time-of-day: 02:00
  - day-of-month: 11
    time-of-day: 03:00
  - day-of-month: 21
    time-of-day: 04:00

Maintenance window support

When performing maintenance activities that involve reconfiguring, restarting or replacing VMs, notably patching or upgrades, use the rvtconfig enter-maintenance-window command to temporarily disable all scheduled tasks on all VMs in a site. You can disable the scheduled tasks for a given number of hours (1 to 24).

Once the maintenance window is finished, run the rvtconfig leave-maintenance-window command. Scheduled tasks will then resume running as per the VMs' configuration.

Tip

While a maintenance window is active, you can still make configuration changes as normal. Uploading configuration that includes (changes to) schedules won’t reactivate the scheduled tasks. Once the maintenance window ends, the tasks will run according to the most recent configuration.

The stats gathering task cannot be disabled and isn’t affected by maintenance windows. However, there is no impact as this task only performs read-only operations and gracefully handles any attempt to connect to a node that is temporarily out of service.
Important

Scheduled tasks that are already running at the time you run rvtconfig enter-maintenance-window are not canceled; they will complete their current run. As such, it is best to run the enter-maintenance-window command at a time when no tasks are scheduled, and/or perform a manual check that tasks aren’t running on the VMs concerned before starting any maintenance activity.

For more details on the enter-maintenance-window and leave-maintenance-window commands, see the rvtconfig page.

Previous page Next page
Rhino VoLTE TAS VMs Version 4.2