Upgrade bundle for REM

An upgrade bundle for REM is a self-contained package with:

  • the orca bundle

  • the new version of Rhino Element Manager

  • the Sentinel plugins if necessary

  • the README

An upgrade bundle has the name

rem-<version>-upgrade.zip

Example: rem-1.5.0.5-upgrade.zip

Item Description

README

Contains information about the REM upgrade: how to use it, what components it changes, and what problems it fixes

orca

The orca tool

helpers directory

Contains set of scripts used by orca

core directory

Contains set of scripts used by orca

workflows directory

Contains set of scripts used by orca

resources directory

Contains the properties used by orca

licenses directory

License information for third party libraries used by the patch runner

packages directory

Contains package files and the package.cfg

The package files directory contains:

  • the packages.cfg file

  • REM product zip, e.g rhino-element-manager-1.5.0.5.zip

  • REM plugins. e.g volte-sentinel-element-manager-2.7.0.7.em.jar, sentinel-gaa-em-2.7.0.7.em.jar

  • optionally, a custom package

Requirements

An upgrade to REM uses the orca tool but since it is controlling REM nodes rather than Rhino, it has different path requirements. See requirements. Applying any upgrade requires administrative access to the hosts, so be sure the credentials used in ssh trusted connections are valid.

Disk space requirements when upgrading REM is minimal, no bigger than 100 Mbytes.

The CATALINA_HOME environment variable should be set correctly on the host running Rhino Element Manager.

What should be present in a REM upgrade

REM contains the Rhino Element Manager used to manage Rhino cluster installations, but besides that REM also can contain plugins. Currently the plugins for the VoLTE solution are:

  • sentinel-volte-element-manager

  • sentinel-gaa, also known as the NAF filter

  • sis-em

The sentinel-volte-element-manager contains:

  • Provisioning REST API

  • XCAP Server

  • Specific functionalities to manage and configure Sentinel Features

A REM upgrade can be used to upgrade any or all of those. The only requirement is that they should be compatible among themselves and the REM package should be compatible with the installed Rhino version.

Creating a REM upgrade package

Before start:

  • get the new REM version you want to create the upgrade package

  • get the required plugins

  • make sure the plugins are compatible by testing them together

  • make sure the plugins are compatible with the installed products (sentinel-volte-element-manager and the NAF filter).

The orca bundler provide the features to create a REM upgrade package by including the REM and any optional plugin.

To create an upgrade package do:

  • get the latest version of orca-bundler.zip from operational-tools

  • decompress it

  • run generate-orca-bundle rem-upgrade to bundle REM and plugins together.

$ unzip orca-bundler.zip
$ ./generate-orca-bundle rem-upgrade --rem-package <the REM package> --plugin <plugin one> --plugin <plugin two>

The --rem-package and --plugin are optional, but at least one of them should be given to generate a bundle and the --plugin parameter can be as many as necessary.

Optionally, a custom package can be added to the upgrade bundle, using the --custom-package parameter. During the upgrade procedure the custom packages will be installed if they are present in the packages.cfg. This will take place after all the other upgrade steps have taken place, but before Tomcat is restarted.

Note
Custom packages

The custom packages must conform to the following specification:

  • be a .zip file

  • contain an executable file named install (no extension) in the root directory of the zip.

orca extracts the packages to a temporary directory on the REM node and runs the install script from this directory. It calls the install script with one or two command-line parameters. The first one is the full path to the REM home directory. The second one is an optional properties file provided when the upgrade is run, using the --custom-package-properties property.

It is recommended that the package developer edits the README in the upgrade bundle to include the extra details on the custom package(s).

Applying the upgrade to REM

Applying an upgrade to a production environment is simple. The normal steps are:

  • create the upgrade bundle for REM

  • copy the upgrade bundle for REM to the management host

  • ensure the management host has ssh access to the cluster hosts on which Rhino Element Manager is installed

  • decompress the upgrade bundle

  • read the README for instructions and for details of the upgrade

Running an upgrade for REM

Run

./orca --hosts remhost1,remhost2 upgrade-rem packages

In summary orca will:

  • check the connection to the hosts

  • For each host

    • check for the existence of any plugins, match them against the specified ones and ask for confirmation to proceed

    • backup any files to be changed by the upgrade (package and/or plugins)

    • stop the web-server on that host

    • upgrade the package and plugins as appropriate

    • optionally install any custom packages (if included in the upgrade bundle)

    • restart the web-server on that host

You can skip the prompt questions by passing the --no-prompt parameter.

./orca --hosts remhost1,remhost2 upgrade-rem packages --no-prompt

The default backup path is $HOME/rem-backup. You can specify another one using the parameter --backup-dir

./orca --hosts remhost1,remhost2 upgrade-rem packages --no-prompt --backup-dir <other backup dir>

If the included custom package requires a properties file, it can be passed in using the --custom-package-properties parameter.

./orca --hosts remhost1,remhost2 upgrade-rem packages --custom-package-properties file.properties

To rollback the upgrade

To rollback to the most recent backup:

./orca --hosts remhost1,remhost2 rollback-rem

To rollback to a specific backup (must be done on each host separately):

  • First identify the backup ID to roll back to by examining the status output.

./orca -H remhost1 status
...
backups =
  (#1) 20180117-123500 contains REM:2.6.1.0 Plugins:sentinel-gaa-em-2.8.0.1,sentinel-volte-element-manager-2.8.0.1
...

The backup ID is the number in brackets, without the # (so 1 in the above example).

  • Then, run the rollback-rem command with the --target <backup ID> parameter:

./orca --hosts remhost1 rollback-rem --target 1

To cleanup old backups

  • First identify the backup ID(s) to clean up by examining the status output.

./orca --hosts remhost1 status
...
backups =
  (#1) 20180117-123500 contains REM:2.6.1.0 Plugins:sentinel-gaa-em-2.8.0.1,sentinel-volte-element-manager-2.8.0.1
...

The backup ID is the number in brackets, without the # (so 1 in the above example).

  • Then, run the cleanup-rem command:

./orca --hosts remhost1 cleanup-rem --backups 1,2

The backups are specified as a comma-separated list of integers without any spaces.

This procedure must be done on each host separately.

Example of rhino-element-manager upgrade

This example shows an upgrade from rhino-element-manager-1.5.0.4 to rhino-element-manager-1.5.0.5

$ ./orca --hosts vm-makara-01 upgrade-rem --rem-package rhino-element-manager-1.5.0.5.zip

Verify connections

Verifying connection to vm-makara-01
Applying REM upgrade rhino-element-manager-1.5.0.5.zip to host vm-makara-01
Running command on host vm-makara-01: upgrade-rem 'tmpj2xCvO'

Check for specific plugin and ask for confirmation

These plugins are installed but no replacement appears to be specified: volte-sentinel-element-manager-2.7.0.9.em.jar
Continue anyway? (y/N):
y

Unzip the package and backup the existing installation

Archive:  /opt/tomcat/apache-tomcat-8.5.29/webapps/rem.war
 inflating: /tmp/remupg.OOPq5R/old/WEB-INF/classes/log4j.properties
Archive:  /tmp/remupg.OOPq5R/rhino-element-manager-1.5.0.5/admin/resources/rem.war
 inflating: /tmp/remupg.OOPq5R/new/WEB-INF/classes/log4j.properties
Stopping Tomcat...
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-8.5.29/temp
Using JRE_HOME:        /opt/java/jdk1.8.0_162
Using CLASSPATH:       /opt/tomcat/apache-tomcat-8.5.29/bin/rem-rmi.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/tomcat-juli.jar
Using CATALINA_PID:    /opt/tomcat/apache-tomcat-8.5.29/tomcat.pid
Tomcat stopped.
Backing up previous version of REM to /home/rhino/rem-backup/20180618-135604...

Install the new version

Installing new version of REM...
Installation complete. Restarting Tomcat...
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-8.5.29/temp
Using JRE_HOME:        /opt/java/jdk1.8.0_162
Using CLASSPATH:       /opt/tomcat/apache-tomcat-8.5.29/bin/rem-rmi.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/tomcat-juli.jar
Using CATALINA_PID:    /opt/tomcat/apache-tomcat-8.5.29/tomcat.pid
Tomcat started.
REM upgrade completed.
Done on vm-makara-01

Example of rhino-element-manager plugin upgrade

This example shows an upgrade from volte-sentinel-element-manager-2.7.0.8.em.jar to volte-sentinel-element-manager-2.7.0.9.em.jar

$ ./orca --hosts vm-makara-01 upgrade-rem --plugin upgrade/volte-sentinel-element-manager-2.7.0.9.em.jar

Verify connections

Verifying connection to vm-makara-01
Applying REM upgrade Plugins:volte-sentinel-element-manager-2.7.0.9.em.jar to host vm-makara-01
Running command on host vm-makara-01: upgrade-rem 'tmpOMDvj0'

Backup the current installation and install the new version

Stopping Tomcat...
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-8.5.29/temp
Using JRE_HOME:        /opt/java/jdk1.8.0_162
Using CLASSPATH:       /opt/tomcat/apache-tomcat-8.5.29/bin/rem-rmi.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/tomcat-juli.jar
Using CATALINA_PID:    /opt/tomcat/apache-tomcat-8.5.29/tomcat.pid
Tomcat stopped.
Installing plugins...
Installation complete. Restarting Tomcat...
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-8.5.29
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-8.5.29/temp
Using JRE_HOME:        /opt/java/jdk1.8.0_162
Using CLASSPATH:       /opt/tomcat/apache-tomcat-8.5.29/bin/rem-rmi.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/bootstrap.jar:/opt/tomcat/apache-tomcat-8.5.29/bin/tomcat-juli.jar
Using CATALINA_PID:    /opt/tomcat/apache-tomcat-8.5.29/tomcat.pid
Tomcat started.
REM upgrade completed.
Done on vm-makara-01
Previous page Next page