Operational Tools 1.1.0 :: Operational Tools :: Applying patches to production system

On this page...

Production Patches
Getting the available patches
Applying the patch in production environments
Troubleshooting
- Running out of Perm Gen space
Example of patch execution on a 3 node cluster

Production Patches

For live systems the patch is shipped with the orca tool that handles the cluster migration workflow. Internally, the orca tool will call the slee-patch-runner and the actions will be the same as explained in What the patch runner does. The difference here is that the patch will be applied against the first node of the new cluster created by the orca tool.

A production patch is encapsulated in a zip file and should follow the naming convention:

<product name>-<product version>-patch-bundle-<date and time>-<english code name>.zip

Example: volte-2.7.0.4-patch-bundle-20180517-1403-cdiv-timers.zip

The patch file is self contained and has the artifacts and the necessary tools to apply the patch. For consistency, all patch contents are contained within a directory called "patch" within the zip file.

The contents of a patch bundle file are:

Item	Description
README	Contains the information about the patch: how to use it, what components it changes, and what problems it fixes
orca	The orca tool
helpers directory	Contains set of scripts used by orca
resources directory	Contains the properties used by orca
licenses directory	License information for third party libraries used by the patch runner
the patch zip file	The patch package

Item

Description

README

Contains the information about the patch: how to use it, what components it changes, and what problems it fixes

orca

The orca tool

helpers directory

Contains set of scripts used by orca

resources directory

Contains the properties used by orca

licenses directory

License information for third party libraries used by the patch runner

the patch zip file

The patch package

Getting the available patches

The released patches are published in the Metaswitch artifactory and it’s necessary to have the proper credentials to access those.

Applying the patch in production environments

Applying a patch to a production environment is simple. The normal steps are:

download the patch bundle to the management host
assure the management host has ssh access to the cluster hosts
decompress the patch
read the README for instructions and for details of the patch
- what the patch fixes
- how to install and uninstall
- the actions the patch does
verify that you have a Rhino running with the installed product, i.e Sentinel VoLTE
run

*./orca --hosts host1,host2,host3 apply-patch <path to the patch zip file>*

In summary orca will:

check the connection to the hosts
clone the cluster
migrate the first node
apply the patch against the first node calling the slee-patch-runner and execute the same actions as explained here
migrate the other nodes of the cluster

Applying a patch requires administrative access to the Rhino running the product to be patched.

To keep the patches organized we recommend creating a specific directory in which to decompress the patches.

The patch has to stop all the SLEE services that the component is bound to. In a cluster environment this means the services will be deactivated cluster wide.

To revert the patch

*./orca --hosts host1,host2,host3 revert-patch <path to the patch zip file>*

To rollback the installation

*./orca --hosts host1,host2,host3 rollback -f*

To delete an old installation

*./orca --hosts host1,host2,host3 cleanup --cluster <cluster id>*

Troubleshooting

See Troubleshooting.

Running out of Perm Gen space

Installing a patch is actually a pretty complex operation as far as the Java virtual machine is concerned, and with patches that impact lots of the system, there is a chance that the operation will halt with an exception that describes the system as having run out of Perm Gen space. This is particularly likely if a number of patches are applied without restarting the Rhino between each one, as the loss of Perm Gen space can build up over time.

Unfortunately there is no recovery from this situation (beyond restoring the backup you took before starting the patch operation). To avoid this, it is advisable to stop and restart Rhino before starting the patch operation, which ensures that it has the maximum headroom of Perm Gen space available during the patching operation.

Example of patch execution on a 3 node cluster

Applying the patch with the command

./orca --hosts rhino-vm1,rhino-vm2,rhino-vm3 apply-patch volte-2.7.1.5-patch-20181017-0553-time-stamps-fix.zip --no-pause

Starting on host rhino-vm1
Checking for prepared Rhino clusters
Done on rhino-vm1

Starting on host rhino-vm2
Checking for prepared Rhino clusters
Done on rhino-vm2

Starting on host rhino-vm3
Checking for prepared Rhino clusters
Done on rhino-vm3

Doing Prepare
Copying the database
Done on rhino-vm1

Starting on host rhino-vm2
Doing Prepare
Done on rhino-vm2

Starting on host rhino-vm3
Finished running 236 post-patch checks.
Patch was applied successfully.

Done on rhino-vm1
Starting on host rhino-vm2
Doing Migrate

Stopping node 102 in cluster 109
Waiting up to 120 seconds for calls to drain and SLEE to stop on node 102
Rhino has exited.
Successfully shut down Rhino on node 102. Now waiting for sockets to close...
Starting node 102 in cluster 110
Started Rhino. State is now: Running
Waiting for Rhino to be ready for 75 seconds
Started node 102 successfully
Done on rhino-vm2

Starting on host rhino-vm3
Doing Migrate
Stopping node 103 in cluster 109
Waiting up to 120 seconds for calls to drain and SLEE to stop on node 103
Rhino has exited.
Successfully shut down Rhino on node 103. Now waiting for sockets to close...
Starting node 103 in cluster 110
Started Rhino. State is now: Running
Waiting for Rhino to be ready for 41 seconds
Started node 103 successfully
Done on rhino-vm3

Querying status on hosts [rhino-vm1, rhino-vm2, rhino-vm3]
Global info:
Symmetric activation state mode is currently enabled
Status of host rhino-vm1
Clusters:
 - volte-2.7.1.5-cluster-109
 - volte-2.7.1.5-cluster-110 - LIVE
Live Node:
Rhino node 101: found process with id 23202
Node 101 is Running
Rhino version='2.6', release='1.2', build='201807050952', revision='c5bfb8e'
Exports:
 - volte-2.7.1.5-cluster-109

License information:
1 valid license (of 1 installed), expiry date Fri Nov 09 17:08:54 UTC 2018
Licensed Rhino version(s): 2.*, Development
Java:
found = live cluster
java_home = /opt/java/jdk1.7.0_79
version = java version "1.7.0_79", Java(TM) SE Runtime Environment (build 1.7.0_79-b15), Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
OS:
python = 2.7.12 (default, Nov 20 2017, 18:23:56) , [GCC 5.4.0 20160609]
version = Linux version 4.4.0-130-generic (buildd@lgw01-amd64-039) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018

Services:
name=IM-SSF vendor=OpenCloud version=1.4.7
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5-copy#1
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5
name=sentinel.registrar vendor=OpenCloud version=2.7.1
name=sentinel.registrar vendor=OpenCloud version=current
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1
name=volte.sentinel.sip vendor=OpenCloud version=current
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1
name=volte.sentinel.ss7 vendor=OpenCloud version=current
Status of host rhino-vm2
Clusters:
 - volte-2.7.1.5-cluster-109
 - volte-2.7.1.5-cluster-110 - LIVE
Live Node:
Rhino node 102: found process with id 13391
Node 102 is Running
Rhino version='2.6', release='1.2', build='201807050952', revision='c5bfb8e'
Exports:

License information:
1 valid license (of 1 installed), expiry date Fri Nov 09 17:08:54 UTC 2018
Licensed Rhino version(s): 2.*, Development
Java:
found = live cluster
java_home = /opt/java/jdk1.7.0_79
version = java version "1.7.0_79", Java(TM) SE Runtime Environment (build 1.7.0_79-b15), Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
OS:
python = 2.7.12 (default, Nov 20 2017, 18:23:56) , [GCC 5.4.0 20160609]
version = Linux version 4.4.0-128-generic (buildd@lcy01-amd64-019) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018

Services:
name=IM-SSF vendor=OpenCloud version=1.4.7
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5-copy#1
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5
name=sentinel.registrar vendor=OpenCloud version=2.7.1
name=sentinel.registrar vendor=OpenCloud version=current
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1
name=volte.sentinel.sip vendor=OpenCloud version=current
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1
name=volte.sentinel.ss7 vendor=OpenCloud version=current
Status of host rhino-vm3
Clusters:
 - volte-2.7.1.5-cluster-109
 - volte-2.7.1.5-cluster-110 - LIVE

Live Node:
Rhino node 103: found process with id 31276
Node 103 is Running
Rhino version='2.6', release='1.2', build='201807050952', revision='c5bfb8e'

Exports:

License information:
1 valid license (of 1 installed), expiry date Fri Nov 09 17:08:54 UTC 2018
Licensed Rhino version(s): 2.*, Development

Java:
found = live cluster
java_home = /opt/java/jdk1.7.0_79
version = java version "1.7.0_79", Java(TM) SE Runtime Environment (build 1.7.0_79-b15), Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

OS:
python = 2.7.12 (default, Nov 20 2017, 18:23:56) , [GCC 5.4.0 20160609]
version = Linux version 4.4.0-128-generic (buildd@lcy01-amd64-019) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018

Services:
name=IM-SSF vendor=OpenCloud version=1.4.7
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5-copy#1
name=sentinel.registrar vendor=OpenCloud version=2.7.1.5
name=sentinel.registrar vendor=OpenCloud version=2.7.1
name=sentinel.registrar vendor=OpenCloud version=current
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.sip vendor=OpenCloud version=2.7.1
name=volte.sentinel.sip vendor=OpenCloud version=current
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5-copy#1
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1.5
name=volte.sentinel.ss7 vendor=OpenCloud version=2.7.1
name=volte.sentinel.ss7 vendor=OpenCloud version=current

Available actions:
  - prepare
  - prepare-new-rhino
  - cleanup --clusters
  - cleanup --exports
  - rollback

Previous page Next page