Besides the information on the console, orca provides detailed output of the actions taken in the log file. The log file by default is located on the host that executed the command under the path logs.

orca can’t connect to the remote hosts

Check if the trusted connection via ssh is working. The command ssh <the host to connect to> should work without asking for a password.

You can add a trusted connection by executing the steps below

  • Create SSH key by using default locations and empty passphrase. Just hit enter until you’re done

ssh-keygen -t rsa

  • Copy the SSH key onto all VMs that need to be accessed including the node you are on. You will have to enter the password for the user

ssh-copy-id -i $HOME/.ssh/id_rsa.pub sentinel@<VM_ADDRESS>

where VM_ADDRESS is the host name you want the key to be copied to.

To check run:

ssh VM_ADDRESS

It should return a shell of the remote host (VM_ADDRESS).

Tomcat fails to restart

Sometimes Tomcat can fail to restart after an upgrade or rollback. This can be due to a number of different reasons.

Stuck Tomcat process

If you see the following output:

Tomcat appears to still be running with PID <pid>. Start aborted.
If the following process is not a Tomcat process, remove the PID file and try again:
<process information>

it means Tomcat failed to shut down.

To resolve:

  • Log into the host using ssh as the same user that orca uses.

  • Verify the process still exists by running ps -f <pid>, where <pid> is the PID given in the error message:

    • if there is no process with that PID then Tomcat has since exited and you can retry the upgrade or rollback

    • if there is a Tomcat process with that PID, then kill it (see instructions below)

    • if there is another process with that PID, refer to the Stuck pid file section below.

You can identify Tomcat processes by looking for org.apache.catalina.startup.Bootstrap start in the output of ps -f.

To kill a Tomcat process:

  • first try to kill it gracefully with kill <pid>

  • wait for one minute

  • check if the process still exists with ps -f <pid>

  • if the process has not exited, use kill -9 <pid> to forcibly terminate it.

Stuck pid file

If you see the following output:

Unable to remove or clear stale PID file. Start aborted.

or there is a non-Tomcat process running with what Tomcat thinks is its PID, then follow these steps.

  • Log into the host using ssh as the same user that orca uses.

  • Check for running Tomcat processes using ps -ef | grep catalina | grep -v grep. If there are any results, kill them using the procedure detailed above.

  • Verify the file defined by the CATALINA_PID variable in your bin/setenv.sh exists and that the current user can read from and write to the file.

  • Delete this file using rm <file>.

Low /dev/random entropy leading to slow start

If orca reports that Tomcat is started but the REM web service is unavailable in-browser for several minutes, this is normally caused by low /dev/random entropy.

  • Examine the log file logs/catalina.out in your Tomcat installation and look for the following line:

INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [<time>] milliseconds.

If you see large values for <time> (over 10000 milliseconds), then you have low entropy in /dev/random. This occurs when your server is performing a lot of cryptographic operations, such as those required by REM when it starts up, and so is often seen if REM is restarted more than once in a short period.

The fix is to change the entropy source to /dev/urandom as follows:

  • Stop Tomcat: cd to the directory where Tomcat is installed and run bin/catalina.sh stop

  • Edit the file bin/setenv.sh

  • Add -Djava.security.egd=file:/dev/./urandom to the JAVA_OPTS variable and save the file

    • If the variable is not defined in that file, add it on a new line: JAVA_OPTS=-Djava.security.egd=file:/dev/./urandom

  • Restart Tomcat: bin/catalina.sh start

  • Verify that Tomcat is now available quickly (under a minute).

If you are finding REM is restarting often, examine the catalina.out logs for crashes and raise with your Customer Care representative.

Previous page