Below are troubleshooting steps -- symptoms, diagnostic steps, and workarounds or resolutions -- for the Rhino management database persistent store.
Management Database Server Failure
Rhino uses either the PostgreSQL or Oracle Database Server as a non-volatile storage mechanism. It is possible that the database server can terminate due to administrative action or software failure. Failure to communicate with the database server may also occur due to a network fault. This is a problem related to the database only and is not indicative of a problem in Rhino.
A Rhino cluster gracefully handles the failure of a management database server, however if all nodes in the cluster fail and are restarted before the database server is restarted, then the cluster will pause in its booting process until a server is available. If all nodes in the cluster fail during the period the database server was unavailable, then any configuration or profile changes made after the database failure will be lost.
Symptoms
Warning messages in the system console from the log key memdb.storer
Messages similar to the following indicate that a PostgreSQL Database Server has terminated either due to administrative action or failure.
...
WARN [memdb.storer]
<SQLStableStorage Storer thread for rhino_profiles>
[ProfileDatabase/rhino_profiles] Connection failed
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:201)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:329)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:239)
...
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:243)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1122)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:175)
... 8 more
WARN [memdb.storer]
<SQLStableStorage Storer thread for rhino_management>
[ManagementDatabase/rhino_management] Connection failed org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:201)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:388)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:329)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:239)
...
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:243)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1122)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:175)
... 8 more
...Resolution and Mitigation
The management database server should be restarted as soon as possible. Once the database has been restarted and the Rhino cluster has reconnected, messages similar to the following should appear in the Rhino logs.
...
INFO [memdb.storer] <SQLStableStorage Storer thread for rhino_profiles>
[ProfileDatabase/rhino_profiles] Connected to database
INFO [memdb.storer] <SQLStableStorage Storer thread for rhino_management>
[ManagementDatabase/rhino_management] Connected to database
...To reduce the risk of this causing data loss Rhino can be configured to replicate persisted data across multiple PostgreSQL instances. When using Oracle as the persistence database, replication is provided by the Oracle RAC clustering mechanism.
