A key/value store is an adjunct to a Rhino in-memory database that serves to persist local database state to an external database such that another node in the Rhino cluster can recover that state in the event of a node failure.
It provides a form of state replication without the state being directly replicated between Rhino cluster nodes such as with the traditional savanna replication method.
High-level usage
In normal operation in a stable Rhino cluster, a key/value store has a data flow in only one direction — from the local in-memory database to the external database.
In the event of a Rhino node failure, the activities and SBB entities that were persisted to the external database by a key/value store on that node remain in the database but become "non-resident", which simply means that that application state still exists but is not currently owned by or resident in any Rhino node.
Subsequently, another node in the cluster may receive a redirected network event for a call or session that was previously owned by the failed node. As the node will not immediately recognise the call/session, it queries the key/value store to determine if there is any state for the call/session in the external database. If state is found, the node can adopt ownership of the call/session, restore the call/session state, and continue processing the event as if the call/session had always been locally known.
As the event is processed, further state related to the call/session may need to be retrieved from the external database, for example attached SBB entity state and the state of other activities that these SBB entities may be attached to.
A key/value store implementation typically operates with a delay between when transactional data is committed to the in-memory database and when it gets written to the external database. This allows techniques such as write buffering and combining to be utilised to reduce bandwidth needs by discarding all but the latest update to a database key observed over a short period of time. However it also means that transactions that occurred within this delay period right before a node failure will not have been persisted to the external database and will be permanently lost as a result.
