Rhino Changelog

Fixed the parsing of a configured SNMP notification target to allow the - character in the middle of the hostname. (#423328)

Fixed a memory leak in the key/value store that occurred if a non-transactionally started activity was removed in its first generational update (for example, if no SBB processed any events fired on an activity during its lifetime) or if an activity remove transaction completed committing before a previous update transaction to the same activity completed committing. Also added a garbage collector into the key/value store that will clean up this kind of leaked state if it happens in other unknown circumstances. New configuration options for this garbage collector have been added in rhino-config.xml. (RHI-6724)

Fixed another issue where connections to the LDAP server are left open indefinitely, in this case only after an authentication failure while using search mode. (RHI-6585)

Added a system property ldap.casesensitive in LDAP login module to allow case-insensitive search. This property defaults to true to match prior behaviour. (RHI-6606)

Fixed a defect in the LDAP login module when groups without members caused authentication attempts to fall-back to a file based authentication (RHI-6607)

Fixed an issue where connections opened by Rhino to the authentication LDAP server are left open indefinitely. (RHI-6464)

Fixed a concurrency issue in profile table queries operating on non-indexed profile attributes that could lead to SBB transaction failure due to a javax.slee.NoSuchObjectLocalException being thrown mid-query if a profile had been concurrently removed by another transaction (RHI-6429).

Fixed a race condition in Session Ownership Activity end that could cause an UnrecognizedActivityHandleException if the activity became unreferenced while outstanding requests were still in progress (RHI-6355).

Fixed installer handling of defaults for JAVA_HOME and LOCALIPS. Removed non-settable values from defaults file. (RHI-5973)

Fixed a node hang if a service was started on a node after it had already been started on other nodes but the replicated ServiceActivity for that service no longer existed. (RHI-5987)

JVM GC log files are now be rotated rather than overwritten when a node in a production Rhino install is restarted via the keep alive loop. Previously rotation was only happening during the initial boot sequence. (RHI-6181)

Fixed two bugs identified in SessionOwnershipActivity creation. The invoking SAS trail was not propagating to the newly created SessionOwnershipActivity. A race could occur between the create, attach, and querying the Session Ownership Store and a Session Ownership event being delivered. This could result in the first Session Ownership event being delivered without invoking the attached SBB entity. (RHI-6211)

Fixed an untended retention of some OCBB state when idle OCBB persistent state objects were stored in their object pool. (RHI-6293)

It is now possible to change the state of service, resource adaptor entity, or the SLEE on a non-existent cluster node where the node had previously terminated when the stateful entity was in transient activation state such as STOPPING (RHI-5852).

Session Ownership Activities will now implicitly end in the SLEE if they are unreferenced by SBB entities, SLEE facilities, etc (ie. in the same way that Null Activities can implicitly end). An SBB must attach to a Session Ownership Activity or create a reference to it elsewhere, for example by using the Activity Context Naming Facility to bind it to a name if it wishes to keep the activity alive. Session Ownership Activities can of course still be ended explicitly by using the endActivity() method. (RHI-6200)

Fixed a race condition during service activation between the start of the replicated ServiceActivity and ResourceAdaptor.serviceActive() callbacks being made on the nodes not involved with the creation of the ServiceActivity. The race condition meant that a resource adaptor entity responding to the callback could fire an event to an SBB that attempted to attach to the ServiceActivity, but could not do so because the activity was not yet visible on the node. Rhino will now wait and ensure that the required replicated activity state exists on the node before proceeding to make the resource adaptor callbacks. (RHI-5196)

Fixed PolledMemoryAppender resource leak which occurred on failed socket binding attempts. (RHI-5788)

The ServiceActivityFactory will no longer return a null ServiceActivity in cases where the activity has not yet started or has already ended when the activity object is requested. A FacilityException will be thrown instead. (RHI-5843)

Fixed a NullPointerException in the SAS configuration manager that occurred in a booting node if some other node updated SAS config state before the booting node had installed the initial SAS config. (RHI-5845)

Fixed a minor key/value store bug where it sometimes did not release a reference count to state discarded after an application disabled state persistence. This caused the state object(s) to end up as JVM heap garbage rather than being returned to their corresponding object pools for reuse. (RHI-5847)

Fixed various issues with memdb internal sizing statistics:

Fixed the cyclic raising and clearing of alarms that occurred when a key/value or session ownership store’s Cassandra connection lost quorum and was unable to execute queries. (RHI-5853)

Fixed the corruption of per-node service activation state that occurred for services with an install level of VERIFIED when symmetric activation state mode was enabled. (RHI-5856)

Fixed an SBB lifecycle management issue in the event router which could cause SBB entities to have stale internal state while handling an event due to sbbLoad() not being invoked on them first when it should have been. (RHI-5867)

Fix a case where the key/value store could persist stale generations to the backend database, overwriting more recent state. (RHI-5868)

Fixed a regression in the Cassandra Key/Value Store’s global stats parameter set persistTime sample stat introduced in Rhino 2.6.2.3. The sample stat will now report total persist-phase time as intended rather than the persistence time for individual sets of related transactions. (RHI-5879)

The DeploymentMBean.setEnvEntries method will no longer throw an exception if given an unmodifiable map as an argument. (RHI-5929)

JVM GC log files will now be rotated rather than overwritten when a node in a production Rhino install restarts. (RHI-5821)

A "ZERO-length generation snapshots" warning message that could occur during some activity-related housekeeping operations has been demoted to a debug level message as it’s completely benign. (RHI-5894)

Fixed a miscount that caused the key/value store table statistic KeyUpdatesPending to become negative in some cases when an application disabled state persistence during a session. (RHI-5836)

The creation of a remote timer during a transaction is now handled asynchronously during commit rather than synchronously. This allows the event router thread to be released back to the pool faster, increasing event router performance. (RHI-5839)

Concurrency improvements have been made to memdb to reduce lock contention under heavy load, thereby improving performance. (RHI-5840)

Fixed a memory leak in the key/value store when a database entry became implicitly persist-disabled, due to being related to another persist-disabled transaction, after the entry had been removed from the database but before the updates for that key had been considered for persistence by the key/value store. (RHI-5838)

Add a new production safe debug log key to MemDB - memdb.mvcc.summary. This log key provides visibility into MemDB’s Multi-Version Concurrency Control sweeping. (RHI-5835)

Fixed an AssertionError that occurred during key/value store persistence if, for some database key X:

Fixed a key/value store bug that resulted in updates awaiting persistence for any given key that appeared in multiple memdb tables to interfere with each other. This could have resulted in state being persisted to the wrong backend database table, or not being persisted at all. (RHI-5831)

Fixed a memdb / key/value store integration issue that meant that memdb could miss a deregistration of transaction interest callback from the key/value store when read-only transactions (from the perspective of a memdb instance) committed. These callbacks are critically important to the proper functioning of memdb’s internal garbage collection. (RHI-5832)

Improved the performance of determining the earliest memdb generation that a key/value store was still referencing. The previous algorithm could lead to unexpected CPU load increases under certain load conditions. (RHI-5826)

Fixed an event router infinite looping bug that could occur if an SBB threw a runtime exception from a lifecycle callback method such as sbbLoad() or sbbStore(). (RHI-4339)

Updates for stats client sessions with a very small polling period are now rate limited to avoid saturation of cluster communications, which has the potential to lead to group heartbeat failures and node loss. (RHI-5779)

Fixed potential active namespace corruption issues when a management client invokes concurrent management operations that can affect the currently active namespace. (RHI-5780)

Fixed an initialisation error in the Cluster State Change Facility implementation that prevented resource adaptor entities from receiving correct cluster state information if symmetric activation state mode was in force when a node started. (RHI-5802)

Fixed an AssertionError that occurred in the key/value store in some scenarios that included persist-disabled SBB entity trees. (RHI-5823)

Changes to the periodic alarm reporting period will now be reflected across all cluster nodes immediately rather than requiring a restart of nodes not handling the change request in the first instance before the change is applied. (RHI-3511)

Fixed a key/value store pending write scanner issue that caused it to select some transactions for persistence when they should have inherited persist-disabled status from related transactions. (RHI-5793)

Fixed a key/value store issue that caused memdb to return state from the key/value store that was not identifiable as more recent than any other previously retrieved state for the same key, which meant that OCBB objects that had cycled through the object pool could end up being used with stale CMP state. (RHI-5653)

Fixed a deadlock that could occur in the remote timer server when a node leaves the cluster, leading to the node where the deadlock occurs also failing and leaving the cluster a short time later. (RHI-5754)

Fixed remote timer server behaviour handling connections to nodes that fail without shutting down their TCP connections cleanly. Connections will now be closed if a timer event report is not acknowledged within the required timeout period so that normal failover handling will occur for subsequent timer events destined for the same node. (RHI-5756)

Changed SessionOwnershipActivity.endActivity() behaviour so that it won’t end the activity until all in-flight result events have been queued. (RHI-5759)

Using the housekeeping operation to remove an activity context name binding from a non-resident activity will now check that the activity pkey supplied matches with the binding, rather than just blindly removing the binding regardless of what activity pkey is provided. (RHI-5763)

Fixed a key/value store transaction scanning issue that caused transactions committed out of natural order with mixed persistence-enabling flags to become stuck and never persisted, leading to eventual out of memory errors. (RHI-5766)

Fixed key/value store persistence issues when committed state that was selected for persistence was concurrently GC’ed by it’s source - either a memdb or activity handler instance. Activity handler persistence failures in particular led to a permanent jam in the key/value store that created a memory leak and eventual heap exhaustion. (RHI-5777)

Fixed a key/value store memory leak that occurred when an update scheduled for external database persistence was discarded rather than persisted due to the transaction batch containing a complete create/remove cycle. (RHI-5784)

The deprecated configuresas command has been removed from rhino-console. (RHI-5420)

The Session Ownership Facility tryAdopt methods will now execute exclusively in the calling thread rather than also utilising the facility’s thread pool for various intermediate functions. (RHI-5739)

Implemented a new cluster monitor that will raise an alarm if it detects other event-router nodes have a system clock that differs from the local node more than a configured threshold. (RHI-5762)

The rhino-console getactivityinfo command now includes any activity context name bindings associated with the activity in the output, and the findactivitybindings command no longer truncates long name binding names. (RHI-5764)

The description of alarms generated by Rhino’s Watchdog subsystem are now included in Rhino’s alarm catalog. (RHI-5649)

Fixed the initialisation, on node restart, of object pools used by in-memory database objects for the Activity Context Naming Facility, and their subsequent cleanup when namespaces are removed. (RHI-5751)

Fixed a key/value store issue that caused it to erroneously discard backing database entry removals in some cases when a key update that had already been persisted to the database became persist-disabled. This could later cause problems when state that should not exist was able to be retrieved from the database and reinstantiated into the in-memory database. (RHI-5752)

Fixed a NullPointerException that could manifest in the Cassandra key/value store during pending transaction scanning under certain load conditions. (RHI-5746)

SLEE timers can now be armed redundantly when using key-value store replication and external session ownership. Redundantly armed timers will be armed on one other cluster node and timer events will be directed to the node owning the SBB entity tree at the time of firing. (RHI-5479)

SBBs now have access to a Convergence Name Session Ownership Record, a special type of session ownership record tied to the convergence name of an SBB entity tree. Access to this record is available using the RhinoSbbContext API for SBBs and SbbPartContext API for SBB parts. (RHI-5517)

The resource adaptor entity Session Ownership Facility has new methods that allow a node to attempt the atomic adoption of Convergence Name Session Ownership Records. (RHI-5541)

Added a new resource adaptor extension property (defined in RhinoExtensionProperties) that indicates if the new session ownership record adoption support added in Rhino 2.6.2 is available and enabled in the namespace the resource adaptor is running in. A resource adaptor can use this property to determine if session ownership behaviour requiring this functionality should be enabled or disabled at runtime. (RHI-5673)

Added a new SBB/SBB Part API to permanently disable key/value store persistence of application-related state (as opposed to disabling initial persistence which was only ever intended to be a temporary postponement of state persistence). (RHI-5727)

Added a new built-in ServiceNodeActivity and ServiceNodeStartedEvent, similar to the SLEE-defined ServiceActivity and ServiceStartedEvent, but with the following differences:

The housekeeping operations that were previously available to find all non-resident activities and SBB entities from a key/value store have been deprecated and no longer perform that function. This is due to the fact that Cassandra copes poorly with such requests when the key/value store is being used in a production capacity.

Added MBean methods, console commands, and an Ant task to configure a service’s initial persistence status. (RHI-5296)

Improved the resiliency of the key/value store transaction scanning thread to scan failures. (RHI-5385)

The options for showing removed objects in the rhino-console findactivities and findsbbs commands have changed. In addition the option for showing non-resident objects with these commands has been deprecated and no longer performs any function. (RHI-5403)

Added a remove(key) operation to the resource adaptor entity Replicated Storage Facility that allows a table entry to be removed from storage without having to read that entry first. (RHI-5453)

Added a new maxPersistDeferredTransactionAge configuration option to the key/value store that specifies the maximum age that persist-deferred transactions will be retained for before persistence is forced upon them. (RHI-5455)

The Cassandra key/value store now has a configurable maximum cap on the amount of data that it will hold on to pending persistence to the backing database, with a default limit of 125% of served memdb instances capacity. (RHI-5469)

Added database read/write response times to key/value and session ownership store statistics. (RHI-5485)

The default Rhino transaction timeout used for most management operations has been increased from 2 minutes to 3 minutes. (RHI-5532)

The default read and write timeouts for the Cassandra key/value store and session ownership store have been increased to 2000ms. (RHI-5578)

Improved distributed lock use when managing object pool configurations during node start. (RHI-5588)

Improved how stats parameter sets and parameter set types are configured and registered with the internal SNMP stats manager so that these operations no longer use excessive system resources or continue for a prolonged period of time after node boot when a large number of parameter sets and types exist. (RHI-5590)

The distributed locks held while a service is activating have been revised to reduce the likelihood of lock timeouts occurring in other threads and/or other nodes when a service that takes a long time to activate is asked to do so by a management client. (RHI-5611)

The SNMP config update thread now sets an explicit transaction timeout based on the lock acquisition timeout period and number of possible retries it may do, to avoid transaction timeout when this time period exceeds the default transaction timeout period. The maximum number of retries is now also configurable by a system property. (RHI-5619)

Reduced the time taken to dynamically allocate SNMP counter mappings, such that very large statistics parameter sets that need counter mappings allocated will be now be registered significantly faster. (RHI-5621)

Added a -contains <xyz> filtering option to the rhino-console listlogkeys command. (RHI-5632)

SBB entity trees retrieved from a key/value store on failover will now be validated for completeness before event delivery. SBB entity trees that fail validation will not be delivered the event but rather be detached from all activities and removed from the database. (RHI-5644)

The size of the executor thread pool used by the session ownership store is now configurable in rhino-config.xml. (RHI-5664)

Increased default session ownership thread pool size from 5 to 30 to improve performance of synchronous operations (RHI-5740).

Added a -H option to rhino-stats that displays the hostname in the statistics, allowing stats to be tracked to their origin host. (RHI-5695)

Alarms are now raised by the Cassandra key/value and session ownership stores if a host in a Cassandra cluster the store is connected to unexpectedly leaves the cluster. (RHI-5737)

Updated SNMP MIB counter descriptors to be unique by prefixing the parameter set name. (RHI-4819)

The NodeHousekeepingMBean getSbbInfo operation will now no longer internally wrap an UnrecognizedSbbEntityException in a ManagementException. The UnrecognizedSbbEntityException will now be thrown unwrapped as intended. (RHI-5364)

Fixed a lock timeout issue that could cause a node to fail to restart if a large number of key/value store-enabled namespaces were configured in the cluster. (RHI-5422)

Fixed import failures when an import containing the setservicereplicationselectors task is run multiple times in succession. The task will now ignore errors due to the target service already being deployed if the intended replication selectors match with the current service settings. (RHI-5440)

Fixed NullPointerException when checking if a failed key/value store update session should be rechecked. (RHI-5444)

When a key/value store-replicated service is deactivated on all nodes where it is running, any nodes that fail during the draining period will have their outstanding SBB entity trees adopted by one of the remaining nodes before the drain completes. (RHI-5462)

Fixed a bug in the SAS configuration Ant task <configuresas> that caused it to remove the configured SAS servers if no updated SAS servers were specified. (RHI-5477)

The getSbbInfo and getSbbDiagnostics housekeeping commands will, when invoked against a non-resident SBB entity, no longer cause that SBB entity to become resident on the node running the management operation. In addition, the activity handles of non-resident attached activities will now be rendered correctly in the TabularData returned from these methods. (RHI-5488)

Fixed uncaught exception alarm caused by a permission check failure during log rollover. (RHI-5492)

The getSbbInfo, getSbbDiagnostics, getAttachedActivities, and removeSbb housekeeping operations all now have a new variant that includes a nonResident parameter, which must be set to true for the operation to interact with non-resident key/value store replicated SBB entities. This fixes issues where these operations were operating on state implicitly retrieved locally by the key/value store in the management node rather than the current in-memory state of the SBB entities owning node. (RHI-5495)

The Cassandra key/value store will now initialise and start properly if configured in rhino-config.xml with no configuration properties specified, ie. all default values will be assumed. (RHI-5500)

Fixed a race condition that could lead to a NullPointerException when attempting to flush key/value store state, eg. during resource adaptor entity deactivation or SLEE stopping. (RHI-5503)

Fixed a bug where SBB Parts were not releasing their component references to dependent event types when unverified. (RHI-5520)

Creating or deleting a session ownership record containing no additional keys now completes successfully rather than failing with a NoSuchElementException. (RHI-5521)

Fixed an IllegalArgumentException that occurred when trying to deploy a service containing an SBB that defined usage parameter interfaces using only the extension deployment descriptor and didn’t define any of the interfaces as the root usage parameter set type. (RHI-5525)

Fixed an issue with the resource adaptor entity Cluster State Change Facility where the facility on a given node would not report updates from new nodes added to the cluster in some circumstances. (RHI-5533)

Fixed a NullPointerException when attempting to set SBB tracer levels in a service with an install level of INSTALLED. The operation now fails with a more appropriate error. (RHI-5542)

Fixed the NodeHousekeepingMBean.getTimerInfo(timerID) operation so that it works correctly for timers set on key/value store replicated activities. (RHI-5545)

Cancelling a timer on a key/value store-replicated activity forcefully using a housekeeping operation will no longer leave a dangling transaction on the nodes where the timer is not present. (RHI-5547)

Fixed a NullPointerException that occurred during event processing if a replicated activity restored from a key/value store contained attached SBBs from services that were no longer deployed. (RHI-5570)

Fixed a concurrency issue that could lead to a NullPointerException in the statistics manager if parameters were subscribed or unsubscribed from a stats session that was already collecting data. (RHI-5572)

Fixed a NullPointerException when creating a SessionOwnershipRecord when a map that does not support null values (eg. java.util.TreeMap) was passed as an argument to the builder’s setAttributes method. (RHI-5581)

Fixed potential distributed lock timeouts in the deployed component file manager when large clusters are deploying complex applications. (RHI-5592)

Fixed possible distributed lock timeouts reading and/or updating per-node activation state during a concurrent restart of many cluster nodes. (RHI-5595)

Fixed non-replicated activity GC that could report a confusing IllegalStateException if it was triggered early enough during node boot. (RHI-5600)

Fixed a distributed lock manager race condition that could grant an exclusive lock to a waiting thread in error if an acquisition attempt on a highly contended lock timed out but was immediately retried, for example as happens in the SNMP configuration update thread during large parameter set configuration storms. This error could furthermore result in the lock becoming permanently jammed if the erroneously acquired lock was released before the lock was officially granted by the lock manager. (RHI-5614)

Cassandra database tables used in the key/value and session ownership stores will now garbage-collect tombstones after a one-hour grace period rather than the general database default of ten days. This period can be configured with the new tombstoneGCGracePeriod configuration property for the key/value and session ownership stores in rhino-config.xml. (RHI-5629)

Fixed an issue with the key/value store pending transaction scanner that caused it to miss some related transactions in certain circumstances, leading to those related transaction potentially being persisted out of order. (RHI-5631)

Fixed failover of the replicated SLEE-generated service activities so now they will only get adopted by nodes where the corresponding service is running. If a service is not running on any remaining nodes then the activity for that service is removed. This also fixes an AssertionError that would occur, for example, on node recovery when the restarting node considered itself to be the first node where a service was starting again and tried to recreate the service activity, but failed to do so because the activity already existed on another node where it shouldn’t. (RHI-5636)

Fixed an issue preventing removal of key/value store-replicated SBB entity trees that were broken after failover by incomplete or inconsistent state being available in the key/value store. (RHI-5643)

If a service was deactivated on the node owning the associated SLEE-generated service activity, and the service was still running on other cluster nodes, ownership of the service activity was not being reassigned to one of these other nodes, rather the activity remained owned by the deactivating node. This could cause future problems such as AssertionErrors, for example, if the other nodes were restarted. This issue is now fixed. (RHI-5658)

Profile specifications with large numbers of profile attributes will now deploy correctly. Previously a SLEE 1.0 profile specification with more than 255 profile attributes, or a SLEE 1.1 profile specification with more than 253 profile attributes, would unexpectedly fail at the code generation stage. (RHI-5659)

Fixed a security permissions issue that caused nodes to fail to restart when using JDK 1.7 if a profile table using the non-default pessimistic locking existed. (RHI-5676)

Fixed a problem with a service returning to the Inactive state when the service is deactivated on a node and the ServiceActivity, with replicated SBBs attached to it, needs to be reassigned to another node where the service is still running. (RHI-5682)

The configuration manager will now only initialise empty configuration types on boot of the first node in a virgin cluster (for global config types) or on namespace creation (for namespaced config types). This eliminates a class of bug that would, for example, cause a node booting into an existing cluster where all licenses had been manually uninstalled to reinstall the default platform license, but not tell other nodes about it, leading to inconsistent configuration state on the other nodes. (RHI-5689)

Fixed the inability to use some housekeeping operations against SBBs that perform JNDI lookups for resource adaptor entity bindings in setSbbContext() when the service they are contained in is inactive on the node performing the operation. (RHI-5691)

The Cassandra key/value and session ownership stores will now refrain from terminating database sessions on potentially transient connection issues such as query timeouts or single host failures. (RHI-5698)

The rhino-console listserviceralinks command will now also report resource adaptor entity links for SBB Part components in the service. In addition the command will also traverse link and shadow components to find all resource adaptor entity links for the service. (RHI-5699)

Fixed the rhino-console command completer for the third argument of the shadowcomponent command. (RHI-5701)

Fixed a memory leak that occurred in Rhino if a stats client started collection for a statistics session that had no stat parameter subscriptions. Also fixed a reconnection issue in rhino-stats where it failed to restart stats collection after certain connection failover scenarios, meaning the client didn’t output stats any more after the failover. (RHI-5717)

Fixed a NegativeArraySizeException error in rhino-stats when fetching statistics for which a subscription does not exist. (RHI-5731)

Fixed a sorting comparator bug in the rhino-console listactivealarms command that could lead to an IllegalArgumentException in particular dataset cases. (RHI-5732)

Added a new per-node configuration file rhino.java.security to override two Java security properties. This file specifies the following default values:

networkaddress.cache.ttl=30
networkaddress.negative.cache.ttl=10

Added new alarmType parameter (1.3.6.1.4.1.19808.2.102.14) to SNMP Alarm Notifications. This OID contains the Alarm Type listed in the alarm catelog. (RHI-5516)

Fixed rare BufferOverflowException triggered by concurrent SNMP counter access. (RHI-5493)

Fixed a bug where SBB parts were not releasing their component references to dependent event types when unverified. (RHI-5520)

Creating or deleting a session ownership record containing no additional keys now completes successfully rather than failing with a NoSuchElementException. (RHI-5521)

Fixed an IllegalArgumentException that occurred when trying to deploy a service containing an SBB that defined usage parameter interfaces using only the extension deployment descriptor and didn’t define any of the interfaces as the root usage parameter set type. (RHI-5525)

Fixed an issue with the resource adaptor entity cluster state change facility where the facility on a given node would not report updates from new nodes added to the cluster in some circumstances. (RHI-5533)

Fixed a NullPointerException when attempting to set SBB tracer levels in a service with an install level of INSTALLED. The operation now fails with a more appropriate error. (RHI-5542)

Fixed the NodeHousekeepingMBean.getTimerInfo(timerID) operation so that it works correctly for timers set on key/value store replicated activities. (RHI-5545)

Cancelling a timer on a key/value store-replicated activity forcefully using a housekeeping operation will no longer leave a dangling transaction on the nodes where the timer is not present. (RHI-5547)

Fixed a NullPointerException that occurred during event processing if a replicated activity restored from a key/value store contained attached SBBs from services that were no longer deployed. (RHI-5570)

Fix a NullPointerException when creating a SessionOwnershipRecord when a map that does not support null values (eg. java.util.TreeMap) was passed as an argument to the builder’s setAttributes method. (RHI-5581)

Fix a distributed lock manager race condition that could grant an exclusive lock to a waiting thread in error if an acquisition attempt on a highly contended lock timed out but was immediately retried, for example as happens in the SNMP configuration update thread during large parameter set configuration storms. This error could furthermore result in the lock becoming permanently jammed if the erroneously acquired lock was released before the lock was officially granted by the lock manager. (RHI-5614)

Added a remove(key) operation to the Replicated Storage Facility available to resource adaptors that allows a table entry to be removed from storage without having to read that entry first. (RHI-5453)

Add Rhino support for additional SAS event fields message_id and call_info_id. (RHI-5523)

Added a new built-in ServiceNodeActivity and ServiceNodeStartedEvent, similar to the SLEE-defined ServiceActivity and ServiceStartedEvent, but with the following differences:

Fixed NullPointerException when checking if a failed KV store update session should be rechecked. (RHI-5444)

Fixed permission issue that could result in lost log messages during a log rollover. (RHI-5456)

Fixed an IllegalAccessError exception that occurred when a failed timer event was refired. (RHI-5473)

Fixed abnormal execution alarm raised regarding log4j2 rolling file manager access permission. (RHI-5492)

Fixed a bug in the SAS configuration Ant task <configuresas> that caused it to remove the configured SAS servers if no updated SAS servers are specified. (RHI-5477)

New session ownership subsystem backed by Cassandra. See Replication Support Services for more information. (RHI-5018)

New "key/value store" replication mode for in-memory database state backed by Cassandra. See Replication Support Services for more information. (RHI-5012)

The DataStax Java Driver for the Cassandra database is now bundled with Rhino and support for Cassandra persistence instances has been added. (RHI-5007)

A new service extension deployment descriptor option has been added that allows a service to delay persistence of its initial transactional state changes to a key/value store’s external database until such time as the service elects to enable persistence. (RHI-5120)

New resource adaptor extension configuration properties have been added that provide a resource adaptor with knowledge about its replicated storage replication method and the availability of the session ownership subsystem. (RHI-5249)

A new DTD version of the oc-resource-adaptor-jar.xml extension deployment descriptor has been added that allows a resource adaptor to more extensively configure the options used during the creation of its replicated storage resources. (RHI-5266)

The replicated storage resource and availability of the session ownership subsystem have been added as configurable options when creating a user-defined namespace. The settings provided for a namespace are output to the Rhino log when the namespace is created or reinitialised on node restart. (RHI-5263)

The structure of rhino-config.xml has been updated slightly to support the new key/value store and session ownership configuration.

The Rhino production installer has been updated to include prompts for the initial configuration of the new optional key/value store and session ownership subsystems for Rhino’s default namespace. (RHI-5207)

Fixed an NPE that occurred trying to reassign to another node a service activity that had previously been administratively removed when the SLEE on the node that had owned the activity transitioned from RUNNING to STOPPING. (RHI-5322)

The activity handler txEnd and nonTxEnd statistics are now correctly recorded. (RHI-5336)

Fixed an internal corruption of metadata that occurred if a component referenced by more than one linked component had one of those linked components removed. This could have led to, for example, service activation callbacks for the remaining linked components not being made to resource adaptors when required by a target service component. (RHI-5347)

Fixed a lock manager issue that affected a deactivate-and-activate service operation when more than one service was being deactivated, and improved the resiliency of stopping tasks in general in the wake of unexpected errors during their execution. (RHI-5362)

Fixed some numerical overflow errors in stats-related classes that could lead to a NegativeArraySizeException in rhino-stats in some circumstances. (RHI-5375)

Fixed an off-by-one error when determining if a late SBB timer should generate a timer event when the PRESERVE_LAST timer option was used. The error meant that a timer event was not generated in the case the last repetition of the timer (or the only repetition, in the case of a non-periodic timer) fired late. (RHI-5379)

SLEE 1.1 profile table creation will no longer fail with an NPE if debug logging for the "profile.profile" log key is enabled. (RHI-5424)

Fixed exception when activating rhino-extensions examples service (RHI-5436)

Improved error handling for adding and removing SAS servers. (RHI-5301)

Stats parameter set descriptions output by rhino-stats -l now include a "Short name" column and the content of the "Label" column has been corrected. (RHI-5394)

Removed heap limit from rhino-console and rhino-import client tools to prevent OutOfMemoryError during install. (RHI-5418)

Added a new @CMPFieldReplication annotation usable by SBBs and SBB Parts that allows the replication of individual CMP fields to be completely suppressed. (RHI-5245)

SBBs and SBB Parts can now tag CMP fields with one or more names using the @CMPFieldTag annotation then programmatically iterate over tagged CMP fields using new methods in the CMPFields extension interface. A CMPField interface has also been added as an alternative access mechanism to individual CMP fields to facilitate this new feature. (RHI-5081)

Previously, SBB CMP fields using the @PassByReference annotation with scope PERMANENT could not access the SBB object’s encodable context. They now can. (RHI-5406)

Added a deregisterProfileMBeans command to rhino-console that allows rogue profile MBeans such as those for uncommitted profiles to be cleaned up. (RHI-5383)

The Rhino cluster ID value is now available directly, as a java.lang.Integer value, to resource adaptors, by way of a new extension configuration property, and SBBs and SBB Parts, via the java:comp/env/rhino/cluster-id JNDI name. (RHI-5240)

Added a new Cluster State Change Facility resource adaptor listener interface that provides more information about the cluster’s current state with respect to the original listener interface. (RHI-5239)

New logging subsystem based on Log4j 2. See the logging documentation.

SLEE Tracing now uses the logging subsystem directly.

Metaswitch Service Assurance Server (SAS) support. See the SAS tracing documentation.

Profile MBeans now isolate transactional state between different management clients. Changes being made by one client no longer leak into the view of other clients. (RHI-4317)

The configurestagingqueues Ant task no longer transposes maximumAge and maximumSize values. (RHI-4380)

Object pool stats are now grouped by namespace, and their parameter set names have been improved. (RHI-4429)

The various ProfileProvisioningMBean getProfiles* methods now respect the state of any user transaction currently in progress by the client. (RHI-4458)

Added ResourceExecutor stats parameter set type to TransactionManager. These statistics were previously being combined with SynchronizationExecutor. (RHI-4533)

Reworked the export of custom security policies assigned to components so that these are now set during import as part of the corresponding deployable unit install rather than as a post-deployment target. This means that the polices are now set during import long before the components are put to any real use, which is required if, for example, a resource adaptor entity needs the custom policy in order to initialise correctly. (RHI-4537)

The object pool configuration for a profile table named "default" is now separate from the default profile table object pool configuration. (RHI-4584)

Fixed some erroneous transactional behaviour during namespace creation and removal that occurred if the bounding transaction happened to fail during commit. (RHI-4712)

Non-replicating activity handlers will now correctly inform their associated resource adaptor if an activity under their control is forcefully ended by a platform administrator. (Replicating activity handlers were already behaving correctly here.) (RHI-4743)

Locks held by Profile MBeans in the writeable state on nodes that leave the cluster will now be cleaned up by remaining nodes in the cluster, rather than leaking and leaving those profiles uneditable until cluster restart. (RHI-4815)

The cascade uninstaller tool will now correctly manage deactivation of services and resource adaptor entities with different activation states on different nodes, as well as correctly dealing with the symmetric activation state mode. (RHI-4836)

CMP reset methods now manage transactional state correctly. (RHI-4857)

Fixed a potential MemDB state object sizing issue if toString() was invoked on a persistent state object mid-transaction. (RHI-4858)

Tracer management commands operating on services that are undeployed will now fail with a more helpful error message. (RHI-4958)

Profile table access is no longer corrupted if multiple profile tables are created from the same profile specification. (RHI-4959)

Fixed behaviour of SBB CMP fields storing SBB local objects when using transactional pass-by-reference in the case where an SBB entity is removed after being stored in a CMP field. (RHI-5142)

Fixed a code generation issue that caused deployment of an SBB to fail if the SBB didn’t define an Activity Context Interface but the SBB abstract class defined a non-abstract asSbbActivityContextInterface method. (RHI-5258)

Fixed an NPE that occurred trying to reassign to another node a service activity that had previously been administratively removed when the SLEE on the node that had owned the activity transitioned from RUNNING to STOPPING. (RHI-5322)

Changed default for rhino.skip_lifecycle_checks to false. This is discussed further in the Resource adaptor entity lifecycle states documentation. (RHI-4382)

System property "rhino.tracer.defaultlevel" has been removed. Trace levels can be set via the logging subsystem. (RHI-4947)

Trace notifications are now rate limited to a rate of 10/second with a burst limit of 1000, using the Log4j 2 BurstFilter. (RHI-4937)

The tamper-detection mechanism for audit logs now uses a checksum approach. (RHI-4627)

FreeMemory watchdog condition now triggers when 95% of heap is used. This is configurable via the watchdog.oom_percent_used system property. The old system property, watchdog.oom_min_free is deprecated. (RHI-5255)

Rhino Stats default memory size has been increased to fix problems with GUI mode. (RHI-5184)

An originating node ID field has been added to each of the respective vendor data extension classes included by Rhino in trace, alarm, and state change JMX notifications. (RHI-1962)

Stats parameter sets related to object pools for profile tables are now named after the profile table they relate to for easy identification, rather than an internal application identifier. (RHI-2963)

Add log messages to report transaction manager synchronization and resource executors queue size configuration. (RHI-4502)

Stats parameter set types, and consequently SNMP parameter set type OID mappings, no longer include namespace qualifiers embedded into their name. Namespace qualification is now handled externally from the name. (RHI-4783)

An export can now be imported into a different namespace using the import.namespace Ant property. (RHI-4381)

The cluster state change facility will now only notify resource adaptor entities in the affected namespace, rather than all resource adaptor entities, when the SLEE running state in that namespace changes. (RHI-5021)

Rhino’s config manager now handles namespace-specific configuration items (ie. object pools configs, SNMP parameter set type configs for applications, and limiter endpoint configs) on a per-namespace basis. Configuration items scoped to namespaces are now managed using namespace-specific container configuration MBeans. (RHI-4711)

Names of Rhino internal classes now reflect their original source location. (RHI-4167)

Garbage collection defaults have changed:

The config.log log file has been removed and messages that were previously in this file are now in rhino.log. (RHI-4625)

The namespace names 'default' and 'global' (case insensitive) are now reserved for internal use. It is no longer possible to create user-defined namespaces with these names. (RHI-4909)

Fixed bug in activity attachment mapping that caused various failures including event delivery spinning on activity-end cleanup. (RHI-5097)

CMP fields of type java.math.BigInteger or java.math.BigDecimal will no longer cause a deployment failure during code generation. (RHI-4516)

Fixed an issue in export scripts which created resource adaptor entities before the relevant security permissions had been set. (RHI-4500)

Corrected header attached to CSV alarm log, and fixed timestamp field formatting when an alarm originates from a user-defined namespace. (RHI-4631)

Fixed the exporter when multiple namespace names are sanitised to the same value. The exporter will now generate unique filenames in the exported data when sanitised names clash, rather than simply overwriting previously written data. (RHI-4705)

Fixed some erroneous transactional behaviour during namespace creation and removal that occurred if the bounding transaction happened to fail during commit. (RHI-4712)

The internal distributed management lock is now correctly released if an attempt is made to remove a configuration item of an unknown type. (RHI-4795)

Fixed an activity end race condition in the activity handler that could lead to the pessimistic or replicated lock for an activity not being released when the enclosing transaction ends. (RHI-4983)

Profile, Usage, and Usage Notification Manager MBeans related to a profile table will now be correctly deregistered from the MBean server when a profile table is renamed. (RHI-4975)

The rhino-console installlocaldu command now supports an optional -url argument to specify an alternate URL identifier for the deployable unit. (RHI-4939)

Rhino client tools now allow the GC_OPTIONS environment variable to be overridden. (RHI-5101)

The <setactivenamespace> Ant task now also updates the rhino.namespace system property so that any new connection object instances inherit the same namespace context. (RHI-4764)

The system property "file.encoding" is now globally readable by default. This is needed by some JDBC drivers. (RHI-4907)

Fixed a NullPointerException thrown by the createpersistenceinstance and updatepersistenceinstance Ant management tasks. (RHI-4482)

The namespaces feature is now supported for production use.

Fixed double-event scheduling from multiple suspend-resume cycles and interleaved transactions suspending, resuming and firing events. (RHI-3207)

Fixed a problem that could cause corruption of internal state when a node failed during deployment. (RHI-4327)

Fixed a NullPointerException that occurred when accessing service state in symmetric activation state mode. (RHI-4377)

Fixed a race condition that could report a profile as still being edited by some other management client if a client tried to edit a profile immediately after committing or rolling back changes to that same profile. (RHI-4442)

Fixed a NullPointerException that occurred when deploying linked components. (RHI-4395)

Fix after-transaction logic to always clean up activity event queues. Prevents stuck activities from transactions containing resources that fail between commit and cleanup. (RHI-4326)

Services and resource adaptor entities that are in the STOPPING state when a node is started will now correctly revert to the INACTIVE state during bootup when the symmetric activation state mode is not enabled. (RHI-4421)

Fixed a race condition between the creation of a profile in a user transaction and a concurrent long-running timeout and rollback of that transaction. This race condition meant that some metadata state for the failed profile creation was not cleaned up, preventing any further profile creation attempts for that profile to succeed. (RHI-4424)

Fixed compilation issues when deploying SBBs with a very large number of CMP fields. (RHI-4392)

Fix malformed ObjectName generation that prevented interaction with object pool configurations for components in user-defined namespaces. The "use-defaults" property of object pool configurations has been removed because it did not function in an obvious manner. (RHI-4391)

The namespace that each management operation was performed on is now included in the management.csv audit log file, present as the 7th field in the CSV. Rhino will use the old audit log file format if the "rhino.audit.log_format" system property is set to "2.4" (but this should not be set if namespaces are being used because it will not be possible to tell which namespace an operation was run in). (RHI-4352)

ServiceUsageMBean, ProfileTableUsageMBean and ResourceUsageMBean are now included in the management audit log. (RHI-4427)

The snapshot tool can now be used from a remote host. (RHI-4406)

Fixed Java GC options in the Windows Rhino SDK scripts. (RHI-4409)

Improved the mechanism used to detect and protect against concurrent profile edits occurring over the profile management interface. This change should prevent unexpected lock timeouts if the compare-and-set locking operation takes longer than one second, eg. slow hardware. (RHI-4239)

Added ability to optionally exclude individual configuration types when importing a Rhino export. This is particularly useful for cases where the target Rhino instance does not have a JDBC driver for the persistence configuration being imported. (RHI-4386)

Fixed an UnrecognizedLinkNameException when listing resource adaptor link names. (RHI-4211)

Fixed a NullPointerException when a management client tried to get the current trace level for a tracer. (RHI-4278)

Fixed event router warnings when stage items were rejected for execution, typically because of overload. (RHI-4241)

Fixed a bug that prevented the per-node SNMP agent from being deactivated. (RHI-4318)

Fixed a bug that prevented an SNMP OID mapping from being registered if it was cleared when Rhino started. (RHI-4308)

In the case where a resource adaptor entity or profile table is created from a linked or shadowed component, only the install level of the resolved target component is now relevant when deciding whether or not to allow the operation to proceed. The install level of the virtual component itself is irrelevant. (RHI-4243)

Fixed a potential concurrency race condition that could lead to the SLEE getting stuck in the STOPPING state if a service was deactivated on some nodes when the symmetric activation state mode was enabled. (RHI-4254)

Fixed a number of issues affecting the cascade uninstaller when dealing with linked, shadowed, and copied components. (RHI-4207)

Fixed some issues with the cascade uninstaller failing to correctly determine the necessary uninstallation order in some complex scenarios, particularly involving copied components. (RHI-4213)

Removed a spurious message regarding the removal of copied components when running the cascade uninstaller in assumed-yes mode. (RHI-4215)

The cascade uninstaller will now successfully clean up copied or linked components that have active resource adaptor entities or profile tables created from them. (RHI-4240)

Fixed a bug that prevented passwords being updated with the rhino-user utility. (RHI-4201)

Fixed a possible change to the active namespace after using the exportall command in rhino-console. (RHI-4252)

Some issues affecting the scoping and cleanup of statistics parameter sets used by namespaces have been fixed. (RHI-4212)

SLEE, service, and resource adaptor entity state change notifications now include vendor-specific data that identifies which namespace the notification originated from. (RHI-4193)

Trace and usage notifications now include vendor-specific data that identifies which namespace the notification originated from. (RHI-4264)

SLEE 1.0-style alarm notifications now include vendor-specific data that specifies the namespace that the notification originated from. (RHI-4265)

Improved thread safety of RhinoConnection for use by multi-threaded clients. (RHI-4289)

Improved external database connection logging. (RHI-4186)

The Rhino SDK now has the "CMSClassUnloadingEnabled" JVM option enabled by default, to prevent out-of-memory errors under certain conditions. (It is already enabled by default for a production Rhino installation.) (RHI-3490)

Reduce repetitious log messages about deployable unit verification when importing. (RHI-4210)

Set default heap size for Rhino client tools to 64MB. (RHI-3176)

Scattercast cluster communications mode is now a supported configuration in production environments. Scattercast management must now be done using management commands, not by editing configuration files. See the Rhino Administration and Deployment Guide for details. (RHI-3551)

The database configuration for external persistence of Rhino’s in-memory database state, along with JDBC resource configuration for applications, can now be dynamically updated at runtime rather than requiring a node or cluster restart. (RHI-3785)

Full JVM statistics are now available under the JVM parameter set. (RHI-3890)

Cluster-symmetric activation state is now supported as an option for services and resource adaptor entities. When this option is enabled, an activation state change that affects a service or resource adaptor entity will be reflected on all nodes in the cluster, and if a new node enters the cluster it will inherit the cluster-wide activation state for each of these components. (RHI-3934)

Rhino now supports multiple deployment namespaces. Namespaces are created by an administrator, and each namespace is isolated from all others. Each namespace contains its own deployable units and components, resource adaptor entities, profile tables, SLEE and component activation states, etc. Namespace management is available in the Rhino command line console but not yet in Rhino Element Manager. (RHI-3926)

Update minimum supported version of Java 8 to 1.8.0_60. Earlier versions contain bugs that affect Rhino operation. (RHI-3745)

Rhino SDK requires Java 8 to run the embedded Rhino Element Manager. (RHI-4178)

Alarms have been added for the following conditions:

The Rhino command-line client now reports alarms when they occur and includes the SLEE state and alarm count in the prompt. This new behaviour can be disabled by setting the system property "rhino.console.disable-listeners" to true. (RHI-2101)

The deactivate-and-activate-services operation is now a lot more atomic. Dropped call issues that previously could have occurred if a service took a long time to activate should no longer occur. (RHI-3783)

Added a component classloader auditing function that will look for potential classloader problems amongst the component’s dependencies. (RHI-4050)

Added command to reboot Rhino nodes to their current state. (RHI-3873)

SNMP index names are no longer truncated. The old behaviour can be restored by uncommenting the "snmp.name.max-length=19" line in read-config-variables. (RHI-3796)

Add new Threshold Rule management API call to retrieve all configured rules at once. (RHI-4049)

Reduced severity of log message for cleanup of leaked OCIO buffers. This does not affect the reliability of the node. (RHI-3200)

New sample Systemd service descriptor for RHEL 7 and related systems. (RHI-3715)

Fixed the potential for the SLEE, a service, or a resource adaptor entity to get stuck in the STOPPING state if a lock timeout happened to occur during the deactivation operation. (RHI-3453)

Services and resource adaptor entities will no longer get stuck in the STOPPING state if a temporary database error occurs when trying to reset their state back to INACTIVE. (RHI-3650)

Fix GroupHeartBeat failure on boot into a network containing a one way partition (RHI-3663)

The SLEE will no longer get stuck in the STOPPING state if a start operation is attempted while STOPPING. (RHI-3792)

UsageMBeans no longer return partial results if a node responds with an error. The error is now passed back to the client as a ManagementException as per the API (RHI-3850)

Added support for SBB Parts to the getsecuritypolicy and setsecuritypolicy Rhino Console commands. (RHI-3908)

Cluster will more gracefully handle one way network partitions. (RHI-3924)

Fixed snapshot decode of exported profile tables containing CMP fields that define serialisability only by way of a DatatypeCodec. (RHI-3986)

Fixed stall in savanna groups when rapidly shutting down many nodes in sequence (RHI-3989)

SBB part security permissions are now correctly applied to generated service code. (RHI-3994)

Fix NPE on node start when -d option is used alone with start-rhino.sh. (RHI-4000)

The object pool for a profile table defined by a SLEE 1.1 profile specification will now correctly initialise on profile table creation if the initial pooled size is greater than zero. (RHI-4001)

Fix a NullPointerException if linked/shadowed services are deployed when setting a security policy. (RHI-4021)

The ResourceAdaptor.serviceActive() callback will now be made after an activating service has also been started, if the SLEE is in the RUNNING state, rather than before. This removes a race condition that previously existed if the resource adaptor fired an event intended for the activating service in the callback method - depending on timing the event may or may not have been delivered to the service. (RHI-4029)

Database flush now properly flushes on all nodes. Database flush now respects timeout. (RHI-4030)

Fixed an NPE that occurred when running rhino-export if a component with an install level of INSTALLED depended on another component that was not yet installed. (RHI-4190)

The Rhino SDK start-rhino scripts now support command-line options to force the SLEE into the Running or Stopped state after the Rhino SDK has completed initialisation. (RHI-3189)

When running the Rhino SDK on Windows, fixed the generation of *.class.$$ files that polluted the RhinoSDK directory whenever a deployable unit was installed. (RHI-4187)