Rhino Changelog

Changes to the periodic alarm reporting period will now be reflected across all cluster nodes immediately rather than requiring a restart of nodes not handling the change request in the first instance before the change is applied. (RHI-3511)

Fixed a race condition during service activation between the start of the replicated ServiceActivity and ResourceAdaptor.serviceActive() callbacks being made on the nodes not involved with the creation of the ServiceActivity. The race condition meant that a resource adaptor entity responding to the callback could fire an event to an SBB that attempted to attach to the ServiceActivity, but could not do so because the activity was not yet visible on the node. Rhino will now wait and ensure that the required replicated activity state exists on the node before proceeding to make the resource adaptor callbacks. (RHI-5196)

Fixed a distributed lock manager race condition that could grant an exclusive lock to a waiting thread in error if an acquisition attempt on a highly contended lock timed out but was immediately retried, for example as happens in the SNMP configuration update thread during large parameter set configuration storms. This error could furthermore result in the lock becoming permanently jammed if the erroneously acquired lock was released before the lock was officially granted by the lock manager. (RHI-5614)

Fixed failover of the replicated SLEE-generated service activities so now they will only get adopted by nodes where the corresponding service is running. If a service is not running on any remaining nodes then the activity for that service is removed. This also fixes an AssertionError that would occur, for example, on node recovery when the restarting node considered itself to be the first node where a service was starting again and tried to recreate the service activity, but failed to do so because the activity already existed on another node where it shouldn’t. (RHI-5636)

The description of alarms generated by Rhino’s Watchdog subsystem are now included in Rhino’s alarm catalog. (RHI-5649)

If a service was deactivated on the node owning the associated SLEE-generated service activity, and the service was still running on other cluster nodes, ownership of the service activity was not being reassigned to one of these other nodes, rather the activity remained owned by the deactivating node. This could cause future problems such as AssertionErrors, for example, if the other nodes were restarted. This issue is now fixed. (RHI-5658)

Profile specifications with large numbers of profile attributes will now deploy correctly. Previously a SLEE 1.0 profile specification with more than 255 profile attributes, or a SLEE 1.1 profile specification with more than 253 profile attributes, would unexpectedly fail at the code generation stage. (RHI-5659)

The configuration manager will now only initialise empty configuration types on boot of the first node in a virgin cluster (for global config types) or on namespace creation (for namespaced config types). This eliminates a class of bug that would, for example, cause a node booting into an existing cluster where all licenses had been manually uninstalled to reinstall the default platform license, but not tell other nodes about it, leading to inconsistent configuration state on the other nodes. (RHI-5689)

Fixed a memory leak that occurred in Rhino if a stats client started collection for a statistics session that had no stat parameter subscriptions. Also fixed a reconnection issue in rhino-stats where it failed to restart stats collection after certain connection failover scenarios, meaning the client didn’t output stats any more after the failover. (RHI-5717)

Fixed a sorting comparator bug in the rhino-console listactivealarms command that could lead to an IllegalArgumentException in particular dataset cases. (RHI-5732)

The ServiceActivityFactory will no longer return a null ServiceActivity in cases where the activity has not yet started or has already ended when the activity object is requested. A FacilityException will be thrown instead. (RHI-5843)

The ServiceActivityFactory will no longer return a null ServiceActivity in cases where the activity has not yet started or has already ended when the activity object is requested. A FacilityException will be thrown instead. (RHI-5843)

A "ZERO-length generation snapshots" warning message that could occur during some activity-related housekeeping operations has been demoted to a debug level message as it’s completely benign. (RHI-5894)

Added a new built-in ServiceNodeActivity and ServiceNodeStartedEvent, similar to the SLEE-defined ServiceActivity and ServiceStartedEvent, but with the following differences:

Added a new per-node configuration file rhino.java.security to override two Java security properties. This file specifies the following default values:

networkaddress.cache.ttl=30
networkaddress.negative.cache.ttl=10

Fixed permission issue that could result in lost log messages during a log rollover. (RHI-5456)

Fixed uncaught exception alarm caused by a permission check failure during log rollover. (RHI-5492)

Fixed an internal corruption of metadata that occurred if a component referenced by more than one linked component had one of those linked components removed. This could have led to, for example, service activation callbacks for the remaining linked components not being made to resource adaptors when required by a target service component. (RHI-5347)

Fixed a lock manager issue that affected a deactivate-and-activate service operation when more than one service was being deactivated, and improved the resiliency of stopping tasks in general in the wake of unexpected errors during their execution. (RHI-5362)

SLEE 1.1 profile table creation will no longer fail with a NullPointerException if debug logging for the profile.profile log key is enabled. (RHI-5424)

Fixed a bug where SBB parts were not releasing their component references to dependent event types when unverified. (RHI-5520)

Removed heap limit from rhino-console and rhino-import client tools to prevent OutOfMemoryError during install. (RHI-5418)

Fixed exception when activating rhino-extensions examples service. (RHI-5436)

Fixed rare BufferOverflowException triggered by concurrent SNMP counter access. (RHI-5493)

Fixed an IllegalArgumentException that occurred when trying to deploy a service containing an SBB that defined usage parameter interfaces using only the extension deployment descriptor and didn’t define any of the interfaces as the root usage parameter set type. (RHI-5525)

Fixed a NullPointerException when attempting to set SBB tracer levels in a service with an install level of INSTALLED. The operation now fails with a more appropriate error. (RHI-5542)

Fixed a bug in the SAS configuration Ant task <configuresas> that caused it to remove the configured SAS servers if no updated SAS servers are specified. (RHI-5477)

Added a deregisterProfileMBeans command to rhino-console that allows rogue profile MBeans such as those for uncommitted profiles to be cleaned up. (RHI-5383)

Added new alarmType parameter (1.3.6.1.4.1.19808.2.102.14) to SNMP Alarm Notifications. This OID contains the Alarm Type listed in the alarm catalog. (RHI-5516)

Added Rhino support for additional SAS event fields message_id and call_info_id. (RHI-5523)

Improved error handling when adding and removing SAS servers. (RHI-5301)

New logging subsystem based on Log4j 2. See the logging documentation.

SLEE Tracing now uses the logging subsystem directly.

Metaswitch Service Assurance Server (SAS) support. See the SAS tracing documentation.

Profile MBeans now isolate transactional state between different management clients. Changes being made by one client no longer leak into the view of other clients. (RHI-4317)

The configurestagingqueues Ant task no longer transposes maximumAge and maximumSize values. (RHI-4380)

Object pool stats are now grouped by namespace, and their parameter set names have been improved. (RHI-4429)

The various ProfileProvisioningMBean getProfiles* methods now respect the state of any user transaction currently in progress by the client. (RHI-4458)

Added ResourceExecutor stats parameter set type to TransactionManager. These statistics were previously being combined with SynchronizationExecutor. (RHI-4533)

Reworked the export of custom security policies assigned to components so that these are now set during import as part of the corresponding deployable unit install rather than as a post-deployment target. This means that the polices are now set during import long before the components are put to any real use, which is required if, for example, a resource adaptor entity needs the custom policy in order to initialise correctly. (RHI-4537)

The object pool configuration for a profile table named "default" is now separate from the default profile table object pool configuration. (RHI-4584)

Fixed some erroneous transactional behaviour during namespace creation and removal that occurred if the bounding transaction happened to fail during commit. (RHI-4712)

Non-replicating activity handlers will now correctly inform their associated resource adaptor if an activity under their control is forcefully ended by a platform administrator. (Replicating activity handlers were already behaving correctly here.) (RHI-4743)

Locks held by Profile MBeans in the writeable state on nodes that leave the cluster will now be cleaned up by remaining nodes in the cluster, rather than leaking and leaving those profiles uneditable until cluster restart. (RHI-4815)

The cascade uninstaller tool will now correctly manage deactivation of services and resource adaptor entities with different activation states on different nodes, as well as correctly dealing with the symmetric activation state mode. (RHI-4836)

CMP reset methods now manage transactional state correctly. (RHI-4857)

Fixed a potential MemDB state object sizing issue if toString() was invoked on a persistent state object mid-transaction. (RHI-4858)

Tracer management commands operating on services that are undeployed will now fail with a more helpful error message. (RHI-4958)

Profile table access is no longer corrupted if multiple profile tables are created from the same profile specification. (RHI-4959)

Fixed behaviour of SBB CMP fields storing SBB local objects when using transactional pass-by-reference in the case where an SBB entity is removed after being stored in a CMP field. (RHI-5142)

Fixed a code generation issue that caused deployment of an SBB to fail if the SBB didn’t define an Activity Context Interface but the SBB abstract class defined a non-abstract asSbbActivityContextInterface method. (RHI-5258)

Fixed an NPE that occurred trying to reassign to another node a service activity that had previously been administratively removed when the SLEE on the node that had owned the activity transitioned from RUNNING to STOPPING. (RHI-5322)

Changed default for rhino.skip_lifecycle_checks to false. This is discussed further in the Resource adaptor entity lifecycle states documentation. (RHI-4382)

System property "rhino.tracer.defaultlevel" has been removed. Trace levels can be set via the logging subsystem. (RHI-4947)

Trace notifications are now rate limited to a rate of 10/second with a burst limit of 1000, using the Log4j 2 BurstFilter. (RHI-4937)

The tamper-detection mechanism for audit logs now uses a checksum approach. (RHI-4627)

FreeMemory watchdog condition now triggers when 95% of heap is used. This is configurable via the watchdog.oom_percent_used system property. The old system property, watchdog.oom_min_free is deprecated. (RHI-5255)

Rhino Stats default memory size has been increased to fix problems with GUI mode. (RHI-5184)

An originating node ID field has been added to each of the respective vendor data extension classes included by Rhino in trace, alarm, and state change JMX notifications. (RHI-1962)

Stats parameter sets related to object pools for profile tables are now named after the profile table they relate to for easy identification, rather than an internal application identifier. (RHI-2963)

Add log messages to report transaction manager synchronization and resource executors queue size configuration. (RHI-4502)

Stats parameter set types, and consequently SNMP parameter set type OID mappings, no longer include namespace qualifiers embedded into their name. Namespace qualification is now handled externally from the name. (RHI-4783)

An export can now be imported into a different namespace using the import.namespace Ant property. (RHI-4381)

The cluster state change facility will now only notify resource adaptor entities in the affected namespace, rather than all resource adaptor entities, when the SLEE running state in that namespace changes. (RHI-5021)

Rhino’s config manager now handles namespace-specific configuration items (ie. object pools configs, SNMP parameter set type configs for applications, and limiter endpoint configs) on a per-namespace basis. Configuration items scoped to namespaces are now managed using namespace-specific container configuration MBeans. (RHI-4711)

Names of Rhino internal classes now reflect their original source location. (RHI-4167)

Garbage collection defaults have changed:

The config.log log file has been removed and messages that were previously in this file are now in rhino.log. (RHI-4625)

The namespace names 'default' and 'global' (case insensitive) are now reserved for internal use. It is no longer possible to create user-defined namespaces with these names. (RHI-4909)

Fixed bug in activity attachment mapping that caused various failures including event delivery spinning on activity-end cleanup. (RHI-5097)

CMP fields of type java.math.BigInteger or java.math.BigDecimal will no longer cause a deployment failure during code generation. (RHI-4516)

Fixed an issue in export scripts which created resource adaptor entities before the relevant security permissions had been set. (RHI-4500)

Corrected header attached to CSV alarm log, and fixed timestamp field formatting when an alarm originates from a user-defined namespace. (RHI-4631)

Fixed the exporter when multiple namespace names are sanitised to the same value. The exporter will now generate unique filenames in the exported data when sanitised names clash, rather than simply overwriting previously written data. (RHI-4705)

Fixed some erroneous transactional behaviour during namespace creation and removal that occurred if the bounding transaction happened to fail during commit. (RHI-4712)

The internal distributed management lock is now correctly released if an attempt is made to remove a configuration item of an unknown type. (RHI-4795)

Fixed an activity end race condition in the activity handler that could lead to the pessimistic or replicated lock for an activity not being released when the enclosing transaction ends. (RHI-4983)

Profile, Usage, and Usage Notification Manager MBeans related to a profile table will now be correctly deregistered from the MBean server when a profile table is renamed. (RHI-4975)

The rhino-console installlocaldu command now supports an optional -url argument to specify an alternate URL identifier for the deployable unit. (RHI-4939)

Rhino client tools now allow the GC_OPTIONS environment variable to be overridden. (RHI-5101)

The <setactivenamespace> Ant task now also updates the rhino.namespace system property so that any new connection object instances inherit the same namespace context. (RHI-4764)

The system property "file.encoding" is now globally readable by default. This is needed by some JDBC drivers. (RHI-4907)

Fixed a NullPointerException thrown by the createpersistenceinstance and updatepersistenceinstance Ant management tasks. (RHI-4482)

The namespaces feature is now supported for production use.

Fixed double-event scheduling from multiple suspend-resume cycles and interleaved transactions suspending, resuming and firing events. (RHI-3207)

Fixed a problem that could cause corruption of internal state when a node failed during deployment. (RHI-4327)

Fixed a NullPointerException that occurred when accessing service state in symmetric activation state mode. (RHI-4377)

Fixed a race condition that could report a profile as still being edited by some other management client if a client tried to edit a profile immediately after committing or rolling back changes to that same profile. (RHI-4442)

Fixed a NullPointerException that occurred when deploying linked components. (RHI-4395)

Fix after-transaction logic to always clean up activity event queues. Prevents stuck activities from transactions containing resources that fail between commit and cleanup. (RHI-4326)

Services and resource adaptor entities that are in the STOPPING state when a node is started will now correctly revert to the INACTIVE state during bootup when the symmetric activation state mode is not enabled. (RHI-4421)

Fixed a race condition between the creation of a profile in a user transaction and a concurrent long-running timeout and rollback of that transaction. This race condition meant that some metadata state for the failed profile creation was not cleaned up, preventing any further profile creation attempts for that profile to succeed. (RHI-4424)

Fixed compilation issues when deploying SBBs with a very large number of CMP fields. (RHI-4392)

Fix malformed ObjectName generation that prevented interaction with object pool configurations for components in user-defined namespaces. The "use-defaults" property of object pool configurations has been removed because it did not function in an obvious manner. (RHI-4391)

The namespace that each management operation was performed on is now included in the management.csv audit log file, present as the 7th field in the CSV. Rhino will use the old audit log file format if the "rhino.audit.log_format" system property is set to "2.4" (but this should not be set if namespaces are being used because it will not be possible to tell which namespace an operation was run in). (RHI-4352)

ServiceUsageMBean, ProfileTableUsageMBean and ResourceUsageMBean are now included in the management audit log. (RHI-4427)

The snapshot tool can now be used from a remote host. (RHI-4406)

Fixed Java GC options in the Windows Rhino SDK scripts. (RHI-4409)

Improved the mechanism used to detect and protect against concurrent profile edits occurring over the profile management interface. This change should prevent unexpected lock timeouts if the compare-and-set locking operation takes longer than one second, eg. slow hardware. (RHI-4239)

Added ability to optionally exclude individual configuration types when importing a Rhino export. This is particularly useful for cases where the target Rhino instance does not have a JDBC driver for the persistence configuration being imported. (RHI-4386)

Fixed an UnrecognizedLinkNameException when listing resource adaptor link names. (RHI-4211)

Fixed a NullPointerException when a management client tried to get the current trace level for a tracer. (RHI-4278)

Fixed event router warnings when stage items were rejected for execution, typically because of overload. (RHI-4241)

Fixed a bug that prevented the per-node SNMP agent from being deactivated. (RHI-4318)

Fixed a bug that prevented an SNMP OID mapping from being registered if it was cleared when Rhino started. (RHI-4308)

In the case where a resource adaptor entity or profile table is created from a linked or shadowed component, only the install level of the resolved target component is now relevant when deciding whether or not to allow the operation to proceed. The install level of the virtual component itself is irrelevant. (RHI-4243)

Fixed a potential concurrency race condition that could lead to the SLEE getting stuck in the STOPPING state if a service was deactivated on some nodes when the symmetric activation state mode was enabled. (RHI-4254)

Fixed a number of issues affecting the cascade uninstaller when dealing with linked, shadowed, and copied components. (RHI-4207)

Fixed some issues with the cascade uninstaller failing to correctly determine the necessary uninstallation order in some complex scenarios, particularly involving copied components. (RHI-4213)

Removed a spurious message regarding the removal of copied components when running the cascade uninstaller in assumed-yes mode. (RHI-4215)

The cascade uninstaller will now successfully clean up copied or linked components that have active resource adaptor entities or profile tables created from them. (RHI-4240)

Fixed a bug that prevented passwords being updated with the rhino-user utility. (RHI-4201)

Fixed a possible change to the active namespace after using the exportall command in rhino-console. (RHI-4252)

Some issues affecting the scoping and cleanup of statistics parameter sets used by namespaces have been fixed. (RHI-4212)

SLEE, service, and resource adaptor entity state change notifications now include vendor-specific data that identifies which namespace the notification originated from. (RHI-4193)

Trace and usage notifications now include vendor-specific data that identifies which namespace the notification originated from. (RHI-4264)

SLEE 1.0-style alarm notifications now include vendor-specific data that specifies the namespace that the notification originated from. (RHI-4265)

Improved thread safety of RhinoConnection for use by multi-threaded clients. (RHI-4289)

Improved external database connection logging. (RHI-4186)

The Rhino SDK now has the "CMSClassUnloadingEnabled" JVM option enabled by default, to prevent out-of-memory errors under certain conditions. (It is already enabled by default for a production Rhino installation.) (RHI-3490)

Reduce repetitious log messages about deployable unit verification when importing. (RHI-4210)

Set default heap size for Rhino client tools to 64MB. (RHI-3176)

Scattercast cluster communications mode is now a supported configuration in production environments. Scattercast management must now be done using management commands, not by editing configuration files. See the Rhino Administration and Deployment Guide for details. (RHI-3551)

The database configuration for external persistence of Rhino’s in-memory database state, along with JDBC resource configuration for applications, can now be dynamically updated at runtime rather than requiring a node or cluster restart. (RHI-3785)

Full JVM statistics are now available under the JVM parameter set. (RHI-3890)

Cluster-symmetric activation state is now supported as an option for services and resource adaptor entities. When this option is enabled, an activation state change that affects a service or resource adaptor entity will be reflected on all nodes in the cluster, and if a new node enters the cluster it will inherit the cluster-wide activation state for each of these components. (RHI-3934)

Rhino now supports multiple deployment namespaces. Namespaces are created by an administrator, and each namespace is isolated from all others. Each namespace contains its own deployable units and components, resource adaptor entities, profile tables, SLEE and component activation states, etc. Namespace management is available in the Rhino command line console but not yet in Rhino Element Manager. (RHI-3926)

Update minimum supported version of Java 8 to 1.8.0_60. Earlier versions contain bugs that affect Rhino operation. (RHI-3745)

Rhino SDK requires Java 8 to run the embedded Rhino Element Manager. (RHI-4178)

Alarms have been added for the following conditions:

The Rhino command-line client now reports alarms when they occur and includes the SLEE state and alarm count in the prompt. This new behaviour can be disabled by setting the system property "rhino.console.disable-listeners" to true. (RHI-2101)

The deactivate-and-activate-services operation is now a lot more atomic. Dropped call issues that previously could have occurred if a service took a long time to activate should no longer occur. (RHI-3783)

Added a component classloader auditing function that will look for potential classloader problems amongst the component’s dependencies. (RHI-4050)

Added command to reboot Rhino nodes to their current state. (RHI-3873)

SNMP index names are no longer truncated. The old behaviour can be restored by uncommenting the "snmp.name.max-length=19" line in read-config-variables. (RHI-3796)

Add new Threshold Rule management API call to retrieve all configured rules at once. (RHI-4049)

Reduced severity of log message for cleanup of leaked OCIO buffers. This does not affect the reliability of the node. (RHI-3200)

New sample Systemd service descriptor for RHEL 7 and related systems. (RHI-3715)

Fixed the potential for the SLEE, a service, or a resource adaptor entity to get stuck in the STOPPING state if a lock timeout happened to occur during the deactivation operation. (RHI-3453)

Services and resource adaptor entities will no longer get stuck in the STOPPING state if a temporary database error occurs when trying to reset their state back to INACTIVE. (RHI-3650)

Fix GroupHeartBeat failure on boot into a network containing a one way partition (RHI-3663)

The SLEE will no longer get stuck in the STOPPING state if a start operation is attempted while STOPPING. (RHI-3792)

UsageMBeans no longer return partial results if a node responds with an error. The error is now passed back to the client as a ManagementException as per the API (RHI-3850)

Added support for SBB Parts to the getsecuritypolicy and setsecuritypolicy Rhino Console commands. (RHI-3908)

Cluster will more gracefully handle one way network partitions. (RHI-3924)

Fixed snapshot decode of exported profile tables containing CMP fields that define serialisability only by way of a DatatypeCodec. (RHI-3986)

Fixed stall in savanna groups when rapidly shutting down many nodes in sequence (RHI-3989)

SBB part security permissions are now correctly applied to generated service code. (RHI-3994)

Fix NPE on node start when -d option is used alone with start-rhino.sh. (RHI-4000)

The object pool for a profile table defined by a SLEE 1.1 profile specification will now correctly initialise on profile table creation if the initial pooled size is greater than zero. (RHI-4001)

Fix a NullPointerException if linked/shadowed services are deployed when setting a security policy. (RHI-4021)

The ResourceAdaptor.serviceActive() callback will now be made after an activating service has also been started, if the SLEE is in the RUNNING state, rather than before. This removes a race condition that previously existed if the resource adaptor fired an event intended for the activating service in the callback method - depending on timing the event may or may not have been delivered to the service. (RHI-4029)

Database flush now properly flushes on all nodes. Database flush now respects timeout. (RHI-4030)

Fixed an NPE that occurred when running rhino-export if a component with an install level of INSTALLED depended on another component that was not yet installed. (RHI-4190)

The Rhino SDK start-rhino scripts now support command-line options to force the SLEE into the Running or Stopped state after the Rhino SDK has completed initialisation. (RHI-3189)

When running the Rhino SDK on Windows, fixed the generation of *.class.$$ files that polluted the RhinoSDK directory whenever a deployable unit was installed. (RHI-4187)