About the Rf Control Resource Adaptor

The Rf Control Resource Adaptor (Rf Control RA) is a component that interfaces Rhino TAS applications, with a Charging Data Function (CDF), for offline charging, using the Diameter Rf protocol. It provides applications a full implementation of the "Client accounting state machine" as per RFC 6733. Applications create Accounting Requests and pass them to the Rf Control RA. The Rf Control RA is responsible for delivery of Accounting Requests to the CDF.

Topics

This document includes the following topics:

Topic Explains…​

Configuring the Rf Control RA

How to configure an Rf Control RA entity

Monitoring the Rf Control RA

How to monitor the behaviour of an Rf Control RA entity

Managing the Rf Control RA

How to manage an Rf Control RA entity

Using the Rf Control API

How to use the Rf Control RA API in an application

Service Assurance Server (SAS) Tracing

The Rf Control resource adaptor is integrated with the Metaswitch Service Assurance Server (SAS). SAS provides a mechanism to record and search detailed end-to-end tracing of call handling.

See the SAS Facility section in the Rhino extended API documentation for more information about SAS tracing.

Note

SAS tracing is supported on Rhino versions 2.6 and later.

The Rf Control resource adaptor can be used with earlier Rhino versions if the Rhino API Compatibility SLEE library is deployed.

The resource adaptor automatically performs SAS tracing, if it is enabled in Rhino.

Rf Control sessions (outgoing requests)

For Rf Control sessions, the resource adaptor will create a new SAS trail and will associate it with the trail on the invoking thread.

SAS markers

When creating a SAS trail, the resource adaptors will raise a SAS marker for trail correlation. They raise a GENERIC_CORRELATOR_MARKER containing the Session-Id AVP.

SAS events

Rf Control SAS events mini-bundle
---
version: 0.1
events:
  FAILED_SESSION_ENDING:
    summary: Failed to send Rf ACR as the session is ending
    level: 40
    details: | Session ID: {{var_data[0]}}
  FAILED_START_NOT_SENT:
    summary: 'Failed to send Rf {{var_data[0]}} ACR as a START_RECORD ACR has not yet been sent'
    level: 40
    details: | Session ID: {{ var_data[1] }}
  FAILED_STOP_SENT:
    summary: 'Failed to send Rf {{var_data[0]}} ACR as a STOP_RECORD ACR has already been sent'
    level: 40
    details: | Session ID: {{ var_data[1] }}
  FAILED_START_ALREADY_SENT:
    summary: 'Failed to send Rf START_RECORD ACR as a Diameter session has already been established'
    level: 40
    details: | Session ID: {{ var_data[0] }}
  FAILED_EVENT_ON_EST_SESSION:
    summary: 'Failed to send Rf EVENT_RECORD ACR as the Diameter session has already been used for session charging'
    level: 40
    details: | Session ID: {{ var_data[0] }}
  BUFFER_FAILED_REQUEST:
    summary: 'Buffering failed Rf {{ var_data[0] }} ACR'
    level: 40
    details: | The request will be reattempted later.
  BUFFER_REPLAY:
    summary: 'Replaying buffered Rf request on session: {{var_data[0]}}'
    level: 40
  FAILED_TO_ADD_TO_BUFFER:
    summary: 'Failed to store Rf request in buffer'
    level: 20
    details: | Error details: {{var_data[0]}}

Configuring the Rf Control Resource Adaptor

Rf Control Resource Adaptor configuration

The Rf Control Resource Adaptor (Rf Control RA) has 2 types of configuration:

  1. General Resource Adaptor configuration

  2. Additional configuration, which includes the Diameter configuration that extends the general Diameter Resource Adaptor configuration

Configuration of an Rf Control RA Entity

The Rf Control RA has the following configuration parameters.

Parameter Description Default value
ConfigurationProfile

The diameter configuration profile. See Configuring Diameter Resource Adaptors

RfControlDiameterConfiguration/client

ConfigurationProfilePollTime

The time (milliseconds) between checking for configuration changes in the diameter configuration profile (as indicated by the ConfigurationProfile parameter). Value may be 0 (don’t poll) or at least 1000L.

30000L

3GPPVersion

The diameter version used for messages between the Rf Control RA and the CDF. See Diameter 3GPP configuration properties.

Vcb0

ExtensionAvpSet

The diameter extension AVP configuration profile. See Configuring Extension AVPs for more information on how to add extension AVPs.

DiameterExtensions/Charging

ExtensionAvpSetPollTime

The time (milliseconds) between checking for configuration changes in the extension AVP profile (as indicated by the ExtensionAvpSet parameter). Value may be 0 (don’t poll) or at least 1000L.

0L

IOClientWorkers

Number of IO threads serving outgoing connections. 0 use value of Runtime.getRuntime().availableProcessors()

0

ConnectTimeout

Timeout (milliseconds) to wait for an outgoing transport connection to be established

30000L

RequestTimeout

Timeout (milliseconds) to wait for a response before firing a Timeout event

2000L

ReconnectDelay

Delay (milliseconds) before reconnecting (RFC6733 timer Tc)

5000L

ForceReconnectAfterDPR

if true, always attempt to reconnect even if the server sent a DPR with Disconnect-Cause set to DO_NOT_WANT_TO_TALK_TO_YOU

false

ThreadPoolSize

Size of thread pool for decoding incoming diameter messages. -1 disable thread pool and use IO threads and 0 use value of Runtime.getRuntime().availableProcessors()

-1

DestinationRealm

Destination realm to use for sending messages, if not already set by client.

'opencloud'

DestinationHost

Destination host to use for sending messages, if not already set by client.

empty

Tip See Configuring Diameter Resource Adaptors for details on the Diameter configuration.

Rf Control RA Diameter Configuration

The ConfigurationProfile configuration parameter identifies the profile that contains additional configuration for the Rf Control RA and includes the Diameter configuration. The default value of the ConfigurationProfile parameter (RfControlDiameterConfiguration/client) means the client profile in the RfControlDiameterConfiguration profile table is used. The profile has the following fields.

Field Name Description Default Value
OnDiskBufferDirectoryName

Name of directory that the SQLite database will store the records in when no CDF is reachable. The Rf Control RA requires write permission on the path specified and the path has to be present in the security-permissions in the resource-adaptor-jar.xml file.

'${rhino.dir.home}/rf-control'

MaxRecordsInMemoryQueue

Maximum number of records waiting to be written in the in-memory queue. 0 indicates the queue is limited by the memory available.

34000L

MaxDbCommitIntervalInMs

The interval (milliseconds) the Persist thread will wait, after successfully writing all records from the memory queue to the disk buffer.

5000L

DbErrorRetry

The interval (milliseconds) the Replay thread and the Persist thread will wait, before trying to access the disk buffer, after a previous access failure.

30000L

ReplayRunInterval

The time (milliseconds) the replay thread will wait, before trying to access the disk buffer, after sending all previous records to the CDF.

30000L

Product

The product name

OpenCloud Diameter

ProductVendorID

Vendor ID; must be 0 for IETF-specified applications, or an IANA-assigned private enterprise number.

19808L

ApplicationId

Diameter Application ID

3L

ApplicationVendorID

Diameter Application ID.

19808

Version

Version number to use in all Firmware-Revision AVPs generated by the Diameter Rf stack. Version must be a positive integer.

1

Host

Host to use in all Origin-Host AVPs. Host must be resolvable.

diameterclient

Realm

Realm to use in all Origin-Realm AVPs.

example

Peer Table

A valid XML document for peer table configuration.

<![CDATA[<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE peer-table PUBLIC "-//Open Cloud Ltd.//DTD Diameter Peer Table Configuration 1.1.0//EN" "http://www.opencloud.com/dtd/diameter-peer-table-1.1.0.dtd">
     <peer-table>
     <default-options>
         <option>
             <option-name>TCP_NODELAY</option-name>
             <option-type>java.lang.Boolean</option-type>
             <option-value>true</option-value>
         </option>
     </default-options>
     <peer connectAtStartup="true">
         <uri>aaa://RfCDF:5898;transport=tcp</uri>
         <address>127.0.0.1</address>
         <option>
             <option-name>TCP_NODELAY</option-name>
             <option-type>java.lang.Boolean</option-type>
             <option-value>false</option-value>
         </option>
     </peer>
 </peer-table>]]>
Realm Table

A valid XML document for realm table configuration.

<![CDATA[<?xml version="1.0" encoding="UTF-8"?>
         <!DOCTYPE realm-table PUBLIC "-//Open Cloud Ltd.//DTD Diameter Realm Table Configuration 1.0//EN" "http://www.opencloud.com/dtd/diameter-realm-table-1.0.dtd">
         <realm-table>
             <realm>
                 <realm-name>opencloud</realm-name>
                 <application-route>
                     <application>
                         <application-id>3</application-id>
                         <vendor-id>0</vendor-id> <!-- optional, default is zero -->
                     </application>
                     <action>LOCAL</action>
                     <peer-ref>
                         <hostname>RfCDF</hostname>
                         <metric>1</metric>
                     </peer-ref>
                 </application-route>
             </realm>
             <default-route>
                 <peer-ref>
                     <hostname>RfCDF</hostname>
                     <metric>1</metric>
                 </peer-ref>
             </default-route>
         </realm-table>]]>
PeerPort

Port of remote Diameter peer

5898

PeerHost

Host of remote Diameter peer

CDF

PeerConnectAtStartup

Whether to connect to the remote peer at startup. must be true or false

true

Action

Must be LOCAL for this version of the RA

LOCAL

Transports

Transport(s) to listen on. Values can be tcp, sctp, or tcp,sctp

tcp

Configuration for Multiple CDF Nodes

Failover between CDF nodes is supported by the Diameter Rf stack. You configure the Rf Control RA by creating realm and peer tables in the Rf Control RA Diameter Configuration appropriate for your deployment. For further information on configuration and diameter stack routing see:

Monitoring the Rf Control Resource Adaptor

Statistics

The health of the Rf Control Resource Adaptor entity can be monitored using the Rhino statistics tools. For example, to monitor the statistics from the command line (<entity-name> is the Resource Adaptor entity name):

rhino-stats -m SLEE-Usage.RAEntities.<entity-name>.(default)
Note The Rf Control Resource Adaptor updates the default parameter set.

The statistics collected by the Rf Control Resource Adaptor are:

Name Description Incremented when …​

recordsReceived

Number of records received via provider interface

The client post a record to the RA

recordsFailedToPersist

Number of record write failures that occur

The Persist thread fails to put a record in the disk buffer

recordsPersisted

Number of records persisted to database on CDF failure

The Persist thread successfully writes a record to the disk buffer

recordsPending

Number of records persisted waiting to be sent to the CDF

The Persist thread write records to the disk buffer. Decremented when the Replay thread deletes the record from the disk buffer

recordsReadFromDb

Number of records read from database

The Replay thread reads a record from the disk buffer

recordsRetransmitted

Number of records retransmitted due to failover

The Replay thread receives an ACA, for an ACR, that has the potentially retransmitted flag set in the header

successfulAnswers

Number of successful answers

The RA receives an ACA for an ACR

sendFailures

Number of temporary send failures

The Diameter Rf stack

  • receives an ACA with status code of DIAMETER_OUT_OF_SPACE.

  • receives an error response with result code of DIAMETER_TOO_BUSY or DIAMETER_LOOP_DETECTED.

failedAnswers

Number of permanent failed answers with record discarded

The Diameter Rf stack

  • received a response with result code other than DIAMETER_OUT_OF_SPACE or SUCCESSFUL.

  • receives an error response with result code other than DIAMETER_TOO_BUSY or DIAMETER_LOOP_DETECTED.

recordsDiscarded

Number of records discarded due to non-failed-answer issues

The Persist thread cannot write the message to the memory buffer as it is full.

The Replay thread cannot read or create an ACR from the stored messages. This indicates that there are corrupt records.

processingTime

Processing times for ACR/ACA pairs. It is the time (milliseconds) between sending an ACR and receiving an ACA in response.

The Diameter Rf stack received an ACA or error message

FailedToStartActivity

The amount of times the Rf Control RA failed to start an activity

The Rf Control RA was unable to start a SLEE activity

FailedToSubmitEvent

The amount of times the Rf Control RA failed to submit an event

The Rf Control RA fails to submit an event for delivery

Tip You can define your own threshhold alarms that are a function of the statistics the Rf Control Resource Adaptor collects. See Threshold Alarms in the Rhino TAS Admin Guide for more information about defining threshold based alarms.

Alarms

The following alarms may be raised with a source of OpenCloud <entity-name> <version> (for a Resource Adaptor entity <entity-name>, of version <version>).

Alarm Level Raised when …​ Cleared when …​

active-reconfiguration

WARNING

A configuration property (such as the host or realm) is changed while the RA entity is active

The RA entity is deactivated

memory-queue-full

CRITICAL

The memory queue reaches the limit set by the configuration parameter MaxRecordsInMemoryQueue. If set to 0 there won’t be an alarm. This alarm indicates the Request handler cannot write records and will discard them.

The Request handler is able write a record to the queue.

write-failure

MAJOR

The Persist thread cannot write to the disk buffer.

The Persist thread is able to write to the disk buffer.

write-thread-failure

CRITICAL

The Persist thread caught an unexpected exception

The RA is restarted

replay-failure

MAJOR

The Replay thread can’t read records from the disk buffer

The Replay thread is able to read records from the disk buffer

replay-thread-failure

CRITICAL

The Replay thread caught an unexpected exception

The RA is restarted

Managing the Rf Control Resource Adaptor

CDF connection

The Rf Control Resource Adaptor (Rf Control RA) will connect to the CDF when it is active. An alarm is raised if the Rf Control RA fails to connect to the CDF. There are several possible reasons connecting to the CDF may fail, including:

  • Incorrect diameter configuration of the Rf Control RA

  • There is a problem with the network

  • The CDF is not accepting connections

  • The host machine the Rf Control RA is running on is not properly configured

The alarm is automatically cleared once the failure is resolved, and the Rf Control RA successfully connects to the CDF.

Disk buffer management

The Rf Control RA uses an SQLite database as a disk buffer when the CDF is not reachable. SQLite is a simple, embedded, SQL database engine ideally suited to be used as an on-disk buffer.

Tip For more information on SQLite see www.sqlite.org

The database name is: buffer-<ra entity>-<rhino node>.db. Example: buffer-rf-control-ra-101.db.

The location of the database is defined by the Rf Control RA Diameter Configuration profile parameter OnDiskBufferDirectoryName (default value is ${rhino.dir.home}/rf-control)

Accounting Request (ACR) messages are stored in the database when the CDF is not reachable.

As a rule of thumb, assume 3kB per ACR (large but reasonable) and 3 ACRs per call (START, INTERIM, STOP). Assuming 500 call attempts per second per Rhino node, the database will grow at a rate of:

  • 4.5 MB per second

  • 270 MB per minute

  • approximately 16 GB per hour

If the Rf Control RA fulls the entire disk partition it is buffering to, an alarm of type write-failure or write-thread-failure will be raised (depending on the cause). The write-failure alarm will be cleared when the database is available for writing. The write-thread-failure alarm will require a Rf Control RA restart.

Note The platform operator is responsible for increasing the available disk space

During the write-failure and write-thread-failure condition the Rf Control RA will store ACR records in a memory buffer. If the database or the CDF connection issue is not resolved, the memory queue will eventually become full and the Rf Control RA will start discarding records.

When the CDF becomes available again, the Replay thread will attempt to transmit stored ACR records. The database will automatically re-use the disk space that successfully replayed ACR records had occupied.

Tip The number of stored and transmitted records can be monitored by viewing the RF Control RA’s statistics.

The SQLite database supports VACUUMING, which rebuilds the database file, repacking it into a minimal amount of disk space. VACUUMing is only necessary if the disk partition that the database is stored in is too small for normal operation use. For example, the database fits into the partition but the database size is deemed "too large". This procedure is nevertheless attempted automatically after all the records have been replayed by the replay thread, that is the database is nominally empty, since the impact on performance in this case is negligible. However, this attempt may fail if there happens to be a concurrent access to the database by another thread. This means that manual intervention may be necessary if disk space is getting low.

The database has incremental vacuum support enabled, since running an incremental vacuum should have less of a performance impact than a full one if there are a lot of records in the database, but it may be less effective.

An incremental vacuum procedure can be performed as follows:

$ sqlite3 buffer-rfcontrol-101.db "PRAGMA incremental_vacuum;"

or

$ sqlite3 buffer-rfcontrol-101.db
sqlite> PRAGMA incremental_vacuum;
sqlite> .quit

If this doesn’t produce the desired results, a full VACUUM procedure can be performed as follows:

$ sqlite3 buffer-rfcontrol-101.db "VACUUM;"

or

$ sqlite3 buffer-rfcontrol-101.db
sqlite> VACUUM;
sqlite> .quit
Note The vacuum procedure may fail if there is concurrent access to the database. To guarantee a successful vacuuming the Rf Control RA may have to be deactivated before performing the procedure.

For more information on VACUUM see SQLite VACUUM.

We recommend that the disk partition is large enough to not require VACUUMing.

If the operator has an independent alarm configured to monitor the disk partition use, then it is possible that a VACUUM may be necessary to reduce the reported disk usage (after a CDF outage and replay completion). Such an alarm should clear once the VACUUM has completed.

Using the Rf Control API

The Rf Control RA provides a Rhino TAS API that allows a client application to send Diameter Rf messages. The following example shows a Service with a single SBB using the API.

@SleeService(
    id = @ComponentId(name="ExampleRfControl", vendor="OpenCloud", version="1.0"),
    rootSbb = @RootSBB(sbb = @ComponentId(name="ExampleRfControl",
                                          vendor="OpenCloud",
                                          version="1.0"))
)
@SBB(
    id = @ComponentId(name="ExampleRfControl", vendor="OpenCloud", version="1.0"),
    sbbClasses = @SBBClasses(),
    raTypes = {
        @RATypeBinding(
            raType = @ComponentId(name = "rf-control-ratype",
                                  vendor = "OpenCloud",
                                  version = "1.0.0"),
            activityContextInterfaceFactoryName =
                        "rf-control-ra/activitycontextinterfacefactory",
            resourceAdaptorEntityLink = "rf-control-ra",
            resourceAdaptorObjectName = "rf-control-ra/provider"
        )
    }
)
public abstract class ExampleRfControl implements Sbb {

    @Override
    public void setSbbContext(SbbContext context) {
        tracer = context.getTracer("example.rf-control");
        this.context = context;
        try {

            /* Get Rf control facilities */

            final Context env = (Context) new InitialContext().lookup("java:comp/env");
            this.profileFacility = (ProfileFacility) env.lookup(ProfileFacility.JNDI_NAME);
            this.rfControlProvider = (RfControlProvider) env.lookup("rf-control-ra/provider");
            this.rfControlACIFactory = (RfControlActivityContextInterfaceFactory)
                    env.lookup("rf-control-ra/activitycontextinterfacefactory");
        }
        catch (NamingException e) {
            throw new RuntimeException("Got NamingException trying to locate facilities", e);
        }
    }

    private void exampleCall() {

        final RfMessageFactory rfMessageFactory = rfControlProvider.getRfMessageFactory();

        tracer.fine("Creating the the message");
        final AccountingRequest startAcr, interimAcr1, interimAcr2, stopAcr;
        try {

            /* Create simple Rf messages */

            startAcr = buildACR(roMessageFactory, rfMessageFactory,
                                AccountingRecordType.START_RECORD);
            interimAcr1 = buildACR(roMessageFactory, rfMessageFactory,
                                   AccountingRecordType.INTERIM_RECORD);
            interimAcr2 = buildACR(roMessageFactory, rfMessageFactory,
                                   AccountingRecordType.INTERIM_RECORD);
            stopAcr = buildACR(rfMessageFactory, AccountingRecordType.STOP_RECORD);
        }
        catch (Exception e) {
            tracer.warning("Failed to send ACR due to an exception", e);
            sendHttpResponse(requestActivity, aci, HTTP_SERVER_ERROR, "Internal Error",
                    "Failed to send ACR due to an exception: " + e.toString());
            return;
        }

        tracer.fine("Sending the the message");

        try {

            /* Send Rf messages */

            final RfControlActivity rfControlActivity = rfControlProvider.newSession();
            rfControlActivity.sendAccountingRequest(startAcr);
            rfControlActivity.sendAccountingRequest(interimAcr1);
            rfControlActivity.sendAccountingRequest(interimAcr2);
            rfControlActivity.sendAccountingRequest(stopAcr);
        }
        catch (Exception e) {
            tracer.warning("Failed to send ACR due to an exception", e);
            return;
        }
    }

    private AccountingRequest buildACR(RfMessageFactory rfMessageFactory,
                                       AccountingRecordType type) throws Exception
    {
        final AccountingRequest acr = rfMessageFactory.createRfAccountingRequest(type);
        final ServiceInformation serviceInformation =
                rfMessageFactory.createServiceInformation();
        final ImsInformation imsInformation =
                rfMessageFactory.createImsInformation(NodeFunctionality.AS);

        final SubscriptionId subscriptionId =
                rfMessageFactory.createSubscriptionId(SubscriptionIdType.END_USER_E164,
                                                      "tel:1234");

        serviceInformation.setSubscriptionId(subscriptionId);
        serviceInformation.setImsInformation(imsInformation);

        acr.setServiceInformation(serviceInformation);

        return acr;
    }
}

The client application needs to have an RA Type Binding and RA Entity link binding to retrieve the Rf Control Provider via JNDI. Once this setup has occurred the primary interfaces for the application are the:

  • RfMessageFactory (to create Rf requests), and

  • RfControlActivity (to send requests on a logical Rf session) .

The RfMessageFactory exposes the underlying Diameter Rf factory to create all necessary requests and AVPs. Once an AccountingRequest has been created and populated, it can be sent via an RfControlActivity.

For more detailed information see:

Resource Adaptor Type API