Description

The slee-data-transformation tool is responsible for converting the configuration data between major versions of Sentinel products.

The tool is an engine that executes a set of transformation steps defined in rules and operates on Profiles and RA properties. The rules are Java classes that use the transformation API and are executed against the exported profiles done via rhino-export and on the files generated by the slee-data-migration tool.

As explained in SLEE Configuration and Sentinel upgrades the configuration data are retrieved from the rhino-export and from the slee-data-migration.

The transformation API can be found in Data Transformation API

Usage options

The tool was built to be used as part of Sentinel Upgrades, but can be used in isolation if required, and indeed developers who are writing rules will probably make extensive use of it whilst perfecting their rules.

For convenience, the tool is packaged as a single jar file slee-data-transformation-standalone.jar with no need for any supporting libraries or shell scripts. It is run using a command:

java -jar slee-data-transformation-standalone.jar <required files, plus options>

There are 3 files which must be provided

  • the rules jar file, which holds the compiled version of the rules written using the transformation API

  • the path to the directory that holds the result of rhino-export from the system being upgraded

  • the pathname of the json file produced by slee-data-migration from the system being upgraded

A further 2 files are optional (when doing a major upgrade orca always passes them, but when developing rules they may be unnecessary)

  • the path to the directory that holds the result of rhino-export from the uplevel system

  • the pathname of the json file produced by slee-data-migration from the uplevel system

The command options are:

  • -l LOG_FILE or --log-file LOG_FILE - pathname of a file to write logging information to

  • -o OUTPUT_DIR or --output-dir OUTPUT_DIR - path of a directory to write the transformed data to. If this is not provided then the entire transformation will still be done, producing the same error and warnings as it goes, but no data files will be written out, thus providing the ability to do a dry-run of the transformation

  • -v or --verbose - causes verbose output, detailing all the warnings that occur during the transformation.

As a convenience (principally for the orca-bundler tool), the data-transformation tool will also allow just a single input rules jar file to be passed, without passing any export files. In this mode, the transformation is meaningless, (and thus you cannot combine this single parameter option with the -o option to write out the result) but the tool will still verify that the rules files can be loaded, and set the exit code to a non-zero value if that is not possible.

The Transformation API

The Transformation API is contained in an abstract Java class called TransformationApi, and a developer uses the API by providing a concrete Java class that extends TransformationApi, and providing their implementation of the rules() method.

public class SimpleTransformation extends TransformationApi {
    ...
}

By extending the API class in this way, all the methods of the API are readily available as local methods.

It is important to understand how the transformation process makes use of the rules class provided by the developer. The engine first loads in the data files from rhino-export and slee-data-migration, and then makes a call into the rules() method of the developer’s class. This method returns an array of rules that are later processed and interpreted by the engine. The rules are applied as a whole, not necessarily sequentially in the order they were written, and the rules express the intent of what is to be done, rather than being direct calls to API methods.

As a concrete example of this, the rules may express actions that end up both changing and deleting a particular piece of data. The net effect of this will always be that the data ends up being deleted - it does not matter what order the rules are presented in, and the change action will always be ineffective. (Yes, you should get warnings that tell you that the change rule was ignored in this case, but it’s entirely possible that with a slightly different input configuration both rules could have applied to different pieces of data, and so not have been in conflict).

As another example, rules are always expressed using the original name of something, even if there is another rule present that applies renaming to that object.

The API consists of a number of parts:

  • the rules() methods, that every user of the API must override, that returns the array of rules that drive the transformation

  • factory methods for various object classes used as parameters by the API, used for generating or matching purposes

  • methods that define rules that provide contexts that control when their child actions apply

  • methods that define rules that express actions that are performed on the data triggered by their enclosing contexts

  • creation methods

  • query methods that allow reading values from the original data, or from the uplevel data

  • some miscellaneous methods

  • the ValueFunction class whose subclasses can be used to provide dynamic value evaluation to rules

  • some pre-written ValueFunction implementations that provide common manipulations used in transformations, such as adding or removing array items

Many of the methods in the API, especially the factory methods and the rule definitions, have multiple overloaded implementations of the same named method, taking different numbers or types of parameters. This allows for flexibility in how the API is used, whilst still giving compile time checks that the appropriate type of data is being provided.

The various parts are used together as follows:

  • the engine calls the rules() method, which returns an array of rules

    • each rule in the array is a context rule, containing further context and action rules

    • rules have varying parameters, which are either inbuilt Java types, or objects that the factory methods create

    • the choice of rules to return can be influenced by the results of the query methods

    • some rules take ValueFunction parameters, which allow a value to be calculated dynamically

      • the dynamic function has access to the context of the rule it was being used as, and can also call the query methods

      • pre written functions are provided for common manipulation requirements

A note about code layout

The way that rules are represented in the API is as methods that take an arbitrary number of parameters, corresponding to the arbitrary number of children contained in a context rule.

All examples given in the documentation follow the following conventions, which you are recommended to follow in your own use of the API

  • a context rule is written on multiple lines

    • the first line has the context rule method name, the opening parenthesis, and the fixed parameters that describe the context, ending with a trailing comma

    • each child rule is given its own line, and indented by 2 spaces from the enclosing context rule

    • each child rule line, except the last, ends in a comma (since these are really just separate parameters, and Java does not allow a trailing comma after the last parameter)

    • child contexts of course cause their own child rules to be indented by a further 2 spaces

    • after all the child rules, the final closing parenthesis gets a line to itself, lined up with the first character of the context rule name

The examples in these documents have been pulled from a working example of a use of the API. As such, the rules are part of an array of such rules, and the various child rules may not always include the last child rule for that particular context. Thus many of the fragments shown here may end with a trailing comma, which is not strictly part of the fragment when viewed in isolation.

The rules() method

This is the method that all API users must implement. Its purpose is to return the rules that the engine will execute to drive the transformation.

The return value is an array, which makes the common use case very simple, since the Java language has special syntax for declaring arrays.

@Override
public Rule[] rules(RulesContext rulesContext) {
    Rule[] rules = {
      profileTable("example",
        doNotImportProfileTable()
      )
    };

    return rules;
}

For more complex cases, where you may want to build up the exact list of rules that are returned in a dynamic fashion, an alternative approach is to use a collection such as an ArrayList, and then to convert that to an array using toRulesArray() when you are ready to return it.

@Override
public Rule[] rules(RulesContext rulesContext) {
    List<Rule> list = new ArrayList<>();

    list.add(
      profileTable("example",
        doNotImportProfileTable()
      )
    );

    return toRulesArray(list);
}

The rules can be built using information about the exported data by using the various query methods in the Transformation API. In addition, the rules() methods is passed the RulesContext object, which provides access to information that details the full set of components that are part of the downlevel and uplevel installations.

The RulesContext

The RulesContext object has methods getDownlevelComponentsByDU() and getUplevelComponentsByDU() that give access to mappings that detail the full set of components that are part of the downlevel and uplevel installations. These mappings can be used to determine exactly which versions of a given component are present, and thus to provide rules that are specific to the actual versions present.

Since the mappings provide ComponentID objects, that provide the version numbers as a multi-component dotted string, the Version class can be used to used to make comparing such strings easy.

@Override
public Rule[] rules(RulesContext rulesContext) {
    List<Rule> list = new ArrayList<>();

    final String componentOfInterest = "SentinelDiameterMediationFeatureSPI";

    Version downlevelVersion =
        findVersion(rulesContext.getDownlevelComponentsByDU(), componentOfInterest);
    Version uplevelVersion =
        findVersion(rulesContext.getUplevelComponentsByDU(), componentOfInterest);

    if (downlevelVersion.greaterThanOrEqual(new Version("1.2.3")) &&
        downlevelVersion.lessThanOrEqual(new Version("1.3.5")) &&
        uplevelVersion.greaterThanOrEqual(new Version("2"))) {

        list.add(
          profileTable("example",
            doNotImportProfileTable()
          )
        );
    }

    return toRulesArray(list);
}

private Version findVersion(Map<DeployableUnitID, Set<ComponentID>> dus, String componentOfInterest) {
    String versionString = null;
    for (DeployableUnitID duId : dus.keySet()) {
        ComponentID componentID = (ComponentID) dus.get(duId);

        if (componentID.getName().equals(componentOfInterest)) {
            versionString = componentID.getVersion();
        }
    }

    return new Version(versionString);
}

The renamedOids() method

This is an optional method that can be provided if an upgrade requires that OID parameter set type names, or the counters they contain, need to be renamed. The return from this method is a mapping from the old to the new names.

An OID rename is likely to be needed if the name of a feature changes.

The OID name is of the form Metrics.Services.SbbID[name=volte.sentinel.sip,vendor=OpenCloud,version=2.7.0-copy#1].lifecycle. You will notice that that contains a version number. For ease of pasting the data, the name passed in the mapping holds that full string, but the version number part is ignored when doing a comparison. That is to say, versions in both the old and the new names are ignored. In practice, the only part of the parameter set type name that should change is the bit specifying the feature name volte.sentinel.sip in this example.

To specify a counter within a OID parameter set type, the rename map returned by this method should have an old name using the OID parameter set type name as a prefix, followed by a dot, and the counter name, thus: Metrics.Services.SbbID[name=volte.sentinel.sip,vendor=OpenCloud,version=2.7.0-copy#1].lifecycle.sbbRollbacks. The new name in the mapping is just the name of the counter after the rename, such as sbbRollbacksDone.

Where a counter is renamed within a OID that is also renamed, the mapping should always use the old name of the OID when specifying the counter.

    public Map<String, String> renamedOids(RulesContext rulesContext) {
        Map<String, String> renamed = new HashMap<>();

        // Rename an OID
        renamed.put("Metrics.Services.SbbID[name=volte.sentinel.sip,vendor=OpenCloud,version=2.7.0-copy#1].lifecycle",
                    "Metrics.Services.SbbID[name=volte.sentinel.sippy,vendor=OpenCloud,version=2.7.0-copy#1].lifecycle");

        // Rename a counter
        renamed.put("Metrics.Services.SbbID[name=volte.sentinel.sip,vendor=OpenCloud,version=2.7.0-copy#1].lifecycle.sbbRollbacks",
                    "sbbRollbacksDone");

        return renamed;
    }

A first look at real rules

In this simple example, the engine will find every profile whose name ends with "MySuffix" in profile table "myTable", and set the value of its "myAttribute" attribute to "myNewValue".

profileTable("myTable",
  profileName(regex(".*MySuffix"),
    setAttribute("myAttribute", "myNewValue")
  )
)

How each of the API features shown in this small fragment work together is explained in more detail below.

The factory methods

The context rules in the API take matching parameters, and many of the action rules take a generator parameter - which is an object that can be asked for the exact value to be used when the action is performed. These matching and generator objects are implemented by the various XXXMatcher classes and ObjectGenerator inner classes respectively, and spreading such objects throughout your code can make it hard to read.

To simplify things, there are a number of factory methods that take simple parameters and return the appropriate matcher or generator, and you will normally never have to deal with the real classes themselves.

array()

The array() factory methods are suitable to pass to rules that deal with attributes that have an array type.

Overrides are provided that construct an array from:

  • nothing at all (an empty array)

  • an arbitrary number of strings

  • an arbitrary number of objects

  • an arbitrary number of other generators

setAttribute("names", array()),
setAttribute("names", array("just one value")),
setAttribute("names", array("hello", "world")),

anyString()

The anyString() factory method produces a matcher object that matches any string. It is a better alternative than using a regex matcher that can also be constructed to match anything.

profileSpec("Name", anyString(), "1.0.5",
  renameProfileSpec("z")
),

regex()

The regex() factory method creates a StringMatcher object which matches string values against the given regular expression. Note: in order to match, the regular expression must match the full value, rather than just a substring.

profileTable(regex("any[thing|one]"),
  setAttribute("a", "b")
),

Note that regular expressions are used for 'matching', not merely for 'finding' - the pattern given needs to define the whole of the name that it will match, not just part of it. Matching is done in a case sensitive manner.

component()

The component() factory methods produce ComponentMatcher objects used by ProfileSpec context rules.

Each SLEE component is identified by name, vendor and version, and there are variants of this factory method that allow each of these to be specified as a constant string, or as a regex or anyString object.

profileSpec(component("Name", "Vendor", regex("1\\.0\\.\\d*")),
  renameProfileSpec("z")
),

However, use of the component method is optional - all rules that take a component can also take the name, vendor, and version values as individual parameters.

The context rules

The context rules all specify some matching requirement, and then contain the list of child rules that are applied when that matching requirement is met. The child rules may themselves be any mixture of action rules, and further context rules (though not all nested contexts make sense). Whether the child rules get executed is controlled by whether the context can be matched, but child action rules may sometimes take parameters that allows them to affect items that are themselves outside of their immediately enclosing context.

profileSpec()

The profileSpec() context rule is used to identify which profile specs need to be matched for the child rules to be applied.

The spec can be given either using the result of a call to the component() factory method, or the three parameters that would have been passed to that method can be passed to this rule directly.

profileSpec(component("a", "b", "c"),
  renameProfileSpec("z"),
  changeAttributeType("a", "type2"),
  changeAttributeType("b", String.class)
),
profileSpec(component("a", "b", regex("ccc")),
  renameProfileSpec("z")
),
profileSpec(component("a", regex("bbb"), "c"),
  renameProfileSpec("z")
),
profileSpec(component("a", regex("bbb"), regex("ccc")),
  renameProfileSpec("z")
),
profileSpec(component(regex("aaa"), "b", "c"),
  renameProfileSpec("z")
),
profileSpec(component(regex("aaa"), "b", regex("ccc")),
  renameProfileSpec("z")
),
profileSpec(component(regex("aaa"), regex("bbb"), "c"),
  renameProfileSpec("z")
),
profileSpec(component(regex("aaa"), regex("bbb"), regex("ccc")),
  renameProfileSpec("z")
),
profileSpec("a", "b", "c",
  renameProfileSpec("z")
),
profileSpec("a", "b", regex("ccc"),
  renameProfileSpec("z")
),
profileSpec("a", regex("bbb"), "c",
  renameProfileSpec("z")
),
profileSpec("a", regex("bbb"), regex("ccc"),
  renameProfileSpec("z")
),
profileSpec(regex("aaa"), "b", "c",
  renameProfileSpec("z")
),
profileSpec(regex("aaa"), "b", regex("ccc"),
  renameProfileSpec("z")
),
profileSpec(regex("aaa"), regex("bbb"), "c",
  renameProfileSpec("z")
),
profileSpec(regex("aaa"), regex("bbb"), regex("ccc"),
  renameProfileSpec("z")
),

profileTable()

The profileTable() context rule is used to identify which profile table needs to be matched for the child rules to be applied.

The name of the table can be specified directly as a string, or via a matcher such as returned by the regex factory.

A profileTable context rule can be nested inside a profileSpec context if desired.

profileTable("aa",
  renameProfileSpec("z")
),
profileTable(regex("xx"),
  renameProfileSpec("z")
),

profileName()

The profileName() context rule is used to identify which profile needs to be matched for the child rules to be applied.

The name of the profile can be specified directly as a string, or via a matcher such as returned by the regex factory.

A profileName context rule can be nested inside profileSpec or profileTable contexts if desired.

profileTable("table",
  profileName("any",
    doNotImportProfile()
  ),
  profileName(regex("[Aa][Nn][Yy]"),
    doNotImportProfile()
  )
),

ifAttributeValue()

The ifAttributeValue() context rule will have its child rules applied only if the specified profile attribute matches the specified value.

The name of the attribute is always specified as a string, the value can be specified directly as a string, or via a matcher such as returned by the regex factory.

An ifAttributeValue context rule can be nested inside any of the profile…​ or ifAttribute…​ contexts. In fact the way to match on multiple attributes at once is to nest ifAttributeValue rules.

ifAttributeValue("a", "99",
  setAttribute("p", "q")
),
ifAttributeValue("a", regex("99|100|101"),
  setAttribute("p", "q")
),

ifAttributeValueIsNull()

The ifAttributeValueIsNull() context rule is a specialized form of the ifAttributeValue() rule that matches on a value being null.

ifAttributeValueIsNull("x",
  setAttributeToNull("y")
)

raEntity()

The raEntity() context rule is used to select on entity names.

The name of the entity can be specified directly as a string, or via a matcher such as returned by the regex factory.

raEntity("a",
  doNotImportRaEntity(),
),

ifPropertyValue()

The ifPropertyValue() context rule will have its child rules applied only if the specified RA configuration property matches the specified value.

The name of the property is always specified as a string, the value can be specified directly as a string, or via a matcher such as returned by the regex factory.

An ifPropertyValue context rule can be nested inside an raEntity context, or another ifPropertyValue one (which allows matching on multiple properties at once).

ifPropertyValue("property", "value",
  setProperty("any", 99)
),
ifPropertyValue("property", regex("hello*"),
  setProperty("any", 100)
),

ifPropertyValueIsNull()

The ifPropertyValueIsNull() context rule is a specialized form of the ifPropertyValue() rule that matches on a value being null.

ifPropertyValueIsNull("property",
  setProperty("any", 101)
)

The action rules

addAttribute()

The addAttribute() action rule adds a new attribute by specifying its name and type, and optionally a default value to be used when no other more specific value is given, and when an attribute value was not already present within the export data.

The name is always given as a string, the type can be a string, or preferably a Java class whose type will be used automatically (which avoids the possibility of mistyping the string).

The optional default value can be almost any type that is found convenient, and no checking is done that it actually matches the type that you specify. A particularly powerful feature of the API is being able to pass a ValueFunction object as the default value, which allows the value to be calculated when the rule is actually applied.

By default, new attributes are expected to be non-serialised, which is the case for the common types such as strings, integers and the like. For custom types which need to be marked as serialised in the profile export XML, use one of the addAttribute() methods which take a serialisation version parameter, all of which create a serialised attribute.

profileTable("any",
  // The type parameter may be spelt out as a string
  addAttribute("a", "java.lang.String", "c"),
  addAttribute("array", "java.lang.String[]", array("c")),
  addAttribute("a", "java.lang.String", exampleFunction()),
  addAttribute("a", "java.lang.boolean", true),
  addAttribute("a", "java.lang.int", 99),

  // The type parameter can also be passed as a class rather than as a string
  addAttributeAsNull("a", String.class),
  addAttribute("a", String.class, "c"),
  addAttribute("array", String[].class, array("c")),
  addAttribute("a", String.class, exampleFunction()),
  addAttribute("a", boolean.class, true),
  addAttribute("a", int.class, 99),

  // Serialised attributes must explicitly be marked as such
  addAttributeAsNull("a", "com.example.CustomType", "2.4.0"),
  addAttribute("a", "com.example.CustomType", "2.4.0", exampleFunction()),
  addAttribute("a", "com.example.CustomType", "2.4.0", "\01\02\03\04"),
  addAttribute("array", "com.example.CustomType[]", "2.4.0", array("\01", "\02")),

  // For serialised attributes which don't have an explicit serialisation version, pass null as the version
  addAttributeAsNull("a", "com.example.CustomType", null),
  addAttribute("a", "com.example.CustomType", null, exampleFunction()),
  addAttribute("a", "com.example.CustomType", null, "\01\02\03\04"),
  addAttribute("array", "com.example.CustomType[]", null, array("\01", "\02"))
),

addAttributeAsNull()

These rules are specialized forms of the addAttribute() rule, where the default value is a null.

profileTable("any",
  addAttributeAsNull("a", "java.lang.String"),
  addAttributeAsNull("a", String.class)
),

doNotImportAttribute()

The doNotImportAttribute() action deletes a named attribute from the exported data for a profile table.

profileTable("any",
  doNotImportAttribute("a")
),

setAttribute()

The setAttribute() action rule sets a named attribute to a particular value.

Overrides on this method allow the value to be specified in a large variety of ways, and no checking is done that the value is of the specific type that the profile stores. As with addAttribute, the value can also be passed via a ValueFunction object, which allows the value to be calculated when the rule is actually applied.

profileTable("table",
  profileName(regex("qq"),
    setAttribute("names", array()),
    setAttribute("names", array("just one value")),
    setAttribute("names", array("hello", "world")),
    setAttribute("name", exampleFunction()),
    setAttribute("name", false),
    setAttribute("name", 42),

    setAttribute("names", new String[]{"1","2","3"}),
    setAttribute("names", new Integer[]{1,2,3}),
    setAttribute("names", new int[]{1,2,3}),
    setAttribute("names", new long[]{1,2,3}),
    setAttribute("names", new Object[]{1,false,"third",this}),

    // As well as defining our own functions, there are some utility ones
    setAttribute("names", functionArrayAppend("extra")),
    setAttribute("names", functionArrayRemove("excess")),

    setAttribute("name", functionUplevelProfileAttribute()),
    setAttribute("names", functionUplevelProfileAttributeArray()),

    setAttribute("name", functionForDefaultProfileAndOthers("value for default profile", "value for all other profiles")),
    setAttribute("name", functionForDefaultProfileAndOthers("value for default profile", exampleFunction())),
    setAttribute("name", functionForDefaultProfileAndOthers(exampleFunction(), "value for all other profiles")),
    setAttribute("name", functionForDefaultProfileAndOthers(exampleFunction(), exampleFunction())),

    // These two rules are equivalent, so the second is an artificial example
    setAttribute("name", 99),
    setAttribute("name", functionForConstant(99))
  )
),

setAttributeToNull()

This is a specialized form of the setAttribute action rule that uses a value of null.

setAttributeToNull("y")

renameAttribute()

The renameAttribute() rule changes the name of an attribute, without affecting its type or value. Both original name and new name are given as strings.

renameAttribute("s", "t"),

changeAttributeType()

The changeAttributeType() rule changes the type of the name attribute.

The name is specified as a string, and the type as either a string or the Java class of the type.

changeAttributeType("a", "type2"),
changeAttributeType("b", String.class)

doNotImportProfile()

The doNotImportProfile() rule deletes the profile from the export data.

Important Deleting the profile from the export data does not mean that it is always deleted from the system - data that is not in the export can still be present in the uplevel system if the installation of the new version recreates the data that is not provided by the imported export. 
profileTable("table",
  profileName("profile",
    doNotImportProfile()
  )
),

doNotImportProfileTable()

The doNotImportProfileTable() rule removes an entire table from the export data.

Important Like deleting a profile, removing a table from the export may not mean that is it deleted from the system if the uplevel installation creates a new version of the table. 
profileTable("table",
  profileName("any",
    doNotImportProfileTable()
  )
),

renameProfile()

The renameProfile() rule renames a profile.

The new name can be a constant value, or can be a ValueFunction which, since it is passed the GeneratorContext, allows the existing name to be used in the production of the new name.

renameProfile("x"),
renameProfile(new ValueFunction() {
    public Object evaluate(GeneratorContext context) {
        return "y";
    }
})

renameProfileTable()

Renames a profile table. The new name is only supported as a constant string.

renameProfileTable("y"),

renameProfileSpec()

Renames a profile spec. The new name is only supported as a constant string.

renameProfileSpec("z")

addProperty()

Adds a property to an RA entity.

Takes the name of the new property, its type (either as a string, or better as the class of the type), and the default value to be used when no other value is provided, and when the property is not already set in the export data. That value can be provided as whatever type is most convenient, which does not need to match the actual defined type of the property (just to be compatible with it, so for example you can pass a numeric parameter to a property that is stored as a string, and the string representation of the number will be used).

This action rule needs to be inside an raEntity context.

addProperty("name", "type", true),
addProperty("name", "type", -101),
addProperty("name", "type", "hello world"),

addProperty("name", boolean.class, true),
addProperty("name", long.class, -101),
addProperty("name", String.class, "hello world"),

addPropertyAsNull()

This is a specialized form of addProperty that has null as the default value.

addPropertyAsNull("name", "type"),
addPropertyAsNull("name", int.class),

doNotImportProperty()

This action rule deletes an RA configuration property from the exported data.

Important Deleting the property from the export data does not mean that it is always deleted from the system - data that is not in the export can still be present in the uplevel system if the installation of the new version recreates the data that is not provided by the imported export. 
doNotImportProperty("name"),

setProperty()

This action sets the value of an RA configuration property, where that value can be specified in a variety of different types (which do not need to match, just be compatible with the actual type of the property).

setProperty("name", false),
setProperty("name", 666),
setProperty("name", "ceci n'est pas un string"),

setPropertyToNull()

This is a specialized form of the setProperty action, where the value to be set is null.

setPropertyToNull("name"),

renameProperty()

Renames an RA configuration property.

renameProperty("old", "new"),

doNotImportRaEntity()

Deletes an RA entity from the exported data.

Important Deleting the RA entity from the export data does not mean that it is always deleted from the system - data that is not in the export can still be present in the uplevel system if the installation of the new version recreates the data that is not provided by the imported export. 
doNotImportRaEntity(),

renameRaEntity()

Renames an RA entity.

renameRaEntity("new name"),

The creation rules

createTable()

The createTable() action rule is unique in that it is the one action rule that does not need to be enclosed in a context rule.

Its purpose is to create a new profile table with a specified name, that uses the given profile spec. Note that the profile spec to use is specified using individual name, vendor, and version values - it is not possible to pass the result of the component() factory method to this rule, since that rule produces a Matcher object that is used to match a profile spec, not to define one (in particular matches may include regular expressions as any of the individual parts).

createTable("tableName", "specName", "vendor", "1.0"),
profileTable("tableName",
  addAttribute("first", String.class, "use this for default"),
  createProfile("extra profile")
),

createProfile()

The createProfile() action rule is used to create a new named profile within a profile table.

profileTable("table",
  createProfile("p")
),

The query methods

getRaProperty()

Get the current (i.e. prior to upgrade) value of a property.

String raProperty = getRaProperty("diameter-sentinel-internal", "SessionTimeout");

getUplevelRaProperty()

Get the uplevel (i.e. present in the upgraded version) value of a property.

String uplevelRaProperty = getUplevelRaProperty("diameter-sentinel-internal", "SessionTimeout");

getProfileAttribute()

Get the current (i.e. prior to upgrade) value of a profile attribute.

If the particular attribute being queried is of an array type, this method will return null, and give a warning. To get the value of an array attribute, you must use getProfileAttributeArray which has a return type which can correctly express the array contents.

String value = getProfileAttribute("dbquery-config", "", "maxSyncTransactionAge");

getProfileAttributeArray()

Get the current (i.e. prior to upgrade) value of a profile attribute which has an array type.

If the particular attribute being queried is not of an array type, this method will return null, and give a warning. To get the value of a non-array attribute, you must use getProfileAttribute which has a return type which can correctly express the simple contents.

String[] array = getProfileAttributeArray("dbquery-config","postgres-Config", "dataSourceProfileIDs");

getUplevelProfileAttribute()

Get the uplevel (i.e. present in the upgraded version) value of a profile attribute.

If the particular attribute being queried is of an array type, this method will return null, and give a warning. To get the value of an array attribute, you must use getUplevelProfileAttributeArray which has a return type which can correctly express the array contents.

String uplevelValue = getUplevelProfileAttribute("dbquery-config", "", "maxSyncTransactionAge");

getUplevelProfileAttributeArray()

Get the uplevel (i.e. present in the upgraded version) value of a profile attribute which has an array type.

If the particular attribute being queried is not of an array type, this method will return null, and give a warning. To get the value of a non-array attribute, you must use getUplevelProfileAttribute which has a return type which can correctly express the simple contents.

String[] uplevelArray = getUplevelProfileAttributeArray("dbquery-config", "postgres-Config", "dataSourceProfileIDs");

getProfileNames()

Get an array of all the profile names in the specified profile table.

String[] profileNames = getProfileNames("dbquery-config");

getUplevelProfileNames()

Get an array of all the profile names in the uplevel version of the specified profile table.

String[] uplevelProfileNames = getUplevelProfileNames("dbquery-config");

Miscellaneous methods

toRulesArray()

The toRulesArray() method converts an iterable object containing rules to the array required by the API, for cases where it’s easier to build up the rules dynamically.

@Override
public Rule[] rules(RulesContext rulesContext) {
    List<Rule> list = new ArrayList<>();

    list.add(
      profileTable("example",
        doNotImportProfileTable()
      )
    );

    return toRulesArray(list);
}

issueWarning()

The primary use for the issueWarning() method is so that code that uses the API can issue a warning, for example if it discovers that the data it is transforming is too complex to be automatically handled.

A secondary use is as "print statements" for use as a temporary diagnostic aid.

issueWarning("Danger, do not reverse polarity");

Using the ValueFunction

When the data transformation rules require a 'value' to be assigned, that value can be given explicitly in the rule, or can be provided via a ValueFunction object.

The ValueFunction object provides an evaluate method that is called when the rule is being used, and allows the function to calculate the value based on the context where it has been invoked.

The context holds:

/**
 * A GeneratorContext is passed to the {@link ValueFunction#evaluate} and
 * {@link ObjectGenerator#generate(GeneratorContext)} methods.<br>
 * It contains information about the context in which the value is being generated.
 */
public class GeneratorContext {

    /**
     * Information about the transformation rule in which the evaluation is taking place.
     */
    public static class RuleInfo {

        /**
         * The name of the rule.
         */
        public String name;

        /**
         * A description of the rule.
         */
        public String description;
    }

    /**
     * Information about the attribute which is being generated, where appropriate.
     */
    public static class Attribute {

        /**
         * The name of the profile attribute.
         */
        public String name;

        /**
         * The attribute value, expressed as a string, where available. Not appropriate for use with array types.
         */
        public String value;

        /**
         * The attribute value, expressed as a string array, where available. Only appropriate for use with array types.
         */
        public String[] array;
    }

    /**
     * Information about the profile in which the evaluation is taking place, where appropriate.
     */
    public static class Profile {

        /**
         * The name of the profile.
         */
        public String name;
    }

    /**
     * Information about the profile table in which the evaluation is taking place, where appropriate.
     */
    public static class Table {

        /**
         * The name of the profile table.
         */
        public String name;
    }

    /**
     * Information about the transformation rule in which the evaluation is taking place.
     */
    public RuleInfo rule;

    /**
     * Information about the attribute which is being generated, where appropriate.
     * This field will be null if there is no attribute present in this context.
     */
    public Attribute attribute;

    /**
     * Information about the profile in which the evaluation is taking place, where appropriate.
     * This field will be null if there is no profile present in this context.
     */
    public Profile profile;

    /**
     * Information about the profile table in which the evaluation is taking place, where appropriate.
     * This field will be null if there is no profile table present in this context.
     */
    public Table table;

In addition, the method has full access to the whole of the data that is being transformed, via the query functions in the API.

new ValueFunction() {
    @Override
    public Object evaluate(GeneratorContext context) throws TransformationException {
        return String.format("hello %s, I see that your value was %s",
                             context.attribute.name, context.attribute.value);
    }

    @Override
    public String toString() {
        return "exampleFunction()";
    }
};

The evaluate method must be provided. The toString() override is optional, but helps make warning messages that relate to rules that use the ValueFunction easier to follow.

A ValueFunction can provide either a single value or an array as its return value. The same ValueFunction can be passed to multiple rules, where the logic can use the context information to act appropriately.

Of interest, there is one advantage to omitting the toString() override. If you do, and have have your IDE set up appropriately this may be shown to you using the Java 8 lambda syntax, which is easier to follow:

(context) -> {
        return String.format("hello %s, I see that your value was %s",
                             context.attribute.name, context.attribute.value);
}

The provided ValueFunction implementations

Given that ValueFunction objects are so powerful, a number of useful functions have been packaged as part of the API. They follow the simple naming convention of all having the word 'function' at the start of their method name!

Array manipulation

When an attribute is of an array type, common operations are to add or remove items from that array. Both of these are supported via provided ValueFunctions.

Appending adds to the end of the array. Removing removes the first item that matches.

setAttribute("names", functionArrayAppend("extra")),
setAttribute("names", functionArrayRemove("excess")),

Providing a different value to the default profile

The addAttribute rule allows a "default" value to be provided - this is the value to be used for profiles that are added that do not specify a particular value for that attribute.

That is quite different from providing a value for the default profile.

A set of functions functionForDefaultProfileAndOthers() are provided that allow you to specify one value for the default profile, and a different value for all other profiles.

The variations allow the values to be specified as either a string value, or a further ValueFunction.

setAttribute("name", functionForDefaultProfileAndOthers("value for default profile", "value for all other profiles")),
setAttribute("name", functionForDefaultProfileAndOthers("value for default profile", exampleFunction())),
setAttribute("name", functionForDefaultProfileAndOthers(exampleFunction(), "value for all other profiles")),
setAttribute("name", functionForDefaultProfileAndOthers(exampleFunction(), exampleFunction())),

Providing a constant value

The functionForConstant() ValueFunction at first sight appears redundant - every method in the API that accepts a ValueFunction can also be passed a constant. However, if your logic is more complex, you may find wrapping a constant up as a ValueFunction is very useful, since you can then provide common code that does not have to distinguish a constant from other possible inputs.

// These two rules are equivalent, so the second is an artificial example
setAttribute("name", 99),
setAttribute("name", functionForConstant(99))

Documenting what the transformation rules do

There’s no doubt about it, writing transformation rules can be tricky, so you are likely to want to include copious comments in the Java file, to help you keep track of what you did, and to guide future maintainers of the code. That is business as usual for programmers.

However, you also need to document what your rules do for a second set of people - those who know nothing about programming, but who are doing an upgrade, and need to understand, in general, human readable terms, what config is being migrated, and what is not, in the specific upgrade.

Since the best way to keep documentation current is to keep it close to the code itself, this second set of documentation is placed in the same Java file as the code that produces the rules. The build process will pull this documentation out of the Java file, and put it in the user facing upgrade document for the appropriate product.

To mark the bits of the Java file that should be included in the docs in this way, they are tagged using AsciiDoc tags, as illustrated here.

/*
This is a java file, so we put documentation inside multi-line comments
The next line is the special marker to know what text to include
// tag::ocdoc[]
Here we document what the rules do...

The text that we write here will be concatenated with any other text
fragments further on in the file that are also tagged "ocdoc",
so that way we can intersperse the code and the documentation of that code.
// end::ocdoc[]
The previous line holds the end marker of the text to be included.
*/

The tag uses the name ocdoc, to indicate that the text fragment it wraps will be pulled out into the OpenCloud upgrade docs.

You can add markup to multiple fragments of text, throughout the file, and the build process will pull them all into the same document.

The fragments that you write are in AsciiDoc markup - at its simplest that means that you can write your description across multiple lines, and it will all end up joined together into the same paragraph. Leave a blank line between paragraphs. However, you can do lots more with this formatting if you wish.

You are also not limited to just including text from inside of Java comments - there may well be times when you want to include text taken from the actual code in the documentation. For example, if you have a lot of similar rules, you may be accumulating them dynamically based on a list of names used in a loop. That same list can be usefully included in the documentation by placing tags around the relevant lines of code.

/*
// tag::ocdoc[]
The following profile tables are not imported from the 2.6 system,
instead using the default values from the 2.7 install:

// We want the list that follows to be treated as pre-formatted by the docs
[listing,indent=0]
// end::ocdoc[]
*/
        String[] do_not_import = {
          // tag::ocdoc[]
          "sip-sis-ra-address-subscriptions",
          "sip-sis-ra-compositions",
          "sip-sis-ra-experimental-features",
          "sip-sis-ra-ext-platforms",
          "sip-sis-ra-extension-refs"
          // end::ocdoc[]
        };
        for (String tableName : do_not_import) {
            rules.add(profileTable(tableName, doNotImportProfileTable()));
        }

Note also the use of other AsciiDoc markup, specifically [listing,indent=0] to make the document processing display the list of strings in a clean way. The line that precedes that line, starting // is an AsciiDoc comment!

You may have split your code that produces the transformation rules across several Java files. If you have done that, then you may find it easiest to put all the documentation markup in just one of those files. If you do choose to spread it around within more than one Java file, the build process will find the markup in all the Java files, but will process the fragments in alphabetical order of the filenames, which may not be what you wanted.

Debugging the transformation rules

  • When you run the transformation tool, it may output a number of warnings and errors. These are your first port of call when trying to debug your rules.

  • If you have made use of ValueFunction, then these warnings will be more readable if you make sure that you include an override of the toString() method in these.

  • Consider adding some temporary issueWarning calls as you build up your rules, or within your ValueFunction implementations so that you can get an insight into the values your rules are processing.

  • A very useful way to check what your rules have done is to do a textual comparison of the input and output data - the processing tries very hard to ensure that the only differences you see in the xml files should be due to the transformation, and not be down to other processing artifacts. (The xml produced by rhino-export is not written in canonical XML format, and this behaviour is preserved by the transformations).

  • The transformation process produces detailed logs of its processing, though these were designed for debugging the engine rather than debugging the rules that it is running. That means that the output is perhaps more verbose than needed, and may not be intelligible without access to the engine source code.

Previous page Next page