Bootstrap

Bootstrap is the process whereby, after a VM is started for the first time, it is configured with key system-level configuration such as IP addresses, DNS and NTP server addresses, a hostname, and so on. This process runs automatically on the first boot of the VM. For bootstrap to succeed it is crucial that all entries in the SDF (or in the case of a manual deployment, all the bootstrap parameters) are correct.

Successful bootstrap

Once the VM has booted into multi-user mode, bootstrap normally takes about one minute.

SSH access to the VM is not possible until bootstrap has completed. If you want to monitor bootstrap from the console, log in as the sentinel user with the password you set in the SDF and examine the log file bootstrap/bootstrap.log. Successful completion is indicated by the line Bootstrap complete.

Troubleshooting bootstrap

If bootstrap fails, an exception will be written to the log file. If the network-related portion of bootstrap succeeded but a failure occurred afterwards, the VM will be accessible over SSH and logging in will display a warning Automatic bootstrap failed.

Examine the log file bootstrap/bootstrap.log to see why bootstrap failed. In the majority of cases it will be down to an incorrect SDF or a missing or invalid bootstrap parameter. Destroy the VM and recreate it with the correct SDF or bootstrap parameters (it is not possible to run bootstrap more than once).

If you are sure you have the SDF or bootstrap parameters correct, or it is not obvious what is wrong, contact your Customer Care Representative.

Configuration

Configuration occurs after bootstrap. It sets up product-level configuration such as:

  • configuring Rhino and the relevant products (on systems that run Rhino)

  • SNMP-based monitoring

  • SSH key exchange to allow access from other VMs in the cluster to this VM

  • authentication and encryption settings for the Cassandra clusters on the TSN VNFCs

To perform this configuration, the process retrieves its configuration in the form of YAML files from the CDS. The CDS to contact is determined using the cds-addresses parameter from the SDF or bootstrap parameters.

The configuration process constantly looks for new configuration, and reconfigures the system if new configuration has been uploaded to the CDS.

The YAML files describing the configuration should be prepared in advance.

rvtconfig

After spinning up the VMs, configuration YAML files can be validated and uploaded to CDS using the rvtconfig tool. The rvtconfig tool can be run either on the SIMPL VM or any Rhino VM Automation VM.

Note

If the VM image does not have a pre-installed license, then a Rhino license must be included with the configuration YAML files uploaded to CDS using the rvtconfig tool.
Note

CDS should be running before any other nodes are booted. See Setting up CDS for instructions on how to set up a Cassandra service to provide CDS.

Configuration files

The configuration process reads settings from YAML files. Each YAML file refers to a particular set of configuration options, for example, SNMP settings. The YAML files are validated against a YANG schema. The YANG schema is human-readable and lists all the possible options, together with a description. It is therefore recommended to reference the Configuration YANG schema while preparing the YAML files.

Some YAML files are shared between different node types. If a file with the same file name is required for two different node types, the same file must be used in both cases.

You may also include Rhino declarative configuration without writing a custom initconf hook. The configuration is stored in a parent directory named rhino-declarative-config, which may consist of optional sub-directories containing YAML files. Together, this directory should form a complete configuration bundle. Aside from some basic validation of the YAML files themselves, the Rhino declarative configuration cannot be validated as the upload process is unaware of the components of the VM. Therefore, ensure the Rhino declarative configuration only configures components that exist in your application.

For troubleshooting purposes, viewing the initconf log, which reports the status of the declarative configuration import, can help monitor and mitigate any issues. This can be done by running one of the following commands after connecting to the VM via SSH:

  • tail -f initconf/initconf.log to monitor initconf in real time as the Rhino declarative configuration is imported into Rhino.

  • cat initconf/initconf.log to inspect any failures that may have occurred during the import process.

Note

Resource adaptors that store part of their configuration in SLEE profiles, such as the Diameter resource adaptors, will not automatically detect changes to these profiles. If you use such resource adaptors, use an after-slee-start configuration hook to run the following command:

rhino-console updateraentityconfigproperties <ra-entity>

This will force the resource adaptor entity to reload the configuration from the SLEE profile, if the declarative configuration import changed it.
Note

Some resource adaptors only offer partial support for active reconfiguration. In this case, even if you specify the --reload-resource-adaptors argument to rvtconfig upload-config, Rhino will raise an "Active reconfiguration" alarm and will not reload the resource adaptor.

Monitor the VM for Rhino alarms, and manually run the following commands:

deactivateraentity <ra-entity>
activateraentity <ra-entity>

to update the configuration. This will however cause an outage, so use with care.
Note

When uploading configuration files, you must also include a Solution Definition File containing all nodes in the deployment (see below). Furthermore, for any VM which runs Rhino, you must also include a valid Rhino license.

Solution Definition File

You will already have written a Solution Definition File (SDF) as part of the creation of the VMs. As the configuration process discovers other RVT nodes using the SDF, this SDF needs to be uploaded as part of the configuration.

Note

The SDF must be named sdf-rvt.yaml, and must contain all nodes in the deployment.

Successful configuration

The configuration process on the VMs starts after bootstrap completes. It is constantly listening for configuration to be written to CDS (via rvtconfig upload-config). Once it detects configuration has been uploaded, it will automatically download and validate it. Assuming everything passes validation, the configuration will then be applied automatically. This can take up to 20 minutes depending on node type.

The configuration process can be monitored using the report-initconf status tool. The tool can be run via an VM SSH session. Success is indicated by status=vm_converged.

Troubleshooting configuration

Like bootstrap, errors are reported to the log file, located at initconf/initconf.log in the default user’s home directory.

initconf initialization failed due to an error: This indicates that initconf initialization has irrecoverably failed. Contact a Customer Care Representative for next steps.

Task <name> marked as permanently failed: This indicates that configuration has irrecoverably failed. Contact a Customer Care Representative for next steps.

<file> failed to validate against YANG schemas: This indicates something in one of the YAML files was invalid. Refer to the output to check which field was invalid, and fix the problem. For configuration validation issues, the VM doesn’t need to be destroyed and recreated. The fixed configuration can be uploaded using rvtconfig upload-config. The configuration process will automatically try again once it detects the uploaded configuration has been updated.

Note If there is a configuration validation error on the VM, initconf will NOT run tasks until new configuration has been validated and uploaded to the CDS.

Other errors: If these relate to invalid field values or a missing license, it is normally safe to fix the configuration and try again. Otherwise, contact a Customer Care Representative.

Configuration alarms

The configuration process can raise the following SNMP alarms, which are sent to the configured notification targets (all with OID prefix 1.3.6.1.4.1.19808.2):

OID Description Details

12355

Initconf warning

This alarm is raised if a task has failed to converge after 5 minutes. Refer to Troubleshooting configuration to troubleshoot the issue.

12356

Initconf failed

This alarm is raised if the configuration process irrecoverably failed, or if the VM failed to quiesce (shut down prior to an upgrade) cleanly. Refer to Troubleshooting configuration to troubleshoot the issue.

12361

Initconf unexpected exception

This alarm is raised if the configuration process encountered an unexpected exception, or if initconf received invalid configuration.

Examine the initconf logs to determine the cause of the exception. If it is due to a validation error, correct any errors in the configuration and try again. (This won’t normally be the case, as rvtconfig validates the configuration before uploading it.)

If initconf hit an unexpected error when applying the configuration, initconf attempts to retry the failed task up to five times. Even if it eventually succeeds on a subsequent attempt, the eventual configuration of the node might not match the desired configuration exactly, or a component may be left in a partly-failed state. We therefore recommend that you investigate further.

This alarm must be administratively cleared as it indicates an issue that requires manual intervention.

12363

Configuration validation warning

This alarm is raised if the VM’s configuration contains items that require attention, such as expired or expiring REM certificates. The configuration will be applied, but some services may not be fully operational. Further information regarding the configuration warning may be found in the initconf log.

12364

OCSS7 reconfiguration attempt blocked

This alarm is raised if the VM configuration has changed, and the change would result in the OCSS7 SGC being reconfigured.

It is not currently possible to reconfigure OCSS7 through changing the YAML configuration alone. Components other than the OCSS7 SGC will be updated to the new configuration, but the OCSS7 SGC component will retain its existing configuration.

Review the configuration changes and revert the SS7-related changes if they are not required. To apply the changes to the OCSS7 SGC, follow the procedure documented in Reconfiguring the SGC.
Previous page Next page
VM Build Container Version 3.2