This page describes the methodology used when running the benchmarks.
Rationale
Benchmarks were performed using simulated network functions, and a real VoLTE cluster. The simulated network functions are run on separate hosts from the VoLTE cluster. The network functions (HSS (data cached in ShCM), OCS, SCSCF) were simulated to abstract away performance considerations for these functions. Where present, SS7 routing is handled by a pair of real SGC clusters. Network functions reachable via SS7 are simulated.
In our benchmarks the VoLTE cluster processed calls for both originating and terminating triggers. Each call is processed in exactly one trigger — either originating or terminating. SAS tracing is enabled for all benchmarks.
Benchmarks were run at maximum sustainable load level for each node. In this configuration there is no tolerance for node failure, any additional incoming calls will be dropped. To allow for node failure, additional nodes need to be added to provide an acceptable margin (an N+K configuration). As the load distribution from adding K nodes for redundancy over the minimum N nodes for stable operation is strongly dependent on the combined cluster size, we test but do not publish the performance of a cluster sized to support failover.
Capacity overhead to support node failure is calculated based on the maximum acceptable nunber of failed nodes. Typically this is 10% of the cluster, rounded up to the nearest whole number. For example, an installation with up to 10 event processing nodes should have sufficient spare capacity to accept a single node failure. This means that for 10 node cluster, if each node can handle 120SPS, the maximum call rate per deployed node should be 0.9*120 or 108SPS, for a whole-cluster rate of 1080SPS. A three node cluster would only be able to support 240SPS (0.66*120*3) if sized to allow one node to fail.
On virtualised systems, the failure is usually rounded to the nearest multiple of the number of VMs on a single host. For example, a typical deployment with two Rhino nodes per physical host should accept an even number of failed nodes. With the same 10 node cluster and 120SPS the maximum call rate per node is 0.8*120 or 96SPS, for a whole-cluster rate of 960SPS.
Callflows
MMTEL
A representative sample of commonly invoked MMTEL features were used to select the callflows, for these scenarios:
-
51% of calls are originating (VoLTE full preconditions, VoLTE full preconditions with one session refresh)
-
49% of calls are terminating (All other callflows)
|
For the full callflows, see Benchmark Scenarios. |
Each test runs a total of x sessions per second, across all callflows.
| Scenario | Percentage |
|---|---|
VoLTE full preconditions |
50% |
CDIV Success-response |
40% |
CDIV Success-response with OIP |
4.5% |
CDIV-busy-response |
4.5% |
VoLTE full preconditions with 1 session refresh |
1% |
Call setup time (latency) is measured by the simulator playing the initiating role. For all CDIV scenarios, latency is measured from INVITE to final response. For both preconditions scenario, latency is measured from INVITE to ACK.
SCC
A representative sample of commonly invoked SCC features were used to select the callflows, for these scenarios:
-
55% of calls are originating (SCC originating with provisional SRVCC, Access-Transfer originating, Simple IMSSF Originating/Terminating)
-
45% of calls are terminating (SCC TADS, Simple IMSSF Originating/Terminating)
| Scenario | Percentage |
|---|---|
SCC originating with provisional SRVCC |
30% |
SCC TADS |
35% |
Access-Transfer originating |
15% |
Simple IMSSF originating |
10% |
Simple IMSSF terminating |
10% |
Call setup time (latency) is measured by the simulator playing the initiating role. For all SCC scenarios, latency is measured from INVITE to ACK. The access-transfer scenario measures from INVITE to 200.
Cluster configurations
These configurations were tested:
-
3 VoLTE nodes on 3 host machines with replication enabled
-
3 VoLTE nodes on 3 host machines with replication configured but not enabled (default config)
Both configurations include SAS tracing enabled.
|
All hosts are virtual machines in Amazon EC2. |
Test setup
Each test includes a ramp-up period of 15 minutes before full load is reached. This is included as the Oracle JVM provides a Just In Time (JIT) compiler. The JIT compiler compiles Java bytecode to machinecode, and recompiles code on the fly to take advantage of optimizations not otherwise possible. This dynamic compilation and optimization process takes some time to complete. During the early stages of JIT compilation/optimization, the node cannot process full load. JVM garbage collection does not reach full efficiency until several major garbage collection cycles have completed.
15 minutes of ramp up allows for several major garbage collection cycles to complete, and the majority of JIT compilation. At this point, the node is ready to enter full service.
The tests are run for one hour after reaching full load. Load is not stopped between ramp up and starting the test timer.
