This page describes the methodology used when running the benchmarks.
Rationale
Benchmarks were performed using a real IP-SM-GW cluster, Cassandra, and simulated network functions. A real Cassandra cluster is used, as simulating this is impractical.
In our benchmarks the IP-SM-GW cluster processed calls for both originating and terminating triggers as well as third party registrations. Each SMS is processed in exactly one trigger — either originating or terminating.
The simulated network functions are run on separate hosts from the IP-SM-GW cluster. This avoids potential CPU and memory contention, and includes some network induced latency. The network functions (HLR, MSC, SMSC, and SCSCF) were simulated to abstract away other performance considerations for these functions.
Benchmarks were run at the maximum sustainable load level for each node. In this configuration there is no tolerance for node failure, any additional incoming messages will be dropped. To allow for node failure, additional nodes need to be added to provide an acceptable margin (an N+K configuration).
Subscriber definition
We assume that a single subscriber will send two SMS messages during the busy hour, and two registration requests every hour. Registration handling and SMS handling require a similar level of resources.
SMS handling requires two triggers, MO and MT. To help ensure performance does not fall short of expectations we assume that all SMS messages will be on net. This requires that our IP-SM-GW handle both triggers for all sessions. Similarly, we assume all registration attempts are initial, as this maximises the work done by IP-SM-GW per registration.
Cluster configurations
We tested a 3 IP-SM-GW nodes on 3 EC2 virtual machine hosts. The Cassandra database is also configured as a 3 node cluster on 3 virtual hosts.
Test setup
Each test includes a ramp-up period of 4 minutes before full load is reached. This is included as the Oracle JVM provides a Just In Time (JIT) compiler. The JIT compiler compiles Java bytecode to machinecode, and recompiles code on the fly to take advantage of optimizations not otherwise possible. This dynamic compilation and optimization process takes some time to complete. During the early stages of JIT compilation/optimization, the node cannot process full load.
4 minutes of ramp up allows for the majority of JIT compilation. At this point, the node is ready to enter full service.
The tests are run for one hour after reaching full load. Load is not stopped between ramp up and starting the test timer.
