This page describes the methodology used when running the benchmarks.
Rationale
The benchmark was performed using a real Sh Cache Microservice, and simulated network functions. The simulated network functions were run on separate hosts from the Sh Cache Microservice. The network functions (HSS, VoLTE, IP-SM-GW) were simulated to abstract away performance considerations for these functions. A real replicated Cassandra database was used, as simulating this effectively is impractical.
The benchmark was run at the maximum sustainable load level for the SH Cache Microservice node. In this configuration there is no tolerance for node failure, any additional incoming messages will be dropped. To allow for node failure, additional nodes need to be added to provide an acceptable margin (an N+K configuration).
Test setup
20 minutes of ramp up allows for the cache to reach a steady state. At this point data is expiring from the cache as fast as it is being replaced, due to a custom cache retention period. Additionally, this allows the majority of JIT compilation to occur before full load is reached. The JIT compiler compiles Java bytecode to machinecode, and recompiles code on the fly to take advantage of optimizations not otherwise possible. This dynamic compilation and optimization process takes some time to complete. During the early stages of JIT compilation/optimization, the node cannot process full load. At this point, the node is ready to enter full service.
The tests are run for two hours after reaching full load. Load is not stopped between ramp up and starting the test timer.
Scenarios
For the benchmark we used 5 scenarios representing common requests made to the SH Cache Microservice. The SH Cache Microservice implements a REST API for interaction with VoLTE and IP-SM-GW. All scenarios are written assuming no errors occur, and every request recieves a 200 response. Each of the 5 scenarios were assigned a data pool of 200,000 unique subscribers to ensure that there is a meaningful amount of data to cache.
GET MMTel-Services
The client-ShCM scenario makes two GET requests for the MMTEL-Services transparent data XML document. Two requests are made to guarantee that there will be a cache hit on the second request. The first request may or may not be a cache hit.
In the case of a cache miss, the SH Cache Microservice will send a User Data Request to the HSS to populate the cache.
GET MSISDN
The client-ShCM scenario makes two GET requests for the MSISDN. Two requests are made to guarantee that there will be a cache hit on the second request. The first request may or may not be a cache hit.
In the case of a cache miss, the SH Cache Microservice will send a User Data Request to the HSS to populate the cache.
PUT MMTel-Services
The client-ShCM scenario makes a PUT request, with a new MMTEL-Services transparent data XML document, and a GET request for the same information.
The SH Cache Microservice will send a Profile Update Request to the HSS for every PUT request.
