This book contains performance benchmarks using the Sh Cache Microservice
Topics
Descriptions of each of the benchmark scenarios, and notes on the benchmark methodology used |
|
Details of the hardware, software, and configuration used for the benchmarks |
|
Summaries of the benchmarks and links to detailed metrics. |
Other documentation for the Sh Cache Microservice can be found on the Sh Cache Microservice product page.
Benchmark Scenarios and Methodology
Methodology
5 scenarios were used in the client simulator
put-mmtel-services.scen put-stn-sr.scen get-mmtel-services.scen get-msisdn.scen ue-reachability-subscribe.scen
The two PUT scenarios contain a PUT and then a GET for the modified data.
The two GET scenarios contain two GETs each for the same data.
The SUBSCRIBE scenario contains one POST.
If each scenario is run 10 times, that equates to 50 simulator "sessions", and 90 requests over the wire. So 1 SPS equals 1.8 requests per second.
Scenarios
For the benchmark we used 5 scenarios to give a broad load profile. Each scenario includes the client call to ShCM and the call from ShCM to the HSS. The scenario mix was weighted to 84% GETS, 11% PUTs, and 5% Susbcribes, to mimic the likely load profile from VoLTE.
GET MMTel-Services
The client-ShCM scenario:
The ShCM-HSS scenario:
This scenario retrieves a subscriber’s transparent data. This is the data returned by the HSS:
<Sh-Data>
<RepositoryData>
<ServiceIndication>MMTEL-Services</ServiceIndication>
<SequenceNumber>1</SequenceNumber>
<ServiceData>
<MMTelServices xmlns:ss="http://uri.etsi.org/ngn/params/xml/simservs/xcap" xmlns:cp="urn:ietf:params:xml:ns:common-policy" xmlns:ocp="urn:oma:xml:xdm:common-policy" xmlns="http://uri.etsi.org/ngn/params/xml/simservs/xcap">
<version>1</version>
<complete-originating-identity-presentation>
<originating-identity-presentation active="true"/>
<operator-originating-identity-presentation authorized="true">
<restriction-override>override-not-active</restriction-override>
</operator-originating-identity-presentation>
</complete-originating-identity-presentation>
<complete-originating-identity-restriction>
<originating-identity-presentation-restriction active="true">
<default-behaviour>presentation-not-restricted</default-behaviour>
</originating-identity-presentation-restriction>
<operator-originating-identity-presentation-restriction authorized="true">
<mode>temporary</mode>
<restriction>all-private-information</restriction>
</operator-originating-identity-presentation-restriction>
</complete-originating-identity-restriction>
<complete-communication-diversion>
<communication-diversion active="true">
<cp:ruleset>
<cp:rule id="CFU">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61212266406</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfb">
<cp:conditions>
<busy/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfnrc">
<cp:conditions>
<rule-deactivated/>
<not-reachable/>
</cp:conditions>
<cp:actions>
<forward-to>
<target/>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfnr">
<cp:conditions>
<rule-deactivated/>
<no-answer/>
</cp:conditions>
<cp:actions>
<forward-to>
<target/>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
</cp:ruleset>
</communication-diversion>
<operator-communication-diversion authorized="true">
<communication-retention-on-invocation>clear-communication-on-invocation-of-diversion</communication-retention-on-invocation>
<retention-when-diverting-rejected-at-diverted-to-user>no-action-at-diverting-user</retention-when-diverting-rejected-at-diverted-to-user>
<total-number-of-diversions-for-each-communication>3</total-number-of-diversions-for-each-communication>
<cdiv-indication-timer>0</cdiv-indication-timer>
<communication-forwarding-on-no-reply-timer>30</communication-forwarding-on-no-reply-timer>
<cdivn-buffer-timer>0</cdivn-buffer-timer>
<cp:ruleset>
<cp:rule id="operator-default-cfb">
<cp:conditions>
<rule-deactivated/>
<busy/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="operator-default-cfnrc">
<cp:conditions>
<rule-deactivated/>
<not-reachable/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="operator-default-cfna">
<cp:conditions>
<no-answer/>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
</cp:ruleset>
</operator-communication-diversion>
</complete-communication-diversion>
<complete-communication-waiting>
<communication-waiting active="true"/>
<operator-communication-waiting authorized="true">
<calling-user-receives-notification-his-call-is-waiting>false</calling-user-receives-notification-his-call-is-waiting>
</operator-communication-waiting>
</complete-communication-waiting>
<complete-communication-hold>
<operator-communication-hold authorized="true"/>
</complete-communication-hold>
<complete-communication-barring>
<incoming-communication-barring active="false">
<cp:ruleset>
<cp:rule id="User-BAIC">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-BAIC-Roam">
<cp:conditions>
<rule-deactivated/>
<roaming/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
</cp:ruleset>
</incoming-communication-barring>
<outgoing-communication-barring active="false">
<cp:ruleset>
<cp:rule id="User-BAOC">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-boic">
<cp:conditions>
<rule-deactivated/>
<international/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-boice-exhc">
<cp:conditions>
<rule-deactivated/>
<international-exHC/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
</cp:ruleset>
</outgoing-communication-barring>
<operator-incoming-communication-barring authorized="true"/>
<operator-outgoing-communication-barring authorized="true"/>
</complete-communication-barring>
<complete-conference>
<operator-conference authorized="true"/>
</complete-conference>
</MMTelServices>
</ServiceData>
</RepositoryData>
</Sh-Data>
GET MSISDN
The client-ShCM scenario:
The ShCM-HSS scenario:
This scenario retrieves the MSISDN from HSS using the data reference MSISDN.
PUT MMTel-Services
The client-ShCM scenario:
The ShCM-HSS scenario:
This scenario sends a PUR request to the HSS with this data:
<Sh-Data>
<RepositoryData>
<ServiceIndication>MMTEL-Services</ServiceIndication>
<SequenceNumber>1</SequenceNumber>
<ServiceData>
<MMTelServices xmlns:ss="http://uri.etsi.org/ngn/params/xml/simservs/xcap" xmlns:cp="urn:ietf:params:xml:ns:common-policy" xmlns:ocp="urn:oma:xml:xdm:common-policy" xmlns="http://uri.etsi.org/ngn/params/xml/simservs/xcap">
<version>1</version>
<complete-originating-identity-presentation>
<originating-identity-presentation active="true"/>
<operator-originating-identity-presentation authorized="true">
<restriction-override>override-not-active</restriction-override>
</operator-originating-identity-presentation>
</complete-originating-identity-presentation>
<complete-originating-identity-restriction>
<originating-identity-presentation-restriction active="true">
<default-behaviour>presentation-not-restricted</default-behaviour>
</originating-identity-presentation-restriction>
<operator-originating-identity-presentation-restriction authorized="true">
<mode>temporary</mode>
<restriction>all-private-information</restriction>
</operator-originating-identity-presentation-restriction>
</complete-originating-identity-restriction>
<complete-communication-diversion>
<communication-diversion active="true">
<cp:ruleset>
<cp:rule id="CFU">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61212266406</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfb">
<cp:conditions>
<busy/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfnrc">
<cp:conditions>
<rule-deactivated/>
<not-reachable/>
</cp:conditions>
<cp:actions>
<forward-to>
<target/>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="cfnr">
<cp:conditions>
<rule-deactivated/>
<no-answer/>
</cp:conditions>
<cp:actions>
<forward-to>
<target/>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
</cp:ruleset>
</communication-diversion>
<operator-communication-diversion authorized="true">
<communication-retention-on-invocation>clear-communication-on-invocation-of-diversion</communication-retention-on-invocation>
<retention-when-diverting-rejected-at-diverted-to-user>no-action-at-diverting-user</retention-when-diverting-rejected-at-diverted-to-user>
<total-number-of-diversions-for-each-communication>3</total-number-of-diversions-for-each-communication>
<cdiv-indication-timer>0</cdiv-indication-timer>
<communication-forwarding-on-no-reply-timer>30</communication-forwarding-on-no-reply-timer>
<cdivn-buffer-timer>0</cdivn-buffer-timer>
<cp:ruleset>
<cp:rule id="operator-default-cfb">
<cp:conditions>
<rule-deactivated/>
<busy/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="operator-default-cfnrc">
<cp:conditions>
<rule-deactivated/>
<not-reachable/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
<cp:rule id="operator-default-cfna">
<cp:conditions>
<no-answer/>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<forward-to>
<target>tel:+61222022000</target>
<notify-caller>false</notify-caller>
</forward-to>
</cp:actions>
</cp:rule>
</cp:ruleset>
</operator-communication-diversion>
</complete-communication-diversion>
<complete-communication-waiting>
<communication-waiting active="true"/>
<operator-communication-waiting authorized="true">
<calling-user-receives-notification-his-call-is-waiting>false</calling-user-receives-notification-his-call-is-waiting>
</operator-communication-waiting>
</complete-communication-waiting>
<complete-communication-hold>
<operator-communication-hold authorized="true"/>
</complete-communication-hold>
<complete-communication-barring>
<incoming-communication-barring active="false">
<cp:ruleset>
<cp:rule id="User-BAIC">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-BAIC-Roam">
<cp:conditions>
<rule-deactivated/>
<roaming/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
</cp:ruleset>
</incoming-communication-barring>
<outgoing-communication-barring active="false">
<cp:ruleset>
<cp:rule id="User-BAOC">
<cp:conditions>
<rule-deactivated/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-boic">
<cp:conditions>
<rule-deactivated/>
<international/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
<cp:rule id="User-boice-exhc">
<cp:conditions>
<rule-deactivated/>
<international-exHC/>
</cp:conditions>
<cp:actions>
<allow>false</allow>
</cp:actions>
</cp:rule>
</cp:ruleset>
</outgoing-communication-barring>
<operator-incoming-communication-barring authorized="true"/>
<operator-outgoing-communication-barring authorized="true"/>
</complete-communication-barring>
<complete-conference>
<operator-conference authorized="true"/>
</complete-conference>
</MMTelServices>
</ServiceData>
</RepositoryData>
</Sh-Data>
Hardware and Software
Software
JVM Setting for ShCM
Oracle Java jdk1.8.0_172 was used for the testing.
Rhino was assigned 3072m for Heap and 512m for New Size
Aside from the standard Rhino JVM settings these were added:
-
-XX:+TieredCompilation
-
-XX:+ParallelRefProcEnabled
-
-XX:ReservedCodeCacheSize=756m
-
-XX:InitialCodeCacheSize=756m
-
-Dio.netty.leakDetectionLevel=advanced
-
-XX:CMSTriggerInterval=300000
Benchmark Results
Server configuration
Server Config A: 3 c5.2xlarge servers. 1 server with Rhino and Cassandra colocated. Single Cassandra node. 2 servers for client simulator.
Server Config B: 6 c5.2xlarge servers. 1 server for Rhino. 3 servers for Cassandra. Replicated 3-node Cassandra cluster. 2 servers for client simulator.
Server Config C: 3 c5.2xlarge servers. 1 server with Rhino and Cassandra colocated. Single Cassandra node. 2 servers for client simulator. All caching disabled.
Results
SPS - Sessions per second
RPS - Requests per second
Rhino Performance
| Test | SPS | RPS | Duration (hours) | Server config | CPU (average) | Heap (average) |
|---|---|---|---|---|---|---|
1 |
1000 |
1800 |
2 |
A |
280% |
1.4G |
2 |
1100 |
1980 |
2 |
A |
310% |
1.4G |
3 |
1200 |
2160 |
2 |
A |
330% |
1.4G |
4 |
1000 |
1800 |
2 |
B |
300% |
1.5G |
5 |
1200 |
2160 |
2 |
B |
320% |
1.5G |
6 |
1400 |
2520 |
2 |
B |
380% |
1.5G |
7 |
1000 |
1800 |
2 |
C |
360% |
1.5G |
 |
The CPU reading is for the ShCM JVM, not the whole host. |
Client latency in milliseconds grouped by percentile
| Test | SPS | RPS | Duration (hours) | Server config | 50th | 75th | 90th | 95th | 99th | Max |
|---|---|---|---|---|---|---|---|---|---|---|
1 |
1000 |
1800 |
2 |
A |
6 |
17.9 |
58.8 |
119 |
264 |
757 |
2 |
1100 |
1980 |
2 |
A |
20.1 |
84.7 |
292 |
438 |
655 |
1210 |
3 |
1200 |
2160 |
2 |
A |
156 |
686 |
1190 |
1530 |
2810 |
3889 |
4 |
1000 |
1800 |
2 |
B |
4.3 |
5.2 |
15.6 |
37.1 |
119 |
837.9 |
5 |
1200 |
2160 |
2 |
B |
4.2 |
5.8 |
22.4 |
41.9 |
131 |
367.2 |
6 |
1400 |
2520 |
2 |
B |
4.8 |
9.8 |
37.4 |
63.7 |
170 |
476 |
7 |
1000 |
1800 |
2 |
C |
2.8 |
21.5 |
88.4 |
139 |
235 |
717 |
Cassandra Disk Usage
All tests had the Cassandra disk usage monitored. The disk usage profile appeared to be the same for all tests.
The default Cassandra settings allow for the commitlog to grow to 8GB before being cleaned up.
Cassandra disk usage maxed at around 9GB for each test.
For the tests with multiple Cassandra nodes in a cluster, each node/server showed the same disk usage profile.
Conclusion
Testing showed that a single ShCM node operating with a 3-node replicated Cassandra cluster can sustainably handle 2160 requests per seconds from a subscriber data pool of 1m, using around 300% CPU with reasonable request latencies (test 5).
Even in the case the RPS was 2520 (test 6) the latency and CPU usage were satisfactory, inside of acceptable ranges.