Analysis of JVM Off-Heap Caching Libraries

Caching is an integral part of any Java application now a days. Among different variants, in memory caching/on heap caching  is most preferred for its simplicity and ease. Applications tend to dump more and more long lived objects in cache leading to huge heap allocations.With large heaps, at some point – typically starting at around 4 GB – you will start having issues with your garbage collection pauses.

Off Heap Cache shines out as strong alternative in such scenario as:

  1. Garbage Collector does not manage Off Heap memory, so large memory chunks can be allocated without impacting GC cycles.
  2. Its much faster than disk/file based caches,  with only added overhead of serialization and deserialization.

Analysis

We have done a comparative analysis of following Off Heap Caching Solutions :

  • HugeCollections (https://github.com/OpenHFT/HugeCollections)
  • Apache DirectMemory (http://directmemory.apache.org/)
  • MapDB (http://www.mapdb.org/)
  • HazelCast Enterprise (http://hazelcast.com/products/hazelcast-enterprise/)
  • Terracotta BigMemory GO (http://terracotta.org/products/bigmemorygo)

Note: Concurrent HashMap has been tested for bench marking purpose only.

Single Threaded Performance

Aim: Average Get/Put timings in Single threaded non contention environment.

Method:

  1. Put  entries with total size ~ 1 GB in cache with keys : String,  Value : int array.  For better results, we tried values of three sizes : 1 kb, 10 kb, 100kb.
  2. Once the cumulative put succeeded, we initiated GC. This was done to make sure cache data is picked from off heap during Get operation and not from on heap leftovers.
  3. We carried out Get operation on 1 GB of data.
  4. Average was calculated from cumulative Get/Put operation data.

Get Operation Time Comparison

Caching Solutions Average time to GET one entry (100KB)(microseconds) Average time to GET one entry (10KB)(microseconds) Average time to GET one entry (1KB)(microseconds)
ConcurrentHashMap (on Heap) 1.1779 0.55 0.35
Ehcache (BigMemory GO) 418.2016 66.08 26.52
MapDb 6.7374 11.18 3.908
HazelCast 232.6716 51 29
Apache DirectMemory 153.737 21.96 8.6
HugeCollections 125.7 21.15 9.7

 

 

 

Put Operation Time Comparison

Caching Solutions Average time to PUT one entry (100KB)(microseconds) Average time to PUT one entry (10KB)(microseconds) Average time to PUT one entry (1KB)(microseconds)
ConcurrentHashMap (on Heap) 93.5364 11.13 3.79
Ehcache (BigMemory GO) 306.2924 34 7.22
MapDb 767.797 64.26 11
HazelCast 439.3761 53.94 24.15
Apache DirectMemory 229.2358 28.95 14.07
HugeCollections 218.6846 27.08 5.25

Multi Threaded Performance

Aim: Average Get/Put timings in Multi -Threaded Environment under different Put/Get operation load

 

 Method:

  1. Populated Cache with 1 GB entries of size 1 kb each.
  2. Started 15 Threads for Get Requests and 5 for Put requests. Total number of Get/Put operations were 1 Million.
  3. Calculated timings of each Get/Put opeartion and derived the percentiles.
  4. Collected data for  3 different GET:Put operation ratios namely 1:9, 1:1, 9:1.

Total GET Requests = 900000, Threads = 15                Total  PUT Requests = 100000, Threads = 5

 

Get Timings per entry (nanoseconds)

 

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 581 826 879 916 946 977 1019 1077 1168 1268 1439 592899368
EhCache 45401 47022 47705 48192 48683 49374 51013 58441.2 162634 811274 4838657 86147625
MapDb 5321 6059 6856 7867 8543 9303 10467 13995 21386 26354 38834 239799047
HazelCast 35414 41723 52876 61159 91645 167318 296883 468651 674227 917238 1974866 61315442
Apache DirectMemory 15168 15660 15865 15996 16106 16208 16324 16491 30022 106449 197021 168048639
HugeCollections 15370 15559 15666 15754 15840 15949 16107 17077 57579 95532 195640 71666147

 

Put Timings per entry (nanoseconds)

 

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 725 921 1042 1133 1218 1311 1427 1633 2212 3598 4282 580121780
EhCache 28020 29080 41537 61934 70338 83507 106454 147082 346864 1710551 9532719 77605319
MapDb 15626 17268 18527 19805 21445 22978 24480 28571 37420 53448 160910 124020424
HazelCast 30623 42089 57268 68323 94814 125547 176235 295096 618926 1270576 2616803 44979232
Apache DirectMemory 11815 12714 13434 14255 18243 25165 46600 64029 80366 95447 115082 100820385
HugeCollections 6629 7057 17439 18560 32318 35159 36890 54238 68022 72221 1329673 55592628

Total GET Requests = 500000, Threads = 15                  Total  PUT Requests = 500000, Threads = 5

Get Timings per entry (nanoseconds)

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 786 843 883 922 963 1006 1056 1123 1238 1365 1801 743907974
EhCache 45810 46922 47629 48319 49370 51411 55139 95417 344240 1720036 6854181 412693493
MapDb 5421 6976 8431 9354 10487 13085 17904 22071 27303 33862 52967 746301982
HazelCast 47653.9 62279.6 92711.4 133579.8 178993 241938.6 354281.8 544853.4 907082 1613059.4 3238632.6 239053963
Apache DirectMemory 0 15383 15906 16224 16481 28030 31742 55999 104666 214214 253821 1876179849
HugeCollections 16428 16674 16828 16976 17195 18366 40332 43913 54330.1 129110.6 353679.4 67836488

 

Put Timings per entry (nanoseconds)

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 677 751 811 876 940 1008 1088 1200 1399 1727 3381 799844628
EhCache 26567 27627 27710 28067 28421 28857 29604 32416 71149 167201 1192888 509715272
MapDb 10601 10747 10829 10951 11238 14168 19290 25284 30510 37936 46501.02 806254639
HazelCast 28524 36274.8 46106.7 59210 74602 92142.4 118168.3 160275 259808.2 407754.3 818924.4 234964325
Apache DirectMemory 8992 9570 9883 10160 10442 10754 11164 11922 29974.1 72463 146979 1903813119
HugeCollections 5668 5870 5997 6106 6222 6367 6576 6922 30957 64072 75028.04 54467105

 

Total GET Requests = 100000, Threads = 15                    Total  PUT Requests = 900000, Threads = 5

 

Get Timings per entry (nanoseconds)

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 719 788 836 880 927 983 1048 1157 1647 2069 2815.14 13975174
EhCache 73127 76577 106909.2 131563 152691.5 179894.4 264266.3 518611 2003461.2 9352409.7 18285421.4 292189404
MapDb 8504.9 10964 17362.7 20820 25311 27049.4 29699.3 34857.4 50318.2 86729.1 222365.6 121741349
HazelCast 130620 181523.6 264081.2 377895.4 562635.5 871186.2 1329692.6 1990812.2 3199998.4 4639420.2 6995621.5 63787909
Apache DirectMemory 47325.9 48677 49826 56501 97305.5 106499 141004.3 172272.4 232646.1 257973.1 9224188.5 2076653447
HugeCollections 49541.8 54795 57651 91153 102241.5 108895 130797.3 162659.4 200245.2 747494.1 9907957.2 104650300

 

Put Timings per entry (nanoseconds)

Percentile 10% 20% 30% 40% 50% 60% 70% 80% 90% 95% 98% 100%
ConcurrentHashMap 701 765 821 876 931 1000 1080 1184 1359 1568 2591 1022008519
EhCache 26005 27263 27788 28178 28505 28807 29154 29943 47679 88426 258440.3 286049644
MapDb 10785 12263 16570 21933 23621 26725 32315 37481 46818 56137 84434.1 288558886
HazelCast 22285 24965 27408 29366 32993 37720 44605 55810 94477 187548.1 523554.4 140060304
Apache DirectMemory 7091 9141 9446 9670 9887 10123 10428 10979 13243.1 24171 91061.06 2073578000
HugeCollections 5600 5794 5919 6032 6150 6289 6460 6699 7219 25576 67046.02 65894894

Conclusion

  • MapDB seems to consistently perform well in both GET/PUT operations on single/multi-threaded environment.  The project is nearly a year old, but is maturing fast with nearly monthly releases. However, there remains a caveat. Heap usage remains  considerably high when large values(100kb size) are put in cache even after GC.
  • HugeCollections, although being quiet new, shows promising results especially in GET operations.
  • BigMemory GO seems to lag behind a bit in performance, but gives a hybrid operation mode, wherein cache can be configured to use Heap first and spill excess data on Off Heap store.
  • Apache DirectMemory has not seen any considerable development for past few months, so may not be preferred for production use.
  • Serialization/Deserialization of objects plays a crucial  role in performance of these libraries. Performance might vary with different serialization/deserialization mechanisms  and the kind of Object we insert in cache. These aspects need to be properly evaluated in choice of these libraries.

 

FacebookTwitterGoogle+Share
Back to top