I had an OOM issue with my spring boot app container running on k8. It was a classic case where the NMT report committed memory stayed the same but the RSS of the java process kept on increasing. After spending days I have found a couple of good articles that suggested to either use jemalloc or set MALLOC_ARENA_MAX to some lower value. Currently, it is around 128.
Now I did see improvement after setting MALLOC_ARENA_MAX to 4 or 16 but my concern is to fully understand the issue. Here are a couple of questions that I want to wrap my head around in order to connect all the dots.
Q1: When I increase MALLOC_ARENA_MAX, the thread count increases in NMT report. What is the relation between the two?
My understanding: Is it that with more arenas available, the allocator will allocate thread stacks to new arenas & from a code perspective the thread pool might be unbounded & depend on a number of arenas available.
Q2: In articles, it's suggested that with more arenas, glibc memory fragmentation will increase which makes sense. When I do pmap -x 1 I notice many 1012k blocks with RSS around 110 etc.
pmap 1 | grep -c 12K
474
You can see the 1012 blocks and number of threads from below attached NMT is pretty much the same i.e 474, 483. Size is pretty much same too if we do the math: 0.11MB/stack * 474 = 52 which is kind of close to 62.
Native Memory Tracking:
Total: reserved=1719MB, committed=1043MB
- Java Heap (reserved=500MB, committed=500MB)
(mmap: reserved=500MB, committed=500MB)
- Class (reserved=249MB, committed=137MB)
(classes #25385)
( instance classes #24091, array classes #1294)
(malloc=5MB #88845)
(mmap: reserved=244MB, committed=132MB)
( Metadata: )
( reserved=116MB, committed=116MB)
( used=113MB)
( free=3MB)
( waste=0MB =0.00%)
( Class space:)
( reserved=128MB, committed=17MB)
( used=15MB)
( free=2MB)
( waste=0MB =0.00%)
- Thread (reserved=487MB, committed=64MB)
(thread #483)
(stack: reserved=485MB, committed=62MB)
(malloc=2MB #2900)
(arena=1MB #964)
- Code (reserved=250MB, committed=109MB)
(malloc=8MB #28044)
(mmap: reserved=242MB, committed=101MB)
- GC (reserved=93MB, committed=93MB)
(malloc=42MB #164839)
(mmap: reserved=51MB, committed=51MB)
- Compiler (reserved=3MB, committed=3MB)
(malloc=3MB #3219)
- Internal (reserved=6MB, committed=6MB)
(malloc=6MB #7627)
- Other (reserved=83MB, committed=83MB)
(malloc=83MB #311)
- Symbol (reserved=27MB, committed=27MB)
(malloc=25MB #329602)
(arena=2MB #1)
- Native Memory Tracking (reserved=10MB, committed=10MB)
(tracking overhead=10MB)
- Shared class space (reserved=9MB, committed=9MB)
(mmap: reserved=9MB, committed=9MB)
- Module (reserved=1MB, committed=1MB)
(malloc=1MB #3567)
- Synchronizer (reserved=1MB, committed=1MB)
(malloc=1MB #4247)
What I want is to visually see what exactly consumed more RSS, because the NMT report suggests the total committed as 1043MB while ps aux gives 1481MB, a 438MB excess. Would it be difficult to visualize some address space that shows RSS constant increase in pmap output?
Here are the reference docs
https://github.com/cloudfoundry/java-buildpack/issues/320