Deploying spring boot app to cloud foundry fails with JVM memory issue

Question

Deploying spring boot application to Pivotal Cloud Foundry fails with below error. Limit for the instance was set to 2GB and then 4GB. Locally application starts up fine with the 2GB heap size. Application fails during initialisation of the cache manager. Any idea what could be the root cause? not sure what role terracotta plays in cloud foundry and what else to look for?

 2019-05-07T19:24:53.39+0530 [APP/PROC/WEB/0] OUT     at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
   2019-05-07T19:24:53.39+0530 [APP/PROC/WEB/0] OUT Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.cache.CacheManager]: Factory method 'cacheManager' threw exception; nested exception is org.ehcache.StateTransitionException: Cache '<name hidden>' creation in EhcacheManager failed.
....
 Caused by: java.lang.IllegalArgumentException: An attempt was made to allocate more off-heap memory than the JVM can allow. The limit on off-heap memory size is given by the -XX:MaxDirectMemorySize command (or equivalent).
   2019-05-07T19:15:13.15+0530 [APP/PROC/WEB/0] OUT     at org.terracotta.offheapstore.paging.UpfrontAllocatingPageSource.bufferAllocation(UpfrontAllocatingPageSource.java:564)
   2019-05-07T19:15:13.15+0530 [APP/PROC/WEB/0] OUT     at org.terracotta.offheapstore.paging.UpfrontAllocatingPageSource.lambda$allocateBackingBuffers$1(UpfrontAllocatingPageSource.java:513)
   2019-05-07T19:15:13.15+0530 [APP/PROC/WEB/0] OUT     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   2019-05-07T19:15:13.15+0530 [APP/PROC/WEB/0] OUT     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   2019-05-07T19:15:13.15+0530 [APP/PROC/WEB/0] OUT     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   2019

EDIT: fixed by removing off-heap ehcache configuration: maxBytesLocalOffHeap

Answer 1

By default, the Java buildpack will constrain the various memory regions of the JVM. It does this so that it can make sure your Java applications do not exceed the memory limit of the container, which would be bad and would cause your app to crash.

You app is attempting to create a amount of direct memory that is above the limit set by the Java buildpack. As I write this, the Java buildpack is setting -XX:MaxDirectMemorySize=10M.

To make this work, you just need to set an environment variable, either in your manifest.yml or with cf set-env, called JAVA_OPTS and the contents need to include -XX:MaxDirectMemorySize=XXM where XX is the larger value that you want to use.

The Java buildpack will see when you adjust memory settings like this and will reduce the size of other memory regions to compensate. If you are trying to use a large amount of direct memory, this could cause you to not have enough memory for other regions. If that happens, you would need to increase the overall memory limit for your application.

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-open_jdk_jre.md#java-options

Hope that helps!