[Liu-HongYe]

The JVM in the container does not set the memory size limit, which causes the JVM to use 1GB of memory by default. At this time, if the function sets the 256MB memory limit, when the memory uses 256MB, the GC operation cannot be performed in time. The container is killed.
And in the case of the Prewarm container, the memory used by the last call cannot be recycled in time, so the new call available memory is drastically reduced.
I recommend dynamically setting the JVM memory limit by reading the Cgroup configuration during container startup.

[rabbah]

Fixes #5?

[Liu-HongYe]

Fixes #5?

yep!

[mrutkows]

Please see article:
https://medium.com/adorsys/jvm-memory-settings-in-a-container-environment-64b0840e1d9e

"To make the JVM play well with cgroup memory limits a new option -XX:+UseCGroupMemoryLimitForHeap was introduced.

It allows setting the heap to the cgroup memory limit. The JVM reads from /sys/fs/cgroup/memory/memory.limit_in_bytes and uses that value instead of -XX:MaxRAM."

$ docker run --rm -m 1g openjdk:8-jdk cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1073741824$ docker run --rm -m 1g openjdk:8-jdk sh -c "java -XX:+PrintFlagsFinal -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -version | grep -Ei 'maxheapsize|maxram'"
    uintx DefaultMaxRAMFraction   = 4             {product}
    uintx MaxHeapSize            := 268435456     {product}     # = 1073741824 / 4
    uint64_t MaxRAM               = 137438953472  {pd product}
    uintx MaxRAMFraction          = 4             {product}

[chetanmeh]

@mrutkows Currently the java runtime uses openj9 instead of openjdk

openwhisk-runtime-java/core/java8/Dockerfile

Line 18 in a8e666f

FROM adoptopenjdk/openjdk8-openj9:x86_64-ubuntu-jdk8u212-b04_openj9-0.14.2

Open J9 does not support -XX:+UseCGroupMemoryLimitForHeap and instead has a different flag UseContainerSupport. This flag is enabled by default in Docker base images we use in OpenWhisk

$ docker run --rm -m 1g openwhisk/java8action /bin/bash -c 'echo $JAVA_TOOL_OPTIONS'
-XX:+IgnoreUnrecognizedVMOptions -XX:+UseContainerSupport -XX:+IdleTuningCompactOnIdle -XX:+IdleTuningGcOnIdle

See also this blog post.

Also j9 does not support -XX:+PrintFlagsFinal instead provides few different flags to get the defaults. Based on that

$ docker run --rm -m 1g openwhisk/java8action sh -c "java -verbose:sizes -version"
  -Xmca32K        RAM class segment increment
  -Xmco128K       ROM class segment increment
  -Xmcrs200M      compressed references metadata initial size
  -Xmns2M         initial new space size
  -Xmnx128M       maximum new space size
  -Xms8M          initial memory size
  -Xmos6M         initial old space size
  -Xmox512M       maximum old space size
  -Xmx512M        memory maximum
  -Xmr16K         remembered set size
  -Xlp:objectheap:pagesize=4K	 large page size
                  available large page sizes:
                  4K
  -Xlp:codecache:pagesize=4K	 large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  4K
  -Xmso256K       operating system thread stack size
  -Xiss2K         java thread stack initial size
  -Xssi16K        java thread stack increment
  -Xss1M          java thread stack maximum size
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
Eclipse OpenJ9 VM (build openj9-0.9.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20180813_291 (JIT enabled, AOT enabled)
OpenJ9   - 24e53631
OMR      - fad6bf6e
JCL      - a05586ac based on jdk8u181-b13)

Per above -Xmx512M memory maximum it appears that it is configured to use 512M when container is launched with 1 GB memory

So I believe we are good here and do not need to explicitly specify memory settings for build version we are using

[rabbah]

cc @sven-lange-last

[mrutkows]

@chetanmeh was aware of the underlying JVM being used and hoped that J9 had similar options or made similar considerations. In addition, as I have access to some of the IBM JVM people and wanted to better understand where things were at in case we needed to approach them.

[sven-lange-last]

@chetanmeh thanks for investigating. I agree that JVM heap size adoption to the action container memory limit should be addressed with #95. The OpenJ9 version we are using now in the managed Java runtime image should adapt to the cgroup's memory limit automatically, because option -XX:+UseContainerSupport is now default.

I think we should close this PR as well as #5 because this area has been addressed.

fyi @falkzoll

[mrutkows]

I too believe this can be closed along with the corresponding issue. Details of heap treatment are well described in contents linked by Chetan (copied here for convenience):

OpenJ9 VM detects when it is running inside a container that imposes a memory limit, and adjusts the maximum Java heap size appropriately. The following table shows the values that are used when -XX:+UseContainerSupport is set:

Container memory limit | Maximum Java heap size

• Less than 1 GB : 50%
• 1 GB - 2 GB : - 512 MB
• Greater than 2 GB : 75%

Currently, our specific base image uses build 0.14.2 (i.e., x86_64-alpine-jdk8u212-b04_openj9-0.14.2) and this version/changelon indicates the "container aware" heap behavior was added as of "0.11.1" release level:
https://www.eclipse.org/openj9/oj9_whatsnew.html

specifically, including a section with:

We recently added some intelligence to make OpenJ9 "container-aware", 
which allows it to make informed decisions about the size of your Java heap 
if you run your application in a container. ```

[Liu-HongYe]

Okay, I also think we can close this PR. Thx.