Article ID: 130263, created on Feb 8, 2017, last review on Feb 8, 2017

  • Applies to:
  • Operations Automation

Symptoms

Provider's Control Panel is unavailable. It can be fixed by the restart of the pau service on the UI node.

In console.log on the UI server, multiple messages with OutOfMemoryError could be observed:

Caused by: java.lang.OutOfMemoryError: Java heap space

In pui.log, the information of huge amount of alive sessions could be found:

10:20:30.001 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4473 hibernated=6 killed=3 zombie=1
10:20:30.010 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4476 hibernated=6 killed=3 zombie=0
10:20:30.100 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4477 hibernated=6 killed=6 zombie=0

Cause

The Out Of Memory condition that provoked the Control Panel outage was caused by the incredibly high value of User session timeout (minutes) in PCP > System > Settings > System Properties > General. In this case, the value was set to 1440, which is 24 hours.

As a result, abandoned sessions (i.e. a user worked in panel and then closed the tab) still are considered as active until the timeout of one day is reached.

Information regarding each active session is stored in the cleanup task's queue and consumes about 400KB memory per session. So in order to keep alive around 4000 sessions, it is required to reserve about 1.5 GB of heap:

400000*4000 = 1.6*10^9 ~ 1.5GB

Resolution

In order to workaround this misconfiguration, restart the pau service on the UI node.

This issue can be fixed by decreasing the PCP > System > Settings > System Properties > General > User session timeout (minutes) parameter. The default value is 15 minutes. One hour timeout is also widely used.

Moreover, if there is an intention to use CCPv2, the value of 30 minutes or less is recommended.

Another possible solution is to increase the heap size (Xmx) in /usr/local/pem/wildfly-8.2.0.Final/bin/standalone.conf in the following line:

export JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:MaxPermSize=128m -XX:+UseParallelOldGC -Dplesk.root=/usr/local/pem -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true "

Search Words

Xmx

Abandoned sessions cleanup timer

CP outage

User session timeout

java.lang.OutOfMemoryError

OOM

Java heap space

outage

out of memory

heap

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07

Email subscription for changes to this article
Save as PDF