Search Engine: Elastic

Article ID: 132461, created on Mar 29, 2018, last review on Mar 29, 2018

  • Applies to:
  • Operations Automation 7.0
  • Operations Automation 7.3

Symptoms

OACI IM was found unmanageable and according to /var/log/pa/vps.log, heap space of OACI IM JVM instance has been exceeded:

ERROR ServletHandler [Grizzly-worker(1)] - service exception:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3236) ~[na:1.8.0_112]
        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[na:1.8.0_112]
        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[na:1.8.0_112]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[na:1.8.0_112]

The closest request to the crash was related to autoscale and resource usage history retrieval:

INFO  CustomLoggingFilter [Grizzly-worker(1)] - 69613 * Server in-bound request
69613 > GET http://192.168.139.2:4465/paci/v1.0/of/1087335/ve/main/autoscale/history/2017-12-23%2014%3A20%20%2B0300/2018-03-24%2014%3A20%20%2B0300

Target VE was checked, but no outstanding records were found.

IM automatically saves heap dumps upon OutOfMemory error. Name and path to heap dump (*.hprof file) could be retrieved from vps.startup.log on IM node.

Cause

Heap usage distribution shows that the most of it were consumed by a line of text (byte array) which was processed inside of ResourceConsumptionAndAutoscaleHistoryExtImpl class. Checking the value which consumed that memory showed repeating <time-out/> in XML construct:

<resource-consumption-and-autoscale-history>
    <time-out/>
    <time-out/>
    <time-out/>

Cause of this is an infinite loop which occurred due to CCU-17675.

Resolution

Since the loop that caused an issue is infinite, increase of heap will not help. There is no workaround for the issue, but the conditions for its occurrence are rare, so it should not appear frequently. CCU-17675 will be fixed in future product versions.

To obtain more information on CCU-17675 you may contact your Technical Manager.

caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 5356b422f65bdad1c3e9edca5d74a1ae 0871c0b47b3b86ae3b1af4c2942cd0ce 1941880841f714e458ae4dc3d9f3062d dd79f16c76b9dca75205ab5e9f5465ea

Email subscription for changes to this article
Save as PDF