Article ID: 126857, created on Sep 10, 2015, last review on Jul 25, 2016

  • Applies to:
  • Operations Automation 6.0

Symptoms

VPS hanged with "configuring firewall" or any other transitional status. This status appeared again in 20 mins after VE was restarted. Recreation of VE also may hang just as any other operations with VE.

As one of other symptoms, backup creation process for VE may report failure while backup has been done correctly on the storage node. That will generate a lot of unregistered backups that consume disk space.

In logs one may found the errors listed below.

/var/log/terracotta/terracotta-server.log:

    [TC Memory Monitor] WARN tc.operator.event - NODE : localhost  Subsystem: MEMORY_MANAGER Message: Current Memory usage(93%) crossed critical threshold(90%).

/var/log/PACI-vm2vf.log:

    INFO  NativeVm2VfCode [Thread-85851] - [59780:484244] INF pthread_cond_timedwait [110]: cbp 0x7f1e380b4be0 job 0x4a23a [__PrlVm_Commit] index [9060] (__configure_firewall_cb, firewall.c, 248) @[common/generic_sdk_cb_tools.c][139][cbps_timeout_sleep][26479])

    WARN  LocalVm2Vf [Thread-85854] - CORBA exception OBJECT_NOT_EXIST:Server-side Exception:  caught at 'in_progress' method invocation
    WARN  LocalVm2Vf [Thread-85862] - CORBA exception OBJECT_NOT_EXIST:Server-side Exception:  caught at 'in_progress' method invocation

And also exception:

    INFO  CallbackProxy [RequestProcessor-4] - vm2vf.callbackproxy InvocationTargetException exception java.lang.NullPointerException: null

Cause

Above shows that request was sent by IM but when callback returned from vm2vf, object that should receive it, was already destroyed and IM was not able to process callback.

This behavior is caused by CCU-10599 and CCU-11519.

Resolution

Fix the IM configuration file and terracotta startup file manually as a workaround. Set Java Heap limit to 2GB as shown below:

/usr/local/share/PACI-im/scripts/im-daemon:

Line 47:

-Xmx2048m -Xms2048m -XX:+HeapDumpOnOutOfMemoryError ${JAVA_CONSOLE_SETTINGS}\

for terracotta startup script:

# grep -n Xms /usr/local/terracotta-3.6.1/bin/start-tc-server.sh
60:${JAVA_COMMAND} -Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError \

Full restart of IM services and purging terracotta cache are required after that:

service PACI-im stop
service terracotta stop
service activemq-server stop
service terracotta clearcache

service activemq-server start
service terracotta start
service PACI-im start

Search Words

crossed critical threshold

backup

storage node full

OBJECT_NOT_EXIST

Exception: caught at 'in_progress' method invocation

vm2vf.callbackproxy InvocationTargetException exception java.lang.NullPointerException: null

Configuring firewall

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 956c448bddc7e1f3585373687602379f 6f1456866eed87488c0f02b298a741c0

Email subscription for changes to this article
Save as PDF