Control panel became unavailable with two main symptoms:
DB deadlock was detected according to
rest:519961 1:18862:e5947b70 lib ]: [ before_completion] ExSystem: module_id:'db_service', ex_type_id:'1',Message:'DBService invocation error: ERROR: deadlock detected
SaaS core dumps were generated during service controller crashes. Their analysis show source of the crash in the following thread:
#0 0x00fa0430 in __kernel_vsyscall () #1 0x0499e781 in raise () from /lib/libc.so.6 #2 0x049a005a in abort () from /lib/libc.so.6 #3 0x04997a4b in __assert_fail_base () from /lib/libc.so.6 #4 0x04997b06 in __assert_fail () from /lib/libc.so.6 #5 0xf6f89a0e in APS::Controller::RdbmsSessionImp::rollback() () from /usr/local/pem/lib/libaps.so #6 0xf6f8ba59 in APS::Controller::RdbmsSession::rollback() () from /usr/local/pem/lib/libaps.so #7 0xf6f8b843 in APS::Controller::RdbmsSession::~RdbmsSession() () from /usr/local/pem/lib/libaps.so #8 0xf70cb177 in APS::Controller::RESTful::ResourceLinkMgrRequest::provisionResourceLink() () from /usr/local/pem/lib/libaps.so #9 0xf710ce45 in APS::Controller::RESTful::BaseRequest::run() () from /usr/local/pem/lib/libaps.so
SaaS service controller incorrectly handles rollbacks of REST requests under certain circumstances causing database deadlocks:
INF [rest:519961 1:23698:e0a10b70 SAAS ]: [ HTTP] IN 192.168.231.72:46010 Begin request: "POST /aps/2/resources/d9670759-6e27-4a1c-820d-5700eea86c4c/mozyProAccountUserLicense/" DBG [rest:519961 1:23698:e0a10b70 SAAS 1986458941]: [ APS::SaaSContextTransaction::commit] ===> ENTRY DBG [rest:519961 RequestProcessor-1322773 pau ]: com.parallels.pa.service.db.DbServiceInvocationExceptionConverter Failure in DBServicejavax.ejb.EJBTransactionRolledbackException: error executing work
This behavior was investigated and passed for further investigation to the Engineering team as APS-42330: 'core dump on subsequent rollback in SaaS'.
There is no workaround for the issue, except restarting OA services on the management node.
Please contact your Technical Manager or a member of Pooled Technical Associates in order to clarify status of the APS-4233.