Search Engine: Elastic

Article ID: 131712, created on Nov 3, 2017, last review on Nov 3, 2017

  • Applies to:
  • Operations Automation 7.0
  • Operations Automation 7.1
  • Operations Automation 7.2

Symptoms

During the Synchronize resource usage for APS application task run for some APS package, REST API requests invoked at the same time during provisioning / modification requests to APS resources become unstable and may take a very long time or fail:

 Oct 25 06:28:50.929 : INF [openapi:131387 1:18696:7f73171ef700 SAAS ]: [ HTTP] OUT  10.253.20.9  [2082] "POST /backupservice/globals/c29e5371-29e5-4438-9753-713a1875cdad/tenant HTTP/1.1"  [0] "204 No Content"  299901ms

or:

Oct 15 16:28:12.512 : ERR [rest:23287 1:3942:7f51e81c1700 SAAS 799579269]: [ APSC] REST Error: PUT /aps/2/application/globals/c29e5371-29e5-4438-9753-713a1875cdad code=500: ERROR:  canceling statement due to statement timeout
CONTEXT:  while locking tuple (72,52) in relation "aps_resource"

The endpoint has default Apache configuration that allows 10 simultaneous connections:

 [root@endpoint ~]# apachectl -M | egrep "prefork|worker"
 ...
  mpm_prefork_module (static)

 # prefork MPM
 # StartServers: number of server processes to start
 # MinSpareServers: minimum number of server processes which are kept spare
 # MaxSpareServers: maximum number of server processes which are kept spare
 # ServerLimit: maximum value for MaxClients for the lifetime of the server
 # MaxClients: maximum number of server processes allowed to start
 # MaxRequestsPerChild: maximum number of requests a server process serves
 <IfModule prefork.c>
 StartServers       1
 MinSpareServers    1
 MaxSpareServers    5
 ServerLimit       10
 MaxClients        10
 MaxRequestsPerChild  4000
 </IfModule>

Cause

The Synchronize resource usage for APS application task runs in 16 threads by default and, in case of a large amount of processed APS resources, makes the Endpoint unavailable for any other requests, as it allows less threads to be executed at the same time.

The behavior is planned to be optimized in scope of APS-43355 request.

Resolution

  1. Tune Apache configuration on the Endpoint: modify /etc/httpd/conf.d/httpd.conf file and increase ServerLimit and MaxClients parameters to 30.
  2. Scheduled the Synchronize resource usage for APS application task to run during non-business time, when there is a minimum amount of provisioning requests.

0871c0b47b3b86ae3b1af4c2942cd0ce caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 5356b422f65bdad1c3e9edca5d74a1ae 1941880841f714e458ae4dc3d9f3062d 31987597efff5a3a9ce779cc203bbe5e 8c199f0ee4305da1a577740620df4a51

Email subscription for changes to this article
Save as PDF