During the Synchronize resource usage for APS application task run for some APS package, REST API requests invoked at the same time during provisioning / modification requests to APS resources become unstable and may take a very long time or fail:
Oct 25 06:28:50.929 : INF [openapi:131387 1:18696:7f73171ef700 SAAS ]: [ HTTP] OUT 10.253.20.9  "POST /backupservice/globals/c29e5371-29e5-4438-9753-713a1875cdad/tenant HTTP/1.1"  "204 No Content" 299901ms
Oct 15 16:28:12.512 : ERR [rest:23287 1:3942:7f51e81c1700 SAAS 799579269]: [ APSC] REST Error: PUT /aps/2/application/globals/c29e5371-29e5-4438-9753-713a1875cdad code=500: ERROR: canceling statement due to statement timeout CONTEXT: while locking tuple (72,52) in relation "aps_resource"
The endpoint has default Apache configuration that allows 10 simultaneous connections:
[root@endpoint ~]# apachectl -M | egrep "prefork|worker" ... mpm_prefork_module (static) # prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # ServerLimit: maximum value for MaxClients for the lifetime of the server # MaxClients: maximum number of server processes allowed to start # MaxRequestsPerChild: maximum number of requests a server process serves <IfModule prefork.c> StartServers 1 MinSpareServers 1 MaxSpareServers 5 ServerLimit 10 MaxClients 10 MaxRequestsPerChild 4000 </IfModule>
The Synchronize resource usage for APS application task runs in 16 threads by default and, in case of a large amount of processed APS resources, makes the Endpoint unavailable for any other requests, as it allows less threads to be executed at the same time.
The behavior is planned to be optimized in scope of APS-43355 request.
- Tune Apache configuration on the Endpoint: modify
/etc/httpd/conf.d/httpd.conffile and increase ServerLimit and MaxClients parameters to 30.
- Scheduled the Synchronize resource usage for APS application task to run during non-business time, when there is a minimum amount of provisioning requests.