Symptoms

1. /usr/sbin/logrotate consumes a lot of (>50%) of Processor's time on PA Management Node

2. /var/log/ directory contains a lot of log files rotated many times. Example:

   "/var/log/poa.debug.log-20150728.gz-20150801.gz-20150804.gz-20150806.gz"
   "/var/log/poa.debug.log-20150728.gz-20150801.gz-20150806.gz"
   

3. A lot of records in /var/lib/logrotate.status. For example:

   [root@ppa ~]# cat /var/lib/logrotate.status | wc -l
   936747
   

Cause

Misconfiguration of logrotate daemon.

Resolution

On PA management node:

1. Find out all running logrotate processes and stop them:

   # ps faxu | grep -v grep | grep -B3 logrotate
   root      2194  0.0  0.0  20888   624 ?        Ss    2014   2:36 crond
   root     11921  0.0  0.0  80508  1840 ?        S    Aug09   0:00  \_ crond
   root     11922  0.0  0.0   8692   984 ?        Ss   Aug09   0:00  |   \_ /bin/bash /usr/bin/run-parts /etc/cron.daily
   root     12202  0.0  0.0   8692   896 ?        S    Aug09   0:00  |   |   \_ /bin/sh /etc/cron.daily/logrotate
   root     12204 61.3  1.8 525568 452904 ?       DN   Aug09 8246:12  |   |   |   \_ /usr/sbin/logrotate /etc/logrotate.conf
   root     12203  0.0  0.0   8784   568 ?        S    Aug09   0:00  |   |   \_ awk -v progname=/etc/cron.daily/logrotate progname {?????   print progname ":\n"?????   progname="";????       }???? 
   
   #kill -9 12203 12204 12204
   

2. Check /var/lib/logrotate.status and determine problem configuration file:

   # grep -ir 'debug' /etc/logrotate.d/*
   /etc/logrotate.d/poa-debug:/var/log/poa.debug.log* {
   /etc/logrotate.d/syslog:/var/log/poa.debug.log {
   

   String /etc/logrotate.d/poa-debug:/var/log/poa.debug.log* { means that all files matches with pattern will be rotated every day.

3. Check number of poa.debug.log* files:

   # ls -la /var/log/poa.debug* | wc -l
   -bash: /bin/ls: Argument list too long
   0
   

   Output means that directory /var/log/ contains significant number of files and lograte daemon couldn't handle all files during execution of daily rotation.

4. Remove /etc/logrotate.d/poa-debug from the system:

   # rm /etc/logrotate.d/poa-debug
   rm: remove regular file `/etc/logrotate.d/poa-debug'? y