Symptoms
/usr/sbin/logrotate
consumes a lot of (>50%) of Processor's time on PA Management Node/var/log/
directory contains a lot of log files rotated many times. Example:"/var/log/poa.debug.log-20150728.gz-20150801.gz-20150804.gz-20150806.gz" "/var/log/poa.debug.log-20150728.gz-20150801.gz-20150806.gz"
A lot of records in
/var/lib/logrotate.status
. For example:[root@ppa ~]# cat /var/lib/logrotate.status | wc -l 936747
Cause
Misconfiguration of logrotate
daemon.
Resolution
On PA management node:
Find out all running
logrotate
processes and stop them:# ps faxu | grep -v grep | grep -B3 logrotate root 2194 0.0 0.0 20888 624 ? Ss 2014 2:36 crond root 11921 0.0 0.0 80508 1840 ? S Aug09 0:00 \_ crond root 11922 0.0 0.0 8692 984 ? Ss Aug09 0:00 | \_ /bin/bash /usr/bin/run-parts /etc/cron.daily root 12202 0.0 0.0 8692 896 ? S Aug09 0:00 | | \_ /bin/sh /etc/cron.daily/logrotate root 12204 61.3 1.8 525568 452904 ? DN Aug09 8246:12 | | | \_ /usr/sbin/logrotate /etc/logrotate.conf root 12203 0.0 0.0 8784 568 ? S Aug09 0:00 | | \_ awk -v progname=/etc/cron.daily/logrotate progname {????? print progname ":\n"????? progname="";???? }???? #kill -9 12203 12204 12204
Check
/var/lib/logrotate.status
and determine problem configuration file:# grep -ir 'debug' /etc/logrotate.d/* /etc/logrotate.d/poa-debug:/var/log/poa.debug.log* { /etc/logrotate.d/syslog:/var/log/poa.debug.log {
String
/etc/logrotate.d/poa-debug:/var/log/poa.debug.log* {
means that all files matches with pattern will be rotated every day.Check number of
poa.debug.log*
files:# ls -la /var/log/poa.debug* | wc -l -bash: /bin/ls: Argument list too long 0
Output means that directory
/var/log/
contains significant number of files andlograte
daemon couldn't handle all files during execution of daily rotation.Remove
/etc/logrotate.d/poa-debug
from the system:# rm /etc/logrotate.d/poa-debug rm: remove regular file `/etc/logrotate.d/poa-debug'? y