Problem:
The logrotate process caused high CPU utilization, ran for about 5-6 hours and finally exited with a non-zero (failure) exit code.
Background & Analysis:
Observed this problem on one of our RHEL 3 hosts. A new daily job was added and this job specified log rotation for all files in a directory (rather than a file or specific patterns to match files) and used the compress option. Consequently, logrotate compressed files that were already compressed and so on and files in the directory started having very long filenames (e.g. test.dmp.gz.gz.1.gz.1.gz…) and a huge number of files were present in the directory, making logrotate’s job more and more difficult, thereby causing logrotate to consume high CPU (system cpu utilization %) and take longer to complete. Also, logrotate’s state file (/var/lib/logrotate.status) ended up being about 58 MB.
Solution:
-
Modify the daily job to rotate only files matching a specific pattern (in my case, I removed the daily job as it was redundant. The files were created with timestamp prefixes and so rotation wasn’t required).
-
Remove logrotate’s state file (/var/lib/logrotate.status). When done, logrotate will generate a new state file.
NOTE: The removal of logrotate’s state file is akin to running logrotate for the first time. So, you can modify the dates in the state file to reflect the previous day or the previous week before running a daily or weekly job to ensure logrotate rotates the files on its first run. ALternatively, use the logrotate –f option.
Root Cause:
Improper configuration of logrotate for a daily job.
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated as it gives me and others who read this article, an indication of whether this solution has worked for people other than me. Also, feel free to leave comments.
