Logrotate: Take Control of Logs with Custom Rotation, Retention, and Compression

Log files are vital for monitoring, debugging, and auditing systems, but if unmanaged, they quickly grow out of control, wasting disk space and risking system stability.

Logrotate is a simple yet powerful tool that automates log rotation, compression, and cleanup. It helps keep your logs organised, your disk usage under control, and your services running smoothly.

In this guide, you’ll learn how Logrotate works, how to configure it properly, and which best practices to follow. Whether you're managing a single server or an entire fleet, mastering Logrotate is essential for reliable operations.

Introduction #

What is Logrotate? #

Logrotate is a utility designed to manage the automatic rotation, compression, removal, and mailing of log files on a Linux system. It is typically run as a daily cron job and is highly configurable via global and per-log configuration files.

Logrotate is especially useful for:

• Preventing log files from consuming all available disk space
• Organising logs into manageable segments
• Maintaining a set number of logs or logs for a specific duration
• Automating compression and cleanup of old logs

Why Logs Matter #

Logs are essential for understanding the behaviour, health, and security of a system. They provide a chronological record of system and application events, making them a foundational tool for system administration, security auditing, and performance monitoring.

1. Troubleshooting and Debugging

• Logs capture errors, warnings, and informational messages that help identify the root cause of system or application issues.

• Developers and administrators rely on logs to trace faults, monitor service failures, and understand unexpected behaviour.

2. System Monitoring

• Logs provide visibility into the operational status of services, hardware components, and user activity.
• Monitoring tools can analyse logs in real time to identify unusual activity or signs of performance issues.

3. Security and Auditing

• Logs track authentication attempts, access patterns, and system changes, which are vital for identifying security breaches or policy violations.

• Audit trails help maintain accountability by recording user actions and system modifications.

4. Compliance and Legal Requirements

• Certain industries require log retention for defined periods to comply with standards such as GDPR, HIPAA, or PCI-DSS.

• Logs serve as evidence in investigations or audits.

5. Capacity Planning and Optimisation

• Logs can reveal resource usage trends, application load, and user behaviour.

• Analysing these patterns aids in scaling infrastructure and tuning system performance.

6. Historical Reference

• Logs offer a historical record of events that can help understand past incidents or changes.

• Useful for post-mortem analysis after outages or security events.

7. Automation and Alerting

• Logs can trigger automated responses, such as restarting a failed service or sending alerts when thresholds are exceeded.

• Tools like logwatch, logrotate, and monitoring platforms (e.g. Prometheus, ELK stack) rely on logs as input.

How Logrotate Works #

Log rotation typically involves the following steps:

1. Moving or Renaming Current Log Files
   The active log file (for example, /var/log/app.log) is renamed or moved to a new file (like /var/log/app.log.1). This frees up the original filename so the application can keep writing logs without interruption.
2. Creating a New Empty Log File
   After renaming, a new empty log file is created at the original path. This allows the application to continue logging seamlessly. During creation, Logrotate can set file permissions and ownership to ensure proper access.
3. Optionally Compressing Older Logs
   Older log files (such as app.log.2, app.log.3, and so on) may be compressed using gzip or similar tools. Compression saves disk space while keeping historical logs accessible if needed.
4. Deleting Logs Past Retention Period
   To avoid filling the disk, Logrotate deletes the oldest logs once a set retention limit is reached (for example, keeping only the last 4 rotated logs). This keeps storage usage under control over the long term.

Example

Before rotation:
  /var/log/app.log
After 1st rotation:
  /var/log/app.log       (new empty file)
  /var/log/app.log.1     (previous log)
After 2nd rotation:
  /var/log/app.log
  /var/log/app.log.1
  /var/log/app.log.2.gz  (older log compressed)

Installation #

To install:

sudo apt install logrotate -y

Check the version:

logrotate --version

Understanding Logging on Ubuntu #

Ubuntu Linux uses a variety of logging mechanisms to track system activity, application events, errors, and security-related information. Understanding where logs are stored and how they are structured is essential for troubleshooting, monitoring, and maintaining system health.

Log Storage Location #

The primary location for logs on Ubuntu is:

/var/log/

This directory contains both system and application logs. Subdirectories and files within it are structured by purpose or service.

Common Log Files #

Logging Systems Overview #

Ubuntu may use one or more of the following logging systems:

rsyslog

• Traditional syslog daemon used to collect and route logs to appropriate files in /var/log/.

• Configuration: /etc/rsyslog.conf and /etc/rsyslog.d/

systemd-journald

• Part of systemd, it captures logs from the kernel, init system, and services.

• Logs can be viewed using:

journalctl

• By default, system logs created by systemd-journald may be stored only in volatile memory (RAM). This means:
  • Logs are not saved permanently
  • All logs are lost after a reboot
• To keep logs persistently, you must configure persistent storage (/var/log/journal/).

Check journal size limit:

sudo journalctl --disk-usage

Expected output:

Archived and active journals take up 225.4M in the file system.

Application-specific Logging

• Many applications (e.g. NGINX) maintain their own log files within /var/log/.
• Rotation of these logs is usually managed via logrotate with per-application configuration in /etc/logrotate.d/.

Permissions and Access #

• Most log files are owned by root or specific system users.

• Viewing them typically requires elevated privileges, for example:

sudo less /var/log/syslog

Configuration Structure #

Logrotate’s configuration is divided into global settings and per-application rules, providing flexibility and modular management.

Global Configuration #

Path:

/etc/logrotate.conf

Purpose:

• This file contains default global settings for how logs should be rotated.
• It also typically includes a line like the following to load all per-application configs:

include /etc/logrotate.d

Example of global settings:

weekly                     # Rotate logs weekly
rotate 4                   # Keep 4 rotated log files
size 100M                  # Rotate if log file exceeds 100 MB, even before weekly schedule
maxage 30                  # Remove logs older than 30 days (cleanup safety net)
missingok                  # Don't error if a log file is missing
notifempty                 # Don't rotate empty log files
create                     # Create a new empty log file after rotation
su root adm                # Run rotation as user 'root' and group 'adm'
dateext                    # Use date-based suffix (e.g. .log-20250706) instead of numeric
compress                   # Compress rotated logs (using gzip by default)
delaycompress              # Compress one cycle later to avoid interfering with recent logs
include /etc/logrotate.d   # Load additional rotation rules from this directory

These defaults apply to all logs unless overridden in specific per-application config files.

Per-Application Rules #

Directory:

/etc/logrotate.d/

Purpose:

• This directory contains individual config files for different services or applications (e.g. NGINX).
• Each file defines rotation rules specific to a log or service.

Example:

/var/log/myapp/*.log {
  weekly
  rotate 4
  missingok
  notifempty
  create
  compress
  delaycompress
}

Directives #

Logrotate uses a flexible set of directives to control how logs are rotated, compressed, created, and removed.

File Rotation Control

Accepted Units for <n>

Compression

Log File Management

Scripting Hooks

Miscellaneous

Directive Sorting Order

When structuring a logrotate rule block, use the following order to improve readability and reduce maintenance errors:

1. Rotation triggers:
   Frequency (daily, weekly, etc.) and size (size) settings.
2. Rotation count:
   Number of rotations to keep (rotate).
3. File checks:
   Options like missingok and notifempty that control when to rotate.
4. File creation:
   Permissions and ownership settings (create).
5. Compression:
   Compression-related options (compress, delaycompress).
6. Scripts:
   Script blocks (prerotate, postrotate, endscript) at the end for clarity.

Commands #

Syntax #

logrotate [OPTION...] <configfile>

Options:

1. Basic Execution #

Run using default config:

sudo logrotate /etc/logrotate.conf

Run with a specific config file:

sudo logrotate /etc/logrotate.d/nginx

This command only processes the NGINX logrotate rules.

Run with a custom state file:

sudo logrotate -s /tmp/logrotate.state /etc/logrotate.conf

2. Testing and Debugging #

Simulate log rotation (dry-run):

sudo logrotate -d /etc/logrotate.conf

Verbose output:

sudo logrotate -v /etc/logrotate.conf

Dry-run with verbose output:

sudo logrotate -d -v /etc/logrotate.conf

Test specific log config with dry-run:

sudo logrotate -d /etc/logrotate.d/nginx

3. Force Rotation #

Force log rotation regardless of age/size:

sudo logrotate -f /etc/logrotate.conf

Force rotation of a specific config:

sudo logrotate -f /etc/logrotate.d/nginx

Force and show verbose output:

sudo logrotate -f -v /etc/logrotate.conf

4. Help and Information #

Show version:

logrotate --version

Show help and usage information:

logrotate --help

Show brief usage instructions:

logrotate --usage

5. Customisation #

Specify a custom mail command (when using mail directive):

sudo logrotate -m /usr/sbin/sendmail /etc/logrotate.conf

Workflow #

Optimise Global Configuration #

Backup current global logrotate config:

sudo cp /etc/logrotate.conf /etc/logrotate.conf.bak

Open config:

sudo vim /etc/logrotate.conf

Modify with the following settings:

weekly                     # Rotate logs weekly
rotate 4                   # Keep 4 rotated log files
size 100M                  # Rotate if log file exceeds 100 MB, even before weekly schedule
maxage 30                  # Remove logs older than 30 days (cleanup safety net)
missingok                  # Don't error if a log file is missing
notifempty                 # Don't rotate empty log files
create                     # Create a new empty log file after rotation
su root adm                # Run rotation as user 'root' and group 'adm'
dateext                    # Use date-based suffix (e.g. .log-20250706) instead of numeric
compress                   # Compress rotated logs (using gzip by default)
delaycompress              # Compress one cycle later to avoid interfering with recent logs
include /etc/logrotate.d   # Load additional rotation rules from this directory

Identify Critical Logs to Rotate #

logrotate manages log file rotation, compression, removal, and mailing. Its primary goal is to prevent log files from consuming excessive disk space. Identifying which logs require attention involves assessing file size, modification time, and whether they're still in active use.

1. Check Log File Sizes

Identify the top 20 largest log files under /var/log:

sudo du -ah /var/log | sort -rh | head -n 20

Large logs should be evaluated to determine if:

• Rotation is properly configured
• They are being written to excessively
• The service is misbehaving or overly verbose

2. Check Modification Times

Use find to determine which log files are active or stale.

Recently modified (last 24 hours):

sudo find /var/log -type f -mtime -1

These are actively written logs that may require:

• Frequent rotation
• Shorter retention periods

Not modified recently (older than 30 days):

sudo find /var/log -type f -mtime +30

These are stale logs, which might be:

• Archived
• Deleted after review

3. Understand Stale Logs

Stale logs are log files that are no longer being updated or used but still occupy disk space.

Criteria for identifying a stale log:

• Not modified in a long time (e.g. 30+ days)
• No active process is writing to it
• Belongs to a disabled or uninstalled service
• Leftover from previous deployments or test runs

Use lsof to check if any process is using a log file:

sudo lsof | grep '/var/log/'

If the file does not appear, it is likely not in active use.

Create Custom Logrotate Rules #

Step 1: Create a Custom Config File

Use a distinct name to avoid conflicts:

sudo vim /etc/logrotate.d/nginx-custom

Define your own rotation policy in this file.

Step 2: Disable the Default Config File

Rename the default file so it’s no longer loaded by logrotate:

sudo mv /etc/logrotate.d/nginx /etc/logrotate.d/nginx.disabled

This preserves the file for reference and avoids deletion.

Step 3: Test Your Configuration

Run a dry-run to ensure syntax is correct and the intended logs will be rotated:

sudo logrotate -d /etc/logrotate.d/nginx-custom

Check the output carefully. If everything looks correct, logrotate will apply the rules during the next scheduled run.

Step 4 (Optional): Delete Old Logs

To delete all existing NGINX logs:

sudo rm -f /var/log/nginx/*

Only perform this step if you're certain the logs are no longer needed.

Adjust Rotation Frequency #

Step 1: Identify Trigger Mechanism

Check if logrotate is run by cron:

cat /etc/cron.daily/logrotate

If you see:

# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
  exit 0
fi

Then cron is bypassed in favour of systemd, which is the default on modern systems.

Check for a systemd timer:

systemctl list-timers | grep logrotate

Example output:

Sat 2025-07-05 00:00:00 CEST 1h 44min Fri 2025-07-04 00:00:05 CEST      22h ago logrotate.timer                logrotate.service

Step 2: Change Frequency (Systemd-Based Systems)

View current timer unit:

systemctl cat logrotate.timer

Look for the OnCalendar= line:

[Timer]
OnCalendar=daily

To customise it, create an override:

sudo systemctl edit logrotate.timer

Add or update:

[Timer]
OnCalendar=weekly

Common values for OnCalendar:

Step 3: Apply and Verify

Reload systemd and restart the timer:

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart logrotate.timer

Confirm the new schedule:

systemctl list-timers --all | grep logrotate

Step 4: (Optional) Test Manual Execution

Run logrotate manually to confirm it works:

sudo logrotate -f /etc/logrotate.conf

Best Practices #

Logrotate is powerful, but incorrect configuration can lead to lost logs, service downtime, or disk space exhaustion. The following best practices help ensure safe, reliable, and efficient log management.

Recommended Approaches #

1. Use size and time criteria together

• Combine age-based rotation (daily, weekly) with size-based rotation (size 100M).
• This ensures:
  • Logs don’t grow excessively between scheduled rotations.
  • Rotation happens whenever either condition is met.

Example:

weekly
size 100M

This rotates logs weekly or as soon as they exceed 100 MB.

2. Enable compression (compress)

• Always enable compress to save disk space.
• Use delaycompress if your application may still access the previous log file after rotation (common with daemons).

Example:

compress
delaycompress

3. Specify file creation settings (create)

• Use create to define:
  • Permissions
  • Owner
  • Group

This prevents permission issues or application startup failures after rotation.

Example:

create 640 syslog adm

Creates the new log with mode 0640, owner syslog, group adm.

4. Use missingok and notifempty

• missingok: Prevents errors if a log file is missing.
• notifempty: Skips rotation for empty log files.

This reduces unnecessary noise and avoids rotating empty or non-existent logs.

Example:

missingok
notifempty

5. Trigger application log reopening properly

• Applications like NGINX, and MySQL need a signal to start using the new log file after rotation.
• Use postrotate with systemctl reload, kill -USR1, or similar methods.

Example for NGINX:

postrotate
  systemctl kill -s USR1 nginx
endscript

6. Group related logs with sharedscripts

• If you rotate multiple logs in a block, sharedscripts ensures the postrotate block runs once, not once per file.

Example:

sharedscripts

Without it, the postrotate script may execute multiple times, which can break services.

7. Use dateext for clearer archives

• Replaces default numeric suffixes (e.g. .1, .2) with date-based ones (e.g. access.log-20250706).
• This makes logs easier to read and audit.

Example:

dateext

8. Keep status tracking intact

• Logrotate uses /var/lib/logrotate/status to track when logs were last rotated.
• Do not manually delete or edit this file.
• Corruption or deletion may cause:
  • Missed rotations
  • Unexpected behaviour

9. Store custom configs in /etc/logrotate.d/

• Place application-specific rotation configs here.
• Keeps configurations modular and easier to manage.
• Helps prevent issues during package upgrades or audits.

10. Regularly test and validate changes

• Always test changes before applying them.
• Use dry-run mode with -d to simulate the result without making any changes.

Example:

logrotate -d /etc/logrotate.conf

This helps catch syntax errors or unexpected behaviour.

Common Pitfalls #

1. Using copytruncate unnecessarily

• copytruncate works by copying the current log file to a rotated version, then truncating the original file to zero length.
• It does not restart or notify the application writing the logs.
• If the application is actively writing to the log while truncation occurs, some data can be lost during the copy-truncate window.
• Avoid this for high-traffic services like NGINX or Apache. Instead, use a postrotate block to signal the application to safely reopen log files.

2. Forgetting to reload services

• After log rotation, an application must reopen its log files to start writing to the new file.
• If you don’t send a signal (like kill -USR1) or reload the service:
  • It will keep writing to the old (rotated) file, which may be deleted or compressed.
  • Disk space won’t be freed, and logs may appear missing.
• Always use a postrotate block when needed.

3. Over-retention of logs

• Setting rotate 365 or similar without enabling compression will:
  • Keep too many large files
  • Rapidly consume disk space
• Use compress, maxage, and a practical rotate count based on your storage and retention needs.
• Example:

rotate 14
maxage 30
compress

4. Assuming logs rotate automatically

• Log files do not rotate by default unless:
  • They are explicitly listed in /etc/logrotate.conf or
  • A rule exists in /etc/logrotate.d/
• Just placing a log in /var/log does not enable rotation.
• Always verify your config covers the target log files.

5. Relying on logrotate for systemd-journald logs

• Logs managed by systemd-journald (viewed with journalctl) are not rotated by Logrotate.
• Configure them via:
  • /etc/systemd/journald.conf
  • Settings like SystemMaxUse=, MaxRetentionSec=, etc.
• Use journalctl --vacuum-time=30d to manually clean up.

6. Overusing wildcards

• Using *.log in a config might unintentionally match:

  • Temporary or partial logs

  • Application-specific backup files

• This can rotate files you didn’t intend, causing:

  • Data loss

  • Application errors

• Always review which files will be matched, or use precise patterns.

7. Using notifempty without understanding

• notifempty tells Logrotate not to rotate a log file if it is empty.
• However, some applications expect the log file to exist, even if unused.
• If the file is missing after rotation, the application may:
  • Fail to start
  • Stop logging
• In these cases, consider removing notifempty or using create to recreate the log file.

8. Manual deletion of rotated logs

• Deleting rotated logs by hand (e.g. access.log.1) bypasses Logrotate’s tracking.
• This can:
  • Cause logs to rotate incorrectly
  • Break the rotation schedule
• Use maxage, rotate, and prerotate/postrotate hooks for automatic, consistent cleanup.

By following these best practices, you can configure Logrotate to be robust, efficient, and well-suited to the needs of any production system.

Next Steps #

With a solid understanding of Logrotate, it’s time to apply and maintain effective log management. Follow these steps to review, plan, implement, and monitor your log rotation setup for optimal system performance.

1. Review Existing Configuration
   Check the global settings in /etc/logrotate.conf to understand default behaviours. Also, examine service-specific configurations in /etc/logrotate.d/ to see how individual applications manage their logs.

2. Identify Critical Logs to Rotate
   Find which logs consume the most disk space or require strict retention policies. This usually includes application logs, system logs, and security logs.

3. Plan Rotation Policies
   Decide how often logs should rotate (daily, weekly, or based on size). Define how many old logs to keep and whether to compress them. Set appropriate ownership and permissions for new log files to ensure proper access.

4. Implement or Customise Configurations
   Create or modify configuration files as needed, following best practices. Avoid editing default package files directly. Use separate files or version control for easier management and rollback.

5. Test Your Configuration
   Use logrotate -d /etc/logrotate.conf to simulate log rotation without making changes. Verify that logs will rotate correctly and no errors occur.

6. Automate Rotation
   Ensure Logrotate runs automatically, typically via a cron job or systemd timer. Confirm these scheduled tasks are active and correctly configured.

7. Monitor and Adjust
   Regularly check disk usage and log file sizes. Adjust rotation frequency, retention limits, or compression settings as needed to balance performance and storage.

8. Document Your Setup
   Keep clear documentation of your rotation policies and configurations. Use version control systems like Git to track changes and maintain history.