Log Rotation and Retention in Python Logging

Configuring log rotation and retention in Python ensures applications maintain manageable file sizes and prevent disk exhaustion. Python’s built-in logging.handlers module provides two primary classes for this purpose: RotatingFileHandler for size-based rotation and TimedRotatingFileHandler for time-interval rotation. Both handlers include a retention mechanism that automatically purges older log archives once a specified limit is reached.

Size-Based Rotation with RotatingFileHandler

RotatingFileHandler triggers rotation when the active log file reaches a predefined size limit in bytes.

The two key parameters controlling rotation and retention are:

• maxBytes: The maximum file size in bytes before the file is rotated. If set to 0, rotation never occurs.
• backupCount: The number of historical log files to retain. If set to 5, the handler retains app.log, app.log.1, app.log.2, up to app.log.5. When a new rotation occurs, the oldest file is deleted.

import logging
from logging.handlers import RotatingFileHandler

# Configure logger
logger = logging.getLogger("SizeLogger")
logger.setLevel(logging.INFO)

# Rotate when file hits 5 MB, retain the 3 most recent backups
handler = RotatingFileHandler(
    "app_size.log",
    maxBytes=5 * 1024 * 1024,
    backupCount=3,
    encoding="utf-8"
)

formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("Size-based rotation configured.")

Time-Based Rotation with TimedRotatingFileHandler

TimedRotatingFileHandler rotates logs at regular temporal intervals, such as every midnight, hourly, or weekly.

The key parameters include:

• when: The unit of time for rotation. Common options include 'S' (seconds), 'M' (minutes), 'H' (hours), 'D' (days), 'midnight' (roll over at midnight), and 'W0'-'W6' (specific weekday).
• interval: The multiplier for the when parameter (e.g., when='H', interval=6 rotates every six hours).
• backupCount: The retention count determining how many past interval files remain on disk before older ones are deleted.

import logging
from logging.handlers import TimedRotatingFileHandler

# Configure logger
logger = logging.getLogger("TimeLogger")
logger.setLevel(logging.INFO)

# Rotate daily at midnight, retain logs for 7 days
handler = TimedRotatingFileHandler(
    "app_time.log",
    when="midnight",
    interval=1,
    backupCount=7,
    encoding="utf-8"
)

formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("Time-based rotation configured.")

Configuration via dictConfig

In production environments, developers often define rotation and retention policies declaratively using logging.config.dictConfig. This separates logging configuration from application logic.

import logging
import logging.config

LOGGING_CONFIG = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
        },
    },
    "handlers": {
        "rotating_file": {
            "class": "logging.handlers.RotatingFileHandler",
            "filename": "production.log",
            "maxBytes": 10485760,  # 10 MB
            "backupCount": 5,
            "formatter": "standard",
            "encoding": "utf-8",
        },
    },
    "root": {
        "level": "INFO",
        "handlers": ["rotating_file"],
    },
}

logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger(__name__)
logger.info("Production logging initialized.")

By defining backupCount alongside maxBytes or when, applications enforce deterministic storage bounds while retaining sufficient historical context for debugging.