Python Logging

Each level signals how important a message is:

A logger drops messages below its configured level. The default is WARNING, which is why logging.info("...") prints nothing until you configure a lower level with logging.basicConfig(level=logging.INFO). In development you might run at DEBUG; in production, INFO or WARNING.

Choose levels honestly. If every progress message is logged as ERROR, real failures are buried and alerting systems fire constantly. A skipped record with a known reason is a WARNING; a database that can't be reached is an ERROR.

Get a logger per module with logging.getLogger(__name__). The name follows the module path, such as catalog.importer, and loggers form a hierarchy by dots: catalog.importer is a child of catalog, which is a child of the root logger.

logger = logging.getLogger(__name__)    # in catalog/importer.py: "catalog.importer"

Messages travel up the hierarchy to the handlers configured on ancestors, usually the root. That lets the application configure output once, at start-up, while each module just logs. It also allows targeted tuning: logging.getLogger("catalog.importer").setLevel(logging.DEBUG) shows detailed messages from one module only, and logging.getLogger("urllib3").setLevel(logging.WARNING) quietens a chatty library.

Libraries should never call basicConfig() or add handlers to the root logger; that's the application's decision. A library module just creates its logger and logs.

Pass values as arguments rather than building the string yourself:

logger.debug("Processed %s records from %s", count, filename)   # preferred
logger.debug(f"Processed {count} records from {filename}")      # works, but always formats

With the argument style, the message is only formatted if a handler will actually output it, so DEBUG messages cost almost nothing in production. It also keeps the message template constant, which lets log analysis tools group identical messages that differ only in their values.

The format of each line is set by a formatter, through basicConfig(format=...) or on a handler. Common fields include %(asctime)s, %(levelname)s, %(name)s, %(message)s, %(funcName)s and %(lineno)d. Handlers decide where messages go: StreamHandler for the console, FileHandler for a file, RotatingFileHandler to cap file size, and many more.

Inside an except block, logger.exception("message") logs at ERROR level and appends the full traceback, so the log shows exactly where the failure happened:

try:
    year = int(record["year"])
except (KeyError, ValueError):
    logger.exception("Could not parse year for record %s", record.get("id"))

Outside an except block there's no active exception, so use logger.error() instead, or pass exc_info=True explicitly.

Logs are copied, stored and read by many people and systems. Never log passwords, API keys, tokens, full card numbers or unnecessary personal data. Log identifiers (record id 1042) rather than contents (borrower name and address), mask secrets (key ...9f2c) and check what your __repr__ methods include, since objects are often logged whole. Data protection rules can apply to log files just as they do to databases.

Task: add useful import progress and error logs with appropriate levels and no private record contents.

import logging, sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(levelname)s %(message)s")
log = logging.getLogger("catalog")

borrowers = [{"id": 7, "email": "ana@example.com", "fines": "2.50"},
             {"id": 8, "email": "ben@example.com", "fines": "unpaid"}]
total = 0.0
for b in borrowers:
    try:
        total += float(b["fines"])
    except ValueError:
        log.exception("Bad fines value for borrower %s", b["id"])   # id, not email
log.info("Fines total: %.2f", total)

The ERROR line names borrower 8 and includes the traceback, without the email address. For the worked check, logger.exception() captures exception context inside an exception handler, where an exception is currently being handled.

The logging HOWTO and the logging cookbook cover configuration in depth. Continue with pytest in depth and revisit debugging. Practise with the Python exercises and Python quiz.