The Python logging facility

We will use Python's logging to keep track of our code's execution. To use a logger, just the following at the beginning of a script:

import logging

logger = logging.getLogger(__name__)

Then, think of logger as a replacement of the print function. Logging uses several levels to distinguish the importance of each message. They are: DEBUG, INFO, WARNING, ERROR, CRITICAL. For example, to issue a logger message for debug purposes, we use logger.debug("This is a debug message"). If a message is just a piece of information, we use logger.info("This message is info"), if it's error, logger.error("Error message"), etc.

The logging facility needs to be configured first, which we typically do through a config file. An example follows.

File logging_config.yaml:

version: 1
formatters:
  standard:
    format: '%(asctime)s|%(name)s|%(levelname)s|%(message)s'

handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: standard
    stream: ext://sys.stdout
  error_file:
    class: fileHandlers.handlers.MakedirsRotatingFileHandler
    level: ERROR
    formatter: standard
    filename: logs/errors.log
    maxBytes: 10485760  # 10 MB
    backupCount: 20
    encoding: utf8
  output_file:
    class: fileHandlers.handlers.MakedirsRotatingFileHandler
    level: NOTSET
    formatter: standard
    filename: logs/output.log
    # maxBytes: 10485760  # 10 MB
    backupCount: 20  # Rotates this many times when maxBytes is reached
    encoding: utf8
loggers:
  __main__:  # Main module, the one you execute
    level: DEBUG
    # handlers: [console]
    handlers: [error_file, output_file]
    propagate: no  # Do not propagate to `root` level.
  models:  # Set up logger for `models` module - change this to your module name
    level: DEBUG
    propagate: no
  features:  # `features` module - change this to your current module name
    level: DEBUG
    handlers: [error_file, output_file]
    propagate: no
root:
  level: DEBUG
  handlers: [console, error_file, output_file]

Note that fileHandlers.handlers.MakedirsRotatingFileHandler is a custom file handler that automatically creates the parent directories of the log file, if needed.

To load a logging config file:

import yaml
from yaml import Loader
import logging

def get_logging_config(path_conf="config/logging_conf.yaml"):
    # Load logging configs
    with open(path_conf) as fin:
        loggingConf = yaml.load(fin, Loader)
    return loggingConf

# Apply configs:

loggingConf = get_logging_config()
logging.config.dictConfig(loggingConf)