Python 日志模块的定制

Python标准logging模块中主要分为四大块内容：

Logger: 定义应用程序使用的接口

Handler: 将Loggers产生的日志输出到目的地

Filter: 对Loggers产生的日志进行过滤

Formatter: 格式化Loggers产生的日志

其中常用的是Logger, Handler, Formatter。

目标

对logging模块进行封装，使其尽量达到几个目标：

• 易用性：尽量保持python原生标准的用法。
• 扩展性：避免将所有模块集中在一个文件中进行配置，而是对不同模块继承扩展。如此可以确保修改时尽量保持简洁性，而非混淆杂乱。
• 文字加色：同一时间段内出现大量日志输出时，可以更清晰地识别出问题所在。
• 输出多个目的地：在生产代码中，并非所有日志信息都输出到console或者file中，而是日志的重要级别，将那些比较重要的发送到console，将那些不太重要但以后分析问题时可能又会用到的发送到file中。
• 非阻塞发送器：记录日志时如果涉及到网络（如socket, smtp），那么这个记录过程可能会阻塞当前线程，在python 3.2+，我们可以非常容易地设计一个非阻塞的发送器（当然了，非阻塞机制的内部实现大多都是队列）。

目录结构

如下的定制日志主要包括两个包，log_config和tools，目录结构如下：

log_config 模块结构

默认配置

• Logging Level:

　　logging.NOTSET is default.

　　You can change it by passing ‘level=’ to BTLogger instance in your application code.

• Sending Level:

　　Send logging messages to console with levels of logging.INFO and higher, and simultaneously to the disk file with levels of logging.DEBUG and higher.

　　You can change them by passing ‘level=’ to BTStreamHandler/BTFileHandler instance in log_config\__init__.py.

• Log Config:

　　LOG_PATH = r'C:\log'
　　LOG_NAME = r'mylogger.log'

　　You can change them in log_config\__init__.py.

• Console Text Color:

　　DEBUG

　　INFO

　　WARNING

　　ERROR

　　CRITICAL

　　You can change them by passing ‘fore_color=’
and ‘back_color=’ to decorator @text_attribute.set_textcolor(fore_color, back_color)
in log_config\__init__.py.

• There
  also exists a NonBlockingHandler class in log_config\handlers.py, which
  will use features of version 3.2+. If you unfold it, you can get a
  non_blocking_handler.

Demo

ts_log_config.py:

 def log_test():
     import log_config

     logger = log_config.BTLogger('mylogger')

     logger.debug('Debug message')
     logger.info('Info message')
     logger.warning('Warning message')
     logger.error('Error message')
     logger.critical('Critical message')

 def main():
     log_test()

 if __name__ == '__main__':
     main()

outputs:

代码实现

..\log_config\__init__.py:

 """
 Wrap the classes of logging module so as to expose more flexible interfaces that application code directly uses.

 By default, when you do some log operations, it will log messages to console with levels of logging.INFO and higher,
 and simultaneously to the disk file with levels of logging.DEBUG and higher.

 Usages:
 To use log_config module, simply 'import log_config' in your code, and create a BTLogger instance, then, you can use this
 instance to log messages:
     import log_config

     logger = log_config.BTLogger('mylogger')

     logger.debug('Debug message')
     logger.info('Info message')
     logger.warning('Warning message')
     logger.error('Error message')
     logger.critical('Critical message')
 """

 import os
 import sys
 import logging
 from log_config import formatters
 from log_config import handlers
 from log_config import text_attribute
 from tools import os_tools

 __author__ = " "
 __status__ = "debugging"  # {debugging, production}
 __version__ = "0.1.0"  # major_version_number.minor_version_number.revision_number
 __date__ = " "

 """
 # Config root logger
 basicConfig(
     filename=__name__,
     filemode='w',
     format='%(asctime)s: %(levelname)s: %(message)s',
     datefmt='%Y-%m-%d %H:%M:%S',
     level='logging.NOTSET'
 )
 """

 # Config log path and file name.
 LOG_PATH = r'C:\log'
 LOG_NAME = r'mylogger.log'
 filename = 'default.log' if not os_tools.makedir(LOG_PATH) else os.path.abspath(os.path.join(LOG_PATH, LOG_NAME))

 class BTLogger(logging.Logger):
     def __init__(self, name=None, level=logging.NOTSET):
         """
         Initialize the BTLogger with basic settings.

         :param
             name: Specify the logger's name. If name is None, default is root logger.
                    Note:
                         All BTLogger instances with a given name return the same logger
                         instance. This means that logger instances never need to be passed
                         between different parts of an application.
             level: The threshold of logging records, messages which are less severe
                    than level will be ignored.
                    If the level is set to NOTSET, than:
                         1> If current logger is root logger, than all messages will
                         be logged.
                         2> Or, delegate to its parent logger, until an ancestor with
                         a level other than NOTSET is found, or the root is reached.
                    Note:
                         The root logger is created with level logging.WARNING.
         :return:
             None
         """
         logging.Logger.__init__(self, name, level)

         # Create handlers
         # non_blocking_ch = handlers.NonBlockingHandler(handlers.BTStreamHandler(sys.stdout, logging.DEBUG)).get_handler()  # version 3.2+
         console_handler = handlers.BTStreamHandler(stream=sys.stdout, level=logging.INFO)
         file_handler = handlers.BTFileHandler(filename=filename, level=logging.DEBUG, mode='w')

         # Set formatters to handlers.
         # non_blocking_ch.setFormatter(formatters.BTStreamFormatter())  # version 3.2+
         console_handler.setFormatter(formatters.BTStreamFormatter())
         file_handler.setFormatter(formatters.BTFileFormatter())

         # Add the handlers to logger.
         # self.addHandler(non_blocking_ch)  # version 3.2+
         self.addHandler(console_handler)
         self.addHandler(file_handler)

     # Override methods in logging.Logger(debug, info, warning, error, critical).
     # @handlers.NonBlockingHandler.non_blocking     # version 3.2+
     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_WHITE')
     def debug(self, msg, *args, **kwargs):
         self.log(logging.DEBUG, msg, *args, **kwargs)

     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_GREEN')
     def info(self, msg, *args, **kwargs):
         self.log(logging.INFO, msg, *args, **kwargs)

     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_YELLOW')
     def warning(self, msg, *args, **kwargs):
         self.log(logging.WARNING, msg, *args, **kwargs)

     warn = warning

     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
     def error(self, msg, *args, **kwargs):
         self.log(logging.ERROR, msg, *args, **kwargs)

     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
     def critical(self, msg, *args, **kwargs):
         self.log(logging.CRITICAL, msg, *args, **kwargs)

     fatal = critical

..\log_config\formatters.py:

 """
 Formatters specify the output format of logging messages.
 """

 from logging import Formatter

 # Changed in version 3.2+: The 'style' parameter was added.
 '''
 class BTStreamFormatter(Formatter):
     def __init__(self,
                  fmt='[%(asctime)s] [%(levelname)s]: %(message)s',
                  datefmt='%Y-%m-%d %H:%M:%S', style='%'):
         Formatter.__init__(self, fmt, datefmt, style)

 class BTFileFormatter(Formatter):
     def __init__(self,
                  fmt='%(asctime)s: %(levelname)s: %(message)s',
                  datefmt='%Y/%m/%d %H:%M:%S', style='%'):
         Formatter.__init__(self, fmt, datefmt, style)
 '''

 # Be suitable for version 3.2-.
 class BTStreamFormatter(Formatter):
     def __init__(self,
                  fmt='[%(asctime)s] [%(levelname)s]: %(message)s',
                  datefmt='%Y-%m-%d %H:%M:%S'):
         Formatter.__init__(self, fmt, datefmt)

 class BTFileFormatter(Formatter):
     def __init__(self,
                  fmt='%(asctime)s: %(levelname)s: %(message)s',
                  datefmt='%Y/%m/%d %H:%M:%S'):
         Formatter.__init__(self, fmt, datefmt)

..\log_config\handlers.py:

 """
 Handlers send logging messages(created by BTLogger) to the specific destination.
 """

 import sys
 import logging
 from logging import StreamHandler
 from logging import FileHandler

 class BTStreamHandler(StreamHandler):
     """
     Write logging records to a stream(e.g., console).

     Note that this class does not close the stream automatically,
     as sys.stdout or sys.stderr may be used.
     """
     def __init__(self, stream=sys.stderr, level=logging.INFO):
         """
         Initialize the handler.

         :param
             stream: If stream is not specified, sys.stderr is used.
         :return
             None
         """
         StreamHandler.__init__(self, stream)
         self.setLevel(level)    # The threshold of handling records, default is logging.INFO.

 class BTFileHandler(FileHandler):
     """
     Write logging records to disk files.

     Inherited from logging.FileHandler, only modify the default mode('a') to ('w').
     """
     def __init__(self,
                  filename='default.log',
                  level=logging.DEBUG,   # Handle messages with levels of logging.DEBUG and higher to file.
                  mode='w',
                  encoding=None,
                  delay=False):
         FileHandler.__init__(self, filename, mode, encoding, delay)
         self.setLevel(level)

 # New in version 3.2+: QueueHandler, QueueListener.
 # Unfold the following block, and you will get a non_blocking_handler.
 '''
 from queue import Queue
 from logging.handlers import QueueHandler
 from logging.handlers import QueueListener
 from functools import wraps

 class NonBlockingHandler(object):
     """
     Let logging handlers do their work without blocking the thread you're logging from.
     Especially those network-based handler can block, e.g., SocketHandler, SMTPHandler.
     """
     def __init__(self, handler_instance):
         self.queue = Queue(-1)
         self.queue_handler = QueueHandler(self.queue)
         self.handler_instance = handler_instance
         global _queue_listener
         _queue_listener = QueueListener(self.queue, self.handler_instance)

     def get_handler(self):
         return self.handler_instance

     @classmethod
     def non_blocking(cls, func):
         @wraps(func)
         def wrapper(*args, **kwargs):
             _queue_listener.start()
             res = func(*args, **kwargs)
             _queue_listener.stop()
             return res

         return wrapper
 '''

..\log_config\text_attribute.py:

 """
 Set the text attributes in console. Currently, you can only set font color.

 # -----------------------------------------
 # Author:
 # Created:
 # Modified:
 # Version: 0.1.0
 # -----------------------------------------

 Usages:
 When you want to design a function which outputs messages to console with different colors, simply use '@text_attribute.set_textcolor(fore_color, back_color)' to decorate your function, e.g.:
     @text_attribute.set_textcolor('FOREGROUND_BLACK', 'BACKGROUND_RED')
     def func(*args, **kwargs):
         # output some messages to console
         # ...

     func(...)
 """

 import ctypes
 from functools import wraps

 STD_INPUT_HANDLE = -10
 STD_OUTPUT_HANDLE = -11
 STD_ERROR_HANDLE = -12

 _CONSOLE_TEXT_COLOR = {
     'FOREGROUND_RED': 0x04,
     'FOREGROUND_GREEN': 0x02,
     'FOREGROUND_BLUE': 0x01,
     # 'FOREGROUND_INTENSITY': 0x08,
     'FOREGROUND_WHITE': 0x07,
     'FOREGROUND_BLACK': 0x00,
     'FOREGROUND_YELLOW': 0x06,

     'BACKGROUND_RED': 0x40,
     'BACKGROUND_GREEN': 0x20,
     'BACKGROUND_BLUE': 0x10,
     # 'BACKGROUND_INTENSITY': 0x80,
     'BACKGROUND_WHITE': 0x70,
     'BACKGROUND_BLACK': 0x00,
     'BACKGROUND_YELLOW': 0x60
 }

 class TextColor(object):
     """
     Set console text color.
     """
     std_output_handle = ctypes.windll.kernel32.GetStdHandle(STD_OUTPUT_HANDLE)

     @staticmethod
     def text_color(fore_color, back_color):
         fore_color = 'FOREGROUND_WHITE' if fore_color not in _CONSOLE_TEXT_COLOR else fore_color
         back_color = 'BACKGROUND_BLACK' if back_color not in _CONSOLE_TEXT_COLOR else back_color
         return _CONSOLE_TEXT_COLOR[fore_color] | _CONSOLE_TEXT_COLOR[back_color]

     @staticmethod
     def set(fore_color='FOREGROUND_WHITE', back_color='BACKGROUND_BLACK', handle=std_output_handle):
         is_set = ctypes.windll.kernel32.SetConsoleTextAttribute(handle, TextColor.text_color(fore_color, back_color))
         return is_set

     @staticmethod
     def unset():
         return TextColor.set()

 def set_textcolor(fore_color='FOREGROUND_WHITE', back_color='BACKGROUND_BLACK'):

     def decorate(func):
         """
         A decorator which can set the color of console message, include foreground and background.
         :param
             func: The original function which will be wrapped by a wrapper.
         :return:
             wrapper
         """
         @wraps(func)
         def wrapper(*args, **kwargs):
             TextColor.set(fore_color, back_color)
             res = func(*args, **kwargs)
             TextColor.unset()
             return res

         return wrapper

     return decorate

..\tools\__init__.py: None

..\tools\os_tools.py:

 """
 Originated from os module.

 # -----------------------------------------
 # Author:
 # Created:
 # Modified:
 # Version: 0.1.0
 # -----------------------------------------
 """

 import os

 def makedir(path):
     """
     Recursive directory creation, include all intermediate-level directories needed to contain the leaf directory.

     If specific path already exists, do nothing. Or path is not a directory, return False.
     """
     try:
         os.makedirs(path, 0o777)
         # os.makedirs(path, 0o777, False) # Version 3.2+
     except OSError:
         if not os.path.isdir(path):      # Path is not a valid directory.
             return False                 # Path already exists.
         return True
     else:                                # Path does not exist.
         return True