来自多个线程的 Python 日志记录

Python logging from multiple threads

提问by nckturner

I have a log.pymodule, that is used in at least two other modules (server.pyand device.py).

我有一个log.py模块，它用于至少两个其他模块（server.py和device.py）。

It has these globals:

它有这些全局变量：

fileLogger = logging.getLogger()
fileLogger.setLevel(logging.DEBUG)
consoleLogger = logging.getLogger()
consoleLogger.setLevel(logging.DEBUG)

file_logging_level_switch = {
    'debug':    fileLogger.debug,
    'info':     fileLogger.info,
    'warning':  fileLogger.warning,
    'error':    fileLogger.error,
    'critical': fileLogger.critical
}

console_logging_level_switch = {
    'debug':    consoleLogger.debug,
    'info':     consoleLogger.info,
    'warning':  consoleLogger.warning,
    'error':    consoleLogger.error,
    'critical': consoleLogger.critical
}

It has two functions:

它有两个功能：

def LoggingInit( logPath, logFile, html=True ):
    global fileLogger
    global consoleLogger

    logFormatStr = "[%(asctime)s %(threadName)s, %(levelname)s] %(message)s"
    consoleFormatStr = "[%(threadName)s, %(levelname)s] %(message)s"

    if html:
        logFormatStr = "<p>" + logFormatStr + "</p>"

    # File Handler for log file
    logFormatter = logging.Formatter(logFormatStr)
    fileHandler = logging.FileHandler( 
        "{0}{1}.html".format( logPath, logFile ))
    fileHandler.setFormatter( logFormatter )
    fileLogger.addHandler( fileHandler )

    # Stream Handler for stdout, stderr
    consoleFormatter = logging.Formatter(consoleFormatStr)
    consoleHandler = logging.StreamHandler() 
    consoleHandler.setFormatter( consoleFormatter )
    consoleLogger.addHandler( consoleHandler )

And:

和：

def WriteLog( string, print_screen=True, remove_newlines=True, 
        level='debug' ):

    if remove_newlines:
        string = string.replace('\r', '').replace('\n', ' ')

    if print_screen:
        console_logging_level_switch[level](string)

    file_logging_level_switch[level](string)

I call LoggingInitfrom server.py, which initializes the file and console loggers. I then call WriteLogfrom all over the place, so multiple threads are accessing fileLoggerand consoleLogger.

我调用LoggingInitfrom server.py，它初始化文件和控制台记录器。然后我WriteLog从各地调用，因此多个线程正在访问fileLogger和consoleLogger。

Do I need any further protection for my log file? The documentation states that thread locks are handled by the handler.

我的日志文件需要进一步保护吗？文档指出线程锁由处理程序处理。

采纳答案by abarnert

The good news is that you don't need to do anything extra for thread safety, and you either need nothing extra or something almost trivial for clean shutdown. I'll get to the details later.

好消息是你不需要为线程安全做任何额外的事情，你要么不需要额外的东西，要么不需要任何东西来干净关闭。稍后我会详细介绍。

The bad news is that your code has a serious problem even before you get to that point: fileLoggerand consoleLoggerare the same object. From the documentation for getLogger():

坏消息是，您的代码甚至在您到达该点之前就存在严重问题：fileLogger并且consoleLogger是同一个对象。从文档中getLogger()：

Return a logger with the specified name or, if no name is specified, return a logger which is the root logger of the hierarchy.

返回具有指定名称的记录器，或者，如果未指定名称，则返回作为层次结构的根记录器的记录器。

So, you're getting the root logger and storing it as fileLogger, and then you're getting the root logger and storing it as consoleLogger. So, in LoggingInit, you initialize fileLogger, then re-initialize the same object under a different name with different values.

因此，您获取根记录器并将其存储为fileLogger，然后您获取根记录器并将其存储为consoleLogger. 因此，在 中LoggingInit，您初始化fileLogger，然后使用不同的值以不同的名称重新初始化相同的对象。

You canadd multiple handlers to the same logger—and, since the only initialization you actually do for each is addHandler, your code will sort of work as intended, but only by accident. And only sort of. You will get two copies of each message in both logs if you pass print_screen=True, and you will get copies in the console even if you pass print_screen=False.

您可以向同一个记录器添加多个处理程序——而且，由于您实际为每个处理程序所做的唯一初始化是addHandler，您的代码将按预期工作，但只是偶然。而且只是有点。如果通过print_screen=True，您将在两个日志中获得每条消息的两个副本，即使通过，您也会在控制台中获得副本print_screen=False。

There's actually no reason for global variables at all; the whole point of getLogger()is that you can call it every time you need it and get the global root logger, so you don't need to store it anywhere.

实际上根本没有理由使用全局变量；重点getLogger()是您可以在每次需要时调用它并获取全局根记录器，因此您无需将其存储在任何地方。

---

---

A more minor problem is that you're not escaping the text you insert into HTML. At some point you're going to try to log the string "a < b"and end up in trouble.

一个更小的问题是您没有转义插入到 HTML 中的文本。在某些时候，您将尝试记录字符串"a < b"并最终遇到麻烦。

Less seriously, a sequence of <p>tags that isn't inside a <body>inside an <html>is not a valid HTML document. But plenty of viewers will take care of that automatically, or you can post-process your logs trivially before displaying them. But if you really want this to be correct, you need to subclass FileHandlerand have your __init__add a header if given an empty file and remove a footer if present, then have your closeadd a footer.

那么严肃，序列<p>标签，是不是里面<body>里面的<html>是不是一个有效的HTML文档。但是很多查看器会自动处理这些问题，或者您可以在显示日志之前对其进行微不足道的后期处理。但是，如果您真的希望这是正确的，则需要子类化FileHandler并__init__在给定空文件的情况下添加标题并删除页脚（如果存在），然后close添加页脚。

---

---

Getting back to your actual question:

回到你的实际问题：

You do not need any additional locking. If a handler correctly implements createLock, acquire, and release(and it's called on a platform with threads), the logging machinery will automatically make sure to acquire the lock when needed to make sure each message is logged atomically.

您不需要任何额外的锁定。如果处理程序正确实现了createLock, acquire, 和release（并且在具有线程的平台上调用），日志记录机制将自动确保在需要时获取锁，以确保以原子方式记录每条消息。

As far as I know, the documentation doesn't directlysay that StreamHandlerand FileHandlerimplement these methods, it does strongly imply it (the text you mentioned in the questionsays "The logging module is intended to be thread-safe without any special work needing to be done by its clients", etc.). And you can look at the source for your implementation (e.g., CPython 3.3) and see that they both inherit correctly-implemented methods from logging.Handler.

据我所知，文档并没有直接说明StreamHandler并FileHandler实现这些方法，它确实强烈暗示了它（您在问题中提到的文字说“日志记录模块旨在是线程安全的，无需任何特殊工作由其客户完成”等）。并且您可以查看您的实现的源代码（例如，CPython 3.3）并看到它们都从logging.Handler.

---

---

Likewise, if a handler correctly implements flushand close, the logging machinery will make sure it's finalized correctly during normal shutdown.

同样，如果处理程序正确实现了flushand close，日志机制将确保它在正常关闭期间正确完成。

Here, the documentation does explain what StreamHandler.flush(), FileHandler.flush(), and FileHandler.close(). They're mostly what you'd expect, except that StreamHandler.close()is a no-op, meaning it's possible that final log messages to the console may get lost. From the docs:

在这里，文件并解释什么StreamHandler.flush()，FileHandler.flush()以及FileHandler.close()。它们主要是您所期望的，除了这StreamHandler.close()是一个空操作，这意味着最终到控制台的日志消息可能会丢失。从文档：

Note that the close()method is inherited from Handlerand so does no output, so an explicit flush()call may be needed at times.

请注意，该close()方法是从继承而来的Handler，因此没有输出，因此flush()有时可能需要显式调用。

If this matters to you, and you want to fix it, you need to do something like this:

如果这对您很重要，并且您想修复它，则需要执行以下操作：

class ClosingStreamHandler(logging.StreamHandler):
    def close(self):
        self.flush()
        super().close()

And then use ClosingStreamHandler()instead of StreamHandler().

然后使用ClosingStreamHandler()代替StreamHandler()。

FileHandlerhas no such problem.

FileHandler没有这样的问题。

---

---

The normal way to send logs to two places is to just use the root logger with two handlers, each with their own formatter.

将日志发送到两个地方的正常方法是使用带有两个处理程序的根记录器，每个处理程序都有自己的格式化程序。

Also, even if you do want two loggers, you don't need the separate console_logging_level_switchand file_logging_level_switchmaps; calling Logger.debug(msg)is exactly the same thing as calling Logger.log(DEBUG, msg). You'll still need some way to map your custom level names debug, etc. to the standard names DEBUG, etc., but you can just do one lookup, instead of doing it once per logger (plus, if your names are just the standard names with different cast, you can cheat).

此外，即使您确实想要两个记录器，您也不需要单独的console_logging_level_switch和file_logging_level_switch地图；call 和 callLogger.debug(msg)完全一样Logger.log(DEBUG, msg)。您仍然需要某种方式将自定义级别名称debug等映射到标准名称DEBUG等，但是您可以只进行一次查找，而不是每个记录器执行一次（另外，如果您的名称只是标准名称）使用不同的演员，你可以作弊）。

This is all described pretty well in the `Multiple handlers and formatterssection, and the rest of the logging cookbook.

这在`多个处理程序和格式化程序部分以及日志记录手册的其余部分中都有很好的描述。

The only problem with the standard way of doing this is that you can't easily turn off console logging on a message-by-message basis. That's because it's not a normal thing to do. Usually, you just log by levels, and set the log level higher on the file log.

这样做的标准方法的唯一问题是您无法轻松地逐条消息地关闭控制台日志记录。那是因为这不是一件正常的事情。通常，您只需按级别记录，并在文件日志上设置更高的日志级别。

But, if you want more control, you can use filters. For example, give your FileHandlera filter that accepts everything, and your ConsoleHandlera filter that requires something starting with console, then use the filter 'console' if print_screen else ''. That reduces WriteLogto almost a one-liner.

但是，如果您想要更多控制，可以使用过滤器。例如，给您FileHandler一个接受所有内容的过滤器，以及ConsoleHandler一个需要以 开头console的过滤器，然后使用过滤器'console' if print_screen else ''。这减少WriteLog到几乎单行。

You still need the extra two lines to remove newlines—but you can even do thatin the filter, or via an adapter, if you want. (Again, see the cookbook.) And then WriteLogreally isa one-liner.

您还需要额外的两行删除换行符，但你甚至可以做的是在过滤器中，或者通过一个适配器，如果你想要的。（再次，请参阅食谱。）然后WriteLog真的是单行。

回答by andrew cooke

Python logging is thread safe:

Python 日志是线程安全的：

• Is Python's logging module thread safe?

• http://docs.python.org/2/library/logging.html#thread-safety

• Python 的日志记录模块线程安全吗？

• http://docs.python.org/2/library/logging.html#thread-safety

So you have no problem in the Python (library) code.

所以你在 Python（库）代码中没有问题。

The routine that you call from multiple threads (WriteLog) does not write to any shared state. So you have no problem in your code.

您从多个线程 ( WriteLog)调用的例程不会写入任何共享状态。所以你的代码没有问题。

So you are OK.

所以你没事。