scala 如何在单元测试中抑制 Spark 日志记录？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/27248997/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to suppress Spark logging in unit tests?
提问by samthebest
So thanks to easily googleable blogs I tried:
因此，多亏了我尝试过的易于搜索的博客：
import org.specs2.mutable.Specification
class SparkEngineSpecs extends Specification {
sequential
def setLogLevels(level: Level, loggers: Seq[String]): Map[String, Level] = loggers.map(loggerName => {
val logger = Logger.getLogger(loggerName)
val prevLevel = logger.getLevel
logger.setLevel(level)
loggerName -> prevLevel
}).toMap
setLogLevels(Level.WARN, Seq("spark", "org.eclipse.jetty", "akka"))
val sc = new SparkContext(new SparkConf().setMaster("local").setAppName("Test Spark Engine"))
// ... my unit tests
But unfortunately it doesn't work, I still get a lot of spark output, e.g.:
但不幸的是它不起作用，我仍然得到很多火花输出，例如：
14/12/02 12:01:56 INFO MemoryStore: Block broadcast_4 of size 4184 dropped from memory (free 583461216)
14/12/02 12:01:56 INFO ContextCleaner: Cleaned broadcast 4
14/12/02 12:01:56 INFO ContextCleaner: Cleaned shuffle 4
14/12/02 12:01:56 INFO ShuffleBlockManager: Deleted all files for shuffle 4
回答by Emre Sevin?
Add the following code into the log4j.propertiesfile inside the src/test/resourcesdir, create the file/dir if not exist
下面的代码添加到log4j.properties里面的文件src/test/resources目录，创建文件/目录。如果不存在
# Change this to set Spark log level
log4j.logger.org.apache.spark=WARN
# Silence akka remoting
log4j.logger.Remoting=WARN
# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN
When I run my unit tests (I'm using JUnit and Maven), I only receive WARN level logs, in other words no more cluttering with INFO level logs (though they can be useful at times for debugging).
当我运行我的单元测试（我使用 JUnit 和 Maven）时，我只收到 WARN 级别的日志，换句话说，不再有 INFO 级别的日志混乱（尽管它们有时对调试很有用）。
I hope this helps.
我希望这有帮助。
回答by LiMuBei
In my case one of my own libraries brought logback-classic into the mix. This materialized in a warning at the start:
就我而言，我自己的一个库将 logback-classic 融入其中。这在开始时出现了警告：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
I solved this by excluding it from the dependency:
我通过从依赖项中排除它来解决这个问题：
"com.mystuff" % "mylib" % "1.0.0" exclude("ch.qos.logback", "logback-classic")
Now I could add a log4j.propertiesfile in test/resourceswhich now gets used by Spark.
现在我可以添加一个现在被 Spark 使用的log4j.properties文件test/resources。
回答by Philosophus42
After some time of struggling with Spark log output as well, I found a blog postwith a solution I particularly liked.
在用 Spark 日志输出挣扎了一段时间后，我找到了一篇博客文章，其中包含我特别喜欢的解决方案。
If you use slf4j, one can simply exchange the underlying log implementation. A good canidate for the test scope is slf4j-nop, which carfully takes the log output and puts it where the sun never shines.
如果您使用 slf4j，则可以简单地交换底层日志实现。测试范围的一个很好的候选者是 slf4j-nop，它小心翼翼地获取日志输出并将其放在阳光从不发光的地方。
When using Maven you can add the following to the top of your dependencies list:
使用 Maven 时，您可以将以下内容添加到依赖项列表的顶部：
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.12</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.7.12</version>
<scope>test</scope>
</dependency>
Note that it might be important to have it at the beginning of the dependencies list to make sure that the given implementations are used instead of those that might come with other packages (and which you can consider to exclude in order to keep your class path tidy and avoid unexpected conflicts).
请注意，将它放在依赖项列表的开头可能很重要，以确保使用给定的实现而不是其他包可能附带的实现（并且您可以考虑将其排除以保持类路径整洁）并避免意外冲突）。
回答by Daniel Darabos
You can use a separate Logback config for tests. Depending on your environment it's possible that you just need to create conf/logback-test.xmlwith something that hides the logs. I think this should do that:
您可以使用单独的 Logback 配置进行测试。根据您的环境，您可能只需要conf/logback-test.xml使用隐藏日志的内容进行创建。我认为这应该这样做：
<configuration>
<root level="debug">
</root>
</configuration>
As I understand it, this captures all logs (level debugand higher) and assigns no logger to them, so they get discarded. A better option is to configure a file logger for them, so you can still access the logs if you want to.
据我了解，这会捕获所有日志（级别debug和更高级别）并且没有为它们分配记录器，因此它们会被丢弃。更好的选择是为它们配置文件记录器，这样您仍然可以根据需要访问日志。
See http://logback.qos.ch/manual/configuration.htmlfor the detailed documentation.
回答by lockwobr
A little late to the party but I found this in the spark example code:
聚会有点晚了，但我在spark 示例代码中发现了这一点：
def setStreamingLogLevels() {
val log4jInitialized = Logger.getRootLogger.getAllAppenders.hasMoreElements
if (!log4jInitialized) {
// We first log something to initialize Spark's default logging, then we override the
// logging level.
logInfo("Setting log level to [WARN] for streaming example." +
" To override add a custom log4j.properties to the classpath.")
Logger.getRootLogger.setLevel(Level.WARN)
}
}
I also found that with your code if you call setLogLevels like below it cut out alot of out put for me.
我还发现，如果你像下面这样调用 setLogLevels，你的代码会为我减少很多输出。
setLogLevels(Level.WARN, Seq("spark", "org", "akka"))
回答by oskarryn
The easiest solution working for me is:
对我来说最简单的解决方案是：
cp $SPARK_HOME/conf/log4j.properties.template $YOUR_PROJECT/src/test/resources/log4j.properties
sed -i -e 's/log4j.rootCategory=INFO/log4j.rootCategory=WARN/g' $YOUR_PROJECT/src/test/resources/log4j.properties
