scala 使用 sbt-assembly 的 assembly-merge-strategy 问题

assembly-merge-strategy issues using sbt-assembly

提问by sc_ray

I am trying to convert a scala project into a deployable fat jar using sbt-assembly. When I run my assembly task in sbt I am getting the following error:

我正在尝试使用sbt-assembly将 scala 项目转换为可部署的胖罐。当我在 sbt 中运行我的组装任务时，出现以下错误：

Merging 'org/apache/commons/logging/impl/SimpleLog.class' with strategy 'deduplicate'
    :assembly: deduplicate: different file contents found in the following:
    [error] /Users/home/.ivy2/cache/commons-logging/commons-logging/jars/commons-logging-1.1.1.jar:org/apache/commons/logging/impl/SimpleLog.class
    [error] /Users/home/.ivy2/cache/org.slf4j/jcl-over-slf4j/jars/jcl-over-slf4j-1.6.4.jar:org/apache/commons/logging/impl/SimpleLog.class

Now from the sbt-assembly documentation:

现在来自 sbt-assembly 文档：

If multiple files share the same relative path (e.g. a resource named application.conf in multiple dependency JARs), the default strategy is to verify that all candidates have the same contents and error out otherwise. This behavior can be configured on a per-path basis using either one of the following built-in strategies or writing a custom one:

• MergeStrategy.deduplicateis the default described above
• MergeStrategy.firstpicks the first of the matching files in classpath order
• MergeStrategy.lastpicks the last one
• MergeStrategy.singleOrErrorbails out with an error message on conflict
• MergeStrategy.concatsimply concatenates all matching files and includes the result
• MergeStrategy.filterDistinctLinesalso concatenates, but leaves out duplicates along the way
• MergeStrategy.renamerenames the files originating from jar files
• MergeStrategy.discardsimply discards matching files

如果多个文件共享相同的相对路径（例如，多个依赖项 JAR 中名为 application.conf 的资源），默认策略是验证所有候选文件具有相同的内容，否则会出错。可以使用以下任一内置策略或编写自定义策略在每个路径的基础上配置此行为：

• MergeStrategy.deduplicate是上面描述的默认值
• MergeStrategy.first按类路径顺序选择第一个匹配的文件
• MergeStrategy.last选择最后一个
• MergeStrategy.singleOrError以关于冲突的错误消息退出
• MergeStrategy.concat简单地连接所有匹配的文件并包含结果
• MergeStrategy.filterDistinctLines也连接，但沿途省略重复项
• MergeStrategy.rename重命名源自 jar 文件的文件
• MergeStrategy.discard简单地丢弃匹配的文件

Going by this I setup my build.sbt as follows:

按照这个我设置我的 build.sbt 如下：

import sbt._
import Keys._
import sbtassembly.Plugin._
import AssemblyKeys._
name := "my-project"
version := "0.1"
scalaVersion := "2.9.2"
crossScalaVersions := Seq("2.9.1","2.9.2")

//assemblySettings
seq(assemblySettings: _*)

resolvers ++= Seq(
    "Typesafe Releases Repository" at "http://repo.typesafe.com/typesafe/releases/",
    "Typesafe Snapshots Repository" at "http://repo.typesafe.com/typesafe/snapshots/",
    "Sonatype Repository" at "http://oss.sonatype.org/content/repositories/releases/"
)

libraryDependencies ++= Seq(
    "org.scalatest" %% "scalatest" % "1.6.1" % "test",
    "org.clapper" %% "grizzled-slf4j" % "0.6.10",
    "org.scalaz" % "scalaz-core_2.9.2" % "7.0.0-M7",
    "net.databinder.dispatch" %% "dispatch-core" % "0.9.5"
)

scalacOptions += "-deprecation"
mainClass in assembly := Some("com.my.main.class")
test in assembly := {}
mergeStrategy in assembly := mergeStrategy.first

In the last line of the build.sbt, I have:

在 build.sbt 的最后一行，我有：

mergeStrategy in assembly := mergeStrategy.first

Now, when I run SBT, I get the following error:

现在，当我运行 SBT 时，出现以下错误：

error: value first is not a member of sbt.SettingKey[String => sbtassembly.Plugin.MergeStrategy]
    mergeStrategy in assembly := mergeStrategy.first

Can somebody point out what I might be doing wrong here?

有人可以指出我在这里可能做错了什么吗？

Thanks

谢谢

采纳答案by mrArias

I think it should be MergeStrategy.firstwith a capital M, so mergeStrategy in assembly := MergeStrategy.first.

我想应该是MergeStrategy.first有了资本M，所以mergeStrategy in assembly := MergeStrategy.first。

回答by Beryllium

As for the current version 0.11.2 (2014-03-25), the way to define the merge strategy is different.

对于当前版本0.11.2（2014-03-25），定义合并策略的方式有所不同。

This is documented here, the relevant part is:

这在此处记录，相关部分是：

NOTE: mergeStrategy in assembly expects a function, you can't do

mergeStrategy in assembly := MergeStrategy.first

注意：程序集中的 mergeStrategy 需要一个函数，你不能这样做

mergeStrategy in assembly := MergeStrategy.first

The new way is (copied from the same source):

新方法是（从同一来源复制）：

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*)         => MergeStrategy.first
    case PathList(ps @ _*) if ps.last endsWith ".html" => MergeStrategy.first
    case "application.conf" => MergeStrategy.concat
    case "unwanted.txt"     => MergeStrategy.discard
    case x => old(x)
  }
}

This is possibly applicable to earlier versions as well, I don't know exactly when it has changed.

这可能也适用于早期版本，我不知道它何时发生了变化。

回答by mrArias

I have just setup a little sbt project that needs to rewire some mergeStrategies, and found the answer a little outdated, let me add my working code for versions (as of 4-7-2015)

我刚刚设置了一个需要重新连接一些 mergeStrategies 的小 sbt 项目，发现答案有点过时，让我添加我的版本工作代码（截至 2015 年 4 月 7 日）

• sbt 0.13.8
• scala 2.11.6

• assembly 0.13.0

  mergeStrategy in assembly := {
    case x if x.startsWith("META-INF") => MergeStrategy.discard // Bumf
    case x if x.endsWith(".html") => MergeStrategy.discard // More bumf
    case x if x.contains("slf4j-api") => MergeStrategy.last
    case x if x.contains("org/cyberneko/html") => MergeStrategy.first
    case PathList("com", "esotericsoftware", xs@_ *) => MergeStrategy.last // For Log$Logger.class
    case x =>
       val oldStrategy = (mergeStrategy in assembly).value
       oldStrategy(x)
  }
  

• sbt 0.13.8
• Scala 2.11.6

• 组装 0.13.0

  mergeStrategy in assembly := {
    case x if x.startsWith("META-INF") => MergeStrategy.discard // Bumf
    case x if x.endsWith(".html") => MergeStrategy.discard // More bumf
    case x if x.contains("slf4j-api") => MergeStrategy.last
    case x if x.contains("org/cyberneko/html") => MergeStrategy.first
    case PathList("com", "esotericsoftware", xs@_ *) => MergeStrategy.last // For Log$Logger.class
    case x =>
       val oldStrategy = (mergeStrategy in assembly).value
       oldStrategy(x)
  }
  

回答by Alex Punnen

For the new sbt version (sbt-version :0.13.11), I was getting the error for slf4j; for the time being took the easy way out : Please also check the answer here Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.classwhere sbt-dependency-graphtool is mentioned which is pretty cool to do this manually

对于新的 sbt 版本（sbt 版本：0.13.11），我收到了 slf4j 的错误；暂时采取简单的方法：请同时检查此处的答案Scala SBT Assembly 由于在 StaticLoggerBinder.class中的重复数据删除错误而无法合并，其中提到了 sbt-dependency-graph工具，手动执行此操作非常酷

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

回答by linehrr

this is the proper way to merge most of the common java/scala projects. it takes care of META-INF and classes.

这是合并大多数常见 java/scala 项目的正确方法。它负责 META-INF 和类。

also the service registration in META-INF is taken care of.

META-INF 中的服务注册也得到处理。

assemblyMergeStrategy in assembly := {
case x if Assembly.isConfigFile(x) =>
  MergeStrategy.concat
case PathList(ps @ _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
  MergeStrategy.rename
case PathList("META-INF", xs @ _*) =>
  (xs map {_.toLowerCase}) match {
    case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
      MergeStrategy.discard
    case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
      MergeStrategy.discard
    case "plexus" :: xs =>
      MergeStrategy.discard
    case "services" :: xs =>
      MergeStrategy.filterDistinctLines
    case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
      MergeStrategy.filterDistinctLines
    case _ => MergeStrategy.first
  }
case _ => MergeStrategy.first}

回答by Jake

Quick update: mergeStrategy is deprecated. Use assemblyMergeStrategy. Apart from that, earlier responses are still solid

快速更新：不推荐使用 mergeStrategy。使用 assemblyMergeStrategy。除此之外，早期的反应仍然很可靠

回答by Sushruth

Add following to build.sbt to add kafka as source or destination

将以下内容添加到 build.sbt 以将 kafka 添加为源或目标

 assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 //To add Kafka as source
 case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => 
 MergeStrategy.concat
 case x => MergeStrategy.first
 }