Java Maven docker 缓存依赖项

Maven docker cache dependencies

提问by Daniel Watrous

I'm trying to use docker to automate maven builds. The project I want to build takes nearly 20 minutes to download all the dependencies, so I tried to build a docker image that would cache these dependencies, but it doesn't seem to save it. My Dockerfile is

我正在尝试使用 docker 来自动化 maven 构建。我要构建的项目需要将近20分钟才能下载所有依赖项，因此我尝试构建一个可以缓存这些依赖项的docker镜像，但似乎并没有保存它。我的 Dockerfile 是

FROM maven:alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD pom.xml /usr/src/app
RUN mvn dependency:go-offline

The image builds, and it does download everything. However, the resulting image is the same size as the base maven:alpineimage, so it doesn't seem to have cached the dependencies in the image. When I try to use the image to mvn compileit goes through the full 20 minutes of redownloading everything.

图像构建，它确实下载了所有东西。但是，生成的图像与基础maven:alpine图像的大小相同，因此它似乎没有缓存图像中的依赖项。当我尝试使用图像时，mvn compile它经历了整整 20 分钟的重新下载所有内容。

Is it possible to build a maven image that caches my dependencies so they don't have to download everytime I use the image to perform a build?

是否可以构建一个缓存我的依赖项的 Maven 图像，这样他们就不必在我每次使用图像执行构建时都下载？

I'm running the following commands:

我正在运行以下命令：

docker build -t my-maven .

docker run -it --rm --name my-maven-project -v "$PWD":/usr/src/mymaven -w /usr/src/mymaven my-maven mvn compile

My understanding is that whatever RUNdoes during the docker build process becomes part of the resulting image.

我的理解是，RUN在 docker 构建过程中所做的一切都会成为生成图像的一部分。

回答by vstrom coder

If the dependencies are downloaded after the container is already up, then you need to commit the changes on this container and create a new image with the downloaded artifacts.

如果在容器已经启动之后下载依赖项，那么您需要在此容器上提交更改并使用下载的工件创建一个新映像。

回答by Bruno9779

I had this issue just a litle while ago. The are many solutions on the web, but the one that worked for me is simply mount a volume for the maven modules directory:

不久前我遇到了这个问题。网络上有很多解决方案，但对我有用的解决方案只是为 Maven 模块目录安装一个卷：

mkdir /opt/myvolumes/m2

then in the Dockerfile:

然后在 Dockerfile 中：

...
VOLUME /opt/myvolumes/m2:/root/.m2
...

There are better solutions, but not as straightforward.

有更好的解决方案，但没有那么简单。

This blog post goes the extra mile in helping you to cache everything:

这篇博文在帮助您缓存所有内容方面付出了额外的努力：

https://keyholesoftware.com/2015/01/05/caching-for-maven-docker-builds/

https://keyholesoftware.com/2015/01/05/caching-for-maven-docker-builds/

回答by Daniel Watrous

It turns out the image I'm using as a base has a parent image which defines

事实证明，我用作基础的图像有一个父图像，它定义了

VOLUME "$USER_HOME_DIR/.m2"

see: https://github.com/carlossg/docker-maven/blob/322d0dff5d0531ccaf47bf49338cb3e294fd66c8/jdk-8/Dockerfile

见：https: //github.com/carlossg/docker-maven/blob/322d0dff5d0531ccaf47bf49338cb3e294fd66c8/jdk-8/Dockerfile

The result is that during the build, all the files are written to $USER_HOME_DIR/.m2, but because it is expected to be a volume, none of those files are persisted with the container image.

结果是在构建过程中，所有文件都被写入$USER_HOME_DIR/.m2，但由于它预计是一个卷，这些文件都不会与容器映像一起持久化。

Currently in Docker there isn't any way to unregister that volume definition, so it would be necessary to build a separate maven image, rather than use the official maven image.

目前在 Docker 中没有任何方法可以取消注册该卷定义，因此有必要构建一个单独的 maven 镜像，而不是使用官方的 maven 镜像。

回答by Kim

Usually, there's no change in pom.xmlfile but just some other source code changes when you're attempting to start docker image build. In such circumstance you can do this:

通常，pom.xml当您尝试启动 docker 镜像构建时，文件不会发生变化，而只是其他一些源代码发生了变化。在这种情况下，您可以这样做：

FYI:

供参考：

FROM maven:3-jdk-8

ENV HOME=/home/usr/app

RUN mkdir -p $HOME

WORKDIR $HOME

# 1. add pom.xml only here

ADD pom.xml $HOME

# 2. start downloading dependencies

RUN ["/usr/local/bin/mvn-entrypoint.sh", "mvn", "verify", "clean", "--fail-never"]

# 3. add all source code and start compiling

ADD . $HOME

RUN ["mvn", "package"]

EXPOSE 8005

CMD ["java", "-jar", "./target/dist.jar"]

So the key is:

所以关键是：

1. add pom.xmlfile.

2. then mvn verify --fail-neverit, it will download maven dependencies.

3. add all your source file then, and start your compilation(mvn package).

1. 添加pom.xml文件。

2. 然后mvn verify --fail-never它，它将下载 maven 依赖项。

3. 然后添加所有源文件，然后开始编译（mvn package）。

When there are changes in your pom.xmlfile or you are running this script for the first time, docker will do 1 -> 2 -> 3. When there are no changes in pom.xmlfile, docker will skip step 1、2 and do 3 directly.

当你的pom.xml文件有改动或者你是第一次运行这个脚本时，docker会做1->2->3。当pom.xml文件没有变化时，docker会跳过步骤1、2直接做3。

This simple trick can be used in many other package management circumstances(gradle、yarn、npm、pip).

这个简单的技巧可以用于许多其他的包管理环境（gradle、yarn、npm、pip）。

Edit:

编辑：

You should also consider using mvn dependency:resolveor mvn dependency:go-offlineaccordingly as other comments & answers suggest.

您还应该考虑使用mvn dependency:resolve或mvn dependency:go-offline相应地作为其他评论和答案的建议。

回答by Filip Procházka

@Kim is closest, but it's not quite there yet. I don't think adding --fail-neveris correct, even through it get's the job done.

@Kim 最接近，但还没有完全到位。我不认为添加--fail-never是正确的，即使通过它完成了工作。

The verifycommand causes a lot of plugins to execute which is a problem (for me) - I don't think they should be executing when all I want is to install dependencies! I also have a multi-module build and a javascript sub-build so this further complicates the setup.

该verify命令会导致执行很多插件，这是一个问题（对我而言） - 我认为当我只想安装依赖项时，它们不应该执行！我还有一个多模块构建和一个 javascript 子构建，因此这进一步使设置复杂化。

But running only verifyis not enough, because if you run installin the following commands, there will be more plugins used - which means more dependencies to download - maven refuses to download them otherwise. Relevant read: Maven: Introduction to the Build Lifecycle

但是只运行verify是不够的，因为如果你install在下面的命令中运行，将会使用更多的插件——这意味着需要下载更多的依赖——否则 maven 拒绝下载它们。相关阅读：Maven：构建生命周期简介

You basically have to find what properties disable each plugin and add them one-by-one, so they don't break your build.

您基本上必须找到禁用每个插件的属性并一个一个地添加它们，这样它们就不会破坏您的构建。

WORKDIR /srv

# cache Maven dependencies
ADD cli/pom.xml /srv/cli/
ADD core/pom.xml /srv/core/
ADD parent/pom.xml /srv/parent/
ADD rest-api/pom.xml /srv/rest-api/
ADD web-admin/pom.xml /srv/web-admin/
ADD pom.xml /srv/
RUN mvn -B clean install -DskipTests -Dcheckstyle.skip -Dasciidoctor.skip -Djacoco.skip -Dmaven.gitcommitid.skip -Dspring-boot.repackage.skip -Dmaven.exec.skip=true -Dmaven.install.skip -Dmaven.resources.skip

# cache YARN dependencies
ADD ./web-admin/package.json ./web-admin/yarn.lock /srv/web-admin/
RUN yarn --non-interactive --frozen-lockfile --no-progress --cwd /srv/web-admin install

# build the project
ADD . /srv
RUN mvn -B clean install

but some plugins are not that easily skipped - I'm not a maven expert (so I don't know why it ignores the cli option - it might be a bug), but the following works as expected for org.codehaus.mojo:exec-maven-plugin

但是有些插件不是那么容易跳过 - 我不是 Maven 专家（所以我不知道为什么它会忽略 cli 选项 - 它可能是一个错误），但以下内容按预期工作 org.codehaus.mojo:exec-maven-plugin

<project>
    <properties>
        <maven.exec.skip>false</maven.exec.skip>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.3.2</version>
                <executions>
                    <execution>
                        <id>yarn install</id>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                        <phase>initialize</phase>
                        <configuration>
                            <executable>yarn</executable>
                            <arguments>
                                <argument>install</argument>
                            </arguments>
                            <skip>${maven.exec.skip}</skip>
                        </configuration>
                    </execution>
                    <execution>
                        <id>yarn run build</id>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                        <phase>compile</phase>
                        <configuration>
                            <executable>yarn</executable>
                            <arguments>
                                <argument>run</argument>
                                <argument>build</argument>
                            </arguments>
                            <skip>${maven.exec.skip}</skip>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

please notice the explicit <skip>${maven.exec.skip}</skip>- other plugins pick this up from the cli params but not this one (neither -Dmaven.exec.skip=truenor -Dexec.skip=truework by itself)

请注意明确<skip>${maven.exec.skip}</skip>-其他插件（既不接从CLI PARAMS这件事，但没有这一项-Dmaven.exec.skip=true，也不-Dexec.skip=true由本身的工作）

Hope this helps

希望这可以帮助

回答by ikandars

Similar with @Kim answer but I use dependency:resolvemvn command. So here's my complete Dockerfile:

与@Kim 的回答类似，但我使用dependency:resolvemvn 命令。所以这是我完整的 Dockerfile：

FROM maven:3.5.0-jdk-8-alpine

WORKDIR /usr/src/app

# First copy only the pom file. This is the file with less change
COPY ./pom.xml .

# Download the package and make it cached in docker image
RUN mvn -B -f ./pom.xml -s /usr/share/maven/ref/settings-docker.xml dependency:resolve

# Copy the actual code
COPY ./ .

# Then build the code
RUN mvn -B -f ./pom.xml -s /usr/share/maven/ref/settings-docker.xml package

# The rest is same as usual
EXPOSE 8888

CMD ["java", "-jar", "./target/YOUR-APP.jar"]

回答by Krzysztof Czelusniak

I don't think the other answers here are optimal. For example, the mvn verifyanswer executes the following phases, and does a lot more than just resolving dependencies:

我不认为这里的其他答案是最佳的。例如，mvn verifyanswer 执行以下阶段，并且不仅仅是解决依赖关系：

validate - validate the project is correct and all necessary information is available

compile - compile the source code of the project

test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed

package - take the compiled code and package it in its distributable format, such as a JAR.

verify - run any checks on results of integration tests to ensure quality criteria are met

验证 - 验证项目是否正确并且所有必要的信息都可用

compile - 编译项目的源代码

测试 - 使用合适的单元测试框架测试编译的源代码。这些测试不应该要求打包或部署代码

包 - 获取编译后的代码并将其打包为其可分发格式，例如 JAR。

验证 - 对集成测试的结果进行任何检查，以确保满足质量标准

All of these phases and their associated goals don't need to be ran if you only want to resolve dependencies.

如果您只想解决依赖关系，则不需要运行所有这些阶段及其相关目标。

If you only want to resolve dependencies, you can use the dependency:go-offlinegoal:

如果只想解决依赖关系，可以使用dependency:go-offline目标：

FROM maven:3-jdk-12
WORKDIR /tmp/example/

COPY pom.xml .
RUN mvn dependency:go-offline

COPY src/ src/
RUN mvn package

回答by Calvin Zhou

After a few days of struggling, I managed to do this caching later using intermediate contrainer, and I'd like to summarize my findings here as this topic is so useful and being frequently shown in Google search frontpage:

经过几天的努力，我后来设法使用中间约束器进行了缓存，我想在这里总结一下我的发现，因为这个主题非常有用并且经常出现在 Google 搜索首页中：

1. Kim's answer is only working to a certain condition: pom.xml cannot be changed, plus Maven do a regular update daily basis by default
2. mvn dependency:go-offline -B --fail-never has a similar drawback, so if you need to pull fresh code from repo, high chances are Maven will trigger a full checkout every time
3. Mount volume is not working as well because we need to resolve the dependencies during image being built
4. Finally, I have a workable solution combined(May be not working to others):
   • Build an image to resolve all the dependencies first(Not intermediate image)
   • Create another Dockerfile with intermediate image, sample dockerfiles like this:

1. Kim 的回答仅适用于特定条件：pom.xml 不能更改，加上 Maven 默认每天定期更新
2. mvn dependency:go-offline -B --fail-never 没有类似的缺点，所以如果你需要从 repo 中提取新代码，Maven 很有可能每次都会触发一次完整的检出
3. 挂载卷不能正常工作，因为我们需要在构建映像期间解决依赖关系
4. 最后，我有一个可行的解决方案（可能对其他人不起作用）：
   • 先构建一个镜像来解决所有的依赖（不是中间镜像）
   • 使用中间映像创建另一个 Dockerfile，示例 dockerfiles 如下所示：

#docker build -t dependencies .
From ubuntu
COPY pom.xml pom.xml
RUN mvn dependency:go-offline -B --fail-never

From dependencies as intermediate

From tomcat
RUN git pull repo.git (whatsoever)
RUN mvn package

The idea is to keep all the dependencies in a different image that Maven can use immediately

这个想法是将所有依赖项保存在 Maven 可以立即使用的不同映像中

It could be other scenarios I haven't encountered yet, but this solution relief me a bit from download 3GB rubbish every time I cannot imagine why Java became such a fat whale in today's lean world

可能是我还没有遇到过的其他场景，但是这个解决方案让我从下载 3GB 垃圾中解脱出来，每次我无法想象为什么 Java 在今天的精益世界中变成了如此肥大的鲸鱼

回答by Omri Spector

There are two ways to cache maven dependencies:

缓存maven依赖有两种方式：

1. Execute "mvn verify" as part of a container execution, NOT build, and make sure you mount .m2 from a volume.

   This is efficient but it does not play well with cloud build and multiple build slaves

2. Use a "dependencies cache container", and update it periodically. Here is how:

   a. Create a Dockerfile that copies the pom and build offline dependencies:

   FROM maven:3.5.3-jdk-8-alpine
   WORKDIR /build
   COPY pom.xml .
   RUN mvn dependency:go-offline
   

   b. Build it periodically (e.g. nightly) as "Deps:latest"

   c. Create another Dockerfile to actually build the system per commit (preferably use multi-stage) - and make sure it is FROM Deps.

1. 执行“mvn verify”作为容器执行的一部分，而不是构建，并确保从卷挂载 .m2。

   这是有效的，但它不适用于云构建和多个构建从属

2. 使用“依赖项缓存容器”，并定期更新它。方法如下：

   一种。创建一个 Dockerfile 来复制 pom 并构建离线依赖项：

   FROM maven:3.5.3-jdk-8-alpine
   WORKDIR /build
   COPY pom.xml .
   RUN mvn dependency:go-offline
   

   湾 定期（例如每晚）将其构建为“Deps：latest”

   C。创建另一个 Dockerfile 以在每次提交时实际构建系统（最好使用多阶段） - 并确保它是 FROM Deps。

Using this system you will have fast, reconstruct-able builds with a mostly good-enough cache.

使用此系统，您将拥有快速、可重构的构建，并且具有足够好的缓存。

回答by Farzad Vertigo

Using BuildKit

使用 BuildKit

From Docker v18.03onwards you can use BuildKitinstead of volumes that were mentioned in the other answers. It allows mounting caches that can persist between builds and you can avoid downloading contents of the corresponding .m2/repositoryevery time.

从那时Docker v18.03起，您可以使用BuildKit而不是其他答案中提到的卷。它允许挂载可以在构建之间持续存在的缓存，并且您可以避免.m2/repository每次都下载相应的内容。

Assuming that the Dockerfile is in the root of your project:

假设 Dockerfile 位于项目的根目录中：

# syntax = docker/dockerfile:1.0-experimental

FROM maven:3.6.0-jdk-11-slim AS build
COPY . /home/build
RUN mkdir /home/.m2
WORKDIR /home/.m2
USER root
RUN --mount=type=cache,target=/root/.m2 mvn -f /home/build/pom.xml clean compile

target=/root/.m2mounts cache to the specified place in maven image Dockerfile docs.

target=/root/.m2将缓存挂载到 Maven 镜像 Dockerfile docs 中的指定位置。

For building you can run the following command:

对于构建，您可以运行以下命令：

DOCKER_BUILDKIT=1 docker build --rm --no-cache  .   

More info on BuildKit can be found here.

更多关于 BuildKit 的信息可以在这里找到。