在 docker 中运行 cron python 作业

Running cron python jobs within docker

提问by Christopher Rapcewicz

I would like to run a python cron job inside of a docker container in detached mode. My set-up is below:

我想以分离模式在 docker 容器内运行 python cron 作业。我的设置如下：

My python script is test.py

我的 python 脚本是 test.py

  #!/usr/bin/env python
  import datetime
  print "Cron job has run at %s" %datetime.datetime.now()

My cron file is my-crontab

我的 cron 文件是 my-crontab

* * * * * /test.py > /dev/console

and my Dockerfile is

我的 Dockerfile 是

FROM ubuntu:latest
RUN apt-get update && apt-get install -y software-properties-common python-software-properties && apt-get update

RUN apt-get install -y python cron
ADD my-crontab /
ADD test.py /
RUN chmod a+x test.py

RUN crontab /my-crontab
ENTRYPOINT cron -f

What are the potential problems with this approach? Are there other approaches and what are their pros and cons?

这种方法有哪些潜在问题？是否有其他方法，它们的优缺点是什么？

采纳答案by Christopher Rapcewicz

Several issues that I faced while trying to get a cron job running in a docker container were:

我在尝试在 docker 容器中运行 cron 作业时遇到的几个问题是：

1. time in the docker container is in UTC not local time;
2. the docker environment is not passed to cron;
3. as Thomas noted, cron logging leaves a lot to be desired and accessing it through docker requires a docker-based solution.

1. docker 容器中的时间是 UTC 而不是本地时间；
2. docker 环境不会传递给 cron；
3. 正如 Thomas 所指出的，cron 日志记录还有很多不足之处，通过 docker 访问它需要一个基于 docker 的解决方案。

There are cron-specific issues and are docker-specific issues in the list, but in any case they have to be addressed to get cron working.

列表中有特定于 cron 的问题和特定于 docker 的问题，但无论如何都必须解决它们才能使 cron 工作。

To that end, my current working solution to the problem posed in the question is as follows:

为此，我目前对问题中提出的问题的工作解决方案如下：

Create a docker volume to which all scripts running under cron will write:

创建一个 docker 卷，所有在 cron 下运行的脚本都将写入该卷：

# Dockerfile for test-logs

# BUILD-USING:        docker build -t test-logs .
# RUN-USING:          docker run  -d -v /t-logs --name t-logs test-logs
# INSPECT-USING:      docker run -t -i  --volumes-from t-logs ubuntu:latest /bin/bash

FROM stackbrew/busybox:latest

# Create logs volume
VOLUME /var/log

CMD  ["true"]

The script that will run under cron is test.py:

将在 cron 下运行的脚本是test.py：

#!/usr/bin/env python

# python script which needs an environment variable and runs as a cron job
import datetime
import os

test_environ = os.environ["TEST_ENV"]
print "Cron job has run at %s with environment variable '%s'" %(datetime.datetime.now(), test_environ)

In order to pass the environment variable to the script that I want to run under cron, follow Thomas' suggestion and put a crontab fragment for each script (or group of scripts) that has need of a docker environment variable in /etc/cron.dwith a placeholder XXXXXXXwhich must be set.

为了将环境变量传递给我想在 cron 下运行的脚本，请遵循 Thomas 的建议，并为需要 docker 环境变量的每个脚本（或脚本组）放置一个 crontab 片段，其中必须/etc/cron.d包含一个占位符XXXXXXX被设置。

# placed in /etc/cron.d 
# TEST_ENV is an docker environment variable that the script test.py need

TEST_ENV=XXXXXXX
#
* * * * * root python /test.py >> /var/log/test.log

Instead of calling cron directly, wrap cron in a python script that does does things: 1. reads the environment variable from the docker environment variable and sets the environment variable in a crontab fragment.

不是直接调用 cron，而是将 cron 包装在一个执行以下操作的 python 脚本中： 1. 从 docker 环境变量中读取环境变量并在 crontab 片段中设置环境变量。

#!/usr/bin/env python

# run-cron.py
# sets environment variable crontab fragments and runs cron

import os
from subprocess import call
import fileinput

# read docker environment variables and set them in the appropriate crontab fragment
environment_variable = os.environ["TEST_ENV"]

for line in fileinput.input("/etc/cron.d/cron-python",inplace=1):
    print line.replace("XXXXXXX", environment_variable)

args = ["cron","-f", "-L 15"]
call(args)

The Dockerfilethat for the container in which the cron jobs run is as follows:

该Dockerfile说的容器，其中cron作业运行如下：

# BUILD-USING:        docker build -t test-cron .
# RUN-USING docker run --detach=true --volumes-from t-logs --name t-cron test-cron

FROM debian:wheezy
#
# Set correct environment variables.
ENV HOME /root
ENV TEST_ENV test-value

RUN apt-get update && apt-get install -y software-properties-common python-software-properties && apt-get update

# Install Python Setuptools
RUN apt-get install -y python cron

RUN apt-get purge -y python-software-properties software-properties-common && apt-get clean -y && apt-get autoclean -y && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

ADD cron-python /etc/cron.d/
ADD test.py /
ADD run-cron.py /

RUN chmod a+x test.py run-cron.py

# Set the time zone to the local time zone
RUN echo "America/New_York" > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata

CMD ["/run-cron.py"]

Finally, create the containers and run them:

最后，创建容器并运行它们：

1. Create the log volume (test-logs) container: docker build -t test-logs .
2. Run log volume: docker run -d -v /t-logs --name t-logs test-logs
3. Create the cron container: docker build -t test-cron .
4. Run the cron container: docker run --detach=true --volumes-from t-logs --name t-cron test-cron
5. To inspect the log files of the scripts running under cron: docker run -t -i --volumes-from t-logs ubuntu:latest /bin/bash. The log files are in /var/log.

1. 创建日志卷（test-logs）容器： docker build -t test-logs .
2. 运行日志卷： docker run -d -v /t-logs --name t-logs test-logs
3. 创建 cron 容器： docker build -t test-cron .
4. 运行 cron 容器： docker run --detach=true --volumes-from t-logs --name t-cron test-cron
5. 要检查在cron下运行脚本的日志文件：docker run -t -i --volumes-from t-logs ubuntu:latest /bin/bash。日志文件位于/var/log.

回答by Thomas Orozco

Adding crontab fragments in /etc/cron.d/instead of using root's crontabmight be preferable.

添加 crontab 片段/etc/cron.d/而不是使用 rootcrontab可能更可取。

This would:

这个会：

• Let you add additional cron jobs by adding them to that folder.
• Save you a few layers.
• Emulate how Debian distros do it for their own packages.

• 让您通过将它们添加到该文件夹​​来添加其他 cron 作业。
• 为您节省几层。
• 模拟 Debian 发行版如何为他们自己的软件包做这件事。

Observe that the format of those files is a bit different from a crontab entry. Here's a sample from the Debian php package:

请注意，这些文件的格式与 crontab 条目略有不同。这是来自 Debian php 包的示例：

# /etc/cron.d/php5: crontab fragment for php5
#  This purges session files older than X, where X is defined in seconds
#  as the largest value of session.gc_maxlifetime from all your php.ini
#  files, or 24 minutes if not defined.  See /usr/lib/php5/maxlifetime

# Look for and purge old sessions every 30 minutes
09,39 *     * * *     root   [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)

Overall, from experience, running cron in a container does work very well (besides cron logging leaving a lot to be desired).

总的来说，根据经验，在容器中运行 cron 效果很好（除了 cron 日志记录还有很多不足之处）。

回答by Wernight

Single Container Method

单一容器方法

You may run crondwithin the same container that is doing something closely relatedusing a base image that handles PID 0 well, like phusion/baseimage.

您可以crond在使用处理 PID 0 的基本映像（例如phusion/baseimage）执行密切相关的同一容器中运行。

Specialized Container Method

专用容器法

May be cleaner would be to have another Container linked to it that just runs crond. For example:

将另一个 Container 链接到刚刚运行的容器可能会更干净crond。例如：

Dockerfile

Dockerfile

 FROM busybox
 ADD crontab /var/spool/cron/crontabs/www-data
 CMD crond -f

crontab

crontab

 * * * * * echo $USER

Then run:

然后运行：

 $ docker build -t cron .
 $ docker run --rm --link something cron

Note: In this case it'll run the job as www-data. Cannot just mount the crontabfile as volume because it needs to be owned by rootwith only write access for root, else crondwill run nothing. Also you'll have to run crondas root.

注意：在这种情况下，它将作为www-data. 不能只是将crontab文件挂载为卷，因为它需要由root只有写访问权限的拥有root，否则crond将不运行。您还可以运行crond的root。

回答by Alban Mouton

Here is a complement on rosksw answer.

这是对 rossksw 答案的补充。

There is no need to do some string replacement in the crontab file in order to pass environment variables to the cron jobs.

无需在 crontab 文件中进行一些字符串替换即可将环境变量传递给 cron 作业。

It is simpler to store the environment variables in a file when running the contrainer, then load them from this file at each cron execution. I found the tip here.

在运行 contrainer 时将环境变量存储在文件中更简单，然后在每次 cron 执行时从该文件加载它们。我在这里找到了小费。

In the dockerfile:

在 dockerfile 中：

CMD mkdir -p /data/log && env > /root/env.txt && crond -n

In the crontab file:

在 crontab 文件中：

* * * * * root env - `cat /root/env.txt` my-script.sh

回答by Anton I. Sipos

Another possibility is to use Crython. Crython allows you to regularly schedule a python function from within a single python script / process. It even understands cron syntax:

另一种可能性是使用Crython。Crython 允许您从单个 python 脚本/进程中定期调度 python 函数。它甚至理解 cron 语法：

@crython.job(expr='0 0 0 * * 0 *')
def job():
    print "Hello world"

Using crython avoids the various headaches of running crond inside a docker container - your job is now a single process that wakes up when it needs to, which fits better into the docker execution model. But it has the downside of putting the scheduling inside your program, which isn't always desirable. Still, it might be handy in some use cases.

使用 crython 避免了在 docker 容器内运行 crond 的各种头痛 - 您的工作现在是一个在需要时唤醒的单个进程，这更适合 docker 执行模型。但它的缺点是将调度放在您的程序中，这并不总是可取的。不过，在某些用例中它可能很方便。

回答by evtuhovdo

Here's an alternative solution.

这是一个替代解决方案。

in Dockerfile

在 Dockerfile

ADD docker/cron/my-cron /etc/cron.d/my-cron
RUN chmod 0644 /etc/cron.d/my-cron

ADD docker/cron/entrypoint.sh /etc/entrypoint.sh

ENTRYPOINT ["/bin/sh", "/etc/entrypoint.sh"]

in entrypoint.sh

在 entrypoint.sh

 #!/usr/bin/env bash
  printenv | cat - /etc/cron.d/my-cron > ~/my-cron.tmp \
    && mv ~/my-cron.tmp /etc/cron.d/my-cron

cron -f

回答by dogik

We are using below solution. It supports both docker logsfunctionality and ability to hang the cron process in the container on PID 1 (if you use tail -fworkarounds provided above - if cron crashes, docker will not follow restart policy):

我们正在使用以下解决方案。它支持docker logs在 PID 1 上的容器中挂起 cron 进程的功能和能力（如果您使用tail -f上面提供的解决方法 - 如果 cron 崩溃，docker 将不会遵循重启策略）：

cron.sh:

cron.sh:

#!/usr/bin/env bash

printenv | cat - /etc/cron.d/cron-jobs > ~/crontab.tmp \
    && mv ~/crontab.tmp /etc/cron.d/cron-jobs

chmod 644 /etc/cron.d/cron-jobs

tail -f /var/log/cron.log &

cron -f

Dockerfile:

Dockerfile:

RUN apt-get install --no-install-recommends -y -q cron 

ADD cron.sh /usr/bin/cron.sh
RUN chmod +x /usr/bin/cron.sh

ADD ./crontab /etc/cron.d/cron-jobs
RUN chmod 0644 /etc/cron.d/cron-jobs

RUN touch /var/log/cron.log

ENTRYPOINT ["/bin/sh", "/usr/bin/cron.sh"]

crontab:

crontab:

* * * * * root <cmd> >> /var/log/cron.log 2>&1

And please don't forget to add the creepy new line in your crontab

并且请不要忘记在您的 crontab 中添加令人毛骨悚然的新行

回答by OPSXCQ

Don't mix crond and your base image. Prefer to use a native solution for your language (schedule or crython as said by Anton), or decouple it. By decoupling it I mean, keep things separated, so you don't have to maintain an image just to be the fusion between python and crond.

不要混合使用 crond 和您的基本图像。更喜欢为您的语言使用本机解决方案（安东所说的时间表或 crython），或将其解耦。通过解耦，我的意思是，将事物分开，这样您就不必为了成为 python 和 crond 之间的融合而维护图像。

You can use Tasker, a task runner that has cron (a scheduler) support, to solve it, if you want keep things decoupled.

如果您想让事情解耦，您可以使用Tasker，一个具有 cron（调度程序）支持的任务运行器来解决它。

Here an docker-compose.ymlfile, that will run some tasks for you

这是一个docker-compose.yml文件，它将为您运行一些任务

version: "2"

services:
    tasker:
        image: strm/tasker
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
        environment:
            configuration: |
                logging:
                    level:
                        ROOT: WARN
                        org.springframework.web: WARN
                        sh.strm: DEBUG
                schedule:
                    - every: minute
                      task: helloFromPython
                tasks:
                    docker:
                        - name: helloFromPython
                          image: python:3-slim
                          script:
                              - python -c 'print("Hello world from python")'

Just run docker-compose up, and see it working. Here is the Tasker repo with the full documentation:

只需运行docker-compose up，看看它的工作。这是带有完整文档的 Tasker 存储库：

http://github.com/opsxcq/tasker

http://github.com/opsxcq/tasker

回答by Dim

Here is my checklist for debugging cron python scripts in docker:

这是我在 docker 中调试 cron python 脚本的清单：

1. Make sure you run croncommand somewhere. Cron doesn't start automatically. You can run it from a Dockerfile using RUNor CMDor add it to a startup script for the container. In case you use CMDyou may consider using cron -fflag which keeps cron in the foreground and won't let container die. However, I prefer using tail -fon logfiles.
2. Store environment variables in /etc/envoronment. Run this from a bash startscript: printenv > /etc/environment. This is an absolute must if you use environment variables inside of python scripts. Cron doesn't know anything about the environment variables by default. By it can read them from /etc/environment.
3. Test Cron by using the following config:

1. 确保在cron某处运行命令。Cron 不会自动启动。您可以从Dockerfile使用运行RUN或CMD或将其添加到启动脚本的容器。如果您使用，CMD您可以考虑使用cron -f将 cron 保持在前台并且不会让容器死亡的标志。但是，我更喜欢tail -f在日志文件上使用。
2. 将环境变量存储在 /etc/envoronment.conf 中。从一个bash startscript运行此：printenv > /etc/environment。如果您在 python 脚本中使用环境变量，这是绝对必须的。默认情况下，Cron 对环境变量一无所知。通过它可以从/etc/environment.
3. 使用以下配置测试 Cron：

* * * * * echo "Cron works" >>/home/code/test.log
* * * * * bash -c "/usr/local/bin/python3 /home/code/test.py >>/home/code/test.log 2>/home/code/test.log"

The python test file should contain some printstatements or something else that displays that the script is running. 2>/home/code/test.logwill also log errors. Otherwise, you won't see errors at all and will continue guessing.

python 测试文件应包含一些print语句或其他显示脚本正在运行的内容。2>/home/code/test.log也会记录错误。否则，您根本看不到错误并继续猜测。

Once done, go to the container, using docker exec -it <container_name> bashand check:

完成后，转到容器，使用docker exec -it <container_name> bash并检查：

1. That crontab config is in place using crontab -l
2. Monitor logs using tail -f /home/code/test.log

1. 该 crontab 配置使用 crontab -l
2. 使用监控日志 tail -f /home/code/test.log

I have spent hours and days figuring out all of those problems. I hope this helps someone to avoid this.

我花了数小时和数天来解决所有这些问题。我希望这有助于某人避免这种情况。