fshr
The musings of a grumpy hairless ape
Docker Cron Job
A quick run through of setting up a Docker container which runs tasks via Cron.
Why would I want to do this?
Generally where you have a need to run a task inside a Docker container on a repeated, scheduled basis rather than having a continually running process (containers usually work on the basis that they run a single process which starts when the container starts, and stops when the container stops).
For example, my use case is the following…
I have a small Python script which connects to an external web service (Openweather in this case), pulls some data, and then writes that data to an InfluxDB database. I want this script to run on a scheduled basis on each 1/4 hour (hh:00, hh:15, hh:30, hh:45), and preferably don’t want to have to bake the scheduling into the Python script. I also want to run this script inside a Docker container to keep the host as clean as possible (e.g. to avoid having to set up a Python vEnv on the host running the script).
How do I do this?
As the solution is Docker based, I’m assuming use of a ‘dockerfile’ for building a new Docker image, and a Docker compose file to start the container, but feel free to swap out for whatever build/run solution you prefer :-)
Filesystem Layout
This is the basic layout of the build files. Everything under “src” relates to the Python script and can be replaced with whatever needs to be run on a scheduled basis (with appropriate changes to the crontab & dockerfile)
.
├── build.sh
├── crontab
├── docker-compose.yaml
├── dockerfile
└── src
├── py_weather.py
└── requirements.txt
Cron File - crontab
This is the cron file which is going to get copied into the container at build time.
# min hour day month weekday command
0,15,30,45 * * * * /bin/date --rfc-2822 >/proc/1/fd/1 2>/proc/1/fd/2
0,15,30,45 * * * * python3 /app/py_weather.py >/proc/1/fd/1 2>/proc/1/fd/2
Both entries are set to run on each 1/4 hour, the first to output the current date and time, and the second is the actual command to run the Python script.
The “>/proc/1/fd/1 2>/proc/1/fd/2” at the end of each line is to redirect the STDOUT and STDERR from the command to the container console. This makes sure that any log output from Cron goes into the container logs.
Docker File - dockerfile
FROM python:alpine
WORKDIR /app
ENV PYTHONUNBUFFERED=1
COPY src/requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt --no-cache-dir
COPY src/ /app
COPY crontab /etc/crontabs/root
CMD ["crond", "-f"]
Going line by line:
| Command | Description |
|---|---|
| FROM python:alpine | Use the Alpine version of the official Python images |
| WORKDIR /app | Set the working directory to ‘/app’ in the image |
| ENV PYTHONUNBUFFERED=1 | Used to make sure Python output gets properly written to logs |
| COPY src/requirements.txt /app/requirements.txt | Copy the Python requirements file to ‘/app’ |
| RUN pip install -r requirements.txt –no-cache-dir | Install our Python pre-reqs in the image |
| COPY src/ /app | Copy the script files to ‘/app’ |
| COPY crontab /etc/crontabs/root | Copy the crontab to the image |
| CMD [“crond”, “-f”] | Set the command to run on starting the container (run crond in the foreground [-f]) |
Note that depending on what your script is, you’ll probably want to use a different image and dockerfile, however I’d suggest sticking with an Alpine based image as they natively include Cron in the image.
Build Script - build.sh
Just a basic shell script to ease building the new Docker image
docker build . -t pfish/py-weather
Docker Compose File - docker-compose.yaml
A basic Docker compose file to start an instance of our custom image with our script.
version: "3.9" # optional since v1.27.0
services:
py-weather:
image: pfish/py-weather
container_name: py-weather
restart: always # unless-stopped
Building and running the container
Once you’ve got the above structure in place (or your equivalent), first run the build script. This will pull down the appropriate base images and start the build:
psf@host:~/py-weather$ ./build.sh
Sending build context to Docker daemon 10.75kB
Step 1/8 : FROM python:alpine
alpine: Pulling from library/python
ca7dd9ec2225: Already exists
9e124a36b9ab: Pull complete
86456952aa28: Pull complete
4ece9ef7a579: Pull complete
ef27b8598222: Pull complete
Digest: sha256:08b03b140633664ed4a55630de38f847d19059318c2473a5bff592d8a0b051d5
Status: Downloaded newer image for python:alpine
---> 91c0c14478fa
Step 2/8 : WORKDIR /app
---> Running in b12d30d8d507
Removing intermediate container b12d30d8d507
---> 57dc3914f423
Step 3/8 : ENV PYTHONUNBUFFERED=1
---> Running in 05492199b198
Removing intermediate container 05492199b198
---> a37227ece74a
Step 4/8 : COPY src/requirements.txt /app/requirements.txt
---> c12b793f49e8
Step 5/8 : RUN pip install -r requirements.txt --no-cache-dir
---> Running in 675fbd91c217
Collecting influxdb
Downloading influxdb-5.3.1-py2.py3-none-any.whl (77 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 4.8 MB/s eta 0:00:00
Collecting python-dateutil>=2.6.0
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 27.7 MB/s eta 0:00:00
Collecting pytz
Downloading pytz-2022.6-py2.py3-none-any.whl (498 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 498.1/498.1 kB 26.9 MB/s eta 0:00:00
Collecting requests>=2.17.0
Downloading requests-2.28.1-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 31.7 MB/s eta 0:00:00
Collecting six>=1.10.0
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting msgpack
Downloading msgpack-1.0.4.tar.gz (128 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.1/128.1 kB 30.3 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting charset-normalizer<3,>=2
Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 19.7 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 30.5 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 27.8 MB/s eta 0:00:00
Building wheels for collected packages: msgpack
Building wheel for msgpack (pyproject.toml): started
Building wheel for msgpack (pyproject.toml): finished with status 'done'
Created wheel for msgpack: filename=msgpack-1.0.4-cp311-cp311-linux_x86_64.whl size=15530 sha256=ddba13846ac4fb92ef21e1365e2af57bfcd681db7b540b6225f254d5620a87ed
Stored in directory: /tmp/pip-ephem-wheel-cache-6_pkeswd/wheels/be/38/62/bffc8d68ee5e3a6a3080b2f8a520e8302fe333528d93a488af
Successfully built msgpack
Installing collected packages: pytz, msgpack, urllib3, six, idna, charset-normalizer, certifi, requests, python-dateutil, influxdb
Successfully installed certifi-2022.9.24 charset-normalizer-2.1.1 idna-3.4 influxdb-5.3.1 msgpack-1.0.4 python-dateutil-2.8.2 pytz-2022.6 requests-2.28.1 six-1.16.0 urllib3-1.26.13
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 675fbd91c217
---> a2575b953c18
Step 6/8 : COPY src/ /app
---> 8e5736fe534a
Step 7/8 : COPY crontab /etc/crontabs/root
---> 8617a0a4738c
Step 8/8 : CMD ["crond", "-f"]
---> Running in 13e48cd5d624
Removing intermediate container 13e48cd5d624
---> 2151dd0f99b3
Successfully built 2151dd0f99b3
Successfully tagged pfish/py-weather:latest
psf@host:~/py-weather$
Once the build completes, run a Docker compose to bring the container up. For the first run I’d suggest running the container in the foreground (“docker compose up”).
(You can stop a container running in the foreground with “Ctrl-C”)
psf@host:~/py-weather$ docker compose up
[+] Running 2/2
⠿ Network py-weather_python Created 0.3s
⠿ Container py-weather Created 0.1s
Attaching to py-weather
py-weather | Hello!
py-weather | Hello!
py-weather | Hello!
py-weather | Tue, 29 Nov 2022 10:00:00 +0000
py-weather | Hello!
py-weather | Starting run...
py-weather | Querying - http://api.openweathermap.org/data/2.5/weather?appid=<snip>&lat=<snip>&lon=<snip>&units=metric
^CGracefully stopping... (press Ctrl+C again to force)
[+] Running 1/1
⠿ Container py-weather Stopped 10.6s
canceled
psf@host:~/py-weather$
Once you’re happy that the container is running as planned, you can start the container in the background (“docker compose up -d”)
psf@host:~/py-weather$ docker compose up -d
[+] Running 1/1
⠿ Container py-weather Started 1.7s
psf@host:~/py-weather$
You can stop a container running in the background with “docker compose down” in the same directory as your docker-compose.yaml file
psf@host:~/py-weather$ docker compose down
[+] Running 2/2
⠿ Container py-weather Removed 10.6s
⠿ Network py-weather_python Removed 0.4s
psf@host:~/py-weather$
Finally, don’t forget to clean up any excess build files if you’ve been doing a lot of build and test, particularly if you’ve been making changes and/or using different container versions (as you can accumulate a lot of unused images). The quick and dirty way to do this is to run a “docker system prune” which will go through and remove any unused containers, networks, images, and build cache:
psf@host:~/py-weather$ docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:2151dd0f99b3a217ed05b3f490e059c8aef28505518d5c302dd7644b5cb61493
deleted: sha256:8617a0a4738c854c07a809815242c6d13412d8dc21dcce8e8474a5531a40e813
deleted: sha256:2c0dccaed633d05a4bddd2086ccf4fe129453100cce9c1a54cc58e46b1872992
deleted: sha256:8e5736fe534a6c55699d81356c1136331d8cb7e5346f78a41891ac9a5bc3d746
deleted: sha256:eb8ae6549ecfccad62b35cafd82eb75ab17773fafc6d9ac0fe37eb9363accccc
deleted: sha256:a2575b953c18bc9216ab4ac44ce4d2860a160f362981ba142df2339a34a46041
deleted: sha256:6ea49e9c61e9d8eb2c4b16803dd9f55e760f7ac0e662d5df10cedbcb45e0505a
deleted: sha256:c12b793f49e881867bbad24106fef1297f9db647affd5b40616e03e76fc51f39
deleted: sha256:796af62c2375a5fc02ccbd95bf5c9d9c99a628e80ebdd6b1b5b1aaca5f4550f2
Deleted build cache objects:
1jmzbf67oy6nyz4tlgffxtgqt
1ulkem78gwttrskgl4ggybif5
kydsjb9g9yf7yos2a3r80pyoe
nhqwb4gctw8fvyf20shx51nod
ja9yumw36umzsaksxgg8b5vfn
syn335f5agmnrw1msimkoc3m0
x1tr7sn594u44uyjoxp8z8ww1
xbqz5zcxx9s9keelzs828cugq
gmn4lc9eoxnuso9rjse6hvo8z
ceiqlfqnc981bj00dn8zdl4a6
mzm3gs9k5c32v0oq234zhmjpb
Total reclaimed space: 1.676GB
psf@host:~/py-weather$
Note the warnings that this will remove all stopped containers and unused networks!
© Paul Fisher 2025 / CC BY-NC-SA 4.0
/ Disclaimer