A question that has always bothered me was how I should deal with security updates on applications that are running in docker containers. Security updates are often done automatically in Linux distros and that for a good reason. This is easily done via the package managers and there are many services and apps to automate package updates for you. But as docker containers are isolated from the core system and they are based on images, you need to come up with a better solution for these. With Watchtower, you can schedule container updates and minimize the downtime during the update.
Update docker container automatically with Watchtower
Watchtower is an application that will monitor all running docker containers. Once it discovers a change in the image it will pull down the new version automatically and restart the container with the new image. It is an open-source project you can find on GitHub and I’m using this in some installations. It offers you some configuration to schedule the update sequence and includes or excludes specific containers on the server. You can also add a notification that will be able to send you an email. Watchtower itself is also running in a docker container and configured by changing environment variables or command arguments.
Deploy Watchtower via Docker CLI
Watchtower can be easily deployed by executing a simple docker run command.
docker run --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower
You might wonder why there is no log output apart from the welcome message. If you want to increase the logging level or watchtower, you simply just add an argument.
docker run --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug
This will automatically run once every day, but you can also do a quick test run with the following command. The “run-once” argument will only run the container once and then immediately exits.
docker run --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once --debug
Note, if you specify a version tag in the docker images, watchtower will not update the container to a newer version. For example, if you run nginx:1.18, watchtower will not upgrade it to nginx:1.19. If you want to stick a specific version of an application you can simply just specify the version in the tag. For all other containers, I would recommend using the “latest” tag, which is the default by the way.
Include or Exclude docker containers from the update
But sometimes, you want to have the latest version of an image, but you still want to do updates manually for some containers. Or you may want to schedule the update to a specific date or time because you want to avoid downtime. You can include or exclude specific containers from the update procedure with labels. This means you don’t attach the label on the watchtower container, but you can attach the exclude or monitor-only label to other containers.
Let’s run an Nginx server and exclude it from our update procedure.
docker run -d --label=com.centurylinklabs.watchtower.enable= false nginx
If we now run our watchtower container, you can see it doesn’t reveal any running containers, because we have excluded the Nginx container.
Scheduled Updates and clean up old images
You can also schedule the update procedure for a specific time. This uses the argument “–schedule” with a crontab argument. For example, if you want to run the update procedure every day at 4 am, use the following command.
docker run --name watchtower -v /var/run/docker.sock:/var/run/docker.sock --restart unless-stopped containrrr/watchtower --schedule "0 0 4 * * *" --debug
I would also recommend adding two more arguments to the command. The first one “—cleanup” will delete all old docker images. Once Watchtower has updated a container the old image will be still on the server. With the cleanup command, you can get rid of these orphaned images. The second one “–rolling-restart” will update one image at a time instead of stopping and starting all at once. This is very useful if you want to minimize the downtime.
Of course, there are also other arguments and improvements, you can find in the official documentation.