Digital Ocean Rolling Deploy of Docker containers

digital-ocean, docker, go

Introduction

More and more companies are moving to containerized deployments and it is very common to do rolling deploys of a new image across multiple machines piece by piece to avoid downtime. Generally, I see devs using bash scripts for their build and deploy steps but I have been increasingly moving towards replacing bash with go as much as possible. This post is to show what a go script looks like to deploy a new docker image across different servers.

Prerequisites

To complete this tutorial, you will need:

  • Two or more servers with docker installed
  • A Load balancer
  • Basic knowledge of Golang

Show me the code

The entire script is available as a snippet on Gitlab here.

The steps are as follows:

Step 1 — Create a DigitalOcean Access Token

To use the DigitalOcean SDK and container registry you will first need to create an access token. You can create one in the API section found in the left navigation bar in the Token/Keys tab - docs here

Step 2 — Create a DigitalOcean container registry

You will need a container registry to hold all your application images. DigitalOcean provides a container registry option backed by Spaces. Product page here

Step 3 — Get all droplets by tag to deploy to

The first thing we need is to get a list of droplets to deploy to. Assuming the droplets are tagged with nginx-service we can get a list of droplets using DropletService ListByTag(context.Context, string, *ListOptions) ([]Droplet, *Response, error) method. Ref: Line 143 of the script

Step 4 — Get the load balancer

Now we will fetch the load balancer that hosts the droplets from step 4. We need the loadbalancer so that we can remove droplets from it so they don’t get any traffic while the containers are being swapped - this avoids any user disruptions. Unfortunately, there is no list by tag method on the load balancer service - so we have to List all loadbalancers and use the first one that has the tag. Ref Line 174 of the script

Step 5 — Drain the server from the load balancer

To avoid deploying while client requests are in flight you will drain the droplet from the load balancer. This step is simple as we can simple untag the droplet that leads to it getting picked up by the loadbalancer by calling the UntagResources(context.Context, string, *UntagResourcesRequest) (*Response, error) method. Ref Line 61 of the script

Step 6 — Connect the docker client to the remote server

Now we need to create a docker client that connects to the docker daemon running on the droplet that was drained. Docker runs on port 2375 so make sure that the machine that this script runs on can access it on the remote droplet. Ref Line 211

Step 7 — Stop the current running container

Before you can deploy the new image - you will need to stop and remove the current version of the container on the server. First we will get the ID of the container with the service tag nginx-service Line 225 and then we can call the ContainerStop(ctx, containerID, timeout) method to kill the container Line 91

Step 8 — Pull new version of authenticated image

Now we need to pull the new version of the image from DigitalOcean’s container registry. We use the ImagePull method in the Docker SDK with DigitalOcean’s access token as both the username and password for auth. Ref Line 242. This makes sure that the new image is available on the remote server to be run.

Step 9 — Start new container

We can now start the container using the image pulled. We need to create a container config that maps the host port to the container port and tag the container. Ref Line 261. Creating the container config is separate from actually running the container. The config then gets passed to the ContainerStart method ref Line 119

Step 10 — Add droplet back to load balancer

With the image updated the droplet can be safely added back to the load balancer and it can serve client requests using the new version of the image. This is as simple as retagging the droplet so that the loadbalancer picks it up again. Ref Line 125

Conclusion

This post shows how to do a rolling deploy on droplets but is more so to show that Golang is an excellent language to replace adhoc bash scripts. It is also extremely portable because of easy cross-compilation and statically linked binaries. I encourage you - next time you are planning on using bash to script something - ask yourself can I use Go instead?