running caddy as a reverse proxy with cloudflare dns

Purpose

Over the past year, I’ve been using a droplet on Digital Ocean to expose some docker apps in the cloud. I’ve been using the Nginx Proxy + Let’s Encrypt Companion docker images to reverse proxy all of my apps and this combo has worked well overall. But recently, I’ve gotten that itch. You know, the one where everyone on the internet is saying that “Caddy is so easy to use!” and there are a bunch of tutorials showing tiny config files magically giving rise to beautiful Let’s Encrypt certificates. I guess if it’s so easy, I should switch over, right? Well, yes … sort of. All I can say is that I found ZERO tutorials that actually spelled out exactly what I needed to do and there are some really important details that must be included to get it working. And although I’m not at all new to docker or the linux command line, I still put a few more hours than I wanted to into this “oh-so-simple” solution. The following tutorial is to help anyone out there (AND FUTURE ME) with similar needs.¹

Is it as simple as advertised? I would say “no”, though I think once you get through all of the initial work, it’s trivial to scale. I don’t profess to be an expert, but by the end of this you should be able to get your docker apps up and running behind Caddy with proper TLS certificates.

What is a Reverse Proxy and Why Should I Use One?

For all the details, I’ll refer you to this excellent explanation.

In summary, a reverse proxy is a server that sits in front of application servers and basically directs traffic. In this way, all client access comes through port 443 and then the reverse proxy points each request to the appropriate application. My main reasons for using one are (1) having only one client entry point into my server, (2) simplified certificate management and (3) the ability to use subdomains to reach various applications.

What Would I Want to Host?

Docker (and containerization generally) has really supercharged the ability to self-host some of the many fantastic docker apps out there. For instance,

• Still not using a password manager? Go check out the Bitwarden_RS repo and increase your security on the web with random, complex passwords.
• Want to share some notes with a team (or spouse)? Hedgedoc is probably worth a look.
• How about self-hosting your photos? Lychee makes it really easy to have a pro photo presence controlled by YOU.
• Everything else you can imagine.

Yes, there are plenty of “free” services that would love to harvest your data and sell it to advertisers. But with a little bit of effort, you can now do much of this hosting yourself with a $5 per month cloud instance. Sound good? Read on …

Basic Set-Up

Here’s what I’m using here:

OS: Ubuntu 20.04 running on the smallest/cheapest Digital Ocean droplet (any of the major cloud providers should work)

Docker Engine and Docker-Compose are installed.

DNS: Cloudflare - my Caddy docker image specifically uses a cloudflare module. If you want to use a different DNS provider, you would need to build a different custom Caddy image (many modules are available here). Anyway, to follow this tutorial closely, you will need to have a domain managed by Cloudflare’s free DNS service. In this example, I’ll be using a subdomain: test2.atkinson.cloud. ²

Step 1 - Create an A Record and an API Token on Cloudflare

After logging into Cloudflare, you’ll go to the DNS settings for your site and set a DNS “A” record that points your domain or subdomain to your Digital Ocean droplet’s public IP address:³

Next, we need to create an API token which will allow our container to authenticate with our Cloudflare account and confirm that we indeed control the subdomain for which we want a certificate. Under your account profile, find the ‘API Tokens’ Tab and create a token (not a key) with two different permissions: Zone-Zone-Read and Zone-DNS-Edit. More on that here. Once you’ve completed these steps, you can log out of Cloudflare.

Step 2 - Start an application as a docker container

For this tutorial, I’m going to use the Heimdall image from our friends over at LinuxServer.io. Heimdall is a nice customizable landing page for organizing your applications.

I like to make separate directories for each app in my home directory, then have a docker-compose file for each. I realize that this isn’t the most efficient way to get apps running (it’s not one giant docker-compose file), but it helps me keep everything organized SO BACK OFF!

Here’s my docker-compose.yaml file for heimdall:

---
version: "2.1"
services:
  heimdall:
    image: ghcr.io/linuxserver/heimdall
    container_name: heimdall
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
    volumes:
      - ./config:/config
    ports:
      - 8080:80
     # - 4443:443
    restart: unless-stopped

networks:
  default:
    external:
      name: caddy_net

Go ahead and save the file in your heimdall directory. Before running the container, we need to set-up the docker network (“caddy-net”) that heimdall and any other reverse proxied applications will use. (You can see it at the end of the file we just created). To do this, issue the following command:

docker network create caddy_net

Now we can start Heimdall. Use the command docker-compose up -d within the “heimdall” directory to get your container up and running. Check to make sure it is running properly with a quick docker ps -a

Step 3 - Caddy time

After trying a few different methods, I decided to run Caddy itself as a docker container, since I was already used to running the nginx-proxy and let’s encrypt companion that way. I first tried (and failed … a lot) with the official Caddy docker image, but I couldn’t get the cert process with cloudflare working. Eventually, I found the Caddy Cloudflare DNS Module, which allows you to plug in an API token from your Cloudflare account, thus allowing DNS verification for your certs. Okay, sounds good. Now how the eff do I use this thing, since I have very little experience with Go modules?

The short answer is that you build a custom Caddy image with this module included. In your Caddy directory, create the following Dockerfile:

FROM caddy:2.2.1-builder AS builder

RUN xcaddy build v2.2.1 --with github.com/caddy-dns/cloudflare@latest

FROM caddy:2.2.1

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

I’m not an expert with Dockerfiles, but let’s walk through this one. This is a multistage build that first takes the caddy 2.2.1-builder base image and uses a command-line tool called xcaddy to add the Cloudflare DNS module into the image. In the second stage of the build, the Caddy binary produced in first stage is copied into a fresh caddy 2.2.1 image. This new custom Caddy image is the one we will use for our reverse proxy container.

The next file we’ll need is our docker-compose.yaml. nano docker-compose.yaml

#docker-compose.yaml
---
version: "3.4"
services:
  caddy:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: caddy
    restart: always
    ports:
      - 80:80
      - 443:443
      - 2019:2019
    volumes:
      - ./config:/config
      - ./data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile

networks:
  default:
    external:
      name: caddy_net

You can see that the Caddy container will also be part of the caddy_net network, allowing it to communicate with Heimdall. The volumes listed can be configured to go elsewhere on the host side (the part before the colon) if you prefer.

Finally, we’re going to create what’s called a Caddyfile: nano Caddyfile

#Caddyfile

test2.atkinson.cloud {
	reverse_proxy heimdall:80

	tls {
                dns cloudflare <insert_cloudflare_api_token_here>
        }
}

The structure above proxies my “test2” subdomain to heimdall at its non-TLS container port 80 ⁴. Additionally, it takes a cloudflare API token to confirm control of the subdomain, so don’t forget to substitute yours in. This last part of the Caddyfile (and the code behind it) is the only reason for that whole multistage docker build we’re about to do.

Okay, so our caddy directory should now have three files: Dockerfile, docker-compose.yaml, and Caddyfile.

Fire-up Caddy with docker-compose up and watch the magic. Your custom docker container will be built and will likely take a few minutes to complete. After it’s complete, you should begin to see the log output from Caddy as it negotiates successfully with Cloudflare and downloads a cert from Let’s Encrypt.

4. Check Your Site. Rinse and Repeat.

If you browse to your domain/subdomain at this point, you should see the Heimdall main page …

Nice!

Now that seems like kind of a lot of work to set up, especially for the reverse proxy solution that is touted as “so easy and quick.” However, the real benefit of this system is that once it’s set up, all you have to do add a new docker app with a docker-compose file, and then add a few lines into the existing Caddyfile to have a second, third, fourth, etc application up and running quickly. Would I recommend switching to Caddy if you’re currently using the Nginx-Proxy + Let’s Encrypt Companion containers or some other functional solution? No I wouldn’t because, at the end of the day, both work just fine. However, if you’re just starting out with self-hosting on the cloud, I’d say Caddy is a pretty compelling reverse proxy option.

Troubleshooting / Common Points of Failure

• Make sure your subdomain points to the correct IP address
• Double-check your permissions for your API token
• Check that your docker containers are running properly anytime with a docker ps -a
• Make sure you are using a non-TLS port for your applications that are being proxied … remember, we’re letting Caddy take care of the TLS.
• Check that the cloudflare “proxy status” of your subdomain reads “DNS only”, it will be set to “proxied” by default.
• If your subdomain is getting an LE cert properly but you’re not seeing your application homepage, you probably have a problem with the deployment of that application, check its docker-compose-yaml file.
• Don’t forget about the docker logs <container name or id> command for debugging