Docker + Caddy v2 issues with certificates upon deployment

1. Caddy version (caddy version): v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

Linux rishighan 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

#!/bin/bash
# Rishi Ghan
# Deployment script for a docker-composed project 

# usage: ./deploy.sh -s [service name]
#                    -h [hostname]
#                    -u [username]
#                    -r [repository base url to the raw text content]

# Emojis
CLIPBOARD="πŸ“‹"
CHECKMARK="βœ…"
RENAME="🏷️"
SCISSORS="βœ‚οΈ"
DOWNLOAD="πŸ“₯"
BROOM="🧹"
CONSTRUCTION="πŸ—οΈ"
START="🏁"

# ssh config
cat >> ~/.ssh/config  << EOF
VerifyHostKeyDNS yes
StrictHostKeyChecking no
EOF

# params
service_name=''
hostname=''
username=''

while getopts 'd:s:h:u:r:x:p:b:o:' flag; do
    case "${flag}" in
        s) service_name="${OPTARG}" ;;
        h) hostname="${OPTARG}" ;;
        u) username="${OPTARG}" ;;
        r) repository_base_url="${OPTARG}" ;;
        x) zookeeper_client_user="${OPTARG}" ;;
        p) zookeeper_client_password="${OPTARG}" ;;
        b) mongodb_database="${OPTARG}" ;;
        o) mongodb_root_password="${OPTARG}" ;;
        *) printf "Usage..."
           exit 1 ;;
    esac
done

printf "$CLIPBOARD Attempting to create configuration folder...\n"

ssh "$username@$hostname" /bin/bash << EOF
if [ ! -d "$service_name" ]
then
    printf "\n$CLIPBOARD Directory doesn't exist. Creating now...\n"
    mkdir "$service_name"
    printf "$service_name created."
else
    printf "\n$RENAME  $service_name already exists. Removing and recreating...\n"
    rm -Rf "$service_name"
    mkdir "$service_name"
    printf "$CHECKMARK Done.\n"
fi
    printf "\n$CLIPBOARD Changing directory to $service_name...\n"
    cd "$service_name"

    printf "\n$SCISSORS  Pruning Docker images, networks and volumes...\n\n"
    docker system prune -f

    printf "\n$CONSTRUCTION Creating an external docker network...\n\n"
    docker network create main-network
    
    printf "$DOWNLOAD Downloading the docker-compose configuration for $service_name...\n\n"
    printf "$repository_base_url\n\n"
    curl "$repository_base_url"/docker-compose.yml --output docker-compose.yml
    curl "$repository_base_url"/docker-compose.env --output docker-compose.env
    curl "$repository_base_url"/Caddyfile --output Caddyfile

    printf "Writing Zookeeper configuration to docker-compose.env... \n"
    echo -e "\n" >> docker-compose.env
    echo -e "ZOOKEEPER_CLIENT_USER=$zookeeper_client_user" >> docker-compose.env
    echo -e "ZOOKEEPER_CLIENT_PASSWORD=$zookeeper_client_password" >> docker-compose.env

    printf "Writing Mongo configuration to docker-compose.env... \n"
    echo -e "MONGODB_DATABASE=$mongodb_database" >> docker-compose.env
    echo -e "MONGODB_PASSWORD=$mongodb_root_password" >> docker-compose.env

    printf "\n$BROOM Stopping and removing containers and volumes...\n\n"
    docker-compose down -v
    
    printf "\n$DOWNLOAD Pulling the relevant Docker images...\n\n"
    docker-compose pull

    printf "\n$CONSTRUCTION  Creating containers...\n\n"
    docker-compose up --no-start

    printf "\n$START Starting images...\n\n"
    docker-compose start
EOF

c. Service/unit/compose file:

version: "3.3"

services:
  nats:
    image: nats
    container_name: nats
    networks:
      - main-network
  
  caddy:
    image: caddy
    container_name: caddy
    restart: always
    ports:
      - "443:443"
      - "80:80"
    networks:
      - main-network
    volumes:
      - "./Caddyfile:/etc/caddy/Caddyfile"
      - $PWD/.caddy/data:/data
      - $PWD/.caddy/config:/config

  mongo:
    image: bitnami/mongodb
    container_name: db
    restart: always
    env_file: docker-compose.env
    ports:
      - "27017:27017"
    volumes:
      - /usr/local/var/mongodb/:/bitnami/mongodb
    networks:
      - main-network

  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    env_file: docker-compose.env
    ports:
      - '2181:2181'
    volumes:
      - ../zookeeper/data:/bitnami/zookeeper
    networks:
      - main-network

networks:
  main-network:
    external: true

d. My complete Caddyfile or JSON config:

{
    # Global options block. Entirely optional, https is on by default
    email boo@foo.com 
    acme_ca https://acme.zerossl.com/v2/DV90
    acme_eab {
        key_id <key_id>
        hmac <hmac>
    }
}

rishighan.com {
    reverse_proxy rishighan:8999
}

posts.services.rishighan.com {
    reverse_proxy posts-service-api:5000
}

user.services.rishighan.com {
    reverse_proxy user-service-api:3456
}

assets.services.rishighan.com {
    reverse_proxy assets-service-api:6000
}

analytics.services.rishighan.com {
    reverse_proxy analytics-service-api:9876
}

3. The problem I’m having:

Caddy is configured as a docker image in my docker-compose file. I deploy to Digital Ocean, where a script spins up these images.

The caddy image works perfectly fine, generating certificates for the configured domains only with fresh EAB credentials. If I make changes to my project and re-deploy with the same EAB credentials, I get tls handshake error and not awaiting EAB credentials errors. (See logs below)

I use ZeroSSL as my certificate provider.

4. Error messages and/or full log output:

{"level":"info","ts":1600197993.0440996,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1600197993.0478976,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":1600197993.0482848,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1600197993.0483599,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2020/09/15 19:26:33 [INFO][cache:0xc000494660] Started certificate maintenance routine
{"level":"info","ts":1600197993.048977,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1600197993.0491297,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["posts.services.rishighan.com","user.services.rishighan.com","rishighan.com","analytics.services.rishighan.com","assets.services.rishighan.com"]}
2020/09/15 19:26:33 [INFO][posts.services.rishighan.com] Obtain certificate; acquiring lock...
2020/09/15 19:26:33 [INFO][posts.services.rishighan.com] Obtain: Lock acquired; proceeding...
{"level":"info","ts":1600197993.05156,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1600197993.051571,"msg":"serving initial configuration"}
2020/09/15 19:26:33 [INFO][user.services.rishighan.com] Obtain certificate; acquiring lock...
2020/09/15 19:26:33 [INFO][user.services.rishighan.com] Obtain: Lock acquired; proceeding...
2020/09/15 19:26:33 [INFO][rishighan.com] Obtain certificate; acquiring lock...
2020/09/15 19:26:33 [INFO][rishighan.com] Obtain: Lock acquired; proceeding...
2020/09/15 19:26:33 [INFO][analytics.services.rishighan.com] Obtain certificate; acquiring lock...
2020/09/15 19:26:33 [INFO][analytics.services.rishighan.com] Obtain: Lock acquired; proceeding...
2020/09/15 19:26:33 [INFO][assets.services.rishighan.com] Obtain certificate; acquiring lock...
2020/09/15 19:26:33 [INFO][assets.services.rishighan.com] Obtain: Lock acquired; proceeding...
2020/09/15 19:26:33 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:34 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:26:34 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:35 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:26:35 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:35 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:26:35 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:36 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:26:36 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:36 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:26:36 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:37 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:26:37 [ERROR] attempt 1: [user.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 1m0s (4.119763151s/720h0m0s elapsed)...
2020/09/15 19:26:37 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:37 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:26:37 [ERROR] attempt 1: [posts.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 1m0s (4.619054731s/720h0m0s elapsed)...
2020/09/15 19:26:37 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:38 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:26:38 [ERROR] attempt 1: [rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 1m0s (5.124534482s/720h0m0s elapsed)...
2020/09/15 19:26:38 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:38 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:26:38 [ERROR] attempt 1: [analytics.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 1m0s (5.6264752s/720h0m0s elapsed)...
2020/09/15 19:26:38 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:26:39 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:26:39 [ERROR] attempt 1: [assets.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 1m0s (6.128027819s/720h0m0s elapsed)...
2020/09/15 19:27:37 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:37 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:27:37 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:38 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:27:38 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:38 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:27:38 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:39 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:27:39 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:39 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 1/2)
2020/09/15 19:27:39 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:40 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:27:40 [ERROR] attempt 2: [user.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 2m0s (1m7.444232566s/720h0m0s elapsed)...
2020/09/15 19:27:40 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:41 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:27:41 [ERROR] attempt 2: [posts.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 2m0s (1m7.957014777s/720h0m0s elapsed)...
2020/09/15 19:27:41 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:41 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:27:41 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:41 [ERROR] attempt 2: [rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 2m0s (1m8.453652409s/720h0m0s elapsed)...
2020/09/15 19:27:42 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:27:42 [ERROR] attempt 2: [analytics.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 2m0s (1m8.978335448s/720h0m0s elapsed)...
2020/09/15 19:27:42 [INFO] acme: Registering account for frishi@me.com
2020/09/15 19:27:42 [ERROR] Making new ACME client: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  (attempt 2/2)
2020/09/15 19:27:42 [ERROR] attempt 2: [assets.services.rishighan.com] Obtain: acme: error: 400 :: POST :: https://acme.zerossl.com/v2/DV90/newAccount :: urn:ietf:params:acme:error:malformed :: [External Account Binding] The account is not awaiting external account binding, url:  - retrying in 2m0s (1m9.464931251s/720h0m0s elapsed)...
2020/09/15 19:28:07 http: TLS handshake error from 17.149.234.204:59779: no certificate available for 'rishighan.com'

5. What I already tried:

I have mapped caddy config and data folders in an effort to persist the certificates, but upon checking the contents of the folder inside the docker container, I found that both the config and data folders are empty.

The only solution that works for me is generating fresh EAB credentials in the ZeroSSL dashboard and re-deploying.

6. Links to relevant resources:

Proper support for ZeroSSL will come in Caddy v2.2, which should be released in the next week or so.

1 Like

Which version of Caddy are you using? β€œv2” isn’t specific enough.

My apologies, it is v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=