Docker Swarm: no certificate available for 'samvanderkris.xyz'

1. Caddy version (caddy:latest (currently 9ce811ab3540)):

2. How I run Caddy:

I’m running Caddy inside of a single server Docker Swarm to add TLS to a ghost blog I’m hosting. Config files are posted below

stack.yml

version: '3.1'

services:

  proxy:
    image: caddy:2-alpine
    restart: always
    ports:
      - 443:443
      - 80:80
    volumes:
      - ./proxy/Caddyfile:/etc/caddy/Caddyfile

  ghost:
    image: ghost:3-alpine
    restart: always
    environment:
      # see https://docs.ghost.org/docs/config#section-running-ghost-with-config-env-variables
      database__client: mysql
      database__connection__host: db
      database__connection__user: root
      database__connection__password: ${MYSQL_PASS}
      database__connection__database: ghost
      url: ${URL}

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_PASS}

Caddyfile

samvanderkris.xyz {
        reverse_proxy {
                to ghost:2368
        }
}

3. The problem I’m having:

Trying to visit the website in my browser results in an SSL_ERROR_INTERNAL_ERROR_ALERT error.

4. Error messages and/or full log output:

ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4519227,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4709291,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4726567,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4727864,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.474206,"logger":"tls","msg":"cleaned up storage units"}
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:48 [INFO][cache:0xc00003b380] Started certificate maintenance routine
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4794965,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["samvanderkris.xyz"]}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.4874046,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
ghost_proxy.1.x83wckek2wsx@arcadia    | {"level":"info","ts":1594938888.48752,"msg":"serving initial configuration"}
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:48 [INFO][samvanderkris.xyz] Obtain certificate; acquiring lock...
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:48 [INFO][samvanderkris.xyz] Obtain: Lock acquired; proceeding...
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:49 [INFO] [samvanderkris.xyz] acme: Obtaining bundled SAN certificate given a CSR
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:49 [INFO][samvanderkris.xyz] Waiting on rate limiter...
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:49 [INFO][samvanderkris.xyz] Done waiting
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:50 [INFO] [samvanderkris.xyz] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5925104005
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:50 [INFO] [samvanderkris.xyz] acme: Could not find solver for: tls-alpn-01
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:50 [INFO] [samvanderkris.xyz] acme: use http-01 solver
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:50 [INFO] [samvanderkris.xyz] acme: Trying to solve HTTP-01
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:54 http: TLS handshake error from 10.0.0.2:45198: no certificate available for 'samvanderkris.xyz'
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:34:56 http: TLS handshake error from 10.0.0.2:45204: no certificate available for 'samvanderkris.xyz'
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:35:01 http: TLS handshake error from 10.0.0.2:45206: no certificate available for 'samvanderkris.xyz'
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:35:47 http: TLS handshake error from 10.0.0.2:38186: no certificate available for 'git.samvanderkris.xyz'
ghost_proxy.1.x83wckek2wsx@arcadia    | 2020/07/16 22:35:49 http: TLS handshake error from 10.0.0.2:36846: no certificate available for 'git.samvanderkris.xyz'

5. What I already tried:

I tried changing the host in my Caddyfile to some other stuff like proxy, localhost and my server’s external IPv4 address (thought maybe Docker Swarm’s networking messed something up). I assume the no certificate available for 'samvanderkris.xyz' error suggests that something went wrong getting a certificate from Let’s Encrypt, but I don’t really know what else to try here.

I’ve looked around online and found some people with very similar problems, but couldn’t actually find a solution. Any help would be very much appreciated!

Yeah, it looks like the HTTP challenge hasn’t completed yet. (Likely, Let’s encrypt isn’t able to reach your Caddy instance from the outside.) So there is no certificate available yet.

When running Caddy in docker, please don’t forget to persist /data with a volume on the Caddy image. That’s where the certificates and keys are stored by default. Without that persisted, you can easily lose the certs and keys and you may hit Let’s Encrypt rate limits.

Oh yeah, I’ll add a volume for /data. But any ideas on why Let’s Encrypt wouldn’t be able to reach my Caddy instance?

It could be any number of reasons. Is your DNS properly configured to point to your server? Are you using a service like Cloudflare which might introduce a layer in between? If you can provide some more specific info about your setup, it might help.

DNS seems to work normally (I SSH’ed into the server using my domain). And I do have Cloudflare, but it’s set to DNS only, so it shouldn’t add anything inbetween. I can’t really think of anything else that could cause problems.

What about firewall rules? Did you make sure both port 80 and 443 are open?

Nope, it doesn’t even have ufw installed at the moment. And I don’t know a lot about Docker’s networking, but it does look like port 443 and 80 are open.

root@arcadia:/home/sam/ghost# ss -tulpn
Netid                 State                  Recv-Q                 Send-Q                                                             Local Address:Port                                  Peer Address:Port                                                                      
udp                   UNCONN                 0                      0                                                                        0.0.0.0:4789                                       0.0.0.0:*                                                                         
udp                   UNCONN                 0                      0                                                                        0.0.0.0:68                                         0.0.0.0:*                     users:(("dhclient",pid=330,fd=7))                   
udp                   UNCONN                 0                      0                                                                     172.18.0.1:123                                        0.0.0.0:*                     users:(("ntpd",pid=13186,fd=26))                    
udp                   UNCONN                 0                      0                                                                    10.19.90.95:123                                        0.0.0.0:*                     users:(("ntpd",pid=13186,fd=19))                    
udp                   UNCONN                 0                      0                                                                      127.0.0.1:123                                        0.0.0.0:*                     users:(("ntpd",pid=13186,fd=18))                    
udp                   UNCONN                 0                      0                                                                        0.0.0.0:123                                        0.0.0.0:*                     users:(("ntpd",pid=13186,fd=17))                    
udp                   UNCONN                 0                      0                                                                              *:7946                                             *:*                     users:(("dockerd",pid=19108,fd=29))                 
udp                   UNCONN                 0                      0                                        [fe80::500d:e1ff:fe61:2733]%veth3180150:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=29))                    
udp                   UNCONN                 0                      0                                         [fe80::cd4:fbff:fe72:b1a1]%veth45fa1e1:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=31))                    
udp                   UNCONN                 0                      0                                        [fe80::2cd2:89ff:fedf:5b40]%veth49443d7:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=30))                    
udp                   UNCONN                 0                      0                                        [fe80::e805:c0ff:fe46:7bb1]%vethbabbdb6:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=28))                    
udp                   UNCONN                 0                      0                                      [fe80::42:20ff:feb2:2c7b]%docker_gwbridge:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=27))                    
udp                   UNCONN                 0                      0                                               [fe80::dc2e:4cff:fe2b:e030]%ens2:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=22))                    
udp                   UNCONN                 0                      0                                                         [2001:bc8:1830:d2f::1]:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=21))                    
udp                   UNCONN                 0                      0                                                                          [::1]:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=20))                    
udp                   UNCONN                 0                      0                                                                           [::]:123                                           [::]:*                     users:(("ntpd",pid=13186,fd=16))                    
tcp                   LISTEN                 0                      128                                                                      0.0.0.0:22                                         0.0.0.0:*                     users:(("sshd",pid=11424,fd=3))                     
tcp                   LISTEN                 0                      20                                                                     127.0.0.1:25                                         0.0.0.0:*                     users:(("exim4",pid=11223,fd=3))                    
tcp                   LISTEN                 4                      128                                                                            *:443                                              *:*                     users:(("dockerd",pid=19108,fd=59))                 
tcp                   LISTEN                 0                      128                                                                            *:2377                                             *:*                     users:(("dockerd",pid=19108,fd=22))                 
tcp                   LISTEN                 0                      128                                                                            *:7946                                             *:*                     users:(("dockerd",pid=19108,fd=28))                 
tcp                   LISTEN                 4                      128                                                                            *:80                                               *:*                     users:(("dockerd",pid=19108,fd=58))                 
tcp                   LISTEN                 0                      128                                                                         [::]:22                                            [::]:*                     users:(("sshd",pid=11424,fd=4))                     
tcp                   LISTEN                 0                      20                                                                         [::1]:25                                            [::]:*                     users:(("exim4",pid=11223,fd=4))

There’s some more stuff in the Caddy log now by the way (but I think it’s still the same problem?)

root@arcadia:/home/sam/ghost# docker service logs ghost_proxy
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.8833032,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9112601,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:21 [INFO][cache:0xc00003aea0] Started certificate maintenance routine
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.963988,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9645126,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9660468,"logger":"tls","msg":"cleaned up storage units"}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9674098,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["samvanderkris.xyz"]}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9682388,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | {"level":"info","ts":1595001141.9913828,"msg":"serving initial configuration"}
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:22 [INFO][samvanderkris.xyz] Obtain certificate; acquiring lock...
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:22 [INFO][samvanderkris.xyz] Obtain: Lock acquired; proceeding...
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:22 [INFO] [samvanderkris.xyz] acme: Obtaining bundled SAN certificate given a CSR
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:22 [INFO][samvanderkris.xyz] Waiting on rate limiter...
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:22 [INFO][samvanderkris.xyz] Done waiting
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:23 [INFO] [samvanderkris.xyz] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939599930
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:23 [INFO] [samvanderkris.xyz] acme: use tls-alpn-01 solver
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:23 [INFO] [samvanderkris.xyz] acme: Trying to solve TLS-ALPN-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:24 http: TLS handshake error from 127.0.0.1:55042: EOF
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:33 http: TLS handshake error from 10.0.0.2:14882: no certificate available for 'git.samvanderkris.xyz'
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:35 http: TLS handshake error from 10.0.0.2:56098: no certificate available for 'git.samvanderkris.xyz'
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:37 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939599930
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:37 [ERROR] error: one or more domains had a problem:
ghost_proxy.1.ijxdkk8li7lg@arcadia    | [samvanderkris.xyz] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during read (your server may be slow or overloaded), url: 
ghost_proxy.1.ijxdkk8li7lg@arcadia    |  (challenge=tls-alpn-01 remaining=[http-01])
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:37 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939599930
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:39 [INFO] [samvanderkris.xyz] acme: Obtaining bundled SAN certificate given a CSR
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:40 [INFO] [samvanderkris.xyz] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939603352
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:40 [INFO] [samvanderkris.xyz] acme: Could not find solver for: tls-alpn-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:40 [INFO] [samvanderkris.xyz] acme: use http-01 solver
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:52:40 [INFO] [samvanderkris.xyz] acme: Trying to solve HTTP-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:54:21 http: TLS handshake error from 10.0.0.2:31896: no certificate available for 'git.samvanderkris.xyz'
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:54:40 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939603352
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:54:40 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5939603352
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:54:40 [ERROR] error: one or more domains had a problem:
ghost_proxy.1.ijxdkk8li7lg@arcadia    | [samvanderkris.xyz] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://samvanderkris.xyz/.well-known/acme-challenge/cvcwk6IKf3zZJLnBD__vo93LgM5ls4xQ73X_B6im7P0: Timeout after connect (your server may be slow or overloaded), url: 
ghost_proxy.1.ijxdkk8li7lg@arcadia    |  (challenge=http-01 remaining=[])
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:54:42 [ERROR] attempt 1: [samvanderkris.xyz] Obtain: [samvanderkris.xyz] error: one or more domains had a problem:
ghost_proxy.1.ijxdkk8li7lg@arcadia    | [samvanderkris.xyz] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://samvanderkris.xyz/.well-known/acme-challenge/cvcwk6IKf3zZJLnBD__vo93LgM5ls4xQ73X_B6im7P0: Timeout after connect (your server may be slow or overloaded), url: 
ghost_proxy.1.ijxdkk8li7lg@arcadia    |  - retrying in 1m0s (2m20.555692942s/720h0m0s elapsed)...
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:55:43 [INFO] [samvanderkris.xyz] acme: Obtaining bundled SAN certificate given a CSR
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:55:43 [INFO] [samvanderkris.xyz] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78644068
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:55:43 [INFO] [samvanderkris.xyz] acme: use tls-alpn-01 solver
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:55:43 [INFO] [samvanderkris.xyz] acme: Trying to solve TLS-ALPN-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:55:43 http: TLS handshake error from 127.0.0.1:55080: EOF
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:00 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78644068
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:00 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78644068
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:00 [ERROR] error: one or more domains had a problem:
ghost_proxy.1.ijxdkk8li7lg@arcadia    | [samvanderkris.xyz] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during read (your server may be slow or overloaded), url: 
ghost_proxy.1.ijxdkk8li7lg@arcadia    |  (challenge=tls-alpn-01 remaining=[http-01])
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:02 [INFO] [samvanderkris.xyz] acme: Obtaining bundled SAN certificate given a CSR
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:03 [INFO] [samvanderkris.xyz] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/78644140
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:03 [INFO] [samvanderkris.xyz] acme: Could not find solver for: tls-alpn-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:03 [INFO] [samvanderkris.xyz] acme: use http-01 solver
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:03 [INFO] [samvanderkris.xyz] acme: Trying to solve HTTP-01
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:56:10 http: TLS handshake error from 10.0.0.2:2616: no certificate available for 'git.samvanderkris.xyz'
ghost_proxy.1.ijxdkk8li7lg@arcadia    | 2020/07/17 15:57:59 http: TLS handshake error from 10.0.0.2:46970: no certificate available for 'git.samvanderkris.xyz'

This topic was automatically closed after 30 days. New replies are no longer allowed.