Cannot get certificates from either provider, off the shelf IPv6 VPS

1. The problem I’m having:

I am trying to create an IPv6 onlyblog server (AWS Lightsail VPS). I have a really basic Caddy config and an empty Ghost blog behind it.
Logs show failure to get certs issued for both ZeroSSL and LE as below.

2. Error messages and/or full log output:

caddy-1  | {"level":"debug","ts":1707784869.2619658,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
caddy-1  | {"level":"warn","ts":1707784869.761995,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
caddy-1  | {"level":"error","ts":1707784869.7620826,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.example.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"registering account [mailto:domainblaster@pm.me] with server: provisioning client: performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
caddy-1  | {"level":"debug","ts":1707784869.7621198,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
caddy-1  | {"level":"warn","ts":1707784899.7632802,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
caddy-1  | {"level":"error","ts":1707784899.7633023,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.example.com","issuer":"acme.zerossl.com-v2-DV90","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": dial tcp 52.205.231.143:443: i/o timeout"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Docker compose pull

a. System environment:

Ubuntu 22.04 in AWS Lightsail. Docker and compose pulled fresh, new server build.
Added HTTPS to the cloud firewall from “all IPv6” origins which has SSH and HTTP there by default.
When the server came up, installed Docker and compose and apt-upgraded everything.

b. Command:

Compose startup → docker compose up -d

c. Service/unit/compose file:

version: "3.7"

networks:
        web:
                external: true
        internal:
                external: false
                driver: bridge

services:
        caddy:
                image: caddy:2-alpine
                restart: unless-stopped
                ports:
                        - "80:80"
                        - "443:443"
                        - "443:443/udp"
                volumes:
                        - /data/caddy/Caddyfile:/etc/caddy/Caddyfile
                        - /data/caddy/data:/data 
                        - /data/caddy/config:/config 
                networks:
                        - web
                        - internal
        ghost:
                image: ghost:4-alpine
                restart: unless-stopped
                  #ports:
                  #- 2368
                environment:
                  url: https://example.com # Change to your domain
                volumes:
                        - /data/myghostapp:/var/lib/ghost/content
                networks:
                        - internal

d. My complete Caddy config:

{
    email foo@example.com
debug
}
example.com {
    reverse_proxy ghost:2368
}
www.example.com {
    redir https://example.com{uri}
}

5. Links to relevant resources:

I have set up Caddy containers many times on IPv4 servers and not had a problem like this.
I did try an off the shelf Nginx container on 80 (without Caddy) which resolved fine from a browser.

Interesting :thinking: I can reach your server from one of my IPv6 compatible servers

# curl -v v5demo.engdex.com
*   Trying 2a05:d014:c0:6b00:bd25:a7ad:2183:c82d:80...
* TCP_NODELAY set
* Connected to v5demo.engdex.com (2a05:d014:c0:6b00:bd25:a7ad:2183:c82d) port 80 (#0)
> GET / HTTP/1.1
> Host: v5demo.engdex.com
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://v5demo.engdex.com/
< Server: Caddy
< Date: Tue, 13 Feb 2024 08:40:55 GMT
< Content-Length: 0

I’m not sure why Let’s Encrypt nor ZeroSSL are reaching your server. It should work fine.

Oh, wait. Is your server not able to perform outgoing IPv4 connections? That might be the problem. Hmm. I’ve never considered that this could be a problem.

I’ll ask around for help.

Can you try curl -v -6 https://acme-v02.api.letsencrypt.org/directory from your server? Does it connect okay?

Hi, yes it would have trouble going to outbound IPv4 addresses. I just tried curling out to an IPv4-only general web site and it failed.

But yes that curl -6 worked ok.

If it was calling the API and trying IPv4 before 6 for some reason that seems likely it would cause a timeout.

From the Go docs:

	// FallbackDelay specifies the length of time to wait before
	// spawning a RFC 6555 Fast Fallback connection. That is, this
	// is the amount of time to wait for IPv6 to succeed before
	// assuming that IPv6 is misconfigured and falling back to
	// IPv4.
	//
	// If zero, a default delay of 300ms is used.
	// A negative value disables Fast Fallback support.
	FallbackDelay time.Duration

My understanding is Go automatically tries both IPv4 and IPv6 as per RFC 6555: Happy Eyeballs: Success with Dual-Stack Hosts, so it should work :grimacing: I don’t know what’s going on here.

That sort of implies it should try IPv6 first, though the debug log by itself doesn’t show which. That 300ms timeout is probably just session establishment as opposed to a completed API response, so establishing a session wouldn’t rule out an API timeout. :thinking:
I’ll think of a few other things to try. A bit worrying as it’s all fresh install stuff with a really basic config. Thanks…

I found the cause of this, incredibly IPv6 is not enabled in Docker by default. See Enable IPv6 support | Docker Docs
Is there a way to edit the previous text to redact info so this can be left for others?

Ah, I had no idea it was disabled by default. Surprising.

Yes, you can edit your post with the :pencil2: button.

Thanks - though I only have an edit pencil on the most recent answer - I don’t have one there for the older stuff. I would just like to change the email address and domains to example.com kind of thing, if possible.

You should be able to click the ... then the pencil.

The three dots doesn’t give me a pencil on older posts, just as per image.
image

Weird. Well, I edited your post.

1 Like

It must be a timeout, i.e. can only edit recent posts before they become solidified. All looks good - IPv6 problem solved - thanks very much for the assistance!

1 Like