Caddy fails to reverse_proxy to HomeAssistant

1. The problem I’m having:

I cannot access my HomeAssistant page externally by using my domain name (x.gmx.no). Other services like Plex, Immich, Joplin etc works just fine, they are hosted on same domain name, behind the same firewall.
HomeAssistant is also running on a separate VM, but on the same proxmox, and the same network segment. I can access HomeAssistant by using NAT and exposing the 8123 port to internet. This all worked nicely with Caddy just a few months ago, Im not sure what has changed.

2. Error messages and/or full log output:

Caddy log:
2025-01-13T08:59:06.107507167Z INF ts=1736758746.107299 msg=trying to solve challenge identifier=x.gmx.no challenge_type=http-01 ca=https://acme-v02.api.letsencrypt.org/directory

2025-01-13T08:59:16.358874694Z ERR ts=1736758756.3586977 msg=challenge failed identifier=x.gmx.no challenge_type=http-01 problem={"type":"urn:ietf:params:acme:error:connection","title":"","detail":"88.92.119.1: Fetching http://x.gmx.no/.well-known/acme-challenge/LV6jJnhS46Dt9cInBhFfPN51e5VBMRC9RNjZuXvkwl0: Timeout during connect (likely firewall problem)","instance":"","subproblems":null} stacktrace=github.com/mholt/acmez/v3.(*Client).pollAuthorization
	github.com/mholt/acmez/v3@v3.0.0/client.go:557
github.com/mholt/acmez/v3.(*Client).solveChallenges
	github.com/mholt/acmez/v3@v3.0.0/client.go:378
github.com/mholt/acmez/v3.(*Client).ObtainCertificate
	github.com/mholt/acmez/v3@v3.0.0/client.go:136
github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:477
github.com/caddyserver/certmagic.(*ACMEIssuer).Issue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:371
github.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue
	github.com/caddyserver/caddy/v2@v2.9.1/modules/caddytls/acmeissuer.go:249
github.com/caddyserver/certmagic.(*Config).obtainCert.func2
	github.com/caddyserver/certmagic@v0.21.6/config.go:626
github.com/caddyserver/certmagic.doWithRetry
	github.com/caddyserver/certmagic@v0.21.6/async.go:104
github.com/caddyserver/certmagic.(*Config).obtainCert
	github.com/caddyserver/certmagic@v0.21.6/config.go:700
github.com/caddyserver/certmagic.(*Config).ObtainCertAsync
	github.com/caddyserver/certmagic@v0.21.6/config.go:505
github.com/caddyserver/certmagic.(*Config).manageOne.func1
	github.com/caddyserver/certmagic@v0.21.6/config.go:415
github.com/caddyserver/certmagic.(*jobManager).worker
	github.com/caddyserver/certmagic@v0.21.6/async.go:73

2025-01-13T08:59:16.358911668Z ERR ts=1736758756.3587453 msg=validating authorization identifier=x.gmx.no problem={"type":"urn:ietf:params:acme:error:connection","title":"","detail":"88.92.119.1: Fetching http://x.gmx.no/.well-known/acme-challenge/LV6jJnhS46Dt9cInBhFfPN51e5VBMRC9RNjZuXvkwl0: Timeout during connect (likely firewall problem)","instance":"","subproblems":null} order=https://acme-v02.api.letsencrypt.org/acme/order/1919698096/343879972615 attempt=2 max_attempts=3 stacktrace=github.com/mholt/acmez/v3.(*Client).ObtainCertificate
	github.com/mholt/acmez/v3@v3.0.0/client.go:152
github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:477
github.com/caddyserver/certmagic.(*ACMEIssuer).Issue
	github.com/caddyserver/certmagic@v0.21.6/acmeissuer.go:371
github.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue
	github.com/caddyserver/caddy/v2@v2.9.1/modules/caddytls/acmeissuer.go:249
github.com/caddyserver/certmagic.(*Config).obtainCert.func2
	github.com/caddyserver/certmagic@v0.21.6/config.go:626
github.com/caddyserver/certmagic.doWithRetry
	github.com/caddyserver/certmagic@v0.21.6/async.go:104
github.com/caddyserver/certmagic.(*Config).obtainCert
	github.com/caddyserver/certmagic@v0.21.6/config.go:700
github.com/caddyserver/certmagic.(*Config).ObtainCertAsync
	github.com/caddyserver/certmagic@v0.21.6/config.go:505
github.com/caddyserver/certmagic.(*Config).manageOne.func1
	github.com/caddyserver/certmagic@v0.21.6/config.go:415
github.com/caddyserver/certmagic.(*jobManager).worker
	github.com/caddyserver/certmagic@v0.21.6/async.go:73

2025-01-13T08:59:16.358917348Z ERR ts=1736758756.3587656 logger=tls.obtain msg=could not get certificate from issuer identifier=x.gmx.no issuer=acme-v02.api.letsencrypt.org-directory error=HTTP 400 urn:ietf:params:acme:error:connection - 88.92.119.1: Fetching http://x.gmx.no/.well-known/acme-challenge/LV6jJnhS46Dt9cInBhFfPN51e5VBMRC9RNjZuXvkwl0: Timeout during connect (likely firewall problem)

2025-01-13T08:59:16.358942077Z INF ts=1736758756.3588967 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=["x.gmx.no"] ca=https://acme.zerossl.com/v2/DV90 account=espen@gmx.no

2025-01-13T08:59:16.358948008Z INF ts=1736758756.3589046 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=["x.gmx.no"] ca=https://acme.zerossl.com/v2/DV90 account=espen@gmx.no

2025-01-13T08:59:16.358949653Z INF ts=1736758756.3589098 logger=tls.issuance.acme msg=using ACME account account_id=https://acme.zerossl.com/v2/DV90/account/LOCD8dnnPJUqvmp0_A3Z7w account_contact=["mailto:espen@gmx.no"]

2025-01-13T08:59:19.265809788Z INF ts=1736758759.2655487 msg=trying to solve challenge identifier=x.gmx.no challenge_type=http-01 ca=https://acme.zerossl.com/v2/DV90

curl to the acme-challenge:

curl -vL https://x.gmx.no/.well-known/acme-challenge/LV6jJnhS46Dt9cInBhFfPN51e5VBMRC9RNjZuXvkwl0
* Host x.gmx.no:443 was resolved.
* IPv6: (none)
* IPv4: 88.92.119.1
*   Trying 88.92.119.1:443...
* Connected to x.gmx.no (88.92.119.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error

3. Caddy version:

Caddy version:
$ docker exec caddy caddy version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

4. How I installed and ran Caddy:

Its installed in portainer. Searched for “Caddy” in the templates section, and clicked “Deploy container”

a. System environment:

Docker running on Ubuntu VM, with proxmox as hypervisor.

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.

# Unless the file starts with a global options block, the first  
# uncommented line is always the address of your site.

# To use your own domain name (with automatic HTTPS), first make  
# sure your domain's A/AAAA DNS records are properly pointed to  
# this machine's public IP, then replace ":80" below with your  
# domain name.  
{  
        email redacted@123.com  
}

start.gmx.no {  
        reverse_proxy 10.0.10.31:3000  
}  
plex.gmx.no {  
        # We don't need to set the X-Forwarded-For and X-Forwarded-Proto headers  
        # Caddy does this automatically.  
        reverse_proxy 10.0.10.15:32400  
}

prox.gmx.no {  
        reverse_proxy 10.0.10.30:8006  
}

portainer.gmx.no {  
        reverse_proxy 10.0.10.31:9443  
}

x.gmx.no {  
        reverse_proxy 10.0.10.12:8123  
}

img.gmx.no {  
        reverse_proxy 10.0.10.31:2283  
}

joplin.gmx.no {  
        reverse_proxy 10.0.10.31:22300  
}

:80 {  
        # Set this path to your site's directory.  
        root * /usr/share/caddy

        # Enable the static file server.  
        file_server

}

5. Links to relevant resources:

Hi @Morbus,

To me it seems like not able to get a certificate issued from Let’s Encrypt from part of the log here shown.

I find the “Timeout during connect (likely firewall problem)” very often the issue for that “hint”.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.