Caddy through Docker- Issue with Cross Container Communciation and Subdomains

[Bubbling9862 (nate)]

1. Output of caddy version:

v2.6.2

2. How I run Caddy:

I’ve got a homelab computer setup with proxmox. It has a LXC which runs docker which has caddy.

a. System environment:

My LXC is Ubuntu 20.04. The LXC is setup with docker. I’m running caddy via the following command. Note 8405 and 8404 are port forwarded via my router.

b. Command:

Currently using:

docker run -d -p 8405:80 -p 8404:443 -p 8404:443/udp \
    -v /files/docker/caddy/caddy_data:/data \
    -v /files/docker/caddy/site:/srv \
    -v /files/docker/caddy/caddy_config:/config \
    -v /files/docker/caddy/Caddyfile:/etc/caddy/Caddyfile \
    --network=caddy-net \
    --name caddy caddy

c. Service/unit/compose file:

see above

d. My complete Caddy config:

[

bubbling9862.com {
    root * /srv
    file_server browse
}

home.bubbling9862.com {
    reverse_proxy 192.168.1.64:8123
}

vault.bubbling9862.com {
    encode gzip
    
    # The negotiation endpoint is also proxied to Rocket (normally 80), now 1119
    reverse_proxy /notifications/hub/negotiate vaultwarden:1119
    
    # Notifications redirected to the websockets server (normally 3012), now 1120
    reverse_proxy /notifications/hub vaultwarden:1120
    
    # Send all other traffic to the regular Vaultwarden endpoint (normally 80), now 1119
    reverse_proxy vaultwarden:1119
}

3. The problem I’m having:

I’m trying to use caddy in a few ways.

First, I might use it as a simple file server, at least for test purposes. This is currently working. I can get to a html file i placed in my root directory when i go to my domain.

Second, I’m trying to use them with other docker images on my local machine. (vaultwarden). These just cant go through. The subdomain isn’t working.

Third, i’m trying to use it as a reverse_proxy for home assistant (and possibly other things) on another server. This is the one on 196.168.1.64:8123. This doesn’t seem to work. The problem could also be on the homeassistant side (although I think i do have that configured correctly). Since the problem could be on the other end i’m debugging #2 right now but am copying this here in case there are other issues.

4. Error messages and/or full log output:

CURL MAIN (static site, works)

curl -v https://bubbling9862.com
*   Trying 71.244.130.208:443...
* Connected to bubbling9862.com (71.244.130.208) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET / HTTP/1.1
> Host: bubbling9862.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Alt-Svc: h3=":443"; ma=2592000
< Content-Length: 6
< Content-Type: text/html; charset=utf-8
< Etag: "ro6ql66"
< Last-Modified: Sun, 08 Jan 2023 21:05:30 GMT
< Server: Caddy
< Date: Tue, 10 Jan 2023 01:57:55 GMT
<
hello
* Connection #0 to host bubbling9862.com left intact

CURL VAULT

curl -v https://vault.bubbling9862.com
*   Trying 172.67.147.60:443...
* Connected to vault.bubbling9862.com (172.67.147.60) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET / HTTP/1.1
> Host: vault.bubbling9862.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Tue, 10 Jan 2023 01:59:36 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://vault.bubbling9862.com/
< CF-Cache-Status: DYNAMIC
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=K2mvA%2FmvgaUsePkae%2ByRl4U7q2IAFP8zo0w6yPdadBNItzGeQE9eBs5MZ1g6%2F2r%2BTGYB%2BZOjTuRWEME%2BVzZcVFBUvUVpiXyMjtK%2FHJCF86kmoG9Ag3Nwr9QXnPxHXNT%2BYozlXLE%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 7871d0731d3f2363-EWR
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
* Connection #0 to host vault.bubbling9862.com left intact

CURL HOME

curl -v https://home.bubbling9862.com
*   Trying 172.67.147.60:443...
* Connected to home.bubbling9862.com (172.67.147.60) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET / HTTP/1.1
> Host: home.bubbling9862.com
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Tue, 10 Jan 2023 02:00:33 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://home.bubbling9862.com/
< CF-Cache-Status: DYNAMIC
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Jp3t2TyNLlBPwC0gDV6tIwilcTguXwHn6z9sKF84%2BtxDV6IK5bgPMSk%2BNcShzX2eQHJnw3RdI%2Fg6y%2BtzvlW3Qn%2Bmxxm%2Ff1z3Aej5pmIqgdSH7MQPl16HXH1vq%2FBtn9r%2BOMlFug%3D%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 7871d1dbcfa5e73c-EWR
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
* Connection #0 to host home.bubbling9862.com left intact

DEBUG LOG

DBG ts=1673316376.703753 logger=http msg=servers shutting down with eternal grace period
INF ts=1673316376.7038429 logger=tls.cache.maintenance msg=stopped background certificate maintenance cache=0xc0004a1500
INF ts=1673316376.703871 logger=admin msg=stopped previous server address=localhost:2019
INF ts=1673316376.7038753 msg=shutdown complete signal=SIGTERM exit_code=0
INF ts=1673316378.9486134 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile
WRN ts=1673316378.9503522 msg=Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies adapter=caddyfile file=/etc/caddy/Caddyfile line=2
INF ts=1673316378.950927 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]
INF ts=1673316378.9511452 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443
INF ts=1673316378.9512944 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0
INF ts=1673316378.9512708 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc00052b5e0
INF ts=1673316378.9519446 logger=tls msg=cleaning storage unit description=FileStorage:/data/caddy
INF ts=1673316378.9519496 logger=http msg=enabling HTTP/3 listener addr=:443
INF ts=1673316378.9519956 msg=failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
DBG ts=1673316378.9520278 logger=http msg=starting server loop address=[::]:443 tls=true http3=true
INF ts=1673316378.9520352 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]
DBG ts=1673316378.9520516 logger=http msg=starting server loop address=[::]:80 tls=false http3=false
INF ts=1673316378.9520545 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]
INF ts=1673316378.9520566 logger=http msg=enabling automatic TLS certificate management domains=["vault.bubbling9862.com","home.bubbling9862.com","bubbling9862.com"]
DBG ts=1673316378.9522417 logger=tls msg=loading managed certificate domain=vault.bubbling9862.com expiration=1681007255 issuer_key=acme-v02.api.letsencrypt.org-directory storage=FileStorage:/data/caddy
DBG ts=1673316378.9523907 logger=tls.cache msg=added certificate to cache subjects=["vault.bubbling9862.com"] expiration=1681007255 managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=f431003b213b20190554b6b11bddc201f4e5c06d27cd3079696048e966ab076b cache_size=1 cache_capacity=10000
DBG ts=1673316378.9524062 logger=events msg=event name=cached_managed_cert id=903dc72d-9343-4e87-ba5f-5636cc886079 origin=tls data={"sans":["vault.bubbling9862.com"]}
INF ts=1673316378.9524443 logger=tls msg=finished cleaning storage units
DBG ts=1673316378.9526658 logger=tls msg=loading managed certificate domain=home.bubbling9862.com expiration=1681007255 issuer_key=acme-v02.api.letsencrypt.org-directory storage=FileStorage:/data/caddy
DBG ts=1673316378.9529362 logger=tls.cache msg=added certificate to cache subjects=["home.bubbling9862.com"] expiration=1681007255 managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=805e4bebfd1dfc56a4dd383003ddee364930da49ae2642e8afbfc3390954931f cache_size=2 cache_capacity=10000
DBG ts=1673316378.9530241 logger=events msg=event name=cached_managed_cert id=dad286c9-aff5-4fd5-b6e1-0bccc8d1e59c origin=tls data={"sans":["home.bubbling9862.com"]}

DBG ts=1673316378.9577696 logger=tls msg=loading managed certificate domain=bubbling9862.com expiration=1680984178 issuer_key=acme-v02.api.letsencrypt.org-directory storage=FileStorage:/data/caddy

DBG ts=1673316378.958051 logger=tls.cache msg=added certificate to cache subjects=["bubbling9862.com"] expiration=1680984178 managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=0a2bb2e6c7154f0d2398e04ad2e32b3953cfea73b7642c3e72e827b4e54928f8 cache_size=3 cache_capacity=10000
DBG ts=1673316378.9582024 logger=events msg=event name=cached_managed_cert id=046e3eac-b153-4889-95dc-3265b6d3dbc9 origin=tls data={"sans":["bubbling9862.com"]}
INF ts=1673316378.9584095 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json
INF ts=1673316378.9584892 msg=serving initial configuration

5. What I already tried:

I think the issue might be on the docker networking .

I read about docker networking so have tried various settings for #2 for example…
vaultwarden:1119 (my docker image name, my docker external port)*
vaultwarden:80 (my docker image name, my docker internal port)
172.17.0.4:1119 (the IP address shown in portainer, docker external port)
172.17.0.4:80 (the ip address shown in portainer, docker external port)
192.168.1.142:1119 (host comptuer IP address, docker external port)

Update: Now that the containers are connected to ‘caddy-net’ i also tried pointing to the internal IP for that container. Vaultwarden says its 172.18.0.3 on caddy-net so i’ve tried putting 172.18.0.3:80 (docker internal port) and 172.18.0.3:1119 (docker external port), neither works.

Note that if i goto 192.168.1.142:1119 i get the vaultwarden webpage, so i know that is running.

6. Links to relevant resources:

[francislavoie]

@Bubbling9862

Second, I’m trying to use them with other docker images on my local machine. (vaultwarden). These just cant go through. The subdomain isn’t working.

Containers need to be in the same Docker network to be seen by eachother.

You should either use a docker-compose.yml file to manage all your services (easiest option), or create a network with the docker network command and add all your containers to that network.

@Bubbling9862

Note 8405 and 8404 are port forwarded via my router.

Just to clarify, your router is routing incoming connections on ports 80/443 to 8405/8404 on your server?

It’s important that your server is publicly accessible on ports 80/443 for ACME challenges to successfully complete.

@Bubbling9862

4. Error messages and/or full log output:

The web browser just doesn’t resolve. I tried to do the curl commands that you specify here but they just seem to resolve correctly. Cloudflare looks like its working. I have the proxies setup on cloudflare for these subdomains. When I turn off caddy I get a cloudflare error page so caddy is doing something.

The debug logs show that the HTML is being served but don’t say anything for the subdomain sections.

Share the actual logs. We can’t really do much with anecdotes, it’s too open to ambiguity.

Make requests with curl -v to show what’s happening with a client, and show your Caddy container logs.

[Bubbling9862 (nate)]

Okay

1. I’ve added a new network with docker network “caddy-net” and added my containers to that network.

2. Correct, 80 & 443 externally on my router forward to 8405/8404 on my intranet. The ACME challenge seems to complete correctly, as my static site is correctly served as HTTPS.

3. I’ve appended the actual logs and curl output to my original post.

Thank you!

[francislavoie]

@Bubbling9862

I’ve appended the actual logs and curl output to my original post.

So it looks like your apex domain is working fine.

Your subdomain is going through CloudFlare, but it looks like it’s serving infinite redirects. Are you sure you have the correct CloudFlare setting? If CloudFlare is proxying over HTTP to Caddy, then Caddy will respond with an HTTP->HTTPS redirect. Just a guess for what’s happening.

Your Caddy logs only includes the startup sequence, and nothing after startup. It should show some reverse_proxy logs if Caddy gets that far. Did you copy your logs before making requests, or is nothing getting logged when you make requests? If nothing is logged, then you definitely have a problem with CloudFlare not sending requests to your server correctly.

[Bubbling9862 (nate)]

Ah. I think that solved it.

I had my cloudflare settings setup with “proxy status: reverse proxy” because I was trying to do a reverse proxy… but I guess that means its THEM trying to do the reverse proxy.

When I turned the setting for my subdomain to “DNS Only” the requests magically started coming through with caddy, and then allowed caddy to do the main work.

Thanks for the (very speedy) help!