Help using caddy for only services on internal network

Help
1

1. Output of caddy version:

v2.6.2

2. How I run Caddy:

I run caddy to reverse proxy self hosted services to users inside my local network only. I run home assistant, immich, nextcloud, pihole, unifi controller, plex, etc. I do not use caddy for anything exposed to the internet - I use cloudflare tunnels to handle anything exposed. I simply want to use caddy so that internal users can reach the services locally with https rather than the direct http service. There a few practical reasons for this:

  • home assistant has a few features with casting that only work if the interfaces are encrypted
  • esphome flash from pc only works is esphome service is encrypted
  • I want immich mobile app to reach internal server directly when on wifi so I run a custom dns to redirect https://photos.bearcave.com back to the service on the server, and I need caddy to provide https

I feel better secured if my public services are encrypted and proxied by cloudflare, so my home ip is never exposed, and it provides protection from a number of intrusion vectors.

a. System environment:

Raspberry pi 4b running ubuntu server 22.04. All services running on docker compose including caddy.
(I have some additional services running on another machine, also Ubuntu server 22.04, docker compose, same version and build of caddy, running immich and a few other services the exact same way.)

b. Command:

cd ~/caddy && docker compose up -d

c. Service/unit/compose file:

docker-compose.yaml

services:
  caddy:
    image: locally_built_caddy
    build: .
    ports:
      - 80:80
      - 443:443
    volumes:
      - caddy:/data
      - /home/josh/caddy/Caddyfile:/etc/caddy/Caddyfile

volumes:
  caddy:

Dockerfile

FROM caddy:2.6.2-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:2.6.2

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

d. My complete Caddy config:

#{
#  email jpmiller25@gmail.com
#}


#ha.bearcave25.com {
#        tls {
#                dns cloudflare <api key>
#                resolvers 1.1.1.1
#                alpn disable_tlsalpn_challenge
#        }
#        reverse_proxy 192.168.1.25:8123
#}

#halocal.bearcave25.com {
#        tls {
#                dns cloudflare <api key>
#                resolvers 1.1.1.1
#                alpn disable_tlsalpn_challenge
#        }
#        reverse_proxy {
#               to 192.168.1.26:8123
#               transport http {
#                       tls
#                       tls_insecure_skip_verify
#               }
#       }
#}

ha.bearcave25.com {
  reverse_proxy 192.168.1.25:8123
}

halocal.bearcave25.com {
  reverse_proxy 192.168.1.25:8123
}

3. The problem I’m having:

I got this config working today as posted, though I’ve had intermittent cert failures which resulted in a lot of head banging and the results are as you can see in the commented configs. I’m new to this but I’m getting a rough idea of how cert challenges and validation works, and so even though the config works currently I’m not sure how: my public host ha.bearcave25.com does not go through caddy at all. So I don’t know how the http validation or dns validation or anything is even possible. So I would like to set this up to work properly for the future and not continue to have intermittent cert failures when the volume happens to get deleted, or the cert expires etc.

4. Error messages and/or full log output: (docker logs caddy-caddy-1)

{"level":"info","ts":1670084891.0298116,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
Error: adapting config using caddyfile: parsing caddyfile tokens for 'reverse_proxy': /etc/caddy/Caddyfile:22 - Error during parsing: unrecognized subdirective tls_insecure_skip_verify
{"level":"info","ts":1670094225.2722664,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1670094225.2772038,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":7}
{"level":"info","ts":1670094225.281425,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
{"level":"info","ts":1670094225.2822523,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40006caa80"}
{"level":"info","ts":1670094225.2825983,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670094225.2842958,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1670094225.2843902,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670094225.2846422,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"info","ts":1670094225.2848923,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670094225.2851732,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670094225.2858202,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ha.bearcave25.com","halocal.bearcave25.com"]}
{"level":"info","ts":1670094225.2923608,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1670094225.2962658,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670094225.2963228,"msg":"serving initial configuration"}
{"level":"info","ts":1670095215.1291006,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1670095215.129212,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":1670095215.1301827,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40006caa80"}
{"level":"info","ts":1670095215.1316724,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1670095215.131733,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1670095224.9631665,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1670095224.9681933,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":7}
{"level":"info","ts":1670095224.9720767,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1670095224.9727454,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000525110"}
{"level":"info","ts":1670095224.9730427,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670095224.9743416,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1670095224.9745436,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670095224.9750435,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"info","ts":1670095224.9753933,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670095224.9756374,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670095224.9756792,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ha.bearcave25.com","halocal.bearcave25.com"]}
{"level":"info","ts":1670095225.1773355,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670095225.1773977,"msg":"serving initial configuration"}
{"level":"info","ts":1670095225.212762,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1670096338.3547087,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"32950","headers":{"Accept-Encoding":["gzip"],"Content-Length":["798"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1670096338.3569999,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1670096338.3578916,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000714c40"}
{"level":"info","ts":1670096338.357944,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670096338.3587992,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670096338.3588874,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096338.3591013,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096338.3591347,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ha.bearcave25.com"]}
{"level":"info","ts":1670096338.3613043,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000525110"}
{"level":"info","ts":1670096338.363158,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670096338.3632228,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1670096338.3656256,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1670096516.15068,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"49010","headers":{"Accept-Encoding":["gzip"],"Content-Length":["757"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1670096516.1531863,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1670096516.1541455,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40002f4c40"}
{"level":"info","ts":1670096516.1546426,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670096516.1559832,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670096516.1561065,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096516.156347,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096516.1564035,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ha.bearcave25.com"]}
{"level":"info","ts":1670096516.1591444,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000714c40"}
{"level":"info","ts":1670096516.1598146,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670096516.1598616,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1670096516.1617064,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1670096713.8730903,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"52998","headers":{"Accept-Encoding":["gzip"],"Content-Length":["257"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1670096713.8748288,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1670096713.8751671,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1670096713.8752337,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670096713.8755474,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000781260"}
{"level":"info","ts":1670096713.8758523,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096713.8760085,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670096713.8763208,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096713.8763568,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["ha.bearcave25.com"]}
{"level":"info","ts":1670096713.8783796,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x40002f4c40"}
{"level":"info","ts":1670096713.8792162,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670096713.8792696,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1670096713.8831258,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"error","ts":1670096806.022191,"logger":"http.log.error","msg":"dial tcp 192.168.1.25:8123: connect: connection refused","request":{"remote_ip":"192.168.1.141","remote_port":"65198","proto":"HTTP/1.1","method":"GET","host":"ha.bearcave25.com","uri":"/api/websocket","headers":{"User-Agent":["Home Assistant/2022.10.1 (io.robbie.HomeAssistant; build:2022.424; iOS 16.1.1)"],"Sec-Websocket-Key":["ZHpjeHdvZnVxdmdidnp3bA=="],"Origin":["https://ha.bearcave25.com"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits; server_max_window_bits=15"],"Connection":["Upgrade"],"Sec-Websocket-Version":["13"],"Upgrade":["websocket"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"ha.bearcave25.com"}},"duration":0.001291442,"status":502,"err_id":"kunjypu6f","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
{"level":"error","ts":1670096811.147911,"logger":"http.log.error","msg":"dial tcp 192.168.1.25:8123: connect: connection refused","request":{"remote_ip":"192.168.1.141","remote_port":"65201","proto":"HTTP/1.1","method":"GET","host":"ha.bearcave25.com","uri":"/api/websocket","headers":{"Sec-Websocket-Key":["eXZxcHVhcXBjdGFlenluag=="],"User-Agent":["Home Assistant/2022.10.1 (io.robbie.HomeAssistant; build:2022.424; iOS 16.1.1)"],"Origin":["https://ha.bearcave25.com"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits; server_max_window_bits=15"],"Connection":["Upgrade"],"Upgrade":["websocket"],"Sec-Websocket-Version":["13"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"ha.bearcave25.com"}},"duration":0.002061874,"status":502,"err_id":"4gigiuajr","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
{"level":"error","ts":1670096821.2426968,"logger":"http.log.error","msg":"dial tcp 192.168.1.25:8123: connect: connection refused","request":{"remote_ip":"192.168.1.141","remote_port":"65202","proto":"HTTP/1.1","method":"GET","host":"ha.bearcave25.com","uri":"/api/websocket","headers":{"Sec-Websocket-Key":["Y2Vod2l4eGhqeWtqdmV2eA=="],"Sec-Websocket-Version":["13"],"Origin":["https://ha.bearcave25.com"],"Upgrade":["websocket"],"Sec-Websocket-Extensions":["permessage-deflate; client_max_window_bits; server_max_window_bits=15"],"User-Agent":["Home Assistant/2022.10.1 (io.robbie.HomeAssistant; build:2022.424; iOS 16.1.1)"],"Connection":["Upgrade"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"ha.bearcave25.com"}},"duration":0.001185351,"status":502,"err_id":"95fin7ect","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
{"level":"info","ts":1670096933.9769015,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"35890","headers":{"Accept-Encoding":["gzip"],"Content-Length":["450"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1670096933.9786227,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1670096933.9789796,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1670096933.979173,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1670096933.9790175,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000330000"}
{"level":"info","ts":1670096933.9800692,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1670096933.9801571,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096933.9803126,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1670096933.980354,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["halocal.bearcave25.com","ha.bearcave25.com"]}
{"level":"info","ts":1670096933.9840643,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x4000781260"}
{"level":"info","ts":1670096933.9869497,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1670096933.9870222,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1670096933.9954727,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

5. What I already tried:

I successfully set up the cloudflare dns module according to the commented out configs, and that’s actually exactly how I have immich running on another ubuntu server on my network. I literally copied the config and tried to set it up for home assistant today, but started getting errors ( Error code: SSL_ERROR_NEXT_PROTOCOL_NO_PROTOCOL) (from the browser, sorry I didn’t curl this when I was checking it.)
I’d like to know if my current config is robust enough or has potential to fail, and what to do about it. Or should I go back and try to get everything configured with cloudflare dns.

6. Links to relevant resources:

2

Looks like Caddy failed to run because of a config error. I assume this was from a previous attempt at running it though.

Okay, in that case you definitely must use the DNS challenge.

The ACME HTTP and TLS-ALPN challenges require Caddy to be publicly accessible.

Remove this line, enabling the DNS challenge implicitly disables other challenge types.

This might not be necessary, but it’s probably not harmful to make sure your local machine’s DNS resolver doesn’t mess things up, I guess.

HTTP challenge would definitely not work, but DNS challenge could.

The way it works is Caddy makes an API call to CloudFlare to tell it to configure a DNS TXT record on your domain with the challenge value, and then the ACME issuer will do a DNS query to read the challenge.

Make sure to configure this on your host machine for the most optimal HTTP/3 performance.

Also, you’ll want to add - "443:443/udp" to your Docker port mappings for HTTP/3 traffic to work.

This error is pretty self explanatory I think, your Caddy container isn’t able to reach your upstream app at 192.168.1.25:8123. You’ll have to figure out the problem there, whether it’s a networking issue, or if the app just isn’t running, or if it’s on a different port.

Okay, that’s good. I’m not seeing anything in your logs about the issuance process though. Did that happen before these logs you copied?

If the DNS challenge did succeed (within the past ~60 days) then Caddy will still have a valid certificate, and it won’t attempt to get a new one. So it’s unlikely that your tls config was wrong.

Yeah, please try with curl -v and see what you get. Browsers can be flaky with TLS errors, and aren’t usually specific about what the problem is. Also, try with a different browser (if you used Chrome, also try with Firefox or Edge, etc). If it works with another browser, try clearing your browser’s cache.

1 Like
3

This topic was automatically closed after 30 days. New replies are no longer allowed.