Single server solution with caddy

1. The problem I’m having:

I got a caddy proxy on a dedicated server (Raspberry Pi 3) to work in my LAN. The proxy manages two applications: Nextcloud and Homeassistant. Nextcloud runs directly on a NUC and Homeassistant within a virtual machine on the same NUC. Both were accessed via individual IPs: the NUC has as well as the virtual machine a specific IP.
Now I run a caddy application (with Docker) on the same NUC. To supply caddy a specific IP I set up macvlan with another specific IP just for caddy. This IP can be pinged successfully. Also this IP is forwarded by the router, in the same way I have forwarded the IP of my Raspberry Pi before. However, opening the caddy IP (macvlan) within my router does not lead to a working proxy. It seems, that the certificate for caddy cannot be supplied. Trying to open the proxied IPs within a browser result in error 502.

2. Error messages and/or full log output:

uli@linuc:~$ sudo docker logs -f --tail 20 caddy
{"level":"debug","ts":1732717052.489745,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["homeassistant.reitz-kef.de","linuc.reitz-kef.de"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.178.128:8123"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.178.87:80"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1732717052.4901845,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1732717052.4909058,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1732717052.4915576,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1732717052.4925246,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1732717052.4925427,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"debug","ts":1732717052.4928277,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1732717052.4928465,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1732717052.4928544,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1732717052.492933,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["homeassistant.reitz-kef.de","linuc.reitz-kef.de"]}
{"level":"debug","ts":1732717052.4933515,"logger":"tls","msg":"loading managed certificate","domain":"homeassistant.reitz-kef.de","expiration":1740488282,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
{"level":"debug","ts":1732717052.493756,"logger":"tls.cache","msg":"added certificate to cache","subjects":["homeassistant.reitz-kef.de"],"expiration":1740488282,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"5d5d6e9a5372ceac444ca8ced205a402ae71832e3664f1fa2784cf027324a679","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1732717052.4938,"logger":"events","msg":"event","name":"cached_managed_cert","id":"76f8de25-aded-442f-a097-2c546e2523d6","origin":"tls","data":{"sans":["homeassistant.reitz-kef.de"]}}
{"level":"debug","ts":1732717052.4941247,"logger":"tls","msg":"loading managed certificate","domain":"linuc.reitz-kef.de","expiration":1740488282,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
{"level":"debug","ts":1732717052.4943552,"logger":"tls.cache","msg":"added certificate to cache","subjects":["linuc.reitz-kef.de"],"expiration":1740488282,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1732717052.494377,"logger":"events","msg":"event","name":"cached_managed_cert","id":"227658ca-8f05-43ec-8d18-31450b231300","origin":"tls","data":{"sans":["linuc.reitz-kef.de"]}}
{"level":"info","ts":1732717052.49455,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1732717052.49456,"msg":"serving initial configuration"}
{"level":"info","ts":1732717052.4958997,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"0c4b82d6-fd6e-4095-91b0-d64981b7d005","try_again":1732803452.4958987,"try_again_in":86399.999999706}
{"level":"info","ts":1732717052.4959676,"logger":"tls","msg":"finished cleaning storage units"}

caddy just startet

{"level":"debug","ts":1732717113.6099072,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.126:13472: EOF"}
{"level":"debug","ts":1732717116.1082685,"logger":"http.stdlib","msg":"http: TLS handshake error from 154.28.229.157:52088: EOF"}
{"level":"debug","ts":1732717120.5172136,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.73:20490: EOF"}
{"level":"debug","ts":1732717121.536762,"logger":"events","msg":"event","name":"tls_get_certificate","id":"22001f09-e496-4a73-b87d-9659d3b5b5a7","origin":"tls","data":{"client_hello":{"CipherSuites":[49199,49195,49169,49159,49171,49161,49172,49162,5,47,53,49170,10],"ServerName":"","SupportedCurves":[23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1025,1027,513,515,1025,1281,1537],"SupportedProtos":null,"SupportedVersions":[771,770,769],"RemoteAddr":{"IP":"162.243.48.72","Port":57932,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717121.5371258,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.178.224"}
{"level":"debug","ts":1732717121.537169,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"162.243.48.72","remote_port":"57932","server_name":"","remote":"162.243.48.72:57932","identifier":"192.168.178.224","cipher_suites":[49199,49195,49169,49159,49171,49161,49172,49162,5,47,53,49170,10],"cert_cache_fill":0.0002,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1732717121.5373814,"logger":"http.stdlib","msg":"http: TLS handshake error from 162.243.48.72:57932: no certificate available for '192.168.178.224'"}
{"level":"debug","ts":1732717121.7089088,"logger":"http.stdlib","msg":"http: TLS handshake error from 162.243.48.72:57944: client sent an HTTP request to an HTTPS server"}

after some time:

{"level":"debug","ts":1732717249.9343698,"logger":"events","msg":"event","name":"tls_get_certificate","id":"796427dd-f585-4a88-ac5f-813006e8f4bf","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"linuc.reitz-kef.de","SupportedCurves":[29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"62.91.145.78","Port":60582,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717249.93463,"logger":"tls.handshake","msg":"choosing certificate","identifier":"linuc.reitz-kef.de","num_choices":1}
{"level":"debug","ts":1732717249.934663,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"linuc.reitz-kef.de","subjects":["linuc.reitz-kef.de"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717249.9346888,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"62.91.145.78","remote_port":"60582","subjects":["linuc.reitz-kef.de"],"managed":true,"expiration":1740488282,"hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717249.9577844,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.178.87:80","total_upstreams":1}
{"level":"debug","ts":1732717252.9584215,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.178.87:80","duration":3.000460431,"request":{"remote_ip":"62.91.145.78","remote_port":"60582","client_ip":"62.91.145.78","proto":"HTTP/2.0","method":"REPORT","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/uli/bcherei/","headers":{"Accept":["text/xml"],"X-Forwarded-For":["62.91.145.78"],"Priority":["u=4"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Depth":["1"],"Origin":["https://linuc.reitz-kef.de"],"Cache-Control":["no-cache"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Thunderbird/128.4.3"],"Sec-Fetch-Mode":["no-cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["linuc.reitz-kef.de"],"Pragma":["no-cache"],"Te":["trailers"],"Sec-Fetch-Site":["same-origin"],"Accept-Charset":["utf-8,*;q=0.1"],"Content-Type":["text/xml; charset=utf-8"],"Authorization":["REDACTED"],"Content-Length":["205"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"linuc.reitz-kef.de"}},"error":"dial tcp 192.168.178.87:80: i/o timeout"}
{"level":"error","ts":1732717252.9586544,"logger":"http.log.error","msg":"dial tcp 192.168.178.87:80: i/o timeout","request":{"remote_ip":"62.91.145.78","remote_port":"60582","client_ip":"62.91.145.78","proto":"HTTP/2.0","method":"REPORT","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/uli/bcherei/","headers":{"Priority":["u=4"],"Accept-Charset":["utf-8,*;q=0.1"],"Content-Length":["205"],"Authorization":["REDACTED"],"Sec-Fetch-Dest":["empty"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Thunderbird/128.4.3"],"Sec-Fetch-Mode":["no-cors"],"Cache-Control":["no-cache"],"Te":["trailers"],"Sec-Fetch-Site":["same-origin"],"Content-Type":["text/xml; charset=utf-8"],"Depth":["1"],"Pragma":["no-cache"],"Origin":["https://linuc.reitz-kef.de"],"Accept":["text/xml"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"linuc.reitz-kef.de"}},"duration":3.000824801,"status":502,"err_id":"0v0uhq3jd","err_trace":"reverseproxy.statusError (reverseproxy.go:1353)"}
{"level":"debug","ts":1732717430.5314198,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.118:2148: EOF"}
{"level":"debug","ts":1732717430.540681,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.158:16332: EOF"}
{"level":"debug","ts":1732717439.3547647,"logger":"http.stdlib","msg":"http: TLS handshake error from 154.28.229.41:23500: EOF"}
{"level":"debug","ts":1732717586.8900661,"logger":"events","msg":"event","name":"tls_get_certificate","id":"923043fa-dca9-4c26-b435-a3ba332ad2f3","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,52393,49199,49200,52392,49161,49162,49171,49172,156,157,47,53],"ServerName":"linuc.reitz-kef.de","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"62.91.145.78","Port":61915,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717586.8902347,"logger":"tls.handshake","msg":"choosing certificate","identifier":"linuc.reitz-kef.de","num_choices":1}
{"level":"debug","ts":1732717586.890263,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"linuc.reitz-kef.de","subjects":["linuc.reitz-kef.de"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717586.8902893,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"62.91.145.78","remote_port":"61915","subjects":["linuc.reitz-kef.de"],"managed":true,"expiration":1740488282,"hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717587.5543995,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.178.87:80","total_upstreams":1}
{"level":"debug","ts":1732717590.5554643,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.178.87:80","duration":3.000950068,"request":{"remote_ip":"62.91.145.78","remote_port":"61915","client_ip":"62.91.145.78","proto":"HTTP/1.1","method":"OPTIONS","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/UliR","headers":{"Accept-Language":["de-DE"],"Authorization":["REDACTED"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["62.91.145.78"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["linuc.reitz-kef.de"],"User-Agent":["curl/7.54"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"linuc.reitz-kef.de"}},"error":"dial tcp 192.168.178.87:80: i/o timeout"}
{"level":"error","ts":1732717590.5556273,"logger":"http.log.error","msg":"dial tcp 192.168.178.87:80: i/o timeout","request":{"remote_ip":"62.91.145.78","remote_port":"61915","client_ip":"62.91.145.78","proto":"HTTP/1.1","method":"OPTIONS","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/UliR","headers":{"Accept-Encoding":["gzip"],"User-Agent":["curl/7.54"],"Accept-Language":["de-DE"],"Authorization":["REDACTED"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"linuc.reitz-kef.de"}},"duration":3.001216766,"status":502,"err_id":"mmab1fuqu","err_trace":"reverseproxy.statusError (reverseproxy.go:1353)"}
Diese Seite funktioniert nicht
linuc.reitz-kef.de kann diese Anfrage momentan nicht verarbeiten.
HTTP ERROR 502

3. Caddy version:

CADDY_VERSION v2.9.0-beta.3

4. How I installed and ran Caddy:

Intel NUC with core I5 6xxx, 16GB RAM, 250GB SSD
Linux 5.15.0-126-generic on x86_64,
Docker with Portainer Community Edition 2.21.3

b. Command:

I run caddy within Portainer - deployment, start and stop with no error messages in Portainer.

c. Service/unit/compose file:

version: "3"
services:
  web:
    image: caddy:2.9-alpine
    container_name: "caddy"
    ports:
    - '80:80'
    - '443:443'
    - '8448:8448'
    volumes:
    - /caddy/Caddyfile:/etc/caddy/Caddyfile
    - caddy-data:/data
    - /etc/ssl/custom:/etc/ssl/custom
    restart: unless-stopped
    networks:
      macvlan:
        ipv4_address: 192.168.178.224

networks:
  macvlan:
    external: true
  

volumes:
  caddy-data:


### d. My complete Caddy config:

{
        debug
}
linuc.reitz-kef.de {
        tls reitz_u@reitz-kef.de
        reverse_proxy 192.168.178.87
}
homeassistant.reitz-kef.de {
        tls reitz_u@reitz-kef.de
        reverse_proxy 192.168.178.128:8123
}


### 5. Links to relevant resources:
<!-- Optional, but can help get us on the same page quickly. -->

Howdy @pampasadmin!

Macvlan networking is blocked at the kernel level from communicating directly with the host, by design. The expected behaviour is that it can reach other physically-separated machines, but does not have an internal route to the host.

This isn’t something Caddy can help you with. You will need to change your strategy, reconfigure your network, or use other methods (e.g. external switching hardware) to accommodate packet routing between your macvlan and the host.

Hi Matthew,
thanks for the clear statement, that my plan won’t work - I’ll stop spending time on this dead horse.
My main target, is to concentrate on one physical server to limit the zoo of single controlers. The next idea is, to pack caddy in its own virtual machine on the NUC. I’ll give it a try and will report within this topic.
Cheers Uli