Single server solution with caddy

Help
1

1. The problem I’m having:

I got a caddy proxy on a dedicated server (Raspberry Pi 3) to work in my LAN. The proxy manages two applications: Nextcloud and Homeassistant. Nextcloud runs directly on a NUC and Homeassistant within a virtual machine on the same NUC. Both were accessed via individual IPs: the NUC has as well as the virtual machine a specific IP.
Now I run a caddy application (with Docker) on the same NUC. To supply caddy a specific IP I set up macvlan with another specific IP just for caddy. This IP can be pinged successfully. Also this IP is forwarded by the router, in the same way I have forwarded the IP of my Raspberry Pi before. However, opening the caddy IP (macvlan) within my router does not lead to a working proxy. It seems, that the certificate for caddy cannot be supplied. Trying to open the proxied IPs within a browser result in error 502.

2. Error messages and/or full log output:

uli@linuc:~$ sudo docker logs -f --tail 20 caddy
{"level":"debug","ts":1732717052.489745,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["homeassistant.reitz-kef.de","linuc.reitz-kef.de"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.178.128:8123"}]}]}]}],"terminal":true},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.178.87:80"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1732717052.4901845,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1732717052.4909058,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","ts":1732717052.4915576,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1732717052.4925246,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1732717052.4925427,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"debug","ts":1732717052.4928277,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"warn","ts":1732717052.4928465,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1732717052.4928544,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1732717052.492933,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["homeassistant.reitz-kef.de","linuc.reitz-kef.de"]}
{"level":"debug","ts":1732717052.4933515,"logger":"tls","msg":"loading managed certificate","domain":"homeassistant.reitz-kef.de","expiration":1740488282,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
{"level":"debug","ts":1732717052.493756,"logger":"tls.cache","msg":"added certificate to cache","subjects":["homeassistant.reitz-kef.de"],"expiration":1740488282,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"5d5d6e9a5372ceac444ca8ced205a402ae71832e3664f1fa2784cf027324a679","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1732717052.4938,"logger":"events","msg":"event","name":"cached_managed_cert","id":"76f8de25-aded-442f-a097-2c546e2523d6","origin":"tls","data":{"sans":["homeassistant.reitz-kef.de"]}}
{"level":"debug","ts":1732717052.4941247,"logger":"tls","msg":"loading managed certificate","domain":"linuc.reitz-kef.de","expiration":1740488282,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
{"level":"debug","ts":1732717052.4943552,"logger":"tls.cache","msg":"added certificate to cache","subjects":["linuc.reitz-kef.de"],"expiration":1740488282,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059","cache_size":2,"cache_capacity":10000}
{"level":"debug","ts":1732717052.494377,"logger":"events","msg":"event","name":"cached_managed_cert","id":"227658ca-8f05-43ec-8d18-31450b231300","origin":"tls","data":{"sans":["linuc.reitz-kef.de"]}}
{"level":"info","ts":1732717052.49455,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1732717052.49456,"msg":"serving initial configuration"}
{"level":"info","ts":1732717052.4958997,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"0c4b82d6-fd6e-4095-91b0-d64981b7d005","try_again":1732803452.4958987,"try_again_in":86399.999999706}
{"level":"info","ts":1732717052.4959676,"logger":"tls","msg":"finished cleaning storage units"}

caddy just startet

{"level":"debug","ts":1732717113.6099072,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.126:13472: EOF"}
{"level":"debug","ts":1732717116.1082685,"logger":"http.stdlib","msg":"http: TLS handshake error from 154.28.229.157:52088: EOF"}
{"level":"debug","ts":1732717120.5172136,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.73:20490: EOF"}
{"level":"debug","ts":1732717121.536762,"logger":"events","msg":"event","name":"tls_get_certificate","id":"22001f09-e496-4a73-b87d-9659d3b5b5a7","origin":"tls","data":{"client_hello":{"CipherSuites":[49199,49195,49169,49159,49171,49161,49172,49162,5,47,53,49170,10],"ServerName":"","SupportedCurves":[23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1025,1027,513,515,1025,1281,1537],"SupportedProtos":null,"SupportedVersions":[771,770,769],"RemoteAddr":{"IP":"162.243.48.72","Port":57932,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717121.5371258,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"192.168.178.224"}
{"level":"debug","ts":1732717121.537169,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"162.243.48.72","remote_port":"57932","server_name":"","remote":"162.243.48.72:57932","identifier":"192.168.178.224","cipher_suites":[49199,49195,49169,49159,49171,49161,49172,49162,5,47,53,49170,10],"cert_cache_fill":0.0002,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1732717121.5373814,"logger":"http.stdlib","msg":"http: TLS handshake error from 162.243.48.72:57932: no certificate available for '192.168.178.224'"}
{"level":"debug","ts":1732717121.7089088,"logger":"http.stdlib","msg":"http: TLS handshake error from 162.243.48.72:57944: client sent an HTTP request to an HTTPS server"}

after some time:

{"level":"debug","ts":1732717249.9343698,"logger":"events","msg":"event","name":"tls_get_certificate","id":"796427dd-f585-4a88-ac5f-813006e8f4bf","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"linuc.reitz-kef.de","SupportedCurves":[29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"62.91.145.78","Port":60582,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717249.93463,"logger":"tls.handshake","msg":"choosing certificate","identifier":"linuc.reitz-kef.de","num_choices":1}
{"level":"debug","ts":1732717249.934663,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"linuc.reitz-kef.de","subjects":["linuc.reitz-kef.de"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717249.9346888,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"62.91.145.78","remote_port":"60582","subjects":["linuc.reitz-kef.de"],"managed":true,"expiration":1740488282,"hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717249.9577844,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.178.87:80","total_upstreams":1}
{"level":"debug","ts":1732717252.9584215,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.178.87:80","duration":3.000460431,"request":{"remote_ip":"62.91.145.78","remote_port":"60582","client_ip":"62.91.145.78","proto":"HTTP/2.0","method":"REPORT","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/uli/bcherei/","headers":{"Accept":["text/xml"],"X-Forwarded-For":["62.91.145.78"],"Priority":["u=4"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"Depth":["1"],"Origin":["https://linuc.reitz-kef.de"],"Cache-Control":["no-cache"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Thunderbird/128.4.3"],"Sec-Fetch-Mode":["no-cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["linuc.reitz-kef.de"],"Pragma":["no-cache"],"Te":["trailers"],"Sec-Fetch-Site":["same-origin"],"Accept-Charset":["utf-8,*;q=0.1"],"Content-Type":["text/xml; charset=utf-8"],"Authorization":["REDACTED"],"Content-Length":["205"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"linuc.reitz-kef.de"}},"error":"dial tcp 192.168.178.87:80: i/o timeout"}
{"level":"error","ts":1732717252.9586544,"logger":"http.log.error","msg":"dial tcp 192.168.178.87:80: i/o timeout","request":{"remote_ip":"62.91.145.78","remote_port":"60582","client_ip":"62.91.145.78","proto":"HTTP/2.0","method":"REPORT","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/uli/bcherei/","headers":{"Priority":["u=4"],"Accept-Charset":["utf-8,*;q=0.1"],"Content-Length":["205"],"Authorization":["REDACTED"],"Sec-Fetch-Dest":["empty"],"Accept-Language":["de,en-US;q=0.7,en;q=0.3"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Thunderbird/128.4.3"],"Sec-Fetch-Mode":["no-cors"],"Cache-Control":["no-cache"],"Te":["trailers"],"Sec-Fetch-Site":["same-origin"],"Content-Type":["text/xml; charset=utf-8"],"Depth":["1"],"Pragma":["no-cache"],"Origin":["https://linuc.reitz-kef.de"],"Accept":["text/xml"],"Accept-Encoding":["gzip, deflate, br, zstd"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"linuc.reitz-kef.de"}},"duration":3.000824801,"status":502,"err_id":"0v0uhq3jd","err_trace":"reverseproxy.statusError (reverseproxy.go:1353)"}
{"level":"debug","ts":1732717430.5314198,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.118:2148: EOF"}
{"level":"debug","ts":1732717430.540681,"logger":"http.stdlib","msg":"http: TLS handshake error from 104.164.173.158:16332: EOF"}
{"level":"debug","ts":1732717439.3547647,"logger":"http.stdlib","msg":"http: TLS handshake error from 154.28.229.41:23500: EOF"}
{"level":"debug","ts":1732717586.8900661,"logger":"events","msg":"event","name":"tls_get_certificate","id":"923043fa-dca9-4c26-b435-a3ba332ad2f3","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,52393,49199,49200,52392,49161,49162,49171,49172,156,157,47,53],"ServerName":"linuc.reitz-kef.de","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"62.91.145.78","Port":61915,"Zone":""},"LocalAddr":{"IP":"192.168.178.224","Port":443,"Zone":""}}}}
{"level":"debug","ts":1732717586.8902347,"logger":"tls.handshake","msg":"choosing certificate","identifier":"linuc.reitz-kef.de","num_choices":1}
{"level":"debug","ts":1732717586.890263,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"linuc.reitz-kef.de","subjects":["linuc.reitz-kef.de"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717586.8902893,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"62.91.145.78","remote_port":"61915","subjects":["linuc.reitz-kef.de"],"managed":true,"expiration":1740488282,"hash":"d2c0ad1dc6e7d28a7b1b35f0de8c15d5385d1e89b0b024904d0ed5765620a059"}
{"level":"debug","ts":1732717587.5543995,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.178.87:80","total_upstreams":1}
{"level":"debug","ts":1732717590.5554643,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.178.87:80","duration":3.000950068,"request":{"remote_ip":"62.91.145.78","remote_port":"61915","client_ip":"62.91.145.78","proto":"HTTP/1.1","method":"OPTIONS","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/UliR","headers":{"Accept-Language":["de-DE"],"Authorization":["REDACTED"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["62.91.145.78"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["linuc.reitz-kef.de"],"User-Agent":["curl/7.54"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"linuc.reitz-kef.de"}},"error":"dial tcp 192.168.178.87:80: i/o timeout"}
{"level":"error","ts":1732717590.5556273,"logger":"http.log.error","msg":"dial tcp 192.168.178.87:80: i/o timeout","request":{"remote_ip":"62.91.145.78","remote_port":"61915","client_ip":"62.91.145.78","proto":"HTTP/1.1","method":"OPTIONS","host":"linuc.reitz-kef.de","uri":"/remote.php/dav/calendars/UliR","headers":{"Accept-Encoding":["gzip"],"User-Agent":["curl/7.54"],"Accept-Language":["de-DE"],"Authorization":["REDACTED"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"linuc.reitz-kef.de"}},"duration":3.001216766,"status":502,"err_id":"mmab1fuqu","err_trace":"reverseproxy.statusError (reverseproxy.go:1353)"}

Diese Seite funktioniert nicht
linuc.reitz-kef.de kann diese Anfrage momentan nicht verarbeiten.
HTTP ERROR 502

3. Caddy version:

CADDY_VERSION v2.9.0-beta.3

4. How I installed and ran Caddy:

Intel NUC with core I5 6xxx, 16GB RAM, 250GB SSD
Linux 5.15.0-126-generic on x86_64,
Docker with Portainer Community Edition 2.21.3

b. Command:

I run caddy within Portainer - deployment, start and stop with no error messages in Portainer.

c. Service/unit/compose file:

version: "3"
services:
  web:
    image: caddy:2.9-alpine
    container_name: "caddy"
    ports:
    - '80:80'
    - '443:443'
    - '8448:8448'
    volumes:
    - /caddy/Caddyfile:/etc/caddy/Caddyfile
    - caddy-data:/data
    - /etc/ssl/custom:/etc/ssl/custom
    restart: unless-stopped
    networks:
      macvlan:
        ipv4_address: 192.168.178.224

networks:
  macvlan:
    external: true
  

volumes:
  caddy-data:



### d. My complete Caddy config:


{
        debug
}
linuc.reitz-kef.de {
        tls reitz_u@reitz-kef.de
        reverse_proxy 192.168.178.87
}
homeassistant.reitz-kef.de {
        tls reitz_u@reitz-kef.de
        reverse_proxy 192.168.178.128:8123
}



### 5. Links to relevant resources:
<!-- Optional, but can help get us on the same page quickly. -->
2

Howdy @pampasadmin!

Macvlan networking is blocked at the kernel level from communicating directly with the host, by design. The expected behaviour is that it can reach other physically-separated machines, but does not have an internal route to the host.

This isn’t something Caddy can help you with. You will need to change your strategy, reconfigure your network, or use other methods (e.g. external switching hardware) to accommodate packet routing between your macvlan and the host.

3

Hi Matthew,
thanks for the clear statement, that my plan won’t work - I’ll stop spending time on this dead horse.
My main target, is to concentrate on one physical server to limit the zoo of single controlers. The next idea is, to pack caddy in its own virtual machine on the NUC. I’ll give it a try and will report within this topic.
Cheers Uli

4

Hello again,
is there a best practice recommendation for single computer scenario hosting a reverse proxy as well as the respective applications?
Cheers Uli

5

Caddy’s job, goal, and philosophy is to make HTTPS easy. The best practice, then, is whatever makes it easiest for you to keep things secure. We want everything to be secure and trusted all the time, and the biggest obstacle to that - realistically - is convenience. Caddy wants to be easy to understand, easy to deploy, and it wants to automate your routine, mundane HTTPS tasks away so you don’t need to think about them anymore.

So I personally have one instance of Caddy on each of my app servers. Each has its own unique DNS token, to provision certs behind firewalls if necessary. It has its own predictably-named Docker proxy network. Then I run my services in Compose stacks, and have web-facing components join the proxy network for Caddy to provide secure access. This makes things very easy for me to maintain.

A beginner may find this difficult to get started with, though. It may be simpler, easier to setup and more convenient for them to simply run Caddy directly on the host, configure Docker containers to expose local ports, and reverse-proxy to localhost for each of those services.

As for non-Caddy-specific advice, the number of servers you operate depends on how much much compute resources your services require, and the level of fault tolerance you want. First you meet service capacity, then you double it if you need redundancy, or triple it with high availability if it’s mission-critical.

For a home lab, just a single server could very easily suffice. As long as it’s okay for you to have some downtime while you tinker on it, and you’re not running anything insanely compute-intensive, you can easily just throw Caddy and all your services on the same box. No problem.

In my lab, I have two hypervisor hosts for hardware redundancy, and I have three app servers (Linux and 2x Windows) that migrate between the two depending on availability of resources and hardware downtimes.

6

Hello Matthew,
thanks for the elaborated answer. To make sure, that I understand it correctly; I summarize it in my words: caddy as well as the applications are all docker stacks on the respective machine. There is no service directly (i.e. not in docker) running on the machine.
I ran into difficulties, because I had the nextcloud running directly on the server as a starting point. In a further step, I tried to add other services without touching the nextcloud. This seems to be impossible.
Therefore I will put the nextcloud into Docker. Such a move is shurely described in the net and I will carefully work towards ist.
Cheers Uli