Trusted_proxies for unix socket, how to

1. The problem I’m having:

I am trying to let Caddy to listen on a unix socket file, and use cloudflared to expose the socket to internet. But I can’t find any documentation describing how to let Caddy “trust” connection from unix socket (like from network port as trusted_proxies.

2. Error messages and/or full log output:

I am using this echo server to inspect the request being sent to upstream, caddy is listening on /run/container-sockets/caddy/caddy.socket and proxies /echo to the echo server.

1月 20 19:06:56 epyc-server podman[2852822]: 2024-01-20 19:06:56.297418551 +0800 CST m=+0.057713766 image pull 4437922cacc5203f315d4e8104019a728795aa8be87f155bca9a03d9fe85c670 quay.io/karuboniru/caddy-cloudflare:latest
1月 20 19:06:56 epyc-server podman[2852822]: 2024-01-20 19:06:56.504779651 +0800 CST m=+0.265074856 container create bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.url=https://caddyserver.com, org.opencontainers.image.title=Caddy, io.containers.autoupdate=registry, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.version=v2.7.6, org.opencontainers.image.documentation=https://caddyserver.com/docs, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker, org.opencontainers.image.licenses=Apache-2.0, io.buildah.version=1.33.2, PODMAN_SYSTEMD_UNIT=caddy.service, org.opencontainers.image.vendor=Light Code Labs)
1月 20 19:06:57 epyc-server podman[2852822]: 2024-01-20 19:06:57.078085741 +0800 CST m=+0.838380976 container init bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.licenses=Apache-2.0, org.opencontainers.image.title=Caddy, io.buildah.version=1.33.2, org.opencontainers.image.url=https://caddyserver.com, io.containers.autoupdate=registry, org.opencontainers.image.version=v2.7.6, PODMAN_SYSTEMD_UNIT=caddy.service, org.opencontainers.image.vendor=Light Code Labs, org.opencontainers.image.documentation=https://caddyserver.com/docs, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker)
1月 20 19:06:57 epyc-server podman[2852822]: 2024-01-20 19:06:57.083795188 +0800 CST m=+0.844090403 container start bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.version=v2.7.6, PODMAN_SYSTEMD_UNIT=caddy.service, io.containers.autoupdate=registry, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker, org.opencontainers.image.vendor=Light Code Labs, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.documentation=https://caddyserver.com/docs, io.buildah.version=1.33.2, org.opencontainers.image.licenses=Apache-2.0, org.opencontainers.image.url=https://caddyserver.com, org.opencontainers.image.title=Caddy)
1月 20 19:06:57 epyc-server systemd[1]: Started caddy.service - Caddy.
1月 20 19:06:57 epyc-server caddy[2852822]: bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2191367,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"warn","ts":1705748820.2263498,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2348022,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2351968,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000484180"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.235819,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2358527,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.243015,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2439492,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2442906,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.244374,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2443867,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.niconi.org"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.247251,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.247272,"msg":"serving initial configuration"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"warn","ts":1705748820.2475991,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"77f6d13f-5223-435e-a920-90d4c96246c6","try_again":1705835220.2475965,"try_again_in":86399.999999179}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2477596,"logger":"tls","msg":"finished cleaning storage units"}
$ curl --unix-socket /run/container-sockets/caddy/caddy.socket 1.niconi.org/echo/ -H "x-forwarded-for: 1.2.3.4"
{
  "path": "/",
  "headers": {
    "host": "1.niconi.org",
    "user-agent": "curl/8.2.1",
    "accept": "*/*",
    "accept-encoding": "gzip"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "1.niconi.org",
  "ip": "::ffff:10.88.0.225",
  "ips": [],
  "protocol": "http",
  "query": {},
  "subdomains": [
    "1"
  ],
  "xhr": false,
  "os": {
    "hostname": "fbac3204243e"
  },
  "connection": {}
}
# The echo server can handle x-forwarded-for
$ curl 10.88.0.224:8080 -H "x-forwarded-for: 1.2.3.4"
{
  "path": "/",
  "headers": {
    "host": "10.88.0.224:8080",
    "user-agent": "curl/8.2.1",
    "accept": "*/*",
    "x-forwarded-for": "1.2.3.4"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "10.88.0.224",
  "ip": "1.2.3.4",
  "ips": [
    "1.2.3.4"
  ],
  "protocol": "http",
  "query": {},
  "subdomains": [],
  "xhr": false,
  "os": {
    "hostname": "fbac3204243e"
  },
  "connection": {}
}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Docker image built with

FROM docker.io/library/caddy:builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM docker.io/library/caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Running using podman with quadlet, in a selinux confined container. With ports exposed and some unix sockets mounted.

a. System environment:

  • Fedora release 39 (Thirty Nine)
  • podman version 4.8.3

b. Command:

See the container configuration below

c. Service/unit/compose file:

Podman quatlet configuration file

[Unit]
Description=Caddy
After=network-online.target
Wants=network-online.target

[Container]
Image=quay.io/karuboniru/caddy-cloudflare:latest
Secret=Caddyfile,target=/etc/caddy/Caddyfile
Volume=CaddyData:/data:Z,U
Volume=CaddyConfig:/config:Z,U

Volume=/run/container-sockets:/run/container-sockets:idmap
Volume=/run/container-sockets/caddy:/run/container-sockets/caddy:idmap,z

Volume=/media/storage:/mnt/share/storage:ro

SecurityLabelType=container_caddy.process

DropCapability=ALL
AddCapability=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
ReadOnly=true
RunInit=true

PublishPort=80:80/tcp
PublishPort=443:443/tcp
PublishPort=443:443/udp

AutoUpdate=registry

[Service]
Restart=on-failure
ExecStartPre=mkdir -p /run/container-sockets/caddy
ExecStopPost=rm -rf /run/container-sockets/caddy

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

&(site) {
  handle_path /qbt/* {
    reverse_proxy http://systemd-qbittorrent.dns.podman:8080 {
      header_down Set-Cookie "path=/" "path=/qbt/ ; SameSite=Strict; Secure"
    }
  }

  handle_path /qbtee/* {
    reverse_proxy http://systemd-qbittorrenteh.dns.podman:8080 {
      header_down Set-Cookie "path=/" "path=/qbtee/ ; SameSite=Strict; Secure"
    }
  }

  handle /cp/* {
    reverse_proxy unix//run/container-sockets/cockpit.socket
  }

  handle /jellyfin/* {
    reverse_proxy http://systemd-jellyfin.dns.podman:8096
  }

  handle_path /adguard/* {
    reverse_proxy http://systemd-adguardhome.dns.podman:80 {
      header_down Location /index.html /adguard/index.html
    }
  }

  handle_path /ab/* {
    reverse_proxy http://systemd-autobangumi.dns.podman:7892
  }
  
  handle_path /file/* {
    file_server browse {
      root /mnt/share
    }
    basicauth {
      !!REDACTED!!
    }
  }

  handle_path /echo/* {
    reverse_proxy http://systemd-echo.dns.podman:8080 
  }

  @ward {
    path / /css* /js* /img* /api* /fonts* /*/ 
  }

  handle @ward {
    reverse_proxy http://systemd-ward.dns.podman:4000
  }

  handle /* {
    redir {scheme}://{host}{uri}/
  }
}

*.niconi.org {
  tls {
    dns cloudflare !!REDACTED!!
  }

  invoke site
}

http://*.niconi.org {
  bind unix//run/container-sockets/caddy/caddy.socket
  invoke site
}

5. Links to relevant resources:

I’m not sure. I’ve not tried that yet.

Add the log directive to your site. What do you see in your logs? What value does remote_ip have?

1月 21 03:57:24 epyc-server systemd-caddy[3051292]: {"level":"debug","ts":1705780644.023336,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"systemd-echo.dns.podman:8080","duration":0.006861339,"request":{"remote_ip":"@","remote_port":"","client_ip":"","proto":"HTTP/1.1","method":"GET","host":"1.niconi.org","uri":"/","headers":{"User-Agent":["curl/8.2.1"],"Accept":["*/*"]}},"headers":{"X-Powered-By":["Express"],"Content-Type":["application/json; charset=utf-8"],"Content-Length":["421"],"Etag":["W/\"1a5-HfO3QhxeuJWkp953J0ctMUD1Ol8\""],"Date":["Sat, 20 Jan 2024 19:57:24 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"]},"status":200}

So it is "@" in this case, which won’t be accepted by the static ip handler of trusted proxy configuration.

Interesting. Good to know.

I’m not sure what the best fix would be. We didn’t design the APIs with that in mind :thinking: trusted_proxies takes an “IP range source” module (we provide a static one by default) and obviously @ is not an IP, so those modules can’t really ever do anything valid.

I suppose there’s an argument to be made that @ should always be considered trusted. That might be what we should do. I don’t know if there’s any ways that could go wrong or be abused though, so I’m nervous about making that change without more research.

Another thought is we could maybe transform @ to ::1 so that if ::1 is trusted then it passes. That would mean trust is still always opt-in. But ::1 is kinda a lie.

Hard to say which approach is more proper.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.