Trusted_proxies for unix socket, how to

karuboniru · January 20, 2024, 11:04am

1. The problem I’m having:

I am trying to let Caddy to listen on a unix socket file, and use cloudflared to expose the socket to internet. But I can’t find any documentation describing how to let Caddy “trust” connection from unix socket (like from network port as trusted_proxies.

2. Error messages and/or full log output:

I am using this echo server to inspect the request being sent to upstream, caddy is listening on /run/container-sockets/caddy/caddy.socket and proxies /echo to the echo server.

1月 20 19:06:56 epyc-server podman[2852822]: 2024-01-20 19:06:56.297418551 +0800 CST m=+0.057713766 image pull 4437922cacc5203f315d4e8104019a728795aa8be87f155bca9a03d9fe85c670 quay.io/karuboniru/caddy-cloudflare:latest
1月 20 19:06:56 epyc-server podman[2852822]: 2024-01-20 19:06:56.504779651 +0800 CST m=+0.265074856 container create bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.url=https://caddyserver.com, org.opencontainers.image.title=Caddy, io.containers.autoupdate=registry, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.version=v2.7.6, org.opencontainers.image.documentation=https://caddyserver.com/docs, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker, org.opencontainers.image.licenses=Apache-2.0, io.buildah.version=1.33.2, PODMAN_SYSTEMD_UNIT=caddy.service, org.opencontainers.image.vendor=Light Code Labs)
1月 20 19:06:57 epyc-server podman[2852822]: 2024-01-20 19:06:57.078085741 +0800 CST m=+0.838380976 container init bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.licenses=Apache-2.0, org.opencontainers.image.title=Caddy, io.buildah.version=1.33.2, org.opencontainers.image.url=https://caddyserver.com, io.containers.autoupdate=registry, org.opencontainers.image.version=v2.7.6, PODMAN_SYSTEMD_UNIT=caddy.service, org.opencontainers.image.vendor=Light Code Labs, org.opencontainers.image.documentation=https://caddyserver.com/docs, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker)
1月 20 19:06:57 epyc-server podman[2852822]: 2024-01-20 19:06:57.083795188 +0800 CST m=+0.844090403 container start bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965 (image=quay.io/karuboniru/caddy-cloudflare:latest, name=systemd-caddy, org.opencontainers.image.version=v2.7.6, PODMAN_SYSTEMD_UNIT=caddy.service, io.containers.autoupdate=registry, org.opencontainers.image.source=https://github.com/caddyserver/caddy-docker, org.opencontainers.image.vendor=Light Code Labs, org.opencontainers.image.description=a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go, org.opencontainers.image.documentation=https://caddyserver.com/docs, io.buildah.version=1.33.2, org.opencontainers.image.licenses=Apache-2.0, org.opencontainers.image.url=https://caddyserver.com, org.opencontainers.image.title=Caddy)
1月 20 19:06:57 epyc-server systemd[1]: Started caddy.service - Caddy.
1月 20 19:06:57 epyc-server caddy[2852822]: bc2269819990dbb4795f06776141439de4df25af8c46dc06eeb4a06f15ca6965
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2191367,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"warn","ts":1705748820.2263498,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2348022,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2351968,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000484180"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.235819,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2358527,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.243015,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2439492,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2442906,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.244374,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2443867,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.niconi.org"]}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.247251,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.247272,"msg":"serving initial configuration"}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"warn","ts":1705748820.2475991,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"77f6d13f-5223-435e-a920-90d4c96246c6","try_again":1705835220.2475965,"try_again_in":86399.999999179}
1月 20 19:07:00 epyc-server systemd-caddy[2852943]: {"level":"info","ts":1705748820.2477596,"logger":"tls","msg":"finished cleaning storage units"}

$ curl --unix-socket /run/container-sockets/caddy/caddy.socket 1.niconi.org/echo/ -H "x-forwarded-for: 1.2.3.4"
{
  "path": "/",
  "headers": {
    "host": "1.niconi.org",
    "user-agent": "curl/8.2.1",
    "accept": "*/*",
    "accept-encoding": "gzip"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "1.niconi.org",
  "ip": "::ffff:10.88.0.225",
  "ips": [],
  "protocol": "http",
  "query": {},
  "subdomains": [
    "1"
  ],
  "xhr": false,
  "os": {
    "hostname": "fbac3204243e"
  },
  "connection": {}
}
# The echo server can handle x-forwarded-for
$ curl 10.88.0.224:8080 -H "x-forwarded-for: 1.2.3.4"
{
  "path": "/",
  "headers": {
    "host": "10.88.0.224:8080",
    "user-agent": "curl/8.2.1",
    "accept": "*/*",
    "x-forwarded-for": "1.2.3.4"
  },
  "method": "GET",
  "body": "",
  "fresh": false,
  "hostname": "10.88.0.224",
  "ip": "1.2.3.4",
  "ips": [
    "1.2.3.4"
  ],
  "protocol": "http",
  "query": {},
  "subdomains": [],
  "xhr": false,
  "os": {
    "hostname": "fbac3204243e"
  },
  "connection": {}
}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Docker image built with

FROM docker.io/library/caddy:builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM docker.io/library/caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Running using podman with quadlet, in a selinux confined container. With ports exposed and some unix sockets mounted.

a. System environment:

Fedora release 39 (Thirty Nine)
podman version 4.8.3

b. Command:

See the container configuration below

c. Service/unit/compose file:

Podman quatlet configuration file

[Unit]
Description=Caddy
After=network-online.target
Wants=network-online.target

[Container]
Image=quay.io/karuboniru/caddy-cloudflare:latest
Secret=Caddyfile,target=/etc/caddy/Caddyfile
Volume=CaddyData:/data:Z,U
Volume=CaddyConfig:/config:Z,U

Volume=/run/container-sockets:/run/container-sockets:idmap
Volume=/run/container-sockets/caddy:/run/container-sockets/caddy:idmap,z

Volume=/media/storage:/mnt/share/storage:ro

SecurityLabelType=container_caddy.process

DropCapability=ALL
AddCapability=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
ReadOnly=true
RunInit=true

PublishPort=80:80/tcp
PublishPort=443:443/tcp
PublishPort=443:443/udp

AutoUpdate=registry

[Service]
Restart=on-failure
ExecStartPre=mkdir -p /run/container-sockets/caddy
ExecStopPost=rm -rf /run/container-sockets/caddy

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

&(site) {
  handle_path /qbt/* {
    reverse_proxy http://systemd-qbittorrent.dns.podman:8080 {
      header_down Set-Cookie "path=/" "path=/qbt/ ; SameSite=Strict; Secure"
    }
  }

  handle_path /qbtee/* {
    reverse_proxy http://systemd-qbittorrenteh.dns.podman:8080 {
      header_down Set-Cookie "path=/" "path=/qbtee/ ; SameSite=Strict; Secure"
    }
  }

  handle /cp/* {
    reverse_proxy unix//run/container-sockets/cockpit.socket
  }

  handle /jellyfin/* {
    reverse_proxy http://systemd-jellyfin.dns.podman:8096
  }

  handle_path /adguard/* {
    reverse_proxy http://systemd-adguardhome.dns.podman:80 {
      header_down Location /index.html /adguard/index.html
    }
  }

  handle_path /ab/* {
    reverse_proxy http://systemd-autobangumi.dns.podman:7892
  }
  
  handle_path /file/* {
    file_server browse {
      root /mnt/share
    }
    basicauth {
      !!REDACTED!!
    }
  }

  handle_path /echo/* {
    reverse_proxy http://systemd-echo.dns.podman:8080 
  }

  @ward {
    path / /css* /js* /img* /api* /fonts* /*/ 
  }

  handle @ward {
    reverse_proxy http://systemd-ward.dns.podman:4000
  }

  handle /* {
    redir {scheme}://{host}{uri}/
  }
}

*.niconi.org {
  tls {
    dns cloudflare !!REDACTED!!
  }

  invoke site
}

http://*.niconi.org {
  bind unix//run/container-sockets/caddy/caddy.socket
  invoke site
}

5. Links to relevant resources:

francislavoie · January 20, 2024, 4:18pm

I’m not sure. I’ve not tried that yet.

Add the log directive to your site. What do you see in your logs? What value does remote_ip have?

karuboniru · January 20, 2024, 7:58pm

1月 21 03:57:24 epyc-server systemd-caddy[3051292]: {"level":"debug","ts":1705780644.023336,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"systemd-echo.dns.podman:8080","duration":0.006861339,"request":{"remote_ip":"@","remote_port":"","client_ip":"","proto":"HTTP/1.1","method":"GET","host":"1.niconi.org","uri":"/","headers":{"User-Agent":["curl/8.2.1"],"Accept":["*/*"]}},"headers":{"X-Powered-By":["Express"],"Content-Type":["application/json; charset=utf-8"],"Content-Length":["421"],"Etag":["W/\"1a5-HfO3QhxeuJWkp953J0ctMUD1Ol8\""],"Date":["Sat, 20 Jan 2024 19:57:24 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"]},"status":200}

So it is "@" in this case, which won’t be accepted by the static ip handler of trusted proxy configuration.

francislavoie · January 20, 2024, 8:46pm

Interesting. Good to know.

I’m not sure what the best fix would be. We didn’t design the APIs with that in mind trusted_proxies takes an “IP range source” module (we provide a static one by default) and obviously @ is not an IP, so those modules can’t really ever do anything valid.

I suppose there’s an argument to be made that @ should always be considered trusted. That might be what we should do. I don’t know if there’s any ways that could go wrong or be abused though, so I’m nervous about making that change without more research.

Another thought is we could maybe transform @ to ::1 so that if ::1 is trusted then it passes. That would mean trust is still always opt-in. But ::1 is kinda a lie.

Hard to say which approach is more proper.

system · February 19, 2024, 8:47pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.