Preserving source IP in rootless Podman network

Help
1

1. The problem I’m having:

I am trying to set up a web server with Caddy and other apps running in rootless Podman containers. However, I’m having trouble with making sure the source IP address is preserved, and I’m not sure if this is a Podman issue or a Caddy issue since I’m new to all of this.

The simplest way I can think to describe the problem is that when I output

<h1><?php echo $_SERVER['REMOTE_ADDR'] . ", " . $_SERVER['HTTP_X_FORWARDED_FOR']?></h1>

on my PHP app, both of the IP addresses match the container that Caddy is running in. However, based on Caddy’s logs, I think it might be seeing the source IP address.

Because I am new to containers and web servers, I don’t fully understand how all the pieces fit together. I don’t expect people here to help debug Podman, so what might be helpful is to learn a reliable way to understand what Caddy is doing so that I can either debug it or move on to debugging other parts of the setup.

2. Error messages and/or full log output:

Here is the output from curl:

gaufde@MacBook-Pro % curl -vL localhost
*   Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://localhost/
< Server: Caddy
< Date: Mon, 02 Sep 2024 17:53:44 GMT
< Content-Length: 0
< 
* Closing connection
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://localhost/'
*   Trying [::1]:443...
* Connected to localhost (::1) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: [NONE]
*  start date: Sep  2 17:27:03 2024 GMT
*  expire date: Sep  3 05:27:03 2024 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: localhost
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/2 200 
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html; charset=UTF-8
< server: Caddy
< x-powered-by: PHP/8.3.10
< content-length: 35
< date: Mon, 02 Sep 2024 17:53:44 GMT
< 
* Connection #1 to host localhost left intact
<h1>192.168.55.5, 192.168.55.5</h1>%

Here are the logs from Caddy:

{"level":"info","ts":1725299930.0155766,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1725299930.0165663,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000502c80"}
{"level":"info","ts":1725299930.0180442,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1725299930.0181675,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":1725299930.0182993,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"subjects":["localhost"]},{}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/usr/share/caddy"}]},{"handle":[{"handler":"static_response","headers":{"Location":["/"]},"status_code":302}],"match":[{"path":["*.txt","*.md","*.mdown","/content/*","/site/*","/kirby/*"]}]},{"handle":[{"handler":"static_response","headers":{"Location":["{http.request.orig_uri.path}/"]},"status_code":308}],"match":[{"file":{"try_files":["{http.request.uri.path}/index.php"]},"not":[{"path":["*/"]}]}]},{"handle":[{"handler":"rewrite","uri":"{http.matchers.file.relative}"}],"match":[{"file":{"split_path":[".php"],"try_files":["{http.request.uri.path}","{http.request.uri.path}/index.php","index.php"]}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"fastcgi","split_path":[".php"]},"upstreams":[{"dial":"phppod:9000"}]}],"match":[{"path":["*.php"]}]},{"handle":[{"handler":"file_server","hide":[".*","/etc/caddy/Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
{"level":"info","ts":1725299930.018634,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1725299930.018916,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"debug","ts":1725299930.0192504,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1725299930.0193887,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1725299930.0195847,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1725299930.0197492,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1725299930.0199049,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["localhost"]}
{"level":"info","ts":1725299930.0203476,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"1a7f331d-8ef6-423a-97c4-13c2d8cbd1f4","try_again":1725386330.0203469,"try_again_in":86399.999999791}
{"level":"info","ts":1725299930.0206177,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":1725299930.0213974,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [localhost]: no OCSP server specified in certificate","identifiers":["localhost"]}
{"level":"debug","ts":1725299930.0214927,"logger":"tls.cache","msg":"added certificate to cache","subjects":["localhost"],"expiration":1725341224,"managed":true,"issuer_key":"local","hash":"9b2b592ce145600131b94ec95612a32d001178fb26d2c6f975b7d78b2fc0af54","cache_size":1,"cache_capacity":10000}
{"level":"debug","ts":1725299930.021589,"logger":"events","msg":"event","name":"cached_managed_cert","id":"b31eb8c8-e57a-4ec3-9503-d0851dd6540b","origin":"tls","data":{"sans":["localhost"]}}
{"level":"info","ts":1725299930.0218549,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1725299930.022059,"msg":"serving initial configuration"}
{"level":"debug","ts":1725299949.8536317,"logger":"events","msg":"event","name":"tls_get_certificate","id":"fa9e8695-475f-4cd0-a56b-7e68ac531783","origin":"tls","data":{"client_hello":{"CipherSuites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"ServerName":"localhost","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2054,1537,1539,2053,1281,1283,2052,1025,1027,513,515],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"192.168.55.5","Port":47322,"Zone":""},"LocalAddr":{"IP":"192.168.55.5","Port":443,"Zone":""}}}}
{"level":"debug","ts":1725299949.8554409,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1725299949.8559809,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"localhost","subjects":["localhost"],"managed":true,"issuer_key":"local","hash":"9b2b592ce145600131b94ec95612a32d001178fb26d2c6f975b7d78b2fc0af54"}
{"level":"debug","ts":1725299949.8565166,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.55.5","remote_port":"47322","subjects":["localhost"],"managed":true,"expiration":1725341224,"hash":"9b2b592ce145600131b94ec95612a32d001178fb26d2c6f975b7d78b2fc0af54"}
{"level":"debug","ts":1725299949.897513,"logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_ip":"192.168.55.5","remote_port":"47322","client_ip":"192.168.55.5","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/","headers":{"Accept":["*/*"],"User-Agent":["curl/8.4.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"localhost"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":1725299949.8985724,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"phppod:9000","total_upstreams":1}
{"level":"debug","ts":1725299949.8993998,"logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_ip":"192.168.55.5","remote_port":"47322","client_ip":"192.168.55.5","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/index.php","headers":{"Accept":["*/*"],"X-Forwarded-For":["192.168.55.5"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["localhost"],"User-Agent":["curl/8.4.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"localhost"}},"env":{"CONTENT_LENGTH":"","PATH_INFO":"","REMOTE_HOST":"192.168.55.5","REQUEST_SCHEME":"https","HTTPS":"on","HTTP_USER_AGENT":"curl/8.4.0","AUTH_TYPE":"","GATEWAY_INTERFACE":"CGI/1.1","REMOTE_ADDR":"192.168.55.5","DOCUMENT_URI":"/index.php","SCRIPT_FILENAME":"/usr/share/caddy/index.php","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","REMOTE_IDENT":"","QUERY_STRING":"","REQUEST_METHOD":"GET","HTTP_X_FORWARDED_FOR":"192.168.55.5","HTTP_X_FORWARDED_PROTO":"https","SERVER_NAME":"localhost","REQUEST_URI":"/","DOCUMENT_ROOT":"/usr/share/caddy","SCRIPT_NAME":"/index.php","SSL_PROTOCOL":"TLSv1.3","CONTENT_TYPE":"","REMOTE_PORT":"47322","REMOTE_USER":"","SERVER_PROTOCOL":"HTTP/2.0","HTTP_HOST":"localhost","SERVER_PORT":"443","SERVER_SOFTWARE":"Caddy/v2.8.4","HTTP_ACCEPT":"*/*","HTTP_X_FORWARDED_HOST":"localhost"},"dial":"phppod:9000","env":{"SCRIPT_NAME":"/index.php","DOCUMENT_ROOT":"/usr/share/caddy","SSL_PROTOCOL":"TLSv1.3","REMOTE_PORT":"47322","REMOTE_USER":"","SERVER_PROTOCOL":"HTTP/2.0","HTTP_HOST":"localhost","SERVER_PORT":"443","CONTENT_TYPE":"","HTTP_ACCEPT":"*/*","HTTP_X_FORWARDED_HOST":"localhost","SERVER_SOFTWARE":"Caddy/v2.8.4","PATH_INFO":"","REMOTE_HOST":"192.168.55.5","REQUEST_SCHEME":"https","HTTPS":"on","CONTENT_LENGTH":"","GATEWAY_INTERFACE":"CGI/1.1","REMOTE_ADDR":"192.168.55.5","DOCUMENT_URI":"/index.php","SCRIPT_FILENAME":"/usr/share/caddy/index.php","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","HTTP_USER_AGENT":"curl/8.4.0","AUTH_TYPE":"","QUERY_STRING":"","REQUEST_METHOD":"GET","HTTP_X_FORWARDED_FOR":"192.168.55.5","HTTP_X_FORWARDED_PROTO":"https","REMOTE_IDENT":"","REQUEST_URI":"/","SERVER_NAME":"localhost"},"request":{"remote_ip":"192.168.55.5","remote_port":"47322","client_ip":"192.168.55.5","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/index.php","headers":{"X-Forwarded-Host":["localhost"],"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.55.5"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"localhost"}}}
{"level":"debug","ts":1725299949.9903283,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"phppod:9000","duration":0.091183966,"request":{"remote_ip":"192.168.55.5","remote_port":"47322","client_ip":"192.168.55.5","proto":"HTTP/2.0","method":"GET","host":"localhost","uri":"/index.php","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.55.5"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["localhost"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"localhost"}},"headers":{"X-Powered-By":["PHP/8.3.10"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}

3. Caddy version:

gaufde@MacBook-Pro % podman exec web caddy --version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

I am using Podman which comes with a machine :

gaufde@MacBook-Pro % podman machine info
host:
    arch: arm64
    currentmachine: podman-machine-default
    defaultmachine: podman-machine-default
    eventsdir: /var/folders/7m/z_f3vkmd6t30fhydp4z513gw0000gn/T/storage-run-501/podman
    machineconfigdir: /Users/gaufde/.config/containers/podman/machine/applehv
    machineimagedir: /Users/gaufde/.local/share/containers/podman/machine/applehv
    machinestate: Running
    numberofmachines: 1
    os: darwin
    vmtype: applehv
version:
    apiversion: 5.2.1
    version: 5.2.1
    goversion: go1.23.0
    gitcommit: d0582c9e1e6c80cc08c3f042b91993c853ddcbc6
    builttime: Wed Aug 14 11:10:32 2024
    built: 1723659032
    osarch: darwin/arm64
    os: darwin

b. Command:

podman network create --subnet 192.168.55.0/24 --gateway 192.168.55.3 --subnet fd52:2a5a:747e:3acd::/64 --gateway fd52:2a5a:747e:3acd::10 kirbynetwork


podman pod create \
	--name caddypod \
	--publish 80:80 \
	--publish 443:443 \
	--network kirbynetwork
	
podman run --detach \
	--name php \
	--pod new:phppod \
	--volume "$(pwd)"/starterkit:/usr/share/caddy:z \
	--network kirbynetwork \
	7553a1b4ba55    


podman run --detach \
	--volume "$(pwd)"/Caddyfile:/etc/caddy/Caddyfile \
	--volume "$(pwd)"/caddy-data:/data \
	--name web \
	--pod caddypod \
	--volumes-from php \
	6fe3572f7c4f

c. Service/unit/compose file:

d. My complete Caddy config:

{
        debug
	log {
		output file /data/log.json
	}

}

(common) {
	php_fastcgi phppod:9000
	tls internal
	file_server {
		hide .*
	}
}

(kirby) {
	@blocked {
		path *.txt *.md *.mdown /content/* /site/* /kirby/*
	}
	redir @blocked /
}


localhost {
	import common
	import kirby
	root * /usr/share/caddy
}

5. Links to relevant resources:

This person has a very helpful guide to networking with rootless Podman. Based on this info, I believe that the way I have the Podman network set up should work: GitHub - eriksjolund/podman-networking-docs: rootless Podman networking documentation with examples

FYI, the default network mode for rootless Podman is pasta since Podman 5.0. Though, I do have a bit of uncertainty about what using podman network create does.

2

I am now suspecting that the issue is probably on the Podman side. Networking in Podman is a bit confusing, but I have found these as the most promising resources:

I think what would help me from the Caddy side of things is to understand the log output a bit better. The first line includes this info that makes it look like Caddy is seeing the original IP address:

"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]

However, the last couple of lines show:

"HTTP_X_FORWARDED_FOR":"192.168.55.5"

Which is not what I would expect if Caddy were able to see the original IP of the request. Can anyone provide some insight on how to read/understand this information?

3

That comes from the log line:

{"level":"info","ts":1725299930.0155766,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}

This is simply telling you that the admin interface started and it’s only listening for local requests (i.e. it won’t respond to API requests from external hosts). It’s not related to the preservation of the original client IP.

Here’s the first Caddy log event relating to these requests - specifically, it’s the log from Caddy trying to select which certificate to present for this incoming request.

A lot of this is TLS-specific stuff, but we can see:

      "RemoteAddr": {
        "IP": "192.168.55.5",
        "Port": 47322,
        "Zone": ""
      },
      "LocalAddr": {
        "IP": "192.168.55.5",
        "Port": 443,
        "Zone": ""
      }

So as far as Caddy is concerned, the source IP is the local IP and I can only assume this is part of the Podman networking. localhost almost certainly wouldn’t work otherwise, since it goes over the loopback address and when containers have their own network stack they definitely don’t share localhost with the host machine.

I suspect this might only be an issue with localhost, though. If you change your site address to some other DNS name and connect to it from another host, you’ll probably get the right source IP information.

1 Like
4

I don’t think this is the case, DNS has nothing to do with the IP address on the TCP connection. Different layers. Matthew clarified with me he meant that you should try to make the request from elsewhere which would definitely make the source IP be different (e.g. connect with your phone on cell networks or something).

This has to be an issue with Podman’s networking. It might be doing some funny business with IP tables or it might be running somekind of userland proxy which makes requests look like they come from the Podman gateway network.

1 Like
5

@francislavoie and @Whitestrake Thank you both for the help! Matthew, thank you for helping me understand Caddy’s logs. That was very helpful since I was able to trust what Caddy was showing me and then debug Podman.

Francis, you are right this is a Podman networking issue. I had thought that I was using a Pasta network, but when you use a podman network command it is actually creating a custom Podman network which can not pass the source IP address in rootless mode.

I got some input from the Podman experts on this thread: Preferred way to set up a reverse proxy web server with Podman containers · containers/podman · Discussion #23845 · GitHub.

It seems like the best ways to use Caddy as a reverse proxy with Podman are to either launch the containers using rootfull Podman but make sure that the containers are isolated from root and from each other using the --userns=auto flag, or to use socket activation in rootless mode.

Socket activation in Caddy is not currently supported, but it is on the near horizon: Implement issue #6296 passing FDs / socket activation by MayCXC · Pull Request #6543 · caddyserver/caddy · GitHub. Once this gets merged, it would be interesting to see how that works. I’m also not sure yet how well the --userns flag works in rootless mode, and that seems to be an important part of isolating containers from each other.

I had thought that rootless mode was one of the key reasons to use Podman, but it seems like it is actually somewhat debatable what the most secure way to run containers is. Dan Walsh seems to be one of the experts here, and he has said:

I have a blog in progress, the problem is it is arguable which is more secure. $ sudo podman run --userns=auto … Runs each containers in a separate user namespace where each container is running with different UIDs and almost assuradly does not match any other UIDs on the host. BUT the podman command runs as root which means it pulls images as root. Pulling images as root can be risky, because flaws in the pulling could (have in the past) cause root to write anywhere. $ podman run … Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

For more information about users and namespaces in Podman, the best resources I’ve found are: