Trying to get local docker microservices to get encrypted through bridge into an IPVLAN Layer 3

Help
1

1. Caddy version:

As Docker container in version: v2.6.2

2. How I installed, and run Caddy:

docker run -d -p 80:80 -p 443:443 -p 443:443/udp -v ./data:/data -v ./config:/config -v ./Caddyfile:/etc/caddy/Caddyfile -v ./logs:/var/log/caddy --restart=unless-stopped --name=caddy caddy

a. System environment:

Fedora 37
Docker version 20.10.23, build 7155243
docker-compose version 1.29.2

b. Command:

N/A

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

# global options   
{
	email example@tuta.io
	ocsp_stapling off
	debug
}

#internal
home.lab {
	acme_server
	tls internal
}

#external
example.de {
	root * /usr/share/caddy/
	file_server

	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
}

pass.example.de {
	reverse_proxy 192.168.0.28:8010
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
	reverse_proxy https://home.lab {
		header_up Host {upstream_hostport}
	}
}

code.example.de {
	reverse_proxy 192.168.0.28:8443
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
}

msg.example.de {
	reverse_proxy 192.168.0.28:8090
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
}

health.example.de {
	reverse_proxy 192.168.0.28:3000
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
}

active.example.de {
	reverse_proxy 192.168.0.28:3001
	encode zstd gzip
	header {
		Access-Control-Allow-Origin: *
		Access-Control-Allow-Methods "OPTIONS,HEAD,GET,POST,PUT,PATCH,DELETE"
		Strict-Transport-Security max-age=15552000;
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy no-referrer-when-downgrade
	}
}

nc.example.de {
	rewrite /.well-known/carddav /remote.php/dav
	rewrite /.well-known/caldav /remote.php/dav

	reverse_proxy 192.168.0.28:8070

	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		Referrer-Policy no-referrer-when-downgrade
	}
}

chat.example.de {
	reverse_proxy 192.168.0.28:3030
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=15552000;
		Referrer-Policy no-referrer-when-downgrade
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
	}
}

3. The problem I’m having:

I want to be able to get an ancrypted backend for my database and other microservices in the local network, which is sorted inside a docker IPVLAN Layer 3, 192.168.10.0/24. The caddy server is running on the server inside a container in the bridge network, 192.168.0.28, where all other exposed public services are running. I know, that different networks can’t talk to each other. But I need to get TSL for my local services. It seems that I cannot teach caddy to reach the services when I reverse_proxy them to the other network (home.lab => 192.168.10.0/24)

4. Error messages and/or full log output:

DBG ts=1675436037.7031493 logger=events msg=event name=tls_get_certificate id=278d9e17-6dfb-4fee-9220-b13fe986dbd5 origin=tls data={"client_hello":{"CipherSuites":[4866,4867,4865,49199,49195,49200,49196,158,49191,103,49192,107,163,159,52393,52392,52394,49327,49325,49315,49311,49245,49249,49239,49235,162,49326,49324,49314,49310,49244,49248,49238,49234,49188,106,49187,64,49162,49172,57,56,49161,49171,51,50,157,49313,49309,49233,156,49312,49308,49232,61,60,53,47,255],"ServerName":"pass.example.de","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"Conn":{}}}
DBG ts=1675436037.7032573 logger=tls.handshake msg=choosing certificate identifier=pass.example.de num_choices=1
DBG ts=1675436037.7032902 logger=tls.handshake msg=default certificate selection results identifier=pass.example.de subjects=["pass.example.de"] managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=c6dfa3d3e2437781ea7a93781abd31606934d8e4c579f84ba3e6821b59dd9737
DBG ts=1675436037.7033086 logger=tls.handshake msg=matched certificate in cache remote_ip=178.10.14.75 remote_port=34966 subjects=["pass.example.de"] managed=true expiration=1682712462 hash=c6dfa3d3e2437781ea7a93781abd31606934d8e4c579f84ba3e6821b59dd9737
DBG ts=1675436037.716325 logger=http.handlers.reverse_proxy msg=selected upstream dial=192.168.0.28:8010 total_upstreams=1
DBG ts=1675436037.7181365 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=192.168.0.28:8010 duration=0.001585688 request={"remote_ip":"178.10.14.75","remote_port":"34966","proto":"HTTP/1.1","method":"GET","host":"pass.example.de","uri":"/","headers":{"User-Agent":["Uptime-Kuma/1.19.6"],"X-Forwarded-For":["178.10.14.75"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["pass.example.de"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"pass.example.de"}} headers={"Cache-Control":["public, max-age=600"],"Permissions-Policy":["accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"],"Referrer-Policy":["same-origin"],"Content-Security-Policy":["default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ ;"],"Content-Length":["1240"],"X-Xss-Protection":["0"],"Date":["Fri, 03 Feb 2023 14:53:57 GMT"],"Content-Type":["text/html; charset=utf-8"],"Expires":["Fri, 03 Feb 2023 15:03:57 GMT"],"Server":["Rocket"],"X-Frame-Options":["SAMEORIGIN"],"X-Content-Type-Options":["nosniff"]} status=200
DBG ts=1675436157.8709457 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=192.168.0.28:8010 duration=0.001410911 request={"remote_ip":"178.10.14.75","remote_port":"51842","proto":"HTTP/1.1","method":"GET","host":"pass.example.de","uri":"/","headers":{"X-Forwarded-Host":["pass.example.de"],"X-Forwarded-For":["178.10.14.75"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Uptime-Kuma/1.19.6"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"pass.example.de"}} headers={"X-Frame-Options":["SAMEORIGIN"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["0"],"Content-Security-Policy":["default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ ;"],"Content-Length":["1240"],"Content-Type":["text/html; charset=utf-8"],"Cache-Control":["public, max-age=600"],"Expires":["Fri, 03 Feb 2023 15:05:57 GMT"],"Server":["Rocket"],"Permissions-Policy":["accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"],"Referrer-Policy":["same-origin"],"Date":["Fri, 03 Feb 2023 14:55:57 GMT"]} status=200
DBG ts=1675436217.922109 logger=events msg=event name=tls_get_certificate id=fee3119e-7dbb-4be8-b069-7ce3957d318d origin=tls data={"client_hello":{"CipherSuites":[4866,4867,4865,49199,49195,49200,49196,158,49191,103,49192,107,163,159,52393,52392,52394,49327,49325,49315,49311,49245,49249,49239,49235,162,49326,49324,49314,49310,49244,49248,49238,49234,49188,106,49187,64,49162,49172,57,56,49161,49171,51,50,157,49313,49309,49233,156,49312,49308,49232,61,60,53,47,255],"ServerName":"pass.example.de","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"Conn":{}}}
DBG ts=1675436217.9222195 logger=tls.handshake msg=choosing certificate identifier=pass.example.de num_choices=1
DBG ts=1675436217.922252 logger=tls.handshake msg=default certificate selection results identifier=pass.example.de subjects=["pass.example.de"] managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=c6dfa3d3e2437781ea7a93781abd31606934d8e4c579f84ba3e6821b59dd9737
DBG ts=1675436217.922272 logger=tls.handshake msg=matched certificate in cache remote_ip=178.10.14.75 remote_port=51898 subjects=["pass.example.de"] managed=true expiration=1682712462 hash=c6dfa3d3e2437781ea7a93781abd31606934d8e4c579f84ba3e6821b59dd9737
DBG ts=1675436217.9355536 logger=http.handlers.reverse_proxy msg=selected upstream dial=192.168.0.28:8010 total_upstreams=1
DBG ts=1675436217.9371426 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=192.168.0.28:8010 duration=0.00147247 request={"remote_ip":"178.10.14.75","remote_port":"51898","proto":"HTTP/1.1","method":"GET","host":"pass.example.de","uri":"/","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["pass.example.de"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Uptime-Kuma/1.19.6"],"X-Forwarded-For":["178.10.14.75"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"pass.example.de"}} headers={"Content-Security-Policy":["default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ ;"],"Content-Length":["1240"],"Server":["Rocket"],"X-Frame-Options":["SAMEORIGIN"],"X-Content-Type-Options":["nosniff"],"Referrer-Policy":["same-origin"],"X-Xss-Protection":["0"],"Date":["Fri, 03 Feb 2023 14:56:57 GMT"],"Content-Type":["text/html; charset=utf-8"],"Cache-Control":["public, max-age=600"],"Expires":["Fri, 03 Feb 2023 15:06:57 GMT"],"Permissions-Policy":["accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"]} status=200

5. What I already tried:

Phew, I don’t know how many weeks I tried to get this to work. I am afraid i cannot recall all of this. I tried it with 1 and 2 caddy servers, I tried it with self signed certs and parsing external certs, I tried to run one server in both networks… Please ask if I tried specific things, it was too much to recall it by now. I am sorry

6. Links to relevant resources:

I tried the following, it’s linkages and they didn’t help me unfortunately:

2

Why are you turning off OCSP stapling? You should only do so if you have a very good reason to do so. Having it enabled has numerous benefits, such as Caddy being able to recover quickly from the certificate being revoked (it happens, when ACME issuers have bugs), and to improve browser performance (so clients can avoid having to call out to OCSP responders themselves, by trusting the staple).

Caddy’s default distribution ships with an HTTP server, not a TCP server. That means that you can only handle and proxy HTTP traffic with it by default.

If you need to proxy TCP/UDP, then you could use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy to do so.

3

Can’t remember tbh. I thought, I read that somewhere. Will delete it. Thanks for the input

Where’s the difference? If I want to traffic homer dashboard or syncthing files through the local network, shouldn’t that be possile via http? Granted, database traffic is maybe something different, but the rest should be possible through caddy, shouldn’t it? Excuse my ignorance please, I am still learning this stuff

4

Yep, that’s fine.

Yeah, MySQL and such are TCP, non-HTTP.

5

and to come back to the original question, how do I traffic that through different vlans inside docker via caddy?

6

That’s not something you configure in Caddy. That’s dependent on your Docker networking settings.

That’s out of scope of these forums.

7

You’re right. I confused two differen t things. I try to elaborate:
I want to be able to get encrypted local microservices inside a docker’s IPVLAN Layer 3 (ip range: 192.168.10.0/24) while the bridge services with the ip 192.168.0.28 with a variety of ports (3030, 8080, 8443, etc) get reverse proxied from the public. Caddy runs in the bridge network. How do I get TLS inside the IPVLAN from the bridge network, auto certified / encrypted by caddy. Most of the services are http based. let’s ignore tcp traffic for now

8

I’m confused about the terminology and your goals.

Are you talking about encrypting the traffic from clients (e.g. browsers) to Caddy, or are you talking about the reverse proxy traffic from Caddy to your upstream apps?

If you mean from client to Caddy, then that’s what Caddy automates with Automatic HTTPS, and there’s nothing else to do.

If you mean the proxy traffic to upstreams, then why do you need that to be encrypted? Is your network not isolated? Typically traffic is proxied over HTTP because it has less overhead, and the machines are all in a private network.

What’s your threat model here?

9

Let’s do an example:

  • I want microservice snipptbox to be reachable as https://snip.lab
  • Right now I have Pihole’s dns redirect Ip Address 192.168.10.11 from inside the IPVLAN Layer 3 to snip.lab
  • Snippetbox works on port 5000, while IPVLAN doesn’t care for ports, so I can’t change it to :80
  • To reach this container I have to navigate to http://snip.lab:5000/snippets
  • In the Caddyfile I have this: image
  • I read online that networks can’t reach each other and caddy can’t tunnel through different docker networks
  • The logs are empty after trying to reach https://snip.lab
  • What do I have to do to get https://snip.lab while snippetbox is inside docker’s IPVLAN?
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.