Errors when using Caddy-Docker-Proxy with Cloudflare

Help
1

1. The problem I’m having:

Trying to set up Caddy (specifically caddy-docker-proxy) on a Oracle Cloud Server with Cloudflare as a DNS provider running Portainer. I can get as far as connecting, but get either “Invalid SSL Certificate” or “Client sent an HTTP request to an HTTPS server” errors.

Not exhaustive list of solutions I’ve tried:

  • changing Cloudflare security settings (off/flexible/full/full strict)
  • regenerating Cloudflare token
  • setting tls global setting (logs told me the setting was deprecated)
  • changing portainer port to 8000, 9000, 9443
  • changing tls to internal in portainer’s container

2. Error messages and/or full log output:

2023/11/13 10:47:28.639	INFO	docker-proxy	Running caddy proxy server
2023/11/13 10:47:28.645	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/13 10:47:28.646	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/13 10:47:28.646	INFO	docker-proxy	Running caddy proxy controller
2023/11/13 10:47:28.649	INFO	docker-proxy	Start	{"CaddyfilePath": "", "LabelPrefix": "caddy", "PollingInterval": 30, "ProxyServiceTasks": true, "ProcessCaddyfile": true, "ScanStoppedContainers": true, "IngressNetworks": "[caddy]", "DockerSockets": [""], "DockerCertsPath": [""], "DockerAPIsVersion": [""]}
2023/11/13 10:47:28.651	INFO	docker-proxy	IngressNetworksMap	{"ingres": "map[803ea9311e58d654e0866fe4ece1d18eac01ba9ff5f030092c9f14abf9ab2bf1:true caddy:true]"}
2023/11/13 10:47:28.653	INFO	docker-proxy	Connecting to docker events	{"DockerSocket": ""}
2023/11/13 10:47:28.663	INFO	docker-proxy	Swarm is available	{"new": false}
2023/11/13 10:47:28.674	INFO	docker-proxy	New Caddyfile	{"caddyfile": "{\n\tacme_dns cloudflare {env.CF_API_TOKEN}\n\tdebug\n}\nportainer.bustinbung.com {\n\treverse_proxy 172.30.0.3:9443\n\ttls internal\n}\n"}
2023/11/13 10:47:28.675	INFO	docker-proxy	New Config JSON	{"json": "{\"logging\":{\"logs\":{\"default\":{\"level\":\"DEBUG\"}}},\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"portainer.bustinbung.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.30.0.3:9443\"}]}]}]}],\"terminal\":true}]}}},\"tls\":{\"automation\":{\"policies\":[{\"subjects\":[\"portainer.bustinbung.com\"],\"issuers\":[{\"module\":\"internal\"}]}]}}}}"}
2023/11/13 10:47:28.675	INFO	docker-proxy	Sending configuration to	{"server": "localhost"}
2023/11/13 10:47:28.677	INFO	admin.api	received request	{"method": "POST", "host": "localhost:2019", "uri": "/load", "remote_ip": "127.0.0.1", "remote_port": "54882", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["463"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2023/11/13 10:47:28.679	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/11/13 10:47:28.680	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x40000baf80"}
2023/11/13 10:47:28.687	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/13 10:47:28.688	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/13 10:47:28.688	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{"subjects":["portainer.bustinbung.com"]},{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"172.30.0.3:9443"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
2023/11/13 10:47:28.713	WARN	pki.ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2023/11/13 10:47:28.713	INFO	warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
2023/11/13 10:47:28.713	INFO	define JAVA_HOME environment variable to use the Java trust
2023/11/13 10:47:28.792	INFO	certificate installed properly in linux trusts
2023/11/13 10:47:28.793	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/13 10:47:28.793	INFO	tls	cleaning storage unit	{"description": "FileStorage:/data/caddy"}
2023/11/13 10:47:28.793	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/11/13 10:47:28.794	DEBUG	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2023/11/13 10:47:28.794	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/13 10:47:28.795	DEBUG	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2023/11/13 10:47:28.795	INFO	tls	finished cleaning storage units
2023/11/13 10:47:28.795	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/13 10:47:28.795	INFO	http	enabling automatic TLS certificate management	{"domains": ["portainer.bustinbung.com"]}
2023/11/13 10:47:28.798	WARN	tls	stapling OCSP	{"error": "no OCSP stapling for [portainer.bustinbung.com]: no OCSP server specified in certificate", "identifiers": ["portainer.bustinbung.com"]}
2023/11/13 10:47:28.798	DEBUG	tls.cache	added certificate to cache	{"subjects": ["portainer.bustinbung.com"], "expiration": "2023/11/13 22:09:59.000", "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90", "cache_size": 1, "cache_capacity": 10000}
2023/11/13 10:47:28.798	DEBUG	events	event	{"name": "cached_managed_cert", "id": "a2ab5ca2-5197-453b-aed2-dfb144133f49", "origin": "tls", "data": {"sans":["portainer.bustinbung.com"]}}
2023/11/13 10:47:28.799	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/13 10:47:28.799	INFO	admin.api	load complete
2023/11/13 10:47:28.800	INFO	docker-proxy	Successfully configured	{"server": "localhost"}
2023/11/13 10:47:28.824	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/13 10:47:40.749	DEBUG	events	event	{"name": "tls_get_certificate", "id": "98cbcaee-4767-4ab3-a2eb-c1853fabf857", "origin": "tls", "data": {"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[14906,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[39578,772,771],"Conn":{}}}}
2023/11/13 10:47:40.750	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:40.750	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:40.750	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "43019", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:40.775	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:43019: remote error: tls: unknown certificate
2023/11/13 10:47:42.055	DEBUG	events	event	{"name": "tls_get_certificate", "id": "9bacedb5-281f-4ebf-9fb6-dd4375d36226", "origin": "tls", "data": {"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[27242,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[31354,772,771],"Conn":{}}}}
2023/11/13 10:47:42.055	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:42.055	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:42.056	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "50559", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:42.077	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:50559: remote error: tls: unknown certificate
2023/11/13 10:47:43.939	DEBUG	events	event	{"name": "tls_get_certificate", "id": "ce7e364c-1330-4a9a-9a00-76f84a38c95c", "origin": "tls", "data": {"client_hello":{"CipherSuites":[27242,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[23130,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771],"Conn":{}}}}
2023/11/13 10:47:43.939	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:43.939	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:43.939	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "4809", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:43.959	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:4809: remote error: tls: unknown certificate
2023/11/13 10:47:53.518	DEBUG	events	event	{"name": "tls_get_certificate", "id": "0d7c652a-df01-4d3b-a88e-b89a1a7ed06c", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,173,171,52398,52397,52396,157,169,52395,172,170,156,168,61,60,49208,49206,183,179,149,145,53,175,141,49207,49205,182,178,148,144,47,174,140,255],"ServerName":"cloud.bustinbung.com","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:53.519	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "cloud.bustinbung.com"}
2023/11/13 10:47:53.519	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.bustinbung.com"}
2023/11/13 10:47:53.519	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.com"}
2023/11/13 10:47:53.519	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.*"}
2023/11/13 10:47:53.519	DEBUG	tls.handshake	no certificate matching TLS ClientHello	{"remote_ip": "198.137.18.254", "remote_port": "32159", "server_name": "cloud.bustinbung.com", "remote": "198.137.18.254:32159", "identifier": "cloud.bustinbung.com", "cipher_suites": [4866, 4867, 4865, 49196, 49200, 159, 52393, 52392, 52394, 49195, 49199, 158, 49188, 49192, 107, 49187, 49191, 103, 49162, 49172, 57, 49161, 49171, 51, 173, 171, 52398, 52397, 52396, 157, 169, 52395, 172, 170, 156, 168, 61, 60, 49208, 49206, 183, 179, 149, 145, 53, 175, 141, 49207, 49205, 182, 178, 148, 144, 47, 174, 140, 255], "cert_cache_fill": 0.0001, "load_or_obtain_if_necessary": true, "on_demand": false}
2023/11/13 10:47:53.519	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:32159: no certificate available for 'cloud.bustinbung.com'
2023/11/13 10:47:53.519	DEBUG	events	event	{"name": "tls_get_certificate", "id": "eac6c24a-24ab-416a-bd11-63b391859927", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,173,171,52398,52397,52396,157,169,52395,172,170,156,168,61,60,49208,49206,183,179,149,145,53,175,141,49207,49205,182,178,148,144,47,174,140,255],"ServerName":"cloud.bustinbung.com","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:53.520	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "cloud.bustinbung.com"}
2023/11/13 10:47:53.520	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.bustinbung.com"}
2023/11/13 10:47:53.520	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.com"}
2023/11/13 10:47:53.520	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.*"}
2023/11/13 10:47:53.520	DEBUG	tls.handshake	no certificate matching TLS ClientHello	{"remote_ip": "198.137.18.254", "remote_port": "41285", "server_name": "cloud.bustinbung.com", "remote": "198.137.18.254:41285", "identifier": "cloud.bustinbung.com", "cipher_suites": [4866, 4867, 4865, 49196, 49200, 159, 52393, 52392, 52394, 49195, 49199, 158, 49188, 49192, 107, 49187, 49191, 103, 49162, 49172, 57, 49161, 49171, 51, 173, 171, 52398, 52397, 52396, 157, 169, 52395, 172, 170, 156, 168, 61, 60, 49208, 49206, 183, 179, 149, 145, 53, 175, 141, 49207, 49205, 182, 178, 148, 144, 47, 174, 140, 255], "cert_cache_fill": 0.0001, "load_or_obtain_if_necessary": true, "on_demand": false}
2023/11/13 10:47:53.520	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:41285: no certificate available for 'cloud.bustinbung.com'
2023/11/13 10:47:53.564	DEBUG	events	event	{"name": "tls_get_certificate", "id": "fb33f0f0-fc54-4e68-b8e3-e42aa1bb4c83", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,173,171,52398,52397,52396,157,169,52395,172,170,156,168,61,60,49208,49206,183,179,149,145,53,175,141,49207,49205,182,178,148,144,47,174,140,255],"ServerName":"cloud.bustinbung.com","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:53.564	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "cloud.bustinbung.com"}
2023/11/13 10:47:53.564	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.bustinbung.com"}
2023/11/13 10:47:53.564	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.com"}
2023/11/13 10:47:53.564	DEBUG	tls.handshake	no matching certificates and no custom selection logic	{"identifier": "*.*.*"}
2023/11/13 10:47:53.564	DEBUG	tls.handshake	no certificate matching TLS ClientHello	{"remote_ip": "198.137.18.254", "remote_port": "35778", "server_name": "cloud.bustinbung.com", "remote": "198.137.18.254:35778", "identifier": "cloud.bustinbung.com", "cipher_suites": [4866, 4867, 4865, 49196, 49200, 159, 52393, 52392, 52394, 49195, 49199, 158, 49188, 49192, 107, 49187, 49191, 103, 49162, 49172, 57, 49161, 49171, 51, 173, 171, 52398, 52397, 52396, 157, 169, 52395, 172, 170, 156, 168, 61, 60, 49208, 49206, 183, 179, 149, 145, 53, 175, 141, 49207, 49205, 182, 178, 148, 144, 47, 174, 140, 255], "cert_cache_fill": 0.0001, "load_or_obtain_if_necessary": true, "on_demand": false}
2023/11/13 10:47:53.564	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:35778: no certificate available for 'cloud.bustinbung.com'
2023/11/13 10:47:55.194	DEBUG	events	event	{"name": "tls_get_certificate", "id": "95a38a77-b868-458e-aa1b-54e8cf61a5f7", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"portainer.bustinbung.com","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:55.195	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:55.195	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.195	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "172.69.58.68", "remote_port": "21590", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.215	DEBUG	http.stdlib	http: TLS handshake error from 172.69.58.68:21590: remote error: tls: unknown certificate authority
2023/11/13 10:47:55.767	DEBUG	events	event	{"name": "tls_get_certificate", "id": "280ed439-9f9c-483f-905f-2320e96d65fe", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"portainer.bustinbung.com","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:55.767	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:55.767	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.768	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "172.69.58.69", "remote_port": "19616", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.791	DEBUG	http.stdlib	http: TLS handshake error from 172.69.58.69:19616: remote error: tls: unknown certificate authority
2023/11/13 10:47:55.806	DEBUG	events	event	{"name": "tls_get_certificate", "id": "58e48974-5e75-4c22-8351-48137a20e924", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"portainer.bustinbung.com","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
2023/11/13 10:47:55.806	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:47:55.806	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.806	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "172.69.58.234", "remote_port": "20432", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:47:55.828	DEBUG	http.stdlib	http: TLS handshake error from 172.69.58.234:20432: remote error: tls: unknown certificate authority
2023/11/13 10:47:58.650	DEBUG	docker-proxy	Skipping default Caddyfile because no path is set
2023/11/13 10:47:58.650	DEBUG	docker-proxy	Skipping swarm config caddyfiles because swarm is not available
2023/11/13 10:47:58.654	DEBUG	docker-proxy	Skipping swarm services because swarm is not available
2023/11/13 10:48:28.651	DEBUG	docker-proxy	Skipping default Caddyfile because no path is set
2023/11/13 10:48:28.651	DEBUG	docker-proxy	Skipping swarm config caddyfiles because swarm is not available
2023/11/13 10:48:28.664	DEBUG	docker-proxy	Skipping swarm services because swarm is not available
2023/11/13 10:48:40.312	DEBUG	events	event	{"name": "tls_get_certificate", "id": "5220dac9-1d25-4303-b004-8a03797582f8", "origin": "tls", "data": {"client_hello":{"CipherSuites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[19018,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771],"Conn":{}}}}
2023/11/13 10:48:40.313	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:40.313	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:40.313	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "18841", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:40.335	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:18841: remote error: tls: unknown certificate
2023/11/13 10:48:40.889	DEBUG	events	event	{"name": "tls_get_certificate", "id": "06ab5df9-f9b7-435b-8a4a-2c0aac982833", "origin": "tls", "data": {"client_hello":{"CipherSuites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[19018,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"Conn":{}}}}
2023/11/13 10:48:40.889	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:40.889	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:40.889	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "51321", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:40.910	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:51321: remote error: tls: unknown certificate
2023/11/13 10:48:41.070	DEBUG	events	event	{"name": "tls_get_certificate", "id": "ac233d3d-e593-44b5-ac1f-016a806eea82", "origin": "tls", "data": {"client_hello":{"CipherSuites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[31354,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[35466,772,771],"Conn":{}}}}
2023/11/13 10:48:41.070	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:41.070	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.070	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "2287", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.092	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:2287: remote error: tls: unknown certificate
2023/11/13 10:48:41.265	DEBUG	events	event	{"name": "tls_get_certificate", "id": "2ed45cdc-2216-47bc-b79e-28127d8ce664", "origin": "tls", "data": {"client_hello":{"CipherSuites":[43690,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[27242,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[10794,772,771],"Conn":{}}}}
2023/11/13 10:48:41.265	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:41.265	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.265	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "28584", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.285	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:28584: remote error: tls: unknown certificate
2023/11/13 10:48:41.480	DEBUG	events	event	{"name": "tls_get_certificate", "id": "660c90da-5950-480a-9d60-5a64c90f7fba", "origin": "tls", "data": {"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[56026,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"Conn":{}}}}
2023/11/13 10:48:41.481	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:41.481	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.481	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "8360", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.502	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:8360: remote error: tls: unknown certificate
2023/11/13 10:48:41.688	DEBUG	events	event	{"name": "tls_get_certificate", "id": "ff7f43a7-53d2-4ef2-a97a-3badb5387815", "origin": "tls", "data": {"client_hello":{"CipherSuites":[60138,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[43690,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771],"Conn":{}}}}
2023/11/13 10:48:41.688	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
2023/11/13 10:48:41.688	DEBUG	tls.handshake	default certificate selection results	{"identifier": "portainer.bustinbung.com", "subjects": ["portainer.bustinbung.com"], "managed": true, "issuer_key": "local", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.688	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "198.137.18.254", "remote_port": "9994", "subjects": ["portainer.bustinbung.com"], "managed": true, "expiration": "2023/11/13 22:09:59.000", "hash": "9e5e4137dec059a75443769a051134d3af19716478d60aa942e6a45c898d0b90"}
2023/11/13 10:48:41.710	DEBUG	http.stdlib	http: TLS handshake error from 198.137.18.254:9994: remote error: tls: unknown certificate
2023/11/13 10:48:41.844	DEBUG	events	event	{"name": "tls_get_certificate", "id": "521e47c0-fb34-4c7c-9620-28d4b55c7434", "origin": "tls", "data": {"client_hello":{"CipherSuites":[23130,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"portainer.bustinbung.com","SupportedCurves":[23130,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[2570,772,771],"Conn":{}}}}
2023/11/13 10:48:41.845	DEBUG	tls.handshake	choosing certificate	{"identifier": "portainer.bustinbung.com", "num_choices": 1}
/// cut for character limit ///

3. Caddy version:

caddy v2.7.5
caddy-docker-proxy v2.8.9
portainer v2.19.2

4. How I installed and ran Caddy:

I created a custom Caddy image using Dockerfiles. The image is available on Docker Hub here. I’ve configured Caddy through Portainer by connecting directly to the web UI with the server IP.

a. System environment:

OCI Ampere instance (aarch64)
Ubuntu 22.04.3 LTS
Docker v24.0.7

b. Command:

Run through Portainer

c. Service/unit/compose file:

d. My complete Caddy config:

Configured via labels through caddy-docker-proxy

# in labels
caddy.acme_dns=cloudflare {env.CF_API_TOKEN}

# in environment variables
CF_API_TOKEN=<API Token with Zone:Read and DNS:Edit permissions>

5. Links to relevant resources:

2

You have tls internal in here, so Caddy is issuing certs using its internal CA instead of with Let’s Encrypt or ZeroSSL, so the certs it serves aren’t trusted by anyone.

3

I tried using a different image of caddy, without caddy-docker-proxy, but still using the cloudflare plugin. I’ve attached the relevant info below. Still getting the “Bad Gateway” error, and not sure what’s happening.

Docker run command:

docker run -it -d --name caddy	\
    -p 80:80	\
    -p 443:443	\
    -v caddy_data:/data	\
    -v caddy_config:/config	\
    -v $PWD/Caddyfile:/etc/caddy/Caddyfile	\
    -e CLOUDFLARE_EMAIL=justin@bustinbung.com	\
    -e CLOUDFLARE_API_TOKEN=<API TOKEN>	\
    -e ACME_AGREE=true	\
    slothcroissant/caddy-cloudflaredns:latest

Caddyfile:

*.bustinbung.com {
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}

	@portainer host portainer.bustinbung.com
	handle @portainer {
		reverse_proxy https://localhost:9443
	}

	handle {
		abort
	}
}

Output to autosave.json (formatted):

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "transport": {
                                    "protocol": "http",
                                    "tls": {}
                                  },
                                  "upstreams": [
                                    {
                                      "dial": "localhost:9443"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "host": [
                            "portainer.bustinbung.com"
                          ]
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "abort": true,
                                  "handler": "static_response"
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "*.bustinbung.com"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}",
                      "name": "cloudflare"
                    }
                  }
                },
                "module": "acme"
              },
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.CLOUDFLARE_API_TOKEN}",
                      "name": "cloudflare"
                    }
                  }
                },
                "module": "zerossl"
              }
            ],
            "subjects": [
              "*.bustinbung.com"
            ]
          }
        ]
      }
    }
  }
}

Portainer Docker command:

docker run -d -it --name portainer	\
    -p 9443:9443	\
    --restart=always	\
    -v /var/run/docker.sock:/var/run/docker.sock:ro	\
    -v portainer_data:/data	\
    portainer/portainer-ce:latest
4

And relevant logs:

Logs from Caddy container:

2023/11/13 20:40:23.908	INFO	using provided configuration	{"config_file": "/etc/caddy/Caddyfile", "config_adapter": "caddyfile"}
2023/11/13 20:40:23.926	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/13 20:40:23.928	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x40004be800"}
2023/11/13 20:40:23.928	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/13 20:40:23.928	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/13 20:40:23.930	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/13 20:40:23.930	INFO	tls	cleaning storage unit	{"description": "FileStorage:/data/caddy"}
2023/11/13 20:40:23.932	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/11/13 20:40:23.932	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/13 20:40:23.933	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/13 20:40:23.933	INFO	http	enabling automatic TLS certificate management	{"domains": ["*.bustinbung.com"]}
2023/11/13 20:40:23.940	INFO	tls	finished cleaning storage units
2023/11/13 20:40:23.944	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/13 20:40:23.944	INFO	serving initial configuration
2023/11/13 20:40:46.791	ERROR	http.log.error	dial tcp 127.0.0.1:9443: connect: connection refused	{"request": {"remote_ip": "172.71.254.84", "remote_port": "13794", "client_ip": "172.71.254.84", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"Cache-Control": ["max-age=0"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Encoding": ["gzip"], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Site": ["none"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "X-Forwarded-For": ["198.137.18.12"], "Cf-Ray": ["8259d4ebde002226-ORD"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Accept-Language": ["en-US,en;q=0.9"], "Cf-Connecting-Ip": ["198.137.18.12"], "Cdn-Loop": ["cloudflare"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Sec-Fetch-Mode": ["navigate"], "Priority": ["u=0, i"], "Cf-Ipcountry": ["US"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.009275984, "status": 502, "err_id": "p81is10cw", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/13 20:42:05.917	ERROR	http.log.error	dial tcp 127.0.0.1:9443: connect: connection refused	{"request": {"remote_ip": "172.70.126.154", "remote_port": "45490", "client_ip": "172.70.126.154", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"User-Agent": ["curl/8.1.2"], "Cdn-Loop": ["cloudflare"], "Cf-Connecting-Ip": ["198.137.18.12"], "Accept-Encoding": ["gzip"], "X-Forwarded-For": ["198.137.18.12"], "Cf-Ray": ["8259d6da68d32940-ORD"], "X-Forwarded-Proto": ["https"], "Cf-Ipcountry": ["US"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.003956033, "status": 502, "err_id": "ap26tj0i1", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/13 21:18:21.078	ERROR	http.log.error	dial tcp 127.0.0.1:9443: connect: connection refused	{"request": {"remote_ip": "172.68.10.165", "remote_port": "56564", "client_ip": "172.68.10.165", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"X-Forwarded-For": ["5.164.29.116"], "Cf-Ray": ["825a0bf358ed9d87-DME"], "X-Forwarded-Proto": ["https"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Accept-Encoding": ["gzip"], "Cf-Ipcountry": ["RU"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 (scanner.ducks.party)"], "Cdn-Loop": ["cloudflare"], "Cf-Connecting-Ip": ["5.164.29.116"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.018274485, "status": 502, "err_id": "572akg0b1", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/13 21:18:22.316	ERROR	http.log.error	dial tcp 127.0.0.1:9443: connect: connection refused	{"request": {"remote_ip": "172.68.10.188", "remote_port": "12956", "client_ip": "172.68.10.188", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/favicon.ico", "headers": {"Cf-Connecting-Ip": ["5.164.29.116"], "Cf-Ipcountry": ["RU"], "X-Forwarded-For": ["5.164.29.116"], "Cf-Ray": ["825a0bfb79e79d87-DME"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 (scanner.ducks.party)"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Cdn-Loop": ["cloudflare"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.00042544, "status": 502, "err_id": "jbupqiyv2", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/13 23:05:10.880	ERROR	http.log.error	dial tcp 127.0.0.1:9443: connect: connection refused	{"request": {"remote_ip": "172.71.254.27", "remote_port": "50864", "client_ip": "172.71.254.27", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"Sec-Fetch-Dest": ["document"], "Priority": ["u=0, i"], "Accept-Encoding": ["gzip"], "Cf-Ray": ["825aa872691022cd-ORD"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Dnt": ["1"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9"], "Cf-Ipcountry": ["US"], "X-Forwarded-Proto": ["https"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Sec-Ch-Ua-Mobile": ["?0"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Cf-Connecting-Ip": ["198.137.18.12"], "Upgrade-Insecure-Requests": ["1"], "Cdn-Loop": ["cloudflare"], "X-Forwarded-For": ["198.137.18.12"], "Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.017597448, "status": 502, "err_id": "mnzj3y8m6", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
5

The problem wasn’t with your image, it was with your labels (which you didn’t share fully). The tls internal config must have come from your labels. All you needed to do is remove that.

This wouldn’t work because localhost inside a container is “this same container”. You need to use the name of the other Docker container to route to it using Docker’s DNS resolver.

And proxying over HTTPS to another container typically has issues, because Caddy needs to trust the certificate served by the upstream. I strongly suggest using the HTTP port of the upstream instead.

You can remove this completely because Caddy connects to Portainer through the Docker network, you don’t need to expose Portainer to your host machine.

Remove this, it was only relevant for Caddy v1 and older.

6

The problem wasn’t with your image, it was with your labels (which you didn’t share fully). The tls internal config must have come from your labels. All you needed to do is remove that.

I’ve gone ahead and reverted back to my own built image (caddy-proxy-cloudflare) based on your information here.

With the other points you mentioned, I’ll go ahead and share my new configs (labels included). I’m still getting “connection refused” errors.

caddy-docker-proxy run command

docker run -it -d --name caddy  \
    -p 80:80    \
    -p 443:443  \
    -v caddy_data:/data \
    -v /var/run/docker.sock:/var/run/docker.sock        \
    --net caddy \
    -e CADDY_INGRESS_NETWORKS=caddy     \
    # not sure if this email env is needed, but kept
    # anyways just in case
    -e CLOUDFLARE_EMAIL=justin@bustinbung.com   \
    -e CLOUDFLARE_API_TOKEN=<cloudflare_api_token>    \
    --restart unless-stopped    \
    -l caddy.acme_dns="cloudflare {env.CLOUDFLARE_API_TOKEN}"   \
    bustinbung/caddy-proxy-cloudflare:2.7.5

portainer run command

docker run -d -it --name portainer	\
    # keeping https port in case I need to access directly if
    # caddy goes down(?) not sure really
    -p 9443:9443	\
    -p 9000:9000	\
    --restart=always	\
    -v /var/run/docker.sock:/var/run/docker.sock:ro	\
    -v portainer_data:/data	\
    --net caddy	\
    -l caddy=portainer.bustinbung.com	\
    # have also tried {{upstreams 9000}}
    -l caddy.reverse_proxy="http://portainer:9000"	\
    portainer/portainer-ce:latest

caddy error

2023/11/24 00:52:22.624	INFO	docker-proxy	Running caddy proxy server
2023/11/24 00:52:22.626	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/24 00:52:22.626	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/24 00:52:22.626	INFO	docker-proxy	Running caddy proxy controller
2023/11/24 00:52:22.629	INFO	docker-proxy	Start	{"CaddyfilePath": "", "LabelPrefix": "caddy", "PollingInterval": 30, "ProxyServiceTasks": true, "ProcessCaddyfile": true, "ScanStoppedContainers": true, "IngressNetworks": "[caddy]", "DockerSockets": [""], "DockerCertsPath": [""], "DockerAPIsVersion": [""]}
2023/11/24 00:52:22.630	INFO	docker-proxy	Connecting to docker events	{"DockerSocket": ""}
2023/11/24 00:52:22.631	INFO	docker-proxy	IngressNetworksMap	{"ingres": "map[843909078e4052c481e44539764eb0b7f848ff9b8529366136e4a93042019819:true caddy:true]"}
2023/11/24 00:52:22.640	INFO	docker-proxy	Swarm is available	{"new": false}
2023/11/24 00:52:22.650	INFO	docker-proxy	New Caddyfile	{"caddyfile": "{\n\tacme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}\n}\nportainer.bustinbung.com {\n\treverse_proxy http://portainer:9000\n}\n"}
2023/11/24 00:52:22.651	INFO	docker-proxy	New Config JSON	{"json": "{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"portainer.bustinbung.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"portainer:9000\"}]}]}]}],\"terminal\":true}]}}},\"tls\":{\"automation\":{\"policies\":[{\"subjects\":[\"portainer.bustinbung.com\"],\"issuers\":[{\"challenges\":{\"dns\":{\"provider\":{\"api_token\":\"{env.CLOUDFLARE_API_TOKEN}\",\"name\":\"cloudflare\"}}},\"module\":\"acme\"},{\"challenges\":{\"dns\":{\"provider\":{\"api_token\":\"{env.CLOUDFLARE_API_TOKEN}\",\"name\":\"cloudflare\"}}},\"module\":\"zerossl\"}]}]}}}}"}
2023/11/24 00:52:22.651	INFO	docker-proxy	Sending configuration to	{"server": "localhost"}
2023/11/24 00:52:22.652	INFO	admin.api	received request	{"method": "POST", "host": "localhost:2019", "uri": "/load", "remote_ip": "127.0.0.1", "remote_port": "56824", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["624"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2023/11/24 00:52:22.653	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/24 00:52:22.653	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x400049ea00"}
2023/11/24 00:52:22.654	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/24 00:52:22.654	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/24 00:52:22.655	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/24 00:52:22.655	INFO	tls	cleaning storage unit	{"description": "FileStorage:/data/caddy"}
2023/11/24 00:52:22.655	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/11/24 00:52:22.655	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/24 00:52:22.655	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/24 00:52:22.655	INFO	http	enabling automatic TLS certificate management	{"domains": ["portainer.bustinbung.com"]}
2023/11/24 00:52:22.656	INFO	tls	finished cleaning storage units
2023/11/24 00:52:22.657	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/24 00:52:22.657	INFO	admin.api	load complete
2023/11/24 00:52:22.657	INFO	docker-proxy	Successfully configured	{"server": "localhost"}
2023/11/24 00:52:22.662	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/24 00:52:45.408	ERROR	http.log.error	dial tcp 192.168.16.3:9000: connect: connection refused	{"request": {"remote_ip": "172.70.39.116", "remote_port": "28934", "client_ip": "172.70.39.116", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"Priority": ["u=0, i"], "Cdn-Loop": ["cloudflare"], "X-Forwarded-For": ["173.53.89.32"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Site": ["none"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Mode": ["navigate"], "Accept-Language": ["en-US,en;q=0.9"], "Accept-Encoding": ["gzip"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "Sec-Fetch-User": ["?1"], "Cf-Ipcountry": ["US"], "Cf-Ray": ["82adabc7afaf3973-IAD"], "X-Forwarded-Proto": ["https"], "Dnt": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["document"], "Cf-Connecting-Ip": ["173.53.89.32"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Sec-Ch-Ua-Mobile": ["?0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.003473915, "status": 502, "err_id": "xystjb37a", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/24 00:52:45.482	ERROR	http.log.error	dial tcp 192.168.16.3:9000: connect: connection refused	{"request": {"remote_ip": "172.70.39.8", "remote_port": "62622", "client_ip": "172.70.39.8", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/favicon.ico", "headers": {"Cf-Connecting-Ip": ["173.53.89.32"], "X-Forwarded-For": ["173.53.89.32"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Sec-Fetch-Dest": ["image"], "Priority": ["u=1, i"], "Cf-Ipcountry": ["US"], "Accept-Encoding": ["gzip"], "Cf-Ray": ["82adabc828473973-IAD"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "Dnt": ["1"], "Sec-Ch-Ua-Mobile": ["?0"], "Referer": ["https://portainer.bustinbung.com/"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Accept-Language": ["en-US,en;q=0.9"], "Cdn-Loop": ["cloudflare"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.001160758, "status": 502, "err_id": "uc4t1tx1q", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/24 01:01:21.883	INFO	docker-proxy	New Caddyfile	{"caddyfile": "{\n\tacme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}\n}\n"}
2023/11/24 01:01:21.884	INFO	docker-proxy	New Config JSON	{"json": "{}"}
2023/11/24 01:01:21.884	INFO	docker-proxy	Sending configuration to	{"server": "localhost"}
2023/11/24 01:01:21.887	INFO	admin.api	received request	{"method": "POST", "host": "localhost:2019", "uri": "/load", "remote_ip": "127.0.0.1", "remote_port": "50456", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["41"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2023/11/24 01:01:21.892	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/24 01:01:21.893	INFO	http	servers shutting down with eternal grace period
2023/11/24 01:01:21.896	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0x400049ea00"}
2023/11/24 01:01:21.899	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/24 01:01:21.899	INFO	admin.api	load complete
2023/11/24 01:01:21.899	INFO	docker-proxy	Successfully configured	{"server": "localhost"}
2023/11/24 01:01:21.903	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/24 01:01:24.989	INFO	docker-proxy	New Caddyfile	{"caddyfile": "{\n\tacme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}\n}\nportainer.bustinbung.com {\n\treverse_proxy portainer:9000\n}\n"}
2023/11/24 01:01:24.989	INFO	docker-proxy	New Config JSON	{"json": "{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"portainer.bustinbung.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"portainer:9000\"}]}]}]}],\"terminal\":true}]}}},\"tls\":{\"automation\":{\"policies\":[{\"subjects\":[\"portainer.bustinbung.com\"],\"issuers\":[{\"challenges\":{\"dns\":{\"provider\":{\"api_token\":\"{env.CLOUDFLARE_API_TOKEN}\",\"name\":\"cloudflare\"}}},\"module\":\"acme\"},{\"challenges\":{\"dns\":{\"provider\":{\"api_token\":\"{env.CLOUDFLARE_API_TOKEN}\",\"name\":\"cloudflare\"}}},\"module\":\"zerossl\"}]}]}}}}"}
2023/11/24 01:01:24.990	INFO	docker-proxy	Sending configuration to	{"server": "localhost"}
2023/11/24 01:01:24.991	INFO	admin.api	received request	{"method": "POST", "host": "localhost:2019", "uri": "/load", "remote_ip": "127.0.0.1", "remote_port": "35648", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["624"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2023/11/24 01:01:24.991	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/11/24 01:01:24.993	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x40002eec00"}
2023/11/24 01:01:24.996	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/24 01:01:24.996	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/24 01:01:25.002	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/24 01:01:25.003	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/24 01:01:25.003	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/24 01:01:25.003	INFO	http	enabling automatic TLS certificate management	{"domains": ["portainer.bustinbung.com"]}
2023/11/24 01:01:25.008	INFO	autosaved config (load with --resume flag)	{"file": "/config/caddy/autosave.json"}
2023/11/24 01:01:25.008	INFO	admin.api	load complete
2023/11/24 01:01:25.010	INFO	docker-proxy	Successfully configured	{"server": "localhost"}
2023/11/24 01:01:25.010	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/24 01:01:28.300	ERROR	http.log.error	dial tcp 192.168.16.3:9000: connect: connection refused	{"request": {"remote_ip": "172.70.175.12", "remote_port": "55674", "client_ip": "172.70.175.12", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/", "headers": {"Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["none"], "Priority": ["u=0, i"], "Cf-Connecting-Ip": ["173.53.89.32"], "Accept-Encoding": ["gzip"], "Cf-Ray": ["82adb88bbd939c24-IAD"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Accept-Language": ["en-US,en;q=0.9"], "Cdn-Loop": ["cloudflare"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua-Mobile": ["?0"], "Upgrade-Insecure-Requests": ["1"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Dnt": ["1"], "Cache-Control": ["max-age=0"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "Cf-Ipcountry": ["US"], "X-Forwarded-For": ["173.53.89.32"], "Cf-Visitor": ["{\"scheme\":\"https\"}"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.005399272, "status": 502, "err_id": "6m6wjue91", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}
2023/11/24 01:01:28.385	ERROR	http.log.error	dial tcp 192.168.16.3:9000: connect: connection refused	{"request": {"remote_ip": "172.70.175.149", "remote_port": "34694", "client_ip": "172.70.175.149", "proto": "HTTP/2.0", "method": "GET", "host": "portainer.bustinbung.com", "uri": "/favicon.ico", "headers": {"Cf-Ipcountry": ["US"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Sec-Fetch-Mode": ["no-cors"], "Referer": ["https://portainer.bustinbung.com/"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Dest": ["image"], "Sec-Ch-Ua": ["\"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\""], "Accept-Language": ["en-US,en;q=0.9"], "Cf-Connecting-Ip": ["173.53.89.32"], "Accept-Encoding": ["gzip"], "X-Forwarded-For": ["173.53.89.32"], "X-Forwarded-Proto": ["https"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Cf-Ray": ["82adb88c5e659c24-IAD"], "Dnt": ["1"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Priority": ["u=1, i"], "Cdn-Loop": ["cloudflare"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "portainer.bustinbung.com"}}, "duration": 0.000993839, "status": 502, "err_id": "cgyux14ep", "err_trace": "reverseproxy.statusError (reverseproxy.go:1265)"}

I believe this should cover everything, but if you need more info, please let me know, I’d be happy to provide it. Thanks for your help!

7

Yeah you’re getting a lot further now, and the two containers are in the same network because Caddy reloaded its config with the labels from your portainer container.

What’s in your portainer logs? Is it actually listening on port 9000? There might be somekind of configuration you need to enable listening on that port :thinking:

8

Looks like I had enabled the “Force HTTPS” setting within Portainer. Turning the setting off and reloading certificates fixed the issue, and it’s working now. Thanks again! :smile:

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.