1. The problem I’m having:
I use a nginx server in inner net to provide https service for many self-hosted services. And then I use a port-forwarding to outer cloud server and a caddy server to provider https service for outer accesses.
Here is my inner nginx config:
server {
listen 80;
server_name git.example.com;
client_max_body_size 0;
return 301 https://$server_name$request_uri;
}
server {
server_name git.example.com;
listen 443 ssl http2;
client_max_body_size 0;
ssl_certificate "/etc/nginx/cert/example.com.pem";
ssl_certificate_key "/etc/nginx/cert/example.com.key";
location / {
client_max_body_size 512M;
proxy_pass http://gitea:3000;
proxy_set_header Connection $http_connection;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
And here is the outer caddy config:
*.example.com {
tls {
dns <-properly configured cert file config->
protocols tls1.2 tls1.3
ciphers TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
}
@git host git.example.com
handle @git {
reverse_proxy 172.17.0.1:12390 {
transport http {
tls_insecure_skip_verify
}
}
}
}
In outer cloud server, caddy runs in docker and port forwaring is localhost:12390 → inner:443
This works well. And then I hope to alter inner nginx with caddy.
Here is the config for the new caddy server in inner net:
*.example.com {
tls {
dns <- properly configured tls config. with same storage to inner caddy ->
protocols tls1.2 tls1.3
ciphers TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
}
@git host git.example.com
reverse_proxy @git gitea:3000
Then error occurs: Accessing inner caddy directly won’t get any error, page works all well, however when accessing outer caddy I got 502 bad gateway.
I already ensured port forwarding works because other port forwarding rules are working and if I just stop caddy and turn on nginx without modifying or reboot port-forwarding it just goes well again. It’s just normal tcp port forwarding.
2. Error messages and/or full log output:
Outer caddy log output(related):
caddy | 2024/09/19 02:59:09.352 ERROR http.log.error EOF {"request": {"remote_ip": "xxx", "remote_port": "56738", "client_ip": "xxx", "proto": "HTTP/2.0", "method": "POST", "host": "git.example.com", "uri": "/api/actions/runner.v1.RunnerService/FetchTask", "headers": {"Connect-Protocol-Version": ["1"], "Content-Length": ["3"], "X-Runner-Token": ["7efc7f6b88012ed3ab50aed20daeb842ac1eb9d9"], "X-Runner-Version": ["v0.2.10"], "Connect-Timeout-Ms": ["4999"], "User-Agent": ["connect-go/1.15.0 (go1.21.9)"], "Content-Type": ["application/proto"], "Accept-Encoding": ["gzip"], "X-Runner-Uuid": ["8c81f172-d035-4c0d-9947-b282bf873690"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "git.example.com"}}, "duration": 1.014500658, "status": 502, "err_id": "59ggcrzc0", "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"}
caddy | 2024/09/19 02:59:11.348 ERROR http.log.error EOF {"request": {"remote_ip": "xxx", "remote_port": "56738", "client_ip": "xxx", "proto": "HTTP/2.0", "method": "POST", "host": "git.example.com", "uri": "/api/actions/runner.v1.RunnerService/FetchTask", "headers": {"User-Agent": ["connect-go/1.15.0 (go1.21.9)"], "Content-Type": ["application/proto"], "X-Runner-Token": ["7efc7f6b88012ed3ab50aed20daeb842ac1eb9d9"], "X-Runner-Uuid": ["8c81f172-d035-4c0d-9947-b282bf873690"], "X-Runner-Version": ["v0.2.10"], "Content-Length": ["3"], "Connect-Timeout-Ms": ["4999"], "Connect-Protocol-Version": ["1"], "Accept-Encoding": ["gzip"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "git.example.com"}}, "duration": 1.01344524, "status": 502, "err_id": "sqs3s117c", "err_trace": "reverseproxy.statusError (reverseproxy.go:1267)"}
3. Caddy version:
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
4. How I installed and ran Caddy:
with xcaddy:
xcaddy build \
--with github.com/caddy-dns/alidns \
--with github.com/caddy-dns/cloudflare \
--with github.com/greenpau/caddy-security \
--with github.com/yroc92/postgres-storage
in docker image (both inner and outer caddy)
ENTRYPOINT ["/usr/bin/caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]
5. What I already tried
add header_up to outer caddy config
New Caddyfile:
*.example.com {
tls {
dns <-properly configured cert file config->
protocols tls1.2 tls1.3
ciphers TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
}
@git host git.example.com
handle @git {
reverse_proxy 172.17.0.1:12390 {
transport http {
tls_insecure_skip_verify
}
header_up Host {upstream_hostport}
}
}
}
Not working
disable http/2 in outer caddy reverse_proxy transport
handle @git {
reverse_proxy 172.17.0.1:12390 {
transport http {
tls_insecure_skip_verify
versions 1.1
}
}
}
Not working
enable proxy-protocol
For outer caddy:
handle @git {
reverse_proxy 172.17.0.1:12390 {
transport http {
tls_insecure_skip_verify
proxy_protocol v2
}
}
}
For inner caddy:
{
#... other config
servers {
# My port-forwarding agent server in 172.30.162.32/27
trusted_proxies static 172.30.162.32/27
}
auto_https disable_redirects
}
#... other config
Not working