How to debug dial tcp timeout when reverse_proxy

Help
1

1. The problem I’m having:

I’m trying to reverse proxy to supabase api (official not self hosted), I got nothing on curl but there is a dial tcp timeout error in caddy log

*.x.y.z {
	log
	reverse_proxy {http.request.host.labels.3}.supabase.co {
        header_up Host {upstream_hostport}
        # header_up apikey VERY_LONG_JWT_TOKEN
        # header_up Authorization "Bearer <VERY_LONG_JWT_TOKEN>"
        header_up +apikey {http.request.header.apikey}
        header_up +Authorization {http.request.header.authorization}
        header_up X-Real-IP {http.request.remote.host}
        transport http {
            tls
            dial_timeout 10s
        }
    }
}

2. Error messages and/or full log output:

[deploy@VM-16-11-centos ~]$ sudo journalctl -f -u caddy
-- Logs begin at Sat 2023-06-03 14:05:07 CST. --
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451178.9199252,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"info","ts":1692451178.9199562,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"info","ts":1692451178.919964,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.x.y.z"]}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451178.9203305,"logger":"tls","msg":"loading managed certificate","domain":"*.x.y.z","expiration":1700053175,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451178.9206216,"logger":"tls.cache","msg":"added certificate to cache","subjects":["*.x.y.z"],"expiration":1700053175,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"3217d9d98654e568045ec4ddc2c92d30d45885fd7c91897a54874a10b0c712d0","cache_size":1,"cache_capacity":10000}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451178.9206686,"logger":"events","msg":"event","name":"cached_managed_cert","id":"09059c20-564a-4c33-abe1-6ed2efa11ff7","origin":"tls","data":{"sans":["*.x.y.z"]}}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"info","ts":1692451178.920825,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"info","ts":1692451178.9208817,"msg":"serving initial configuration"}
Aug 19 21:19:38 VM-16-11-centos systemd[1]: Started Caddy.
Aug 19 21:19:38 VM-16-11-centos caddy[3475083]: {"level":"info","ts":1692451178.9222844,"logger":"tls","msg":"finished cleaning storage units"}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1350377,"logger":"events","msg":"event","name":"tls_get_certificate","id":"780a66ed-dcdb-44aa-96b0-d86dc30e185d","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"xafadjfuciyyzkgxdzvd.x.y.z","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,515,769,513,770,514,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1352549,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"xafadjfuciyyzkgxdzvd.x.y.z"}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1352634,"logger":"tls.handshake","msg":"choosing certificate","identifier":"*.x.y.z","num_choices":1}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1352744,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"*.x.y.z","subjects":["*.x.y.z"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"3217d9d98654e568045ec4ddc2c92d30d45885fd7c91897a54874a10b0c712d0"}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1352828,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"183.94.51.68","remote_port":"8767","subjects":["*.x.y.z"],"managed":true,"expiration":1700053175,"hash":"3217d9d98654e568045ec4ddc2c92d30d45885fd7c91897a54874a10b0c712d0"}
Aug 19 21:19:43 VM-16-11-centos caddy[3475083]: {"level":"debug","ts":1692451183.1668463,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"xafadjfuciyyzkgxdzvd.supabase.co:0","total_upstreams":1}
Aug 19 21:34:45 VM-16-11-centos caddy[3514582]: {"level":"debug","ts":1692452085.339349,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{http.request.host.labels.3}.supabase.co","duration":10.001054235,"request":{"remote_ip":"183.94.51.68","remote_port":"8786","client_ip":"183.94.51.68","proto":"HTTP/2.0","method":"GET","host":"xafadjfuciyyzkgxdzvd.supabase.co","uri":"/rest/v1/products","headers":{"Accept":["*/*"],"Apikey":["API_KEY"],"Authorization":[],"X-Forwarded-For":["183.94.51.68"],"X-Real-Ip":["183.94.51.68"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["xafadjfuciyyzkgxdzvd.x.y.z"],"User-Agent":["Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"xafadjfuciyyzkgxdzvd.x.y.z"}},"error":"dial tcp 104.18.26.135:0: i/o timeout"}
Aug 19 21:34:45 VM-16-11-centos caddy[3514582]: {"level":"error","ts":1692452085.339417,"logger":"http.log.error","msg":"dial tcp 104.18.26.135:0: i/o timeout","request":{"remote_ip":"183.94.51.68","remote_port":"8786","client_ip":"183.94.51.68","proto":"HTTP/2.0","method":"GET","host":"xafadjfuciyyzkgxdzvd.x.y.z","uri":"/rest/v1/products","headers":{"User-Agent":["Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"],"Accept":["*/*"],"Apikey":["API_KEY"],"Authorization":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"xafadjfuciyyzkgxdzvd.x.y.z"}},"duration":10.001285722,"status":502,"err_id":"vxiskjmmy","err_trace":"reverseproxy.statusError (reverseproxy.go:1246)"}

3. Caddy version:

v2.7.3 h1:eMCNjOyMgB5A1KgOzT2dXKR4I0Va+YHCJYC8HHu+DP0=

caddy list-modules
admin.api.load
admin.api.metrics
admin.api.pki
admin.api.reverse_proxy
caddy.adapters.caddyfile
caddy.config_loaders.http
caddy.listeners.http_redirect
caddy.listeners.proxy_protocol
caddy.listeners.tls
caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.cookie
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.hash
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.filter.query
caddy.logging.encoders.filter.regexp
caddy.logging.encoders.filter.rename
caddy.logging.encoders.filter.replace
caddy.logging.encoders.json
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
events
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.gzip
http.encoders.zstd
http.handlers.acme_server
http.handlers.authentication
http.handlers.copy_response
http.handlers.copy_response_headers
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.invoke
http.handlers.map
http.handlers.metrics
http.handlers.push
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.tracing
http.handlers.vars
http.ip_sources.static
http.matchers.client_ip
http.matchers.expression
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.precompressed.br
http.precompressed.gzip
http.precompressed.zstd
http.reverse_proxy.selection_policies.client_ip_hash
http.reverse_proxy.selection_policies.cookie
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.query
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.selection_policies.weighted_round_robin
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.upstreams.a
http.reverse_proxy.upstreams.multi
http.reverse_proxy.upstreams.srv
pki
tls
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.certificates.load_storage
tls.client_auth.leaf
tls.get_certificate.http
tls.get_certificate.tailscale
tls.handshake_match.remote_ip
tls.handshake_match.sni
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.stek.distributed
tls.stek.standard

  Standard modules: 106

dns.providers.tencentcloud
http.handlers.replace_response

  Non-standard modules: 2

  Unknown modules: 0

4. How I installed and ran Caddy:

install by yum, replace caddy binary with xcaddy build version, start caddy service with systemd

a. System environment:

CentOS 8

b. Test

SUPABASE_URL=“https://.supabase.co”
SUPABASE_URL=“https://.x.y.z”
SUPABASE_ANON_KEY=“”

Append /rest/v1/ to your URL, and then use the table name as the route

curl “$SUPABASE_URL/rest/v1/products”
-H “apikey: $SUPABASE_ANON_KEY”
-H “Authorization: Bearer $SUPABASE_ANON_KEY”

with first SUPABASE_URL, curl get results, with second, curl got nothing and caddy got dial tcp timeout error in log

btw, the server running caddy is reachable to supabase api, the curl above command can get results.

2

You don’t need these, Caddy passes through all headers transparently.

This is also unnecessary, Caddy passes X-Forwarded-For.

Oh, I think the problem is that you didn’t specify a port. You should probably add :443 to your upstream address. When using a placeholder in the upstream address, the port loses its default and you need to specify it explicitly.

Also, you can use {labels.3} instead as a shortcut. See Caddyfile Concepts — Caddy Documentation

3

Wow, thank you a lot.

since I have transport http { tls }, I thought it would use port 443 by default. I’ve read the docs and searched the google, add or remove headers, but never tried to specify it explicitly.

4

You’re right, reverse_proxy (Caddyfile directive) — Caddy Documentation doesn’t explain well what happens when there’s a placeholder. It’s unfortunately pretty complex.

It’s not really viable to figure out if TLS is configured or not when actually proxying because transport is pluggable, so it may not be the http transport; other transports probably don’t have tls config so it’s not a given that we can determine it.

I’ll try to clarify the docs.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.