Caddy proxy gets "no upstreams available" error while curl is OK

1. Caddy version (caddy version): v2.4.6

2. How I run Caddy:

by enable systemctl (just the default settings)

a. System environment:

debain 10
no docker
wireguard

b. Command:

systemctl start caddy

d. My complete Caddyfile or JSON config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

:9000 {
	# Set this path to your site's directory.
	#root * /usr/share/caddy

	# Enable the static file server.
	#file_server

#the name:port is my origin server that doesn't have 80 or 443 port, and its IP changes over time so I use a ddns service to host the server.
	reverse_proxy https://name:port {
		health_uri /
		health_status 2xx

		transport http {
			tls
			tls_trusted_ca_certs /usr/local/nginx/conf/cert/cloudflare_ca.pem
			tls_server_name fangkehou.eu.org
			tls_insecure_skip_verify
		}
	}

	# Another common task is to set up a reverse proxy:
	# reverse_proxy localhost:8080

	# Or serve a PHP site through php-fpm:
	# php_fastcgi localhost:9000
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

3. The problem I’m having:

Hello, I want to use caddy to proxy my server.
I have a dynamic ipv4 server with a ddns link, I want to use my vps to proxy this server.
My vps only has ipv6 address so I use cloudflare warp (wireguard) to connect my ipv4 server.
It works just fine since July, but after I changed my wireguard config yesterday (I changed the DNS server), I’ve got message from caddy that it can’t connect to my server (I’ve tried curl and wireguard works perfectly).
So I wonder if there is something wrong with my config, and how to fix this problem.

4. Error messages and/or full log output:

caddy log(in systemctl status):

12月 31 14:49:54 DiG9 caddy[3883]: {"level":"error","ts":1640958594.0963624,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_addr":"[::1]:36090","proto":"HTTP/1.1","method":"GET","host":"[::0]:9000","uri":"/","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]}},"duration":0.000027208,"status":502,"err_id":"4w1gb3mt7","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}

12月 31 14:47:58 DiG9 caddy[3883]: {"level":"info","ts":1640958478.5356607,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"domain:port","error":"Get \"https://domain:port/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}

curl (connect to caddy):

~$ curl -v [::0]:9000
* Expire in 0 ms for 6 (transfer 0x56540f513fb0)
*   Trying ::...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56540f513fb0)
* Connected to ::0 (::1) port 9000 (#0)
> GET / HTTP/1.1
> Host: [::0]:9000
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Server: Caddy
< Date: Fri, 31 Dec 2021 13:22:12 GMT
< Content-Length: 0
< 
* Connection #0 to host ::0 left intact

curl (to my server):

~$ curl -v --insecure https://domain:port
* Expire in 0 ms for 6 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 0 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
(a lot more the same message)
* Expire in 0 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
(also a lot more)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 1 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 2 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 3 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 3 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 5 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 5 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 4 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 5 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 5 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 8 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 7 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 7 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 8 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 9 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 9 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 8 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 10 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 10 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 16 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 13 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 13 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 16 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 13 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 13 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 16 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 16 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 32 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 50 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 64 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 128 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
(well.... all the same, it seems that I have set too much nameserver, four ipv6 and two ipv4)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 250 ms for 1 (transfer 0x55b842d2bfb0)
* Expire in 200 ms for 1 (transfer 0x55b842d2bfb0)
*   Trying ip...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55b842d2bfb0)
* Connected to domain (ip) port 8849 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
*  start date: Aug 30 02:26:00 2021 GMT
*  expire date: Aug 26 02:26:00 2036 GMT
*  issuer: C=US; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL Certificate Authority; L=San Francisco; ST=California
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: domain:port
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: openresty/1.19.9.1
< Date: Fri, 31 Dec 2021 13:40:29 GMT
< Content-Type: text/html
< Content-Length: 3784
< Last-Modified: Sat, 20 Nov 2021 01:37:43 GMT
< Connection: keep-alive
< ETag: "61985167-ec8"
< Accept-Ranges: bytes
< 
(body)
* Connection #0 to host fangkehou.tpddns.cn left intact

5. What I already tried:

reinstall Caddy, update it to the newest version, restart Nginx, change resolvers, test my origin servers by ping and curl (at the server)

6. Links to relevant resources:

none, I don’t know why this problem happened.

It says no upstreams available because your health check failed, making the only configured upstream to be marked as unhealthy. So there’s no healthy upstreams for Caddy to choose from, so it errors out.

1 Like

Oh, thanks. I have set the health_timeout and it works.

This topic was automatically closed after 30 days. New replies are no longer allowed.