Caddy(docker) reverse proxy with dns-01 on Cloudflare

craffolo · September 16, 2021, 11:45am

1. Caddy version (`caddy version`):

2. How I run Caddy:

Reverse proxy to internal service with dns challenge on my domain managed by cloduflare.

a. System environment:

Raspberry Pi 3 running Raspbian GNU/Linux LTS 20.04.2 LTS

b. Command:

docker-compose up

c. Service/unit/compose file:

#docker-compose.yaml
version: '3'
services:

 caddy-dns:
   container_name: caddy-dns
   build: .
   networks:
    - reverse-proxy
   restart: unless-stopped
   environment:
    - "TZ=Europe/Rome"
    - ACME_AGREE=true
    - CLOUDFLARE_EMAIL=mymail@mail.com
    - CLOUDFLARE_API_TOKEN=aaaaaaaabbbbbbbccccccddddddd
   volumes:
    - /home/pi/compose/caddy/data:/data
    - /home/pi/compose/caddy/config:/config
    - /home/pi/compose/caddy/Caddyfile:/etc/caddy/Caddyfile
    - /home/pi/compose/caddy/key:/key
    - /home/user/caddy/site:/usr/share/caddy
   ports:
    - 80:80
    - 443:443
    - 2019:2019


networks:
  reverse-proxy:
    external: 
      name: reverse-proxy

#dockerfile
FROM caddy:builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

d. My complete Caddyfile or JSON config:

(cloudflare) {
	tls mymail@mail.com {
		dns cloudflare aaaaaaaaabbbbbbbbbbcccccccccdddddddddddd
	}
}

https://grocy.home04.cloud {
	import cloudflare
	reverse_proxy http://192.168.1.7:9283
}

https://tasmoadmin.home04.cloud {
	import cloudflare
	reverse_proxy http://192.168.1.7:9999
}

https://test.home04.cloud {
	import cloudflare
	respond "TEST"
}

3. The problem I’m having:

I have build caddy with cloudflare dns, configured on cloudflare dns th subdomain with A type record pointed to my ip.
I want to connect to may server outside without open port, so i choosed dns challenge. But i cant figure to work. Sometimes the challege go and other no. But in any case i can’t connect to my address. I figure “Error 522 Connection timed out”.

4. Error messages and/or full log output:


caddy-dns    | {"level":"info","ts":1631788496.0762863,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy-dns    | {"level":"warn","ts":1631788496.089521,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy-dns    | {"level":"info","ts":1631788496.0991058,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy-dns    | {"level":"info","ts":1631788496.1007166,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy-dns    | {"level":"info","ts":1631788496.1008477,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy-dns    | {"level":"info","ts":1631788496.1007333,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x201cc80"}
caddy-dns    | {"level":"info","ts":1631788496.1044216,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy-dns    | {"level":"info","ts":1631788496.1047025,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["tasmoadmin.home04.cloud","grocy.home04.cloud","test.home04.cloud"]}
caddy-dns    | {"level":"info","ts":1631788496.1070454,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-dns    | {"level":"info","ts":1631788496.1071415,"msg":"serving initial configuration"}
caddy-dns    | {"level":"info","ts":1631788496.1184165,"logger":"tls","msg":"finished cleaning storage units"}
caddy-dns    | {"level":"info","ts":1631788496.1228895,"logger":"tls.obtain","msg":"acquiring lock","identifier":"tasmoadmin.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788496.125005,"logger":"tls.obtain","msg":"acquiring lock","identifier":"grocy.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788496.164379,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788496.1732118,"logger":"tls.obtain","msg":"lock acquired","identifier":"tasmoadmin.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788496.2006376,"logger":"tls.obtain","msg":"lock acquired","identifier":"grocy.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788496.2232041,"logger":"tls.obtain","msg":"lock acquired","identifier":"test.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788497.4975326,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["tasmoadmin.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788497.497702,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["tasmoadmin.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788498.04707,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["grocy.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788498.047188,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["grocy.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788498.0627294,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"tasmoadmin.home04.cloud","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-dns    | {"level":"info","ts":1631788498.0669165,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788498.0674427,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.home04.cloud"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"mymail@mail.com"}
caddy-dns    | {"level":"info","ts":1631788498.4422772,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"grocy.home04.cloud","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-dns    | {"level":"info","ts":1631788498.5922148,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.home04.cloud","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
caddy-dns    | {"level":"info","ts":1631788524.7609189,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/202900020/24873153120"}
caddy-dns    | {"level":"info","ts":1631788525.7378726,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/044e927bc77c591f0c6e67ad84822e073bba"}
caddy-dns    | {"level":"info","ts":1631788525.7416859,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"tasmoadmin.home04.cloud"}
caddy-dns    | {"level":"info","ts":1631788525.7417982,"logger":"tls.obtain","msg":"releasing lock","identifier":"tasmoadmin.home04.cloud"}
caddy-dns    | {"level":"error","ts":1631788623.783846,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.home04.cloud","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[test.home04.cloud] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/202900130/24873154250) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy-dns    | {"level":"error","ts":1631788624.2832613,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grocy.home04.cloud","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[grocy.home04.cloud] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/202900120/24873154010) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy-dns    | {"level":"info","ts":1631788625.3339524,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Z7dGVt022stg8MimKDI7MA"}
caddy-dns    | {"level":"info","ts":1631788625.3469768,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"tuzRgZ_KftAdB2P4w-QRoQ"}
caddy-dns    | {"level":"error","ts":1631788626.3738816,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.home04.cloud","issuer":"acme.zerossl.com-v2-DV90","error":"provisioning client: HTTP 504: <html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
caddy-dns    | {"level":"error","ts":1631788626.3741493,"logger":"tls.obtain","msg":"will retry","error":"[test.home04.cloud] Obtain: provisioning client: HTTP 504: <html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n","attempt":1,"retrying_in":60,"elapsed":130.150807269,"max_duration":2592000}
caddy-dns    | {"level":"error","ts":1631788626.6836202,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grocy.home04.cloud","issuer":"acme.zerossl.com-v2-DV90","error":"provisioning client: HTTP 504: <html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}
caddy-dns    | {"level":"error","ts":1631788626.683853,"logger":"tls.obtain","msg":"will retry","error":"[grocy.home04.cloud] Obtain: provisioning client: HTTP 504: <html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n","attempt":1,"retrying_in":60,"elapsed":130.483071945,"max_duration":2592000}
caddy-dns    | {"level":"info","ts":1631788688.0496747,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.home04.cloud","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy-dns    | {"level":"info","ts":1631788688.6342053,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"grocy.home04.cloud","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy-dns    | {"level":"error","ts":1631788816.6591487,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.home04.cloud","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[test.home04.cloud] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/26767168/545993728) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy-dns    | {"level":"error","ts":1631788816.7162707,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grocy.home04.cloud","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[grocy.home04.cloud] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/26767178/545993928) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy-dns    | {"level":"info","ts":1631788817.9872618,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"c8tqUCHoVWspsvkKiTBlHQ"}
caddy-dns    | {"level":"info","ts":1631788818.1577652,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"pfZzKOJnCozcrUmjt-eFbg"}
caddy-dns    | {"level":"info","ts":1631788821.4353216,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"grocy.home04.cloud","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy-dns    | {"level":"info","ts":1631788821.5853841,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.home04.cloud","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}

5. What I already tried:

6. Links to relevant resources:

craffolo · September 16, 2021, 5:58pm

UP. Though the dns challenge finish successfully i can’t access by the address my server, so i cant figure where is the problem.

basil · September 16, 2021, 9:16pm

This won’t solve your access problem, but I think your Caddyfile could be simplified. Consider the following:

{
    email mymail@mail.com
    acme_dns cloudflare aaaaaaaaabbbbbbbbbbcccccccccdddddddddddd
}

grocy.home04.cloud {
    reverse_proxy 192.168.1.7:9283
}

tasmoadmin.home04.cloud {
    reverse_proxy 192.168.1.7:9999
}

test.home04.cloud {
    respond "TEST"
}

What are you trying to proxy i.e. what exactly are the upstream services e.g. Nextcloud?

craffolo · September 16, 2021, 10:24pm

I tried also this version but i have the same problem.

The main is HomeAssistant, and then some addon frontend.

craffolo · October 12, 2021, 6:03pm

can anyone help me?

Whitestrake · October 12, 2021, 11:20pm

In your Cloudflare DNS dashboard, is your website grey or orange-cloud?

craffolo · October 13, 2021, 1:39pm

I’ve tried both.
orange-cloud → error 522 Cloudflare page
grey-cloud → redirected to my router page

I’ can’t figure where is the problem, I have tried more configuration.
internal tls, let’s encrypt with cloud flare dns module, cloudlfare origin and I have ever the same result.

Whitestrake · October 14, 2021, 1:00am

So, Cloudflare can’t connect to your origin server.

Likely because you attempted to access your website from inside the same network your server is on, and the traffic went to your router (i.e. towards the internet).

Except that when the router receives the packets, it realises that the router itself is the destination! (i.e. your public IP address).

So it processes them as normal. Which, in a normal port-forwarding situation, would involve passing the traffic back to your web server. Except that because the traffic came from inside the local network, your router treats it as traffic for its own web GUI.

A common issue, solved by enabling hairpin NAT or by using split DNS, assuming your intent is to port forward to the web server. However, going back and looking at your original post…

I suspect you’ve betrayed the source of your problem here.

If you don’t have ports open, you will not be able to serve your website.

There is a way around this, but the DNS challenge on its own is only half the solution. The other half would be cloudflared / Argo Tunnel.

craffolo · October 14, 2021, 6:28pm

Oh yes… in the rush to secure everything I am lost.

Fantastic, thats the solution.
I’ve studied cloudflared it’s wonderfull.

I tried first without caddy for understand the function. And wow it’s worked, i was able to reach a server through my domain without open a port. Fantastic.

But then i tried with caddy it not work.

#cloudflared config
tunnel: x
credentials-file: x

logfile: clodflared.log

ingress:
    - hostname: gorcy.domain.x
      service:  http://127.0.0.1:10000

#caddyfile
gorcy.domain.x:10000   {
	tls {
		dns cloudflare ABCD
		resolvers 1.1.1.1
	}
	reverse_proxy http://192.168.1.7:9283
}

Not sure on this, i just discovered cloudflared, and I am now learning how to use caddy

craffolo · October 14, 2021, 6:31pm

Oh and i forgot to say that on Cloudflare DNS record i have only one CNAME named grocy with target (the tunnel UIDD).cfargotunnel.com

Whitestrake · October 14, 2021, 11:09pm

Unfortunately, “it’s not working” doesn’t give enough information for me to make any guesses at how you could possibly troubleshoot. Is it timing out? Is it returning an error? Is it trying to redirect you?

I have one wild guess based off the configs:

vs.

Caddy is probably trying to use HTTPS on port 10000 because Caddy tries to be secure by default, however, you’ve configured your Argo Tunnel to proxy over HTTP-insecure.

You’ve already configured DNS validation, so instead of downgrading Caddy to HTTP-insecure, I’d suggest upgrading the Argo Tunnel proxy to use HTTPS instead.

P.S.

For the sake of ease of use, I just set:

ingress:
  - service: https://caddy

As my ingress rule; literally a catch-all, since all web traffic will get sorted out by Caddy on my machine regardless of which hostname it’s for. That means I control which domains actually go here in DNS itself, by configuring which ones are CNAMEd to the specific named tunnel in Cloudflare.

This is a lot more similar to how things work without tunnels; using ingress rules with different hostnames is more useful for the kind of scenario where you don’t have Caddy and want cloudflared itself to handle the routing of multiple services.

craffolo · October 14, 2021, 11:31pm

Just tried and now it show me this error

ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" cfRay=69e494bca8b5374a-MXP ingressRule=0 originService=https://127.0.0.1:10000

And on the page " Error 502 Bad gateway"

Wow this is very usefull, but i dont understand how to implement it.

Whitestrake · October 14, 2021, 11:35pm

Easy, just remove the hostname qualifier on your ingress rule.

For you it’d be like:

- service: http://127.0.0.1:10000

That’d send all Argo Tunnel traffic to Caddy regardless of which hostname it’s for.

Then you just configure your hostnames in Caddy exactly like you would normally, all on port 10000.

sub1.example.com:10000 {
  # blah
}

sub2.example.com:10000 {
  # blah
}

craffolo:

Just tried and now it show me this error

ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: remote error: tls: internal error" cfRay=69e494bca8b5374a-MXP ingressRule=0 originService=https://127.0.0.1:10000

And on the page " Error 502 Bad gateway"

Is there any output in Caddy logs for this event?

What was it doing before you made the change, exactly? Were there any cloudflared/Caddy logs from the previous issue?

craffolo · October 14, 2021, 11:44pm

nope, standard

2021/10/14 23:23:44.933 INFO    using adjacent Caddyfile
2021/10/14 23:23:44.936 WARN    input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 1}
2021/10/14 23:23:44.942 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["[::1]:2019", "127.0.0.1:2019", "localhost:2019"]}
2021/10/14 23:23:44.943 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000665ab0"}
2021/10/14 23:23:44.943 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/10/14 23:23:44.949 INFO    http    enabling automatic TLS certificate management   {"domains": ["gorcy.domain.x"]}
2021/10/14 23:23:44.982 INFO    tls     finished cleaning storage units
2021/10/14 23:23:44.982 INFO    autosaved config (load with --resume flag)    
2021/10/14 23:23:44.982 INFO    serving initial configuration

Same log except the error. And on browser now i’ve checked again and it says “Client sent an HTTP request to an HTTPS server.” (you was right ). But before it was something like “can’t resolve hostname”

Whitestrake · October 14, 2021, 11:48pm

Hmm.

What do you get from:

curl -IL --resolve gorcy.domain.x:10000:127.0.0.1 https://gorcy.domain.x:10000

craffolo · October 14, 2021, 11:49pm

curl: (7) Failed to connect to gorcy.domain.x port 10000: Connection refused

Whitestrake · October 14, 2021, 11:53pm

I note that your Caddy is in a Docker container.

Have you configured port 10000 to forward to the Caddy container?

craffolo · October 15, 2021, 12:35am

Yes, i tried also without the external networks “reverse-proxy”.

I noted that if i try to connect internally it show “ERR_SSL_PROTOCOL_ERROR”

craffolo · October 15, 2021, 12:59am

Thanks for your help now had to go sleep, but i would to ask if it’s possible to extend the time of expiration of the topic. Sweat dreams

francislavoie · October 15, 2021, 4:05am

Done.