How to catch dns lookup failures in reverse_proxy directive

Help
1

1. The problem I’m having:

I’m new to caddy and attempting to build a SaaS application that uses the root domain to host my website and provides each customer with a subdomain that reverse proxies back to their own containerized service.

I’m using docker to deploy everything here and everything is running on a shared docker network. I’m able to get the reverse proxy working for subdomains that correlate to live containers in the backend but I’m struggling with how to provide a good experience for someone who visits a subdomain that doesn’t map to a live container in the backend. I’m getting ugly 502 error screens from cloudflare.

I’ve tried looking through a number of different help topics and docs pages around how to use handle_errors and handle_response. For instance this thread felt somewhat similar to my case Custom error page when upstream no available · Issue #4887 · caddyserver/caddy · GitHub.

Unfortunately I’m not able to get it to work.

2. Error messages and/or full log output:

This is pulled directly from my docker compose logs

caddy  | {"level":"debug","ts":1720679940.0913322,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.010998083,"request":{"remote_ip":"162.158.166.79","remote_port":"18166","client_ip":"162.158.166.79","proto":"HTTP/2.0","method":"GET","host":"qw.distill-ai.cloud","uri":"/favicon.ico","headers":{"Accept-Language":["en-US,en;q=0.9"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Dest":["image"],"Priority":["u=1, i"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"X-Forwarded-For":["162.158.166.79"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:5415:1638:3ba3:e349"],"X-Forwarded-Host":["qw.distill-ai.cloud"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["8a16cb395cc09872-SJC"],"Sec-Ch-Ua-Mobile":["?0"],"Referer":["https://qw.distill-ai.cloud/"],"Accept-Encoding":["gzip, br"],"Sec-Fetch-Mode":["no-cors"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"qw.distill-ai.cloud"}},"error":"dial tcp: lookup anythingllm.qw on 127.0.0.11:53: no such host"}
caddy  | {"level":"debug","ts":1720679940.0914972,"logger":"http.log.error","msg":"dial tcp: lookup anythingllm.qw on 127.0.0.11:53: no such host","request":{"remote_ip":"162.158.166.79","remote_port":"18166","client_ip":"162.158.166.79","proto":"HTTP/2.0","method":"GET","host":"qw.distill-ai.cloud","uri":"/favicon.ico","headers":{"X-Forwarded-For":["2600:1700:3ec0:4730:5415:1638:3ba3:e349"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Mode":["no-cors"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, br"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Dest":["image"],"Cf-Ipcountry":["US"],"Cf-Ray":["8a16cb395cc09872-SJC"],"Sec-Ch-Ua-Mobile":["?0"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:5415:1638:3ba3:e349"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["same-origin"],"Referer":["https://qw.distill-ai.cloud/"],"Priority":["u=1, i"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"qw.distill-ai.cloud"}},"duration":0.011358237,"status":502,"err_id":"y7cac1bre","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

I get the typical nasty page from cloudflare on the frontend.

cf502
cf5021713×1213 113 KB

3. Caddy version:

caddy:2.8.4-alpine

4. How I installed and ran Caddy:

Dockerfile

FROM caddy:2.8.4-alpine

COPY Caddyfile /etc/caddy/Caddyfile

a. System environment:

Ubuntu22.04
Docker + Docker Compose
Cloudflare DNS with full (strict) mode. I’m using the cloudflare installed cert and private key which you will see in my Caddyfile

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

My other services are deployed using other compose files but share the network.

services:
  caddy:
    build: .
    container_name: caddy
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    restart: always
    networks:
      - distill_network
    volumes:
      - caddy_data:/data
      - ${CF_CERT_PATH}:/certs/certificate.pem
      - ${CF_PRIVATE_KEY_PATH}:/certs/key.pem

volumes:
  caddy_data:
  caddy_certs:

networks:
  distill_network:
    external: true

here’s my docker ps output for context

CONTAINER ID   IMAGE                                     COMMAND                  CREATED         STATUS                  PORTS                                                                                                                       NAMES
b82191965483   caddy_proxy-caddy                         "caddy run --config …"   3 minutes ago   Up 3 minutes            0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:443->443/udp, :::443->443/udp, 2019/tcp   caddy
125da6282922   ajferrario/distill-website                "/start_nginx.sh"        23 hours ago    Up 23 hours             80/tcp                                                                                                                      website
a72239ba0210   mintplexlabs/anythingllm                  "/bin/bash /usr/loca…"   25 hours ago    Up 25 hours (healthy)   3001/tcp                                                                                                                    anythingllm.test-customer
ca1913cf82d6   mintplexlabs/anythingllm                  "/bin/bash /usr/loca…"   25 hours ago    Up 25 hours (healthy)   3001/tcp                                                                                                                    anythingllm.develop-great-managers

d. My complete Caddy config:

This seemed like a very elegant solution but I’ve tried tons of solutions for catching situations when the wrong subdomain is sent and none have worked :*(

{
	debug
}

distill-ai.cloud {
	tls /certs/certificate.pem /certs/key.pem
	reverse_proxy website
}

*.distill-ai.cloud {
	tls /certs/certificate.pem /certs/key.pem
	map {http.request.host} {subdomain} {
		~(.*)\.distill-ai\.cloud$ "${1}"
	}
	reverse_proxy anythingllm.{subdomain}:3001 {
	}
}

5. Links to relevant resources:

2

What did you try though? handle_errors is what you need. In what way did it not work?

1 Like
3

Ok taking a look at this again in the morning I see that I didn’t give enough detail on my previous post, sorry about that.

Also, thanks for confirming that it should be handle_errors. Based on that I’ve had somewhat of a breakthrough this morning. Using the following Caddyfile. I was able to get to the “It’s a problem!” error response for an invalid subdomain (https://invalid.distill-ai.cloud/).

{
	debug
}

distill-ai.cloud {
	tls /certs/certificate.pem /certs/key.pem
	reverse_proxy website
}

*.distill-ai.cloud {
	tls /certs/certificate.pem /certs/key.pem
	map {http.request.host} {subdomain} {
		~(.*)\.distill-ai\.cloud$ "${1}"
	}
	reverse_proxy anythingllm.{subdomain}:3001
	handle_errors {
		respond "It's a problem!"
	}
}

It’s the same as one I tried yesterday but one external thing changed. My cloudflare proxy was turned off for my DNS A-records when I updated them due to a change in my EC2 instance public IP.

When I turned the cloudflare proxy back on for my A-records and attempted to access the same subdomain I get the cloudflare 502 page I showed in my original post. When I visit a valid subdomain (https://test-customer.distill-ai.cloud/ ) it works and allows the page to come through.

Based on this it seems like it may not be a Caddyfile issue at all. However, I don’t understand how cloudflare can intercept this error before we can in caddy, since as far as I understand all of that is happening inside my machine between caddy, the docker dns, and my other containers.

Let me know if this is out of scope for the caddy community now and I’ll take it elsewhere but I would love any help you can provide.

4

Error logs from trying invalid subdomain (invalid.)

caddy  | {"level":"debug","ts":1720714178.218908,"logger":"events","msg":"event","name":"tls_get_certificate","id":"421b3d96-faee-47a2-8d61-84b53684c0cc","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"invalid.distill-ai.cloud","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.22.86","Port":41890,"Zone":""},"LocalAddr":{"IP":"172.19.0.5","Port":443,"Zone":""}}}}
caddy  | {"level":"debug","ts":1720714178.2190287,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"invalid.distill-ai.cloud"}
caddy  | {"level":"debug","ts":1720714178.2190392,"logger":"tls.handshake","msg":"choosing certificate","identifier":"invalid.distill-ai.cloud","num_choices":1}
caddy  | {"level":"debug","ts":1720714178.2190847,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"invalid.distill-ai.cloud","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"issuer_key":"","hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714178.2190957,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.22.86","remote_port":"41890","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"expiration":2193711481,"hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714178.2248726,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.invalid:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714178.2298172,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.00478562,"request":{"remote_ip":"172.69.22.86","remote_port":"41890","client_ip":"172.69.22.86","proto":"HTTP/2.0","method":"GET","host":"invalid.distill-ai.cloud","uri":"/","headers":{"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["172.69.22.86"],"Cf-Ray":["8a1a0f1db80c15c8-SJC"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, br"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["invalid.distill-ai.cloud"],"Sec-Fetch-User":["?1"],"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Priority":["u=0, i"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invalid.distill-ai.cloud"}},"error":"dial tcp: lookup anythingllm.invalid on 127.0.0.11:53: no such host"}
caddy  | {"level":"debug","ts":1720714178.230183,"logger":"http.log.error","msg":"dial tcp: lookup anythingllm.invalid on 127.0.0.11:53: no such host","request":{"remote_ip":"172.69.22.86","remote_port":"41890","client_ip":"172.69.22.86","proto":"HTTP/2.0","method":"GET","host":"invalid.distill-ai.cloud","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, br"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Cf-Ray":["8a1a0f1db80c15c8-SJC"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invalid.distill-ai.cloud"}},"duration":0.005343427,"status":502,"err_id":"1p9snhjh4","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
caddy  | {"level":"debug","ts":1720714178.317054,"logger":"events","msg":"event","name":"tls_get_certificate","id":"22af3f6c-11ff-406d-9279-40da08d736a6","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"invalid.distill-ai.cloud","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.23.135","Port":56024,"Zone":""},"LocalAddr":{"IP":"172.19.0.5","Port":443,"Zone":""}}}}
caddy  | {"level":"debug","ts":1720714178.3170953,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"invalid.distill-ai.cloud"}
caddy  | {"level":"debug","ts":1720714178.3171039,"logger":"tls.handshake","msg":"choosing certificate","identifier":"invalid.distill-ai.cloud","num_choices":1}
caddy  | {"level":"debug","ts":1720714178.3171468,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"invalid.distill-ai.cloud","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"issuer_key":"","hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714178.3171606,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.23.135","remote_port":"56024","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"expiration":2193711481,"hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714178.3230004,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.invalid:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714178.3266091,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.003528705,"request":{"remote_ip":"172.69.23.135","remote_port":"56024","client_ip":"172.69.23.135","proto":"HTTP/2.0","method":"GET","host":"invalid.distill-ai.cloud","uri":"/favicon.ico","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Site":["same-origin"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Mobile":["?0"],"Referer":["https://invalid.distill-ai.cloud/"],"X-Forwarded-Host":["invalid.distill-ai.cloud"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip, br"],"Cf-Ray":["8a1a0f1e68cd15c8-SJC"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Ch-Ua-Platform":["\"Windows\""],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Mode":["no-cors"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Priority":["u=1, i"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["172.69.23.135"],"X-Forwarded-Proto":["https"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invalid.distill-ai.cloud"}},"error":"dial tcp: lookup anythingllm.invalid on 127.0.0.11:53: no such host"}
caddy  | {"level":"debug","ts":1720714178.3269784,"logger":"http.log.error","msg":"dial tcp: lookup anythingllm.invalid on 127.0.0.11:53: no such host","request":{"remote_ip":"172.69.23.135","remote_port":"56024","client_ip":"172.69.23.135","proto":"HTTP/2.0","method":"GET","host":"invalid.distill-ai.cloud","uri":"/favicon.ico","headers":{"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Priority":["u=1, i"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip, br"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Cf-Ray":["8a1a0f1e68cd15c8-SJC"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Mode":["no-cors"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://invalid.distill-ai.cloud/"],"Accept-Language":["en-US,en;q=0.9"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invalid.distill-ai.cloud"}},"duration":0.003772027,"status":502,"err_id":"p0ysama83","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

logs from successful redirect (test-customer.)

caddy  | {"level":"debug","ts":1720714285.311321,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ac7c5f3e-7e7b-4a39-a6ac-b4ef2e95554d","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"test-customer.distill-ai.cloud","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.134.24","Port":20196,"Zone":""},"LocalAddr":{"IP":"172.19.0.5","Port":443,"Zone":""}}}}
caddy  | {"level":"debug","ts":1720714285.3113654,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"test-customer.distill-ai.cloud"}
caddy  | {"level":"debug","ts":1720714285.3113747,"logger":"tls.handshake","msg":"choosing certificate","identifier":"test-customer.distill-ai.cloud","num_choices":1}
caddy  | {"level":"debug","ts":1720714285.3114429,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"test-customer.distill-ai.cloud","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"issuer_key":"","hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.31147,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.134.24","remote_port":"20196","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"expiration":2193711481,"hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.3178155,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.3211517,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.003245571,"request":{"remote_ip":"172.69.134.24","remote_port":"20196","client_ip":"172.69.134.24","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/","headers":{"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Purpose":["prefetch"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Cf-Ipcountry":["US"],"Sec-Fetch-Mode":["navigate"],"Sec-Purpose":["prefetch;prerender"],"Cf-Ray":["8a1a11bb19711726-SJC"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["none"],"Cdn-Loop":["cloudflare"],"Priority":["u=0, i"],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"If-Modified-Since":["Sun, 07 Jul 2024 22:57:05 GMT"],"Accept-Encoding":["gzip, br"],"X-Forwarded-For":["172.69.134.24"],"Accept-Language":["en-US,en;q=0.9"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"X-Frame-Options":["DENY"],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"],"Vary":["Origin"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=0"],"Last-Modified":["Sun, 07 Jul 2024 22:57:05 GMT"],"Etag":["W/\"695-1908f6a45e8\""],"Keep-Alive":["timeout=5"]},"status":304}
caddy  | {"level":"debug","ts":1720714285.4205174,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.4225698,"logger":"events","msg":"event","name":"tls_get_certificate","id":"94c3bac5-06a0-406f-81d2-d3fc9717245c","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"test-customer.distill-ai.cloud","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.134.25","Port":51424,"Zone":""},"LocalAddr":{"IP":"172.19.0.5","Port":443,"Zone":""}}}}
caddy  | {"level":"debug","ts":1720714285.4226046,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"test-customer.distill-ai.cloud"}
caddy  | {"level":"debug","ts":1720714285.422612,"logger":"tls.handshake","msg":"choosing certificate","identifier":"test-customer.distill-ai.cloud","num_choices":1}
caddy  | {"level":"debug","ts":1720714285.4226568,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"test-customer.distill-ai.cloud","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"issuer_key":"","hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.4226677,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.134.25","remote_port":"51424","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"expiration":2193711481,"hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.4258025,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.005191345,"request":{"remote_ip":"172.69.134.24","remote_port":"20196","client_ip":"172.69.134.24","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/system/logo","headers":{"Accept-Encoding":["gzip, br"],"Sec-Fetch-Mode":["cors"],"Referer":["https://test-customer.distill-ai.cloud/"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Accept":["*/*"],"Purpose":["prefetch"],"X-Forwarded-For":["172.69.134.24"],"Cf-Ray":["8a1a11bbda2d1726-SJC"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Purpose":["prefetch;prerender"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Sec-Fetch-Site":["same-origin"],"Cf-Ipcountry":["US"],"Accept-Language":["en-US,en;q=0.9"],"Priority":["u=1, i"],"Cdn-Loop":["cloudflare"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"Vary":["Origin"],"Access-Control-Expose-Headers":["Content-Disposition,X-Is-Custom-Logo,Content-Type,Content-Length"],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Keep-Alive":["timeout=5"],"X-Powered-By":["Express"],"Content-Type":["image/png"],"Content-Disposition":["attachment; filename=anything-llm.png"],"Content-Length":["6324"],"X-Is-Custom-Logo":["false"],"Connection":["keep-alive"]},"status":200}
caddy  | {"level":"debug","ts":1720714285.4306562,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.4354675,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.004712679,"request":{"remote_ip":"172.69.134.25","remote_port":"51424","client_ip":"172.69.134.25","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/setup-complete","headers":{"Accept-Encoding":["gzip, br"],"Referer":["https://test-customer.distill-ai.cloud/"],"Cf-Ray":["8a1a11bbda2a1726-SJC"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Accept":["*/*"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["empty"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["same-origin"],"Priority":["u=1, i"],"X-Forwarded-For":["172.69.134.25"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Purpose":["prefetch"],"If-None-Match":["W/\"608-WV56SNZheNbCApt6k3VNmfBxqXo\""],"Sec-Purpose":["prefetch;prerender"],"Accept-Language":["en-US,en;q=0.9"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"Connection":["keep-alive"],"Keep-Alive":["timeout=5"],"X-Powered-By":["Express"],"Vary":["Origin"],"Etag":["W/\"608-WV56SNZheNbCApt6k3VNmfBxqXo\""],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"]},"status":304}
caddy  | {"level":"debug","ts":1720714285.4929256,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.4929256,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.4991424,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.005972315,"request":{"remote_ip":"172.69.134.24","remote_port":"20196","client_ip":"172.69.134.24","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/system/support-email","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Windows\""],"If-None-Match":["W/\"15-jZl6nJ8Kl19ZuVWoCBMRtDRBwLI\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["cors"],"Cf-Ray":["8a1a11bc4a971726-SJC"],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Accept":["*/*"],"Sec-Fetch-Dest":["empty"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["172.69.134.24"],"Accept-Language":["en-US,en;q=0.9"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Authorization":["REDACTED"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip, br"],"Referer":["https://test-customer.distill-ai.cloud/"],"Priority":["u=1, i"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Purpose":["prefetch"],"Sec-Purpose":["prefetch;prerender"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"X-Powered-By":["Express"],"Vary":["Origin"],"Etag":["W/\"15-jZl6nJ8Kl19ZuVWoCBMRtDRBwLI\""],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"]},"status":304}
caddy  | {"level":"debug","ts":1720714285.5024285,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.008063261,"request":{"remote_ip":"172.69.134.25","remote_port":"51424","client_ip":"172.69.134.25","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/setup-complete","headers":{"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Fetch-Dest":["empty"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Fetch-Mode":["cors"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Accept":["*/*"],"Sec-Purpose":["prefetch;prerender"],"Accept-Encoding":["gzip, br"],"Sec-Ch-Ua-Mobile":["?0"],"Cf-Ipcountry":["US"],"Cf-Ray":["8a1a11bc4a991726-SJC"],"X-Forwarded-Proto":["https"],"Priority":["u=1, i"],"Purpose":["prefetch"],"If-None-Match":["W/\"608-WV56SNZheNbCApt6k3VNmfBxqXo\""],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"X-Forwarded-For":["172.69.134.25"],"Referer":["https://test-customer.distill-ai.cloud/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"X-Powered-By":["Express"],"Vary":["Origin"],"Etag":["W/\"608-WV56SNZheNbCApt6k3VNmfBxqXo\""],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"]},"status":304}
caddy  | {"level":"debug","ts":1720714285.5328925,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.5346017,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.5372944,"logger":"events","msg":"event","name":"tls_get_certificate","id":"397eba8d-08e2-4445-8148-f6315702fe41","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"test-customer.distill-ai.cloud","SupportedCurves":[29,23,24,25,25497,65074],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"172.69.134.25","Port":51440,"Zone":""},"LocalAddr":{"IP":"172.19.0.5","Port":443,"Zone":""}}}}
caddy  | {"level":"debug","ts":1720714285.538532,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"test-customer.distill-ai.cloud"}
caddy  | {"level":"debug","ts":1720714285.5388563,"logger":"tls.handshake","msg":"choosing certificate","identifier":"test-customer.distill-ai.cloud","num_choices":1}
caddy  | {"level":"debug","ts":1720714285.5396492,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"test-customer.distill-ai.cloud","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"issuer_key":"","hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.5400589,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.134.25","remote_port":"51440","subjects":["cloudflare origin certificate","*.distill-ai.cloud","distill-ai.cloud"],"managed":false,"expiration":2193711481,"hash":"24cb8fd71d3eff4caeb45ac87af70ad679e7426f79c86b92788c60a916865d77"}
caddy  | {"level":"debug","ts":1720714285.5495315,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"anythingllm.test-customer:3001","total_upstreams":1}
caddy  | {"level":"debug","ts":1720714285.5558786,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.022755586,"request":{"remote_ip":"172.69.134.24","remote_port":"20196","client_ip":"172.69.134.24","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/system/support-email","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cache-Control":["max-age=0"],"Sec-Purpose":["prefetch;prerender"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Purpose":["prefetch"],"X-Forwarded-For":["172.69.134.24"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Priority":["u=1, i"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Fetch-Mode":["cors"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Accept":["*/*"],"Referer":["https://test-customer.distill-ai.cloud/"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"If-None-Match":["W/\"15-jZl6nJ8Kl19ZuVWoCBMRtDRBwLI\""],"Sec-Ch-Ua-Mobile":["?0"],"Authorization":["REDACTED"],"Accept-Encoding":["gzip, br"],"Cf-Ray":["8a1a11bc8ad11726-SJC"],"X-Forwarded-Host":["test-customer.distill-ai.cloud"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"],"X-Powered-By":["Express"],"Vary":["Origin"],"Etag":["W/\"15-jZl6nJ8Kl19ZuVWoCBMRtDRBwLI\""]},"status":304}
caddy  | {"level":"debug","ts":1720714285.5588,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.023771318,"request":{"remote_ip":"172.69.134.25","remote_port":"51424","client_ip":"172.69.134.25","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/workspaces","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Sec-Fetch-Dest":["empty"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Purpose":["prefetch;prerender"],"Sec-Fetch-Site":["same-origin"],"Referer":["https://test-customer.distill-ai.cloud/"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["*/*"],"Purpose":["prefetch"],"Accept-Language":["en-US,en;q=0.9"],"Cf-Ray":["8a1a11bc8ad31726-SJC"],"X-Forwarded-For":["172.69.134.25"],"Authorization":["REDACTED"],"Sec-Fetch-Mode":["cors"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Priority":["u=1, i"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Accept-Encoding":["gzip, br"],"Cf-Ipcountry":["US"],"If-None-Match":["W/\"11-qn6bc9+HORbT7CU0gCZtP0Q6Mrw\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"Etag":["W/\"11-qn6bc9+HORbT7CU0gCZtP0Q6Mrw\""],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"],"X-Powered-By":["Express"],"Vary":["Origin"]},"status":304}
caddy  | {"level":"debug","ts":1720714285.5632293,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.013589071,"request":{"remote_ip":"172.69.134.25","remote_port":"51440","client_ip":"172.69.134.25","proto":"HTTP/2.0","method":"GET","host":"test-customer.distill-ai.cloud","uri":"/api/system/welcome-messages","headers":{"Cache-Control":["max-age=0"],"Cf-Ipcountry":["US"],"Authorization":["REDACTED"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Host":["test-customer.distill-ai.cloud"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["empty"],"Referer":["https://test-customer.distill-ai.cloud/"],"Cdn-Loop":["cloudflare"],"Accept":["*/*"],"Sec-Ch-Ua-Mobile":["?0"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Fetch-Mode":["cors"],"If-None-Match":["W/\"25-4uL92ZDbgpxQOgruq7y4SFD25c4\""],"Sec-Purpose":["prefetch;prerender"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Purpose":["prefetch"],"Priority":["u=1, i"],"Accept-Encoding":["gzip, br"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["172.69.134.25"],"Sec-Fetch-Site":["same-origin"],"Cf-Ray":["8a1a11bc8ad61726-SJC"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"test-customer.distill-ai.cloud"}},"headers":{"Keep-Alive":["timeout=5"],"X-Powered-By":["Express"],"Vary":["Origin"],"Etag":["W/\"25-4uL92ZDbgpxQOgruq7y4SFD25c4\""],"Date":["Thu, 11 Jul 2024 16:11:25 GMT"],"Connection":["keep-alive"]},"status":304}
5

I was just looking in my cloudflare DNS logs and I see a number of NOERROR lookups and a number of NXDOMAIN error lookups. Here are 2 examples.

2024-07-11T16:15:00Z,2024-07-11T16:15:59Z,30,NOERROR
2024-07-11T16:15:00Z,2024-07-11T16:15:59Z,0,NXDOMAIN

Given that we’re using handle_errors, I don’t understand how an error is making it back to cloudflare.

6

handle_errors doesn’t have anything to do with DNS, it only catches errors that happen during HTTP handling and allows you to write a custom response instead of the default HTTP status.

If you’re having DNS issues, it’s definitely not a problem with Caddy.

1 Like
7

To be clear, the reason that I think it may still be caddy related is that the dns error is a docker DNS (127.0.0.11) lookup inside my own server which is happening as a part of the name resolution process I’m triggering in my reverse_proxy lookup.

It seems like Caddy is able to control what happens if it is able to get a local docker network IP resolution from the docker dns, but instead of caddy catching the error which it itself is logging somehow cloudflare is intercepting it and changing the returned screen. When I turn off the cloudflare proxy this doesn’t happen.

Based on that, I’m not sure how the cloudflare could be aware of what’s happening. Do you know if Caddy will reach out to other nameservers besides the docker nameserver when attempting to resolve the IP of my reverse_proxy target?

8

I don’t really follow. Please show evidence of that. Make requests with curl -v and show Caddy debug logs.

1 Like
9

The way they got formatted by the site didn’t do us any favors for readability, but previously uploaded Caddy logs actually contain the loglines showing a 127.0.0.11 lookup failure. Here is the first line showing this part of the process

caddy  | {"level":"debug","ts":1720714178.2298172,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"anythingllm.{subdomain}:3001","duration":0.00478562,"request":{"remote_ip":"172.69.22.86","remote_port":"41890","client_ip":"172.69.22.86","proto":"HTTP/2.0","method":"GET","host":"invalid.distill-ai.cloud","uri":"/","headers":{"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["172.69.22.86"],"Cf-Ray":["8a1a0f1db80c15c8-SJC"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua":["\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\""],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, br"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["invalid.distill-ai.cloud"],"Sec-Fetch-User":["?1"],"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"],"Cf-Connecting-Ip":["2600:1700:3ec0:4730:ad02:4c0e:da8c:27c3"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"Priority":["u=0, i"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"invalid.distill-ai.cloud"}},"error":"dial tcp: lookup anythingllm.invalid on 127.0.0.11:53: no such host"}

What we see here is caddy logging that it is unable to find the desired host. My question is why caddy is able to log this lookup failure but I’m still getting a CloudFlare 502 for this request rather than caddy catching the error and letting me send the user some useful response.

Here’s my curl -v log from another container on the same network

root@e41151cd5b9d:/# curl -v http://anythingllm.invalid:3001/
* Could not resolve host: anythingllm.invalid
* Closing connection 0
curl: (6) Could not resolve host: anythingllm.invalid
10

Perhaps check what dns resolver you use. According to Networking overview | Docker Docs containers inherit a copy of the hosts /etc/resolv.conf. And this might point to systemd’s resolver. In my limited experience with systemd resolver is that it is better to deploy a local Unbound or dnsmasq daemon to handle dns resolution.

3 Likes
11

Interesting, thanks for the insight! I will look into that and see if I can either change docker’s behavior or switch to a different technology.

12

This is debug logs from the proxy, it all looks as expected.

If you add the log directive in your site, you can get access logs emitted which will show what Caddy actually responded with (status code etc) after error handling.

I don’t think this is even reaching Caddy. This is curl failing to resolve to an IP before even making the request.

You could test with curl -v --resolve anythingllm.invalid:3001:127.0.0.1 http://anythingllm.invalid:3001 which would force resolve that domain to 127.0.0.1 so it reaches Caddy. Then you can see what it does with it.

1 Like
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.