Matrix Synapse getting no .well-known error - Issues with Federation

Eschguy · February 5, 2023, 12:26am

1. Caddy version:

2.6.2

2. How I installed, and run Caddy:

Official installation

a. System environment:

Ubuntu Server

b. Command:

systemctl start caddy.service

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

(trustedproxy) {
        trusted_proxies 172.16.0.0/24 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22>
}

eschbach.house {
        header /.well-known/matrix/* Content-Type application/json
        header /.well-known/matrix/* Access-Control-Allow-Origin *
        respond /.well-known/matrix/server `{"m.server": "matrix.eschbach.house:443"}`
        respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://matrix.eschbach.house"},"m.identity_server":{"base_url":"https://identity.eschbach.house"}}`
}

matrix.eschbach.house {
        reverse_proxy /_matrix/* 172.16.0.231:8008 {import trustedproxy}
        reverse_proxy /_synapse/client/* 172.16.0.231:8008 {import trustedproxy}
}

3. The problem I’m having:

Server is up and running, but having issues with federation. The federationtester.matrix.org site says I have no .well-known but I have exactly was was put in the docs.

4. Error messages and/or full log output:

Feb 04 18:17:17 caddy caddy[146]: {"level":"error","ts":1675556237.3614252,"logger":"http.log.error","msg":"dial tcp: lookup trustedproxy}: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"41514","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"Accept-Encoding":["gzip"],"Cf-Ray":["794776533f328101-ORD"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Origin":["https://app.element.io"],"Accept-Language":["en-US"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["75.168.220.4"],"Accept":["application/json"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cache-Control":["max-age=0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.00015394,"status":502,"err_id":"xuhfagpvz","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:22 caddy caddy[146]: {"level":"error","ts":1675556242.5334344,"logger":"http.log.error","msg":"dial tcp: lookup trustedproxy}: no such host","request":{"remote_ip":"172.70.131.162","remote_port":"21214","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"Origin":["https://app.element.io"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US"],"Cf-Connecting-Ip":["75.168.220.4"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Accept":["application/json"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["7947767388fe8101-ORD"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000137496,"status":502,"err_id":"cbhk84fvc","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:30 caddy caddy[146]: {"level":"error","ts":1675556250.0287037,"logger":"http.log.error","msg":"dial tcp: lookup {import: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"27110","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"Origin":["https://app.element.io"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["794776a268bb8101-ORD"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000159694,"status":502,"err_id":"0piruwcvh","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:37 caddy caddy[146]: {"level":"error","ts":1675556257.9552946,"logger":"http.log.error","msg":"dial tcp: lookup trustedproxy}: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"27110","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s20_76_0_1_10_1_1_15_0_1","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Origin":["https://app.element.io"],"Cf-Ipcountry":["US"],"Cf-Ray":["794776d429fe8101-ORD"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cf-Connecting-Ip":["75.168.220.4"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"Authorization":[],"Cdn-Loop":["cloudflare"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US"],"Accept":["application/json"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000102002,"status":502,"err_id":"83772xte6","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:43 caddy caddy[146]: {"level":"error","ts":1675556263.5836112,"logger":"http.log.error","msg":"dial tcp: lookup {import: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"41514","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"X-Forwarded-For":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Accept-Language":["en-US"],"Accept-Encoding":["gzip"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["794776f74fb38101-ORD"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000104638,"status":502,"err_id":"y2vcaavq4","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:51 caddy caddy[146]: {"level":"error","ts":1675556271.036854,"logger":"http.log.error","msg":"dial tcp: lookup trustedproxy}: no such host","request":{"remote_ip":"172.70.131.162","remote_port":"21214","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"],"Cf-Ray":["79477725eab78101-ORD"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000107385,"status":502,"err_id":"m3ua9xp31","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:43 caddy caddy[146]: {"level":"error","ts":1675556263.5836112,"logger":"http.log.error","msg":"dial tcp: lookup {import: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"41514","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"X-Forwarded-For":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Accept-Language":["en-US"],"Accept-Encoding":["gzip"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["794776f74fb38101-ORD"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000104638,"status":502,"err_id":"y2vcaavq4","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:51 caddy caddy[146]: {"level":"error","ts":1675556271.036854,"logger":"http.log.error","msg":"dial tcp: lookup trustedproxy}: no such host","request":{"remote_ip":"172.70.131.162","remote_port":"21214","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"],"Cf-Ray":["79477725eab78101-ORD"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000107385,"status":502,"err_id":"m3ua9xp31","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:17:56 caddy caddy[146]: {"level":"error","ts":1675556276.573587,"logger":"http.log.error","msg":"dial tcp: lookup {import: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"27110","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Origin":["https://app.element.io"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Cache-Control":["max-age=0"],"Accept-Language":["en-US"],"Cf-Connecting-Ip":["75.168.220.4"],"Cdn-Loop":["cloudflare"],"Cf-Ray":["7947774888338101-ORD"],"Accept":["application/json"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000093013,"status":502,"err_id":"dhc2jczyn","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}
Feb 04 18:18:02 caddy caddy[146]: {"level":"error","ts":1675556282.0598867,"logger":"http.log.error","msg":"dial tcp: lookup {import: no such host","request":{"remote_ip":"172.70.131.161","remote_port":"41514","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/versions","headers":{"Cf-Ipcountry":["US"],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ray":["7947776acb788101-ORD"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Origin":["https://app.element.io"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15 Tangram/2.0"],"Accept-Language":["en-US"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"duration":0.000099193,"status":502,"err_id":"3k11x5nts","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}

5. What I already tried:

I’ve put the Caddyfile info that’s in the docs ver batum (adjusting for my domain) and it doesn’t work.
I’ve added serve_server_wellknown: true to my homeserver.yaml and it doesn’t work.

6. Links to relevant resources:

https://federationtester.matrix.org/#eschbach.house

https://federationtester.matrix.org/api/report?server_name=eschbach.house

github.com

matrix-org/synapse/blob/develop/docs/reverse_proxy.md#caddy-v2

# Using a reverse proxy with Synapse

It is recommended to put a reverse proxy such as
[nginx](https://nginx.org/en/docs/http/ngx_http_proxy_module.html),
[Apache](https://httpd.apache.org/docs/current/mod/mod_proxy_http.html),
[Caddy](https://caddyserver.com/docs/quick-starts/reverse-proxy),
[HAProxy](https://www.haproxy.org/) or
[relayd](https://man.openbsd.org/relayd.8) in front of Synapse. One advantage
of doing so is that it means that you can expose the default https port
(443) to Matrix clients without needing to run Synapse with root
privileges.

You should configure your reverse proxy to forward requests to `/_matrix` or
`/_synapse/client` to Synapse, and have it set the `X-Forwarded-For` and
`X-Forwarded-Proto` request headers.

You should remember that Matrix clients and other Matrix servers do not
necessarily need to connect to your server via the same server name or
port. Indeed, clients will use port 443 by default, whereas servers default to
port 8448. Where these are different, we refer to the 'client port' and the

This file has been truncated. show original

emilylange · February 5, 2023, 8:37pm

Try opening https://eschbach.house/.well-known/matrix/client in an incognito tab or something similar.

I am getting a http/403 from Cloudflare, so you maybe check your CF dashboard

$ curl https://eschbach.house/.well-known/matrix/server -i
HTTP/2 403 
date: Sun, 05 Feb 2023 20:33:17 GMT
content-type: text/plain; charset=UTF-8
content-length: 16
x-frame-options: SAMEORIGIN
referrer-policy: same-origin
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
server: cloudflare

error code: 1020%

Eschguy · February 5, 2023, 9:42pm

I’m getting an Invalid SSL cert error, which is just weird.

Only thing I could see that would give you a 403 is if you weren’t in the US (I block all non-US connections), so I’ve opened it up where matrix requests can pass through even if it’s not a US request (which makes sense).

francislavoie · February 5, 2023, 10:03pm

That’s because you used invalid syntax for the reverse_proxy. You must use newlines after the {, blocks cannot be inlined.

Would become:

reverse_proxy /_synapse/client/* 172.16.0.231:8008 {
	import trustedproxy
}

Eschguy:

(trustedproxy) {
        trusted_proxies 172.16.0.0/24 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22>
}

FYI, careful when copying from a terminal, notice the > there which is probably your terminal hiding text past that column. You can use something like cat to spit out the config contents before copying it, which shouldn’t have that issue.

I don’t see anything related to TLS issuance problems in your logs. Please take a look again.

Eschguy · February 6, 2023, 7:05pm

That is good to know, changed.

Whoops, thanks.

It’s when following @emilylange’s advice looking at https://eschbach.house/.well-known/matrix/server in a browser window, it says SSL handshake failed (error 525).

francislavoie · February 7, 2023, 2:00am

Neither of us are in the US. I get a 522 connection timeout from Cloudflare when trying to connect.

What’s in your logs at this point? Do you actually have a certificate issued for your domain?

Eschguy · February 7, 2023, 4:53pm

I’ve fixed the firewall rule with Cloudflare, so Matrix requests can be global now.

All of my other services work fine (a Pathfinder VTT, Vaultwarden, Nextcloud, etc.)

Caddy Logs:

Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8396306,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["mirrorless.eschbach.house"],"remaining":14400.160370582}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401008,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["bookshelf.eschbach.house"],"remaining":14400.159900091}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401105,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["foundry.eschbach.house"],"remaining":14400.159890717}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401146,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["hassio.eschbach.house"],"remaining":14400.15988558}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.840118,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["austineschweiler.com"],"remaining":14400.159882177}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401217,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["reading.eschbach.house"],"remaining":14400.159878353}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401315,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["vaultwarden.eschbach.house"],"remaining":14400.159868584}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401353,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["jellyfin.eschbach.house"],"remaining":14400.15986474}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8401387,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["drive.eschbach.house"],"remaining":14400.159861382}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.840514,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["mirrorless.eschbach.house"],"remaining":14400.159486431}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8406103,"logger":"tls.renew","msg":"acquiring lock","identifier":"mirrorless.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8407938,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["bookshelf.eschbach.house"],"remaining":14400.159206682}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.840861,"logger":"tls.renew","msg":"acquiring lock","identifier":"bookshelf.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8409836,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["foundry.eschbach.house"],"remaining":14400.159017019}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.841045,"logger":"tls.renew","msg":"acquiring lock","identifier":"foundry.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8411686,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["hassio.eschbach.house"],"remaining":14400.158832007}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.841237,"logger":"tls.renew","msg":"acquiring lock","identifier":"hassio.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.841357,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["austineschweiler.com"],"remaining":14400.158643502}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8414195,"logger":"tls.renew","msg":"acquiring lock","identifier":"austineschweiler.com"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8415415,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["reading.eschbach.house"],"remaining":14400.158458785}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8416014,"logger":"tls.renew","msg":"acquiring lock","identifier":"reading.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8417222,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["vaultwarden.eschbach.house"],"remaining":14400.158278148}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8417878,"logger":"tls.renew","msg":"acquiring lock","identifier":"vaultwarden.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8419096,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["jellyfin.eschbach.house"],"remaining":14400.158090852}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8419733,"logger":"tls.renew","msg":"acquiring lock","identifier":"jellyfin.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.842095,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["drive.eschbach.house"],"remaining":14400.157905623}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8421602,"logger":"tls.renew","msg":"acquiring lock","identifier":"drive.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.852025,"logger":"tls.renew","msg":"lock acquired","identifier":"mirrorless.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8524263,"logger":"tls.renew","msg":"renewing certificate","identifier":"mirrorless.eschbach.house","remaining":14400.147574641}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8531673,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"mirrorless.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8532789,"logger":"tls.renew","msg":"releasing lock","identifier":"mirrorless.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8533976,"logger":"tls","msg":"reloading managed certificate","identifiers":["mirrorless.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8536649,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [mirrorless.eschbach.house]: no OCSP server specified in certificate","identifiers":["mirrorless.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8537655,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["mirrorless.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8540967,"logger":"tls.renew","msg":"lock acquired","identifier":"austineschweiler.com"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8542058,"logger":"tls.renew","msg":"renewing certificate","identifier":"austineschweiler.com","remaining":14400.14579467}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8548143,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"austineschweiler.com"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8548317,"logger":"tls.renew","msg":"releasing lock","identifier":"austineschweiler.com"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.854859,"logger":"tls","msg":"reloading managed certificate","identifiers":["austineschweiler.com"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8549864,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [austineschweiler.com]: no OCSP server specified in certificate","identifiers":["austineschweiler.com"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8549936,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["austineschweiler.com"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.855006,"logger":"tls.renew","msg":"lock acquired","identifier":"jellyfin.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.855086,"logger":"tls.renew","msg":"renewing certificate","identifier":"jellyfin.eschbach.house","remaining":14400.144914511}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.85561,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"jellyfin.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8556252,"logger":"tls.renew","msg":"releasing lock","identifier":"jellyfin.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8556526,"logger":"tls","msg":"reloading managed certificate","identifiers":["jellyfin.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8557804,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [jellyfin.eschbach.house]: no OCSP server specified in certificate","identifiers":["jellyfin.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8557878,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["jellyfin.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8558002,"logger":"tls.renew","msg":"lock acquired","identifier":"foundry.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8558836,"logger":"tls.renew","msg":"renewing certificate","identifier":"foundry.eschbach.house","remaining":14400.1441171}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8564663,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"foundry.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8564763,"logger":"tls.renew","msg":"releasing lock","identifier":"foundry.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8565083,"logger":"tls","msg":"reloading managed certificate","identifiers":["foundry.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8566425,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [foundry.eschbach.house]: no OCSP server specified in certificate","identifiers":["foundry.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8566496,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["foundry.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8566568,"logger":"tls.renew","msg":"lock acquired","identifier":"vaultwarden.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8567343,"logger":"tls.renew","msg":"renewing certificate","identifier":"vaultwarden.eschbach.house","remaining":14400.143266304}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8572602,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"vaultwarden.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8572702,"logger":"tls.renew","msg":"releasing lock","identifier":"vaultwarden.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8573065,"logger":"tls","msg":"reloading managed certificate","identifiers":["vaultwarden.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8574367,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [vaultwarden.eschbach.house]: no OCSP server specified in certificate","identifiers":["vaultwarden.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8574438,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["vaultwarden.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.85745,"logger":"tls.renew","msg":"lock acquired","identifier":"drive.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8575306,"logger":"tls.renew","msg":"renewing certificate","identifier":"drive.eschbach.house","remaining":14400.142469842}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.858046,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"drive.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.858061,"logger":"tls.renew","msg":"releasing lock","identifier":"drive.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8580887,"logger":"tls","msg":"reloading managed certificate","identifiers":["drive.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8582447,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [drive.eschbach.house]: no OCSP server specified in certificate","identifiers":["drive.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.858253,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["drive.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8582606,"logger":"tls.renew","msg":"lock acquired","identifier":"hassio.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8583515,"logger":"tls.renew","msg":"renewing certificate","identifier":"hassio.eschbach.house","remaining":14400.141649096}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8589597,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"hassio.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.858976,"logger":"tls.renew","msg":"releasing lock","identifier":"hassio.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8590057,"logger":"tls","msg":"reloading managed certificate","identifiers":["hassio.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8591592,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [hassio.eschbach.house]: no OCSP server specified in certificate","identifiers":["hassio.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8591673,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["hassio.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8591747,"logger":"tls.renew","msg":"lock acquired","identifier":"reading.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.859272,"logger":"tls.renew","msg":"renewing certificate","identifier":"reading.eschbach.house","remaining":14400.140728628}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8598783,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"reading.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8598883,"logger":"tls.renew","msg":"releasing lock","identifier":"reading.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8599207,"logger":"tls","msg":"reloading managed certificate","identifiers":["reading.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8600748,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [reading.eschbach.house]: no OCSP server specified in certificate","identifiers":["reading.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8600824,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["reading.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8600893,"logger":"tls.renew","msg":"lock acquired","identifier":"bookshelf.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.860183,"logger":"tls.renew","msg":"renewing certificate","identifier":"bookshelf.eschbach.house","remaining":14400.139817518}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8607473,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"bookshelf.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8607585,"logger":"tls.renew","msg":"releasing lock","identifier":"bookshelf.eschbach.house"}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.8607903,"logger":"tls","msg":"reloading managed certificate","identifiers":["bookshelf.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"warn","ts":1675772549.8609207,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [bookshelf.eschbach.house]: no OCSP server specified in certificate","identifiers":["bookshelf.eschbach.house"]}
Feb 07 06:22:29 caddy caddy[146]: {"level":"info","ts":1675772549.86093,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["bookshelf.eschbach.house"],"new_expiration":1675815750}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.839052,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["eschbach.house"],"remaining":14400.160949114}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8390744,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["matrix.eschbach.house"],"remaining":14400.160925986}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.839083,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["eschbach.house"],"remaining":14400.16091713}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8391929,"logger":"tls.renew","msg":"acquiring lock","identifier":"eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.850759,"logger":"tls.renew","msg":"lock acquired","identifier":"eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.850918,"logger":"tls.renew","msg":"renewing certificate","identifier":"eschbach.house","remaining":14400.149082701}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.851587,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8516037,"logger":"tls.renew","msg":"releasing lock","identifier":"eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8516624,"logger":"tls","msg":"reloading managed certificate","identifiers":["eschbach.house"]}
Feb 07 06:42:29 caddy caddy[146]: {"level":"warn","ts":1675773749.8518064,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [eschbach.house]: no OCSP server specified in certificate","identifiers":["eschbach.house"]}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8518157,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["eschbach.house"],"new_expiration":1675816950}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8518198,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["matrix.eschbach.house"],"remaining":14400.148180528}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.851887,"logger":"tls.renew","msg":"acquiring lock","identifier":"matrix.eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8562071,"logger":"tls.renew","msg":"lock acquired","identifier":"matrix.eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8563735,"logger":"tls.renew","msg":"renewing certificate","identifier":"matrix.eschbach.house","remaining":14400.143627408}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8570445,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"matrix.eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8570917,"logger":"tls.renew","msg":"releasing lock","identifier":"matrix.eschbach.house"}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.857138,"logger":"tls","msg":"reloading managed certificate","identifiers":["matrix.eschbach.house"]}
Feb 07 06:42:29 caddy caddy[146]: {"level":"warn","ts":1675773749.8572798,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [matrix.eschbach.house]: no OCSP server specified in certificate","identifiers":["matrix.eschbach.house"]}
Feb 07 06:42:29 caddy caddy[146]: {"level":"info","ts":1675773749.8572898,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["matrix.eschbach.house"],"new_expiration":1675816950}

Eschguy · February 8, 2023, 11:09pm

I feel like I’m losing my mind, I’m still unable to get this “No .well-known found” thing to work.

emilylange · February 9, 2023, 3:47pm

I would guess you have some problem with either your server’s firewall or Cloudflare’s WAF.

We need more info though, so I need you to run the following command on the server Caddy is running on and share its output

curl -kvL --connect-to eschbach.house:80:127.0.0.1:80 --connect-to eschbach.house:443:127.0.0.1:443 http://eschbach.house/.well-known/matrix/server

I am also a bit confused as to why your eschbach.house certificate hasn’t been logged by any of the Certificate Transparency logs yet, but the output of the command above should clarify that

Eschguy · February 9, 2023, 7:29pm

Appreciate the help! Results below:

* Connecting to hostname: 127.0.0.1
* Connecting to port: 80
*   Trying 127.0.0.1:80...
* Connected to (nil) (127.0.0.1) port 80 (#0)
> GET /.well-known/matrix/server HTTP/1.1
> Host: eschbach.house
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://eschbach.house/.well-known/matrix/server
< Server: Caddy
< Date: Thu, 09 Feb 2023 19:28:24 GMT
< Content-Length: 0
< 
* Closing connection 0
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://eschbach.house/.well-known/matrix/server'
* Connecting to hostname: 127.0.0.1
* Connecting to port: 443
*   Trying 127.0.0.1:443...
* Connected to (nil) (127.0.0.1) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Feb  9 13:20:36 2023 GMT
*  expire date: Feb 10 01:20:36 2023 GMT
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x555b4d4c5e80)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /.well-known/matrix/server HTTP/2
> Host: eschbach.house
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< access-control-allow-origin: *
< alt-svc: h3=":443"; ma=2592000
< content-type: application/json
< server: Caddy
< content-length: 0
< date: Thu, 09 Feb 2023 19:28:24 GMT
< 
* Connection #1 to host (nil) left intact

francislavoie · February 9, 2023, 7:38pm

What does your Caddyfile config looks like now?

What’s gets produced in Caddy’s logs when you make a request? Enable the log directive and the debug global option to see more details.

It seems like Caddy is writing an empty response, seems like a config problem but your original post seems fine. So I assume you changed your config in some way that would have broken it since.

Eschguy · February 9, 2023, 8:29pm

The only real change I’ve made is utilizing the new global trusted_proxies that just dropped (yay, btw).

Caddyfile:

{
        debug
        servers {
                trusted_proxies static 172.16.0.0/24 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.1>
        }
}
eschbach.house {
        log {
                output file /var/log/access.log
        }
        handle_path /.well-known/matrix/* {
                header Access-Control-Allow-Origin *
                header Content-Type application/json
                respond /.well-known/matrix/server `{"m.server": "matrix.eschbach.house:443"}`
                respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://matrix.eschbach.house"},"m.identity_server":{"base_url":"https://identity.eschbach.house"}}`
        }
}

matrix.eschbach.house {
        log {
                output file /var/log/access.log
        }
        reverse_proxy /_matrix/* 172.16.0.231:8008
        reverse_proxy /_synapse/client/* 172.16.0.231:8008
}

Some of the log (hit character limit)

{"level":"info","ts":1675973952.548125,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45634","proto":"HTTP/2.0","method":"POST","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/keys/query","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Authorization":[],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["75.168.220.4"],"Content-Length":["82"],"Accept-Encoding":["gzip"],"Content-Type":["application/json"],"Origin":["https://app.element.io"],"Accept":["application/json"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Dnt":["1"],"Sec-Fetch-Site":["cross-site"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ray":["796f4c7349892324-ORD"],"Priority":["u=4"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.005471735,"size":2899,"status":200,"resp_headers":{"Access-Control-Allow-Origin":["*"],"Content-Type":["application/json"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"]}}
{"level":"info","ts":1675973952.5484884,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45642","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s19_332_2_1_23_1_1_21_0_1","headers":{"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["cross-site"],"Authorization":[],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ray":["796f4c73497f2324-ORD"],"Dnt":["1"],"X-Forwarded-For":["75.168.220.4"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Accept":["application/json"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Origin":["https://app.element.io"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Priority":["u=4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.006383732,"size":284,"status":200,"resp_headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["application/json"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Server":["Caddy","Synapse/1.76.0"]}}
{"level":"info","ts":1675973952.59945,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45618","proto":"HTTP/2.0","method":"POST","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/keys/query","headers":{"Cdn-Loop":["cloudflare"],"Accept":["application/json"],"Sec-Fetch-Mode":["cors"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Site":["cross-site"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Accept-Language":["en-US,en;q=0.5"],"Origin":["https://app.element.io"],"Cf-Connecting-Ip":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Content-Type":["application/json"],"Authorization":[],"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["796f4c73aa462324-ORD"],"Content-Length":["82"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Priority":["u=4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002562086,"size":2899,"status":200,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Content-Type":["application/json"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"]}}
{"level":"error","ts":1675973952.6103637,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45634","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/v3/room_keys/version","headers":{"Dnt":["1"],"Sec-Fetch-Site":["cross-site"],"Cache-Control":["max-age=0"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.5"],"Origin":["https://app.element.io"],"Accept":["application/json"],"Cf-Ray":["796f4c73ba682324-ORD"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Priority":["u=4"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["75.168.220.4"],"Accept-Encoding":["gzip"],"Authorization":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.003578953,"size":51,"status":404,"resp_headers":{"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Origin":["*"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Content-Type":["application/json"]}}
{"level":"info","ts":1675973952.6105292,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45654","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s19_332_2_1_23_1_1_21_0_1","headers":{"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["796f4c73ba5b2324-ORD"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Accept":["application/json"],"Dnt":["1"],"Sec-Fetch-Site":["cross-site"],"Cache-Control":["max-age=0"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Authorization":[],"Cf-Connecting-Ip":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Priority":["u=4"],"Accept-Language":["en-US,en;q=0.5"],"Origin":["https://app.element.io"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.005338302,"size":284,"status":200,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Cache-Control":["no-cache, no-store, must-revalidate"]}}
{"level":"error","ts":1675973952.6109967,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45642","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/v3/room_keys/version","headers":{"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["cross-site"],"Authorization":[],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Origin":["https://app.element.io"],"Dnt":["1"],"Cf-Ray":["796f4c73ba6a2324-ORD"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Cache-Control":["max-age=0"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Sec-Fetch-Dest":["empty"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Priority":["u=4"],"Accept":["application/json"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002396007,"size":51,"status":404,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"]}}
{"level":"error","ts":1675973952.6539912,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45618","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/v3/room_keys/version","headers":{"Cdn-Loop":["cloudflare"],"Sec-Fetch-Mode":["cors"],"Cf-Connecting-Ip":["75.168.220.4"],"Accept":["application/json"],"Sec-Fetch-Dest":["empty"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["75.168.220.4"],"X-Forwarded-Proto":["https"],"Cf-Ray":["796f4c73faf92324-ORD"],"Priority":["u=4"],"Dnt":["1"],"Accept-Language":["en-US,en;q=0.5"],"Origin":["https://app.element.io"],"Sec-Fetch-Site":["cross-site"],"Authorization":[],"Cache-Control":["max-age=0"],"Cf-Ipcountry":["US"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002391605,"size":51,"status":404,"resp_headers":{"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Content-Type":["application/json"]}}
{"level":"info","ts":1675973952.6562002,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45654","proto":"HTTP/2.0","method":"POST","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/keys/query","headers":{"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Content-Type":["application/json"],"Origin":["https://app.element.io"],"Sec-Fetch-Site":["cross-site"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"],"Priority":["u=4"],"Sec-Fetch-Dest":["empty"],"Cf-Ray":["796f4c740b082324-ORD"],"Content-Length":["82"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Sec-Fetch-Mode":["cors"],"Authorization":[]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002045737,"size":2899,"status":200,"resp_headers":{"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"]}}
{"level":"info","ts":1675973952.6603692,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45634","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s19_332_2_1_23_1_1_21_0_1","headers":{"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["796f4c740b132324-ORD"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Dnt":["1"],"Cf-Ipcountry":["US"],"Sec-Fetch-Mode":["cors"],"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["empty"],"X-Forwarded-Proto":["https"],"Priority":["u=4"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Accept-Language":["en-US,en;q=0.5"],"Authorization":[],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["75.168.220.4"],"Sec-Fetch-Site":["cross-site"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002789945,"size":284,"status":200,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Cache-Control":["no-cache, no-store, must-revalidate"]}}
{"level":"info","ts":1675973952.7224867,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45642","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s19_332_2_1_23_1_1_21_0_1","headers":{"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Site":["cross-site"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept":["application/json"],"Cache-Control":["max-age=0"],"X-Forwarded-For":["75.168.220.4"],"Priority":["u=4"],"Cf-Ray":["796f4c746bb32324-ORD"],"Origin":["https://app.element.io"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Sec-Fetch-Mode":["cors"],"Authorization":[],"Cf-Ipcountry":["US"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.003578317,"size":284,"status":200,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Content-Type":["application/json"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
{"level":"info","ts":1675973952.72476,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45618","proto":"HTTP/2.0","method":"POST","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/keys/query","headers":{"Cf-Ray":["796f4c746bb72324-ORD"],"Content-Length":["82"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-US,en;q=0.5"],"Authorization":[],"Cf-Connecting-Ip":["75.168.220.4"],"Accept-Encoding":["gzip"],"Accept":["application/json"],"Origin":["https://app.element.io"],"Dnt":["1"],"Sec-Fetch-Mode":["cors"],"Cdn-Loop":["cloudflare"],"Priority":["u=4"],"Content-Type":["application/json"],"Sec-Fetch-Dest":["empty"],"Cf-Ipcountry":["US"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Sec-Fetch-Site":["cross-site"],"X-Forwarded-For":["75.168.220.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.002106501,"size":2899,"status":200,"resp_headers":{"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"]}}
{"level":"error","ts":1675973952.776794,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45634","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/v3/room_keys/version","headers":{"Priority":["u=4"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Site":["cross-site"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Origin":["https://app.element.io"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Proto":["https"],"Accept":["application/json"],"Authorization":[],"X-Forwarded-For":["75.168.220.4"],"Cf-Ray":["796f4c74cc542324-ORD"],"Dnt":["1"],"Cache-Control":["max-age=0"],"Cf-Connecting-Ip":["75.168.220.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.004279248,"size":51,"status":404,"resp_headers":{"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Content-Type":["application/json"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"]}}
{"level":"info","ts":1675973952.7783368,"logger":"http.log.access.log1","msg":"handled request","request":{"remote_ip":"172.71.254.109","remote_port":"45654","proto":"HTTP/2.0","method":"GET","host":"matrix.eschbach.house","uri":"/_matrix/client/r0/sync?filter=0&timeout=30000&since=s19_332_2_1_23_1_1_21_0_1","headers":{"Accept":["application/json"],"Accept-Language":["en-US,en;q=0.5"],"Dnt":["1"],"X-Forwarded-Proto":["https"],"Priority":["u=4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"],"Origin":["https://app.element.io"],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Site":["cross-site"],"Cache-Control":["max-age=0"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["75.168.220.4"],"Cf-Ray":["796f4c74bc432324-ORD"],"Authorization":[],"Cf-Ipcountry":["US"],"X-Forwarded-For":["75.168.220.4"],"Sec-Fetch-Mode":["cors"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matrix.eschbach.house"}},"user_id":"","duration":0.005551135,"size":284,"status":200,"resp_headers":{"Access-Control-Allow-Headers":["X-Requested-With, Content-Type, Authorization, Date"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Methods":["GET, HEAD, POST, PUT, DELETE, OPTIONS"],"Date":["Thu, 09 Feb 2023 20:19:12 GMT"],"Server":["Caddy","Synapse/1.76.0"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Content-Type":["application/json"]}}

francislavoie · February 10, 2023, 12:28am

This is your problem – you’re stripping /.well-known/matrix from the URL with handle_path, so respond /.well-known/matrix/server doesn’t match anymore.

Either use handle, or change your respond matchers to only match /server and /client instead.

You didn’t have handle_path in your original post. We were working on outdated information. Make sure to include the config next time you ask a followup question!

Eschguy · February 10, 2023, 2:53am

Well that definitely fixed that problem, thank you and apologies, I must have made that change while messing with things.

The issue now is not redirecting correctly.

My Caddyfile entry is now

eschbach.house {
        log {
                output file /var/log/access.log
        }
        header Access-Control-Allow-Origin *
        header Content-Type application/json
        respond /.well-known/matrix/server `{"m.server": "matrix.eschbach.house:443"}`
        respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://matrix.eschbach.house"},"m.identity_server":{"base_url":"https://identity.eschbach.house"}}`
}

matrix.eschbach.house {
        log {
                output file /var/log/access.log
        }
        reverse_proxy /_matrix/* 172.16.0.231:8008
        reverse_proxy /_synapse/client/* 172.16.0.231:8008
}

The issue now is the federation tester is saying:
"Get \"https://eschbach.house/.well-known/matrix/server\": stopped after 10 redirects"

https://federationtester.matrix.org/api/report?server_name=eschbach.house

https://federationtester.matrix.org/#eschbach.house
gives me

Get "https://104.21.37.62:8448/_matrix/key/v2/server": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Get "https://172.67.205.6:8448/_matrix/key/v2/server": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Get "https://[2606:4700:3030::6815:253e]:8448/_matrix/key/v2/server": dial tcp [2606:4700:3030::6815:253e]:8448: i/o timeout (Client.Timeout exceeded while awaiting headers)Get "https://[2606:4700:3032::ac43:cd06]:8448/_matrix/key/v2/server": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

francislavoie · February 10, 2023, 3:08am

How do you have CloudFlare configured? If CloudFlare makes HTTP requests to your server, then Caddy will serve it a redirect.

Eschguy · February 10, 2023, 2:42pm

It’s the same for all my services: Cloudflare proxied with HTTPS rewrites on all requests.

Eschguy · February 13, 2023, 4:46pm

I turned off Cloudflare’s proxy and it’s still timing out after 10 redirects…

emilylange · February 13, 2023, 7:12pm

Check your Cloudflare Zone’s SSL/TLS settings.
The “encryption mode” should be set to Full (strict).

Use https://dash.cloudflare.com/?to=/:account/:zone/ssl-tls as a hotlink to those settings.

Eschguy · February 13, 2023, 7:30pm

That seems to have done it, I had it at Full and have changed it to Full (strict) and now the Federation Tester is coming back positive and I am able to see the rooms at Matrix.org

A huge THANK YOU to both you and @francislavoie, this has definitely been a journey and learning experience.

system · March 15, 2023, 7:30pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.