1. The problem I’m having:
At home I am behind CGNAT, so to expose some services I am running 2 caddy instances: 1 on a VPS and 1 at home with a wireguard VPN between them.
This works well for all GET requests, but only sometimes for PUT and POST requests. That means I can log in and browse my services fine, but hitting ‘save’, watching a movie etc. usually doesn’t work. Such requests seem to time out with no response. If it doesn’t work, it takes a few hours before it works again. I find it difficult to better describe it better.
All the services work fine when accessing them locally. The VPN endpoint IPs are 10.10.10.1 and 10.10.0.1.
I have Authelia running on the VPS in front of the services.
This behaviour is the same for all proxied services.
I didn’t include the actual domain because I am currently not sure about the security of the :80 configuration.
I already tried, with no success
- to disable Authelia (removing the import from the reverse_proxy entires and stopping its container)
- to use a different VPS in front
- numerous header-settings
2. Error messages and/or full log output:
Home: PUT request manually aborted after some time
2023/04/26 06:31:35.986 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "10.0.0.50:8989", "total_upstreams": 1}
2023/04/26 06:32:22.377 ERROR http.handlers.reverse_proxy aborting with incomplete response {"error": "context canceled"}
Same PUT request on VPS:
2023/04/26 06:31:35.972 DEBUG http.handlers.reverse_proxy handling response {"handler": 0}
2023/04/26 06:31:35.972 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "10.10.0.1:80", "total_upstreams": 1}
2023/04/26 06:31:51.007 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "10.10.0.1:80", "duration": 115.859659112, "request": {"remote_ip": "89.204.138.143", "remote_port": "20645", "proto": "HTTP/3.0", "method": "GET", "host": "pong.example.com", "uri": "/signalr/start?transport=serverSentEvents&clientProtocol=2.1&apiKey=snipped&connectionToken=Ud5ReloZ%2F6QyZBRFBtvaeryY7ohQo%2ByQnnfkaUOCfwBy8fjGTHXGNWS7mBvKg46iYBQluvGgZPqZjtJCbILDrT%2FC0wyoBYo6c9eSMt5KXpjf3%2BPfoLyqDKUlO9OgDCojj%2F2mENU0Do9Avd4E5OZ6G0DZdwZ2tyBEC1NxJpnadUQhlVX1%2B4Kl2FPmh4Gqw2cU%2FVGrwKpBwkDCSIL%2FAaaNVUhND7qXPDuzWAbRswHjuN5s4DtOpLhp52VojeWhUiwiQEtfgpvDQ7Q%2B40H9ASs29pCweHNs%2FJCIk%2Bs%2BCaB%2FHq3pbDGuFhK6I4h5%2BuXPkwJi&_=1682483394341", "headers": {"Cookie": [], "Remote-Email": ["x@example.com"], "Content-Type": ["application/json; charset=UTF-8"], "Remote-Groups": ["admins"], "Sec-Fetch-Dest": ["empty"], "Priority": ["u=3, i"], "X-Forwarded-Proto": ["https"], "User-Agent": ["Mozilla/5.0 (iPhone; CPU iPhone OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Mobile/15E148 Safari/604.1"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["cors"], "Accept": ["text/plain, */*; q=0.01"], "Accept-Language": ["de-DE,de;q=0.9"], "X-Forwarded-For": ["89.204.138.143"], "X-Forwarded-Host": ["pong.example.com"], "X-Requested-With": ["XMLHttpRequest"], "Accept-Encoding": ["gzip, deflate, br"], "Referer": ["https://pong.example.com/"], "Remote-User": ["johnny"], "Remote-Name": ["Johnny Jim"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "pong.example.com"}}, "error": "context canceled"}
2023/04/26 06:32:22.363 ERROR http.handlers.reverse_proxy aborting with incomplete response {"error": "context canceled"}
A working PUT request in comparison; at home:
2023/04/26 06:30:01.859 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "10.0.0.50:8989", "total_upstreams": 1}
2023/04/26 06:30:01.881 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "10.0.0.50:8989", "duration": 0.022462352, "request": {"remote_ip": "10.10.10.1", "remote_port": "40546", "proto": "HTTP/1.1", "method": "PUT", "host": "pong.example.com", "uri": "/api/v3/qualityprofile/7?", "headers": {"X-Requested-With": ["XMLHttpRequest"], "Priority": ["u=3, i"], "Remote-Email": ["x@example.com"], "Remote-User": ["johnny"], "Sec-Fetch-Dest": ["empty"], "X-Forwarded-Proto": ["https"], "Content-Length": ["2505"], "X-Api-Key": ["snipped"], "Origin": ["https://pong.example.com"], "Remote-Name": ["Johnny Jim"], "Cookie": [], "Accept": ["application/json, text/javascript, */*; q=0.01"], "Accept-Language": ["de-DE,de;q=0.9"], "X-Forwarded-For": ["89.204.138.143, 10.10.10.1"], "User-Agent": ["Mozilla/5.0 (iPhone; CPU iPhone OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Mobile/15E148 Safari/604.1"], "Remote-Groups": ["admins"], "Sec-Fetch-Mode": ["cors"], "Referer": ["https://pong.example.com/settings/profiles"], "Sec-Fetch-Site": ["same-origin"], "X-Forwarded-Host": ["pong.example.com"], "Accept-Encoding": ["gzip, deflate, br"], "Content-Type": ["application/json"]}}, "headers": {"Date": ["Wed, 26 Apr 2023 04:30:01 GMT"], "Keep-Alive": ["timeout=15,max=99"], "Content-Encoding": ["gzip"], "Vary": ["Accept"], "Cache-Control": ["no-cache, no-store, must-revalidate, max-age=0"], "Pragma": ["no-cache"], "Expires": ["0"], "Access-Control-Allow-Origin": ["*"], "Content-Type": ["application/json; charset=utf-8"], "Set-Cookie": [], "X-Application-Version": ["3.0.10.1567"], "Server": ["Mono-HTTPAPI/1.0"]}, "status": 202}
Same working PUT request on VPS:
023/04/26 06:30:01.845 DEBUG http.handlers.reverse_proxy handling response {"handler": 0}
2023/04/26 06:30:01.846 DEBUG http.handlers.reverse_proxy selected upstream {"dial": "10.10.0.1:80", "total_upstreams": 1}
2023/04/26 06:30:01.895 DEBUG http.handlers.reverse_proxy upstream roundtrip {"upstream": "10.10.0.1:80", "duration": 0.049178459, "request": {"remote_ip": "89.204.138.143", "remote_port": "20645", "proto": "HTTP/3.0", "method": "PUT", "host": "pong.example.com", "uri": "/api/v3/qualityprofile/7?", "headers": {"Priority": ["u=3, i"], "Accept-Encoding": ["gzip, deflate, br"], "Content-Type": ["application/json"], "Remote-User": ["johnny"], "Accept-Language": ["de-DE,de;q=0.9"], "Sec-Fetch-Dest": ["empty"], "Cookie": [], "X-Forwarded-For": ["89.204.138.143"], "X-Forwarded-Host": ["pong.example.com"], "Remote-Groups": ["admins"], "Sec-Fetch-Mode": ["cors"], "Remote-Name": ["Johnny Jim"], "Origin": ["https://pong.example.com"], "Referer": ["https://pong.example.com/settings/profiles"], "Remote-Email": ["x@example.com"], "X-Forwarded-Proto": ["https"], "X-Api-Key": ["snipped"], "User-Agent": ["Mozilla/5.0 (iPhone; CPU iPhone OS 16_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Mobile/15E148 Safari/604.1"], "Accept": ["application/json, text/javascript, */*; q=0.01"], "Sec-Fetch-Site": ["same-origin"], "X-Requested-With": ["XMLHttpRequest"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h3", "server_name": "pong.example.com"}}, "headers": {"Content-Type": ["application/json; charset=utf-8"], "Set-Cookie": [], "Access-Control-Allow-Origin": ["*"], "Cache-Control": ["no-cache, no-store, must-revalidate, max-age=0"], "X-Application-Version": ["3.0.10.1567"], "X-Forwarded-Host": ["pong.example.com"], "Pragma": ["no-cache"], "Server": ["Caddy", "Mono-HTTPAPI/1.0"], "Vary": ["Accept"], "Content-Encoding": ["gzip"], "Date": ["Wed, 26 Apr 2023 04:30:01 GMT"], "Expires": ["0"]}, "status": 202}
3. Caddy version:
VPS:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
home:
caddy version
returns “unknown”, but the installed packages are
caddy-2.6.4-r2 x86_64 {caddy} (Apache-2.0) [installed]
caddy-openrc-2.6.4-r2 x86_64 {caddy} (Apache-2.0) [installed]
4. How I installed and ran Caddy:
a. System environment:
VPS: installed directly on Debian 11
Home: installed directly on Alpine 3.17
b. Command:
VPS: sudo systemctl enable caddy
home: sudo rc-update add caddy
d. My complete Caddy config:
VPS:
{
log {
level DEBUG
output file /var/lib/caddy/log {
roll_size 10mb
roll_keep 5
roll_keep_for 48h
}
format console {
time_local
}
}
email letsencrypt@mail.com
}
(authelia) {
forward_auth localhost:9091 {
uri /api/verify?rd=https://auth.example.com/
copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
}
}
# Authelia Portal.
auth.example.com {
reverse_proxy localhost:9091 {
}
}
pong.example.com {
import authelia
reverse_proxy 10.10.0.1:80 {
transport http {
compression on
}
}
}
home:
{
log {
level DEBUG
output file /var/lib/caddy/log {
roll_size 10mb
roll_keep 5
roll_keep_for 48h
}
format console {
time_local
}
}
}
pong.example.com:80 {
reverse_proxy 10.0.0.50:8989 {
trusted_proxies 10.10.10.1 212.62.184.35
}
header X-Forwarded-Host {host}
}