1. The problem I’m having:
For the past week or two, I’ve been experiencing some strange issues with a handful of my web services, but only when accessing them through iOS devices. A few examples:
- When accessing my Miniflux RSS feeds through the NetNewsWire app, manually refreshing the feeds will trigger the following error.
- Other applications, such as Amperfy, will occasionally hang before throwing similar “cancelled” errors.
- Cryptomator consistently throws “cancelled” errors when attempting to add an existing WebDAV vault, rendering the app unusable. (The debug logs show only
NSURLErrorDomain
code -999 when this happens, which is—you guessed it!—“cancelled”.) - Authelia will occasionally prevent access on a device by responding with “incorrect username and password” to every login attempt. The same credentials work to access the same resource at the same time on non-iOS devices, such as PCs. The logs show timeout requests when Authelia attempts to communicate with Caddy.
This has been a working setup for well over a year, so I’m completely at a loss. None of my configuration has changed, and there aren’t too many common factors between the services that are acting up, so I’m not sure where the problem might be occurring.
2. Error messages and/or full log output:
When refreshing NetNewsWire:
{"level":"debug","ts":1729521246.7065837,"logger":"http.matchers.maxmind_geolocation","msg":"Detected MaxMind data","ip":"[ip-address]","country":"US","subdivisions":"","metro_code":0,"asn":0}
{"level":"debug","ts":1729521246.706681,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"miniflux:8080","total_upstreams":1}
{"level":"debug","ts":1729521246.7065837,"logger":"http.matchers.maxmind_geolocation","msg":"Detected MaxMind data","ip":"[ip-address]","country":"US","subdivisions":"","metro_code":0,"asn":0}
{"level":"debug","ts":1729521246.7068183,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"miniflux:8080","total_upstreams":1}
{"level":"debug","ts":1729521246.7196093,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"miniflux:8080","duration":0.012738377,"request":{"remote_ip":"[ip-address]","remote_port":"64182","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"rss.mydomain.com","uri":"/reader/api/0/stream/items/ids?n=1000&output=json&s=user/-/state/com.google/reading-list&xt=user/-/state/com.google/read","headers":{"User-Agent":["NetNewsWire (RSS Reader; https://netnewswire.com/)"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["rss.mydomain.com"],"Accept":["*/*"],"Authorization":["REDACTED"],"Priority":["u=3"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"rss.mydomain.com"}},"headers":{"Access-Control-Allow-Headers":["Authorization"],"Access-Control-Allow-Methods":["GET, POST, PUT, DELETE, OPTIONS"],"Access-Control-Allow-Origin":["*"],"Content-Type":["application/json"],"X-Content-Type-Options":["nosniff"],"Content-Length":["382"],"Referrer-Policy":["no-referrer"],"Strict-Transport-Security":["max-age=31536000"],"X-Frame-Options":["DENY"],"Date":["Mon, 21 Oct 2024 14:34:06 GMT"]},"status":200}
{"level":"debug","ts":1729521246.7240617,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"miniflux:8080","duration":0.017282085,"request":{"remote_ip":"[ip-address]","remote_port":"64182","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"rss.mydomain.com","uri":"/reader/api/0/stream/items/ids?n=1000&output=json&s=user/-/state/com.google/starred","headers":{"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Host":["rss.mydomain.com"],"Accept":["*/*"],"User-Agent":["NetNewsWire (RSS Reader; https://netnewswire.com/)"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Proto":["https"],"Priority":["u=3"],"Authorization":["REDACTED"],"X-Forwarded-For":["[ip-address]"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"rss.mydomain.com"}},"headers":{"Content-Encoding":["br"],"Content-Type":["application/json"],"Strict-Transport-Security":["max-age=31536000"],"Date":["Mon, 21 Oct 2024 14:34:06 GMT"],"Access-Control-Allow-Headers":["Authorization"],"Access-Control-Allow-Origin":["*"],"Referrer-Policy":["no-referrer"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"Content-Length":["370"],"Access-Control-Allow-Methods":["GET, POST, PUT, DELETE, OPTIONS"]},"status":200}
When attempting to log in with Authelia:
{"level":"debug","ts":1729521451.8113294,"logger":"http.matchers.maxmind_geolocation","msg":"Detected MaxMind data","ip":"[ip-address]","country":"US","subdivisions":"","metro_code":0,"asn":0}
{"level":"debug","ts":1729521451.8133566,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"authelia:9091","total_upstreams":1}
{"level":"debug","ts":1729521451.8140984,"logger":"http.matchers.maxmind_geolocation","msg":"Detected MaxMind data","ip":"[ip-address]","country":"US","subdivisions":"","metro_code":0,"asn":0}
{"level":"debug","ts":1729521451.814136,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"authelia:9091","total_upstreams":1}
{"level":"debug","ts":1729521451.8146186,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.002913376,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/layouts.Login.CM13xCjW.js","headers":{"X-Forwarded-Host":["auth.mydomain.com"],"Sec-Fetch-Dest":["script"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["cors"],"Priority":["u=1, i"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"X-Forwarded-For":["[ip-address]"],"Accept":["*/*"],"If-None-Match":["a02a4b6953c28c720da87bb8484a01a6848f80b7"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Cookie":["REDACTED"],"Origin":["https://auth.mydomain.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["a02a4b6953c28c720da87bb8484a01a6848f80b7"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.8152106,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.003752045,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.AlertTitle.uuJpj6dv.js","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["auth.mydomain.com"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["script"],"Sec-Fetch-Mode":["cors"],"Priority":["u=1, i"],"X-Forwarded-For":["[ip-address]"],"Accept-Language":["en-US,en;q=0.9"],"If-None-Match":["e933362d57ba298fbc3c0c6f7089d68a1d1557b7"],"Sec-Fetch-Site":["same-origin"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Origin":["https://auth.mydomain.com"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Etag":["e933362d57ba298fbc3c0c6f7089d68a1d1557b7"],"Cache-Control":["public, max-age=0, must-revalidate"],"Date":["Mon, 21 Oct 2024 14:37:31 GMT"]},"status":304}
{"level":"debug","ts":1729521451.8158019,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.003387909,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.TextField.Bhz0840J.js","headers":{"Accept-Encoding":["gzip, deflate, br"],"If-None-Match":["9fd9dc7aa85dabd1563a7d24704fcf8f5bacb850"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"Accept-Language":["en-US,en;q=0.9"],"Origin":["https://auth.mydomain.com"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Cookie":["REDACTED"],"X-Forwarded-Host":["auth.mydomain.com"],"Priority":["u=1, i"],"Accept":["*/*"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Dest":["script"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["9fd9dc7aa85dabd1563a7d24704fcf8f5bacb850"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.816225,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.002061873,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.Checkbox.CHnpkqUM.js","headers":{"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Origin":["https://auth.mydomain.com"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, deflate, br"],"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Dest":["script"],"If-None-Match":["c2e1174c88005c1cd5a6003bdac7f99ac753db57"],"Cookie":["REDACTED"],"Priority":["u=1, i"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Host":["auth.mydomain.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["c2e1174c88005c1cd5a6003bdac7f99ac753db57"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.8163154,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.004071747,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.Toolbar.DDc4MPpe.js","headers":{"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Host":["auth.mydomain.com"],"If-None-Match":["1bbaca2d7d809960e801ec29becbb283ba7ac3c4"],"Accept":["*/*"],"Sec-Fetch-Dest":["script"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Proto":["https"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"Origin":["https://auth.mydomain.com"],"Sec-Fetch-Mode":["cors"],"Cookie":["REDACTED"],"Priority":["u=1, i"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["1bbaca2d7d809960e801ec29becbb283ba7ac3c4"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.8164673,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.003027642,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/index.DR-THeLB.js","headers":{"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Accept":["*/*"],"If-None-Match":["abdcdd5023197aa273d30d3d3dea8d1c1ba7aa52"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Host":["auth.mydomain.com"],"Sec-Fetch-Dest":["script"],"Priority":["u=1, i"],"Origin":["https://auth.mydomain.com"],"Cookie":["REDACTED"],"Sec-Fetch-Mode":["cors"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["abdcdd5023197aa273d30d3d3dea8d1c1ba7aa52"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.8166275,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.004127742,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.useFormControl.DXOxQ54l.js","headers":{"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Host":["auth.mydomain.com"],"Sec-Fetch-Mode":["cors"],"Sec-Fetch-Dest":["script"],"Accept":["*/*"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"Origin":["https://auth.mydomain.com"],"Priority":["u=1, i"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"X-Forwarded-For":["[ip-address]"],"X-Forwarded-Proto":["https"],"If-None-Match":["cc45a343f570e9d6f8b417406065492e44e3850d"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["REDACTED"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["cc45a343f570e9d6f8b417406065492e44e3850d"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
{"level":"debug","ts":1729521451.8167155,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"authelia:9091","duration":0.003890877,"request":{"remote_ip":"[ip-address]","remote_port":"64794","client_ip":"[ip-address]","proto":"HTTP/3.0","method":"GET","host":"auth.mydomain.com","uri":"/static/js/mui.FormControlLabel.BPfx_vCT.js","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-For":["[ip-address]"],"Origin":["https://auth.mydomain.com"],"Referer":["https://auth.mydomain.com/?rd=https%3A%2F%2Fredlib.mydomain.com%2F&rm=GET"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 18_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/131.4 Mobile/15E148 Safari/605.1.15"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Host":["auth.mydomain.com"],"Accept":["*/*"],"Sec-Fetch-Mode":["cors"],"Priority":["u=1, i"],"Accept-Language":["en-US,en;q=0.9"],"If-None-Match":["68cb088b76ab7148e6640243b5d6df2f8f6f11fe"],"Cookie":["REDACTED"],"Sec-Fetch-Dest":["script"],"Sec-Fetch-Site":["same-origin"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"auth.mydomain.com"}},"headers":{"Date":["Mon, 21 Oct 2024 14:37:31 GMT"],"Etag":["68cb088b76ab7148e6640243b5d6df2f8f6f11fe"],"Cache-Control":["public, max-age=0, must-revalidate"]},"status":304}
3. Caddy version:
Using Docker Caddy, version 2.8.4, built with Porkbun DNS module and caddy-maxmind-geolocation.
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
All affected applications are running in Docker containers; Miniflux is on the same machine as Caddy, while others are running on a separate machine accessed through WireGuard.
4. How I installed and ran Caddy:
a. System environment:
Debian 12, running Caddy (and other applications) in Docker.
b. Command:
docker compose up -d
c. Service/unit/compose file:
services:
caddy:
container_name: caddy
build: .
restart: always
networks:
- authelia_default
- changedetection_default
- hedgedoc_default
- miniflux_default
- rsshub_default
- umami_default
ports:
- 80:80
- 443:443
- 443:443/udp
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- ./data:/data
- ./logs:/var/log/caddy
- ./site:/srv
- /etc/localtime:/etc/localtime:ro
- config:/config
env_file: .env
labels:
- diun.enable=false
networks:
authelia_default:
external: true
changedetection_default:
external: true
hedgedoc_default:
external: true
miniflux_default:
external: true
rsshub_default:
external: true
umami_default:
external: true
volumes:
config:
d. My complete Caddy config:
{
email hello@mydomain.com
}
(authelia) {
forward_auth authelia:9091 {
uri /api/authz/forward-auth
copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
}
}
(geofilter) {
@geofilter {
not maxmind_geolocation {
db_path "/data/maxmind/GeoLite2-Country.mmdb"
allow_countries US
}
}
respond @geofilter 403
}
(log) {
log {
output file /var/log/caddy/{args[0]}.log
}
}
(robots) {
@robots {
path /robots.txt
}
respond @robots 200 {
body "User-agent: *
Disallow: /"
}
}
(tls) {
tls {
dns porkbun {
api_key {env.PORKBUN_API_KEY}
api_secret_key {env.PORKBUN_API_SECRET_KEY}
}
}
}
*.mydomain.com {
redir https://mydomain.com{$uri}
import tls
import log "access"
}
mydomain.com {
header Strict-Transport-Security "max-age=31536000"
root * /srv
file_server
import geofilter
import robots
import tls
import log "access"
}
auth.mydomain.com {
reverse_proxy authelia:9091
import geofilter
import robots
import tls
import log "authelia"
}
cd.mydomain.com {
import authelia
reverse_proxy changedetection:5000
import geofilter
import robots
import tls
import log "changedetection"
}
cloud.mydomain.com {
header Strict-Transport-Security "max-age=15552000"
redir /.well-known/carddav /remote.php/dav 301
redir /.well-known/caldav /remote.php/dav 301
redir /.well-known/* /index.php{uri} 301
reverse_proxy 10.120.0.2:9090
import geofilter
import tls
import log "nextcloud"
}
docs.mydomain.com {
reverse_proxy hedgedoc:3000
import geofilter
import robots
import tls
import log "hedgedoc"
}
hoarder.mydomain.com {
reverse_proxy 10.120.0.2:3000
import geofilter
import robots
import tls
import log "hoarder"
}
mealie.mydomain.com {
reverse_proxy 10.120.0.2:9935
import geofilter
import robots
import tls
import log "mealie"
}
media.mydomain.com {
reverse_proxy 10.120.0.2:8096
import geofilter
import tls
import log "jellyfin"
}
music.mydomain.com {
import authelia
reverse_proxy 10.120.0.2:4533
import geofilter
import tls
import log "navidrome"
}
readeck.mydomain.com {
reverse_proxy 10.120.0.2:8000
import geofilter
import tls
import log "readeck"
}
redlib.mydomain.com {
import authelia
reverse_proxy 10.120.0.2:8282
import geofilter
import robots
import tls
import log "redlib"
}
rss.mydomain.com {
reverse_proxy miniflux:8080
import geofilter
import robots
import tls
import log "miniflux"
}
rsshub.mydomain.com {
reverse_proxy rsshub:1200
import geofilter
import robots
import tls
import log "rsshub"
}
umami.mydomain.com {
reverse_proxy umami:3000
import geofilter
import robots
import log "umami"
}
I’ve been stuck on this for days with no leads—would be hugely grateful for any input! Thanks so much!