URLs suddenly stops working for a while

Help
1

It stops and suddenly after few minutes it is back up and running. I get logs like

This is just a example for notes app, same happens with Jellyfin multiple times a day

ERR ts=1715264259.2358754 logger=http.log.error msg=dial tcp 192.168.1.9:22300: connect: connection refused request={"remote_ip":"192.168.1.223","remote_port":"54210","client_ip":"192.168.1.223","proto":"HTTP/1.1","method":"GET","host":"notes.gharnas.duckdns.org","uri":"/api/share_users","headers":{"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"X-Api-Auth":["p1rN7xiiJ27lvvxJPWZw5A"],"X-Api-Min-Version":["2.6.0"],"Accept":["*/*"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"notes.gharnas.duckdns.org"}} duration=0.001016577 status=502 err_id=ap0fu3m5i err_trace=reverseproxy.statusError (reverseproxy.go:1267)
ERR ts=1715264387.4663417 logger=http.log.error msg=dial tcp 192.168.1.9:22300: connect: connection refused request={"remote_ip":"192.168.1.223","remote_port":"60292","client_ip":"192.168.1.223","proto":"HTTP/3.0","method":"GET","host":"notes.gharnas.duckdns.org","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua":["\"Chromium\";v=\"124\", \"Brave\";v=\"124\", \"Not-A.Brand\";v=\"99\""],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=0, i"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"notes.gharnas.duckdns.org"}} duration=0.000921424 status=502 err_id=m03wt2mas err_trace=reverseproxy.statusError (reverseproxy.go:1267)
ERR ts=1715264389.7886176 logger=http.log.error msg=dial tcp 192.168.1.9:22300: connect: connection refused request={"remote_ip":"192.168.1.223","remote_port":"60292","client_ip":"192.168.1.223","proto":"HTTP/3.0","method":"GET","host":"notes.gharnas.duckdns.org","uri":"/","headers":{"Cache-Control":["max-age=0"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-User":["?1"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Chromium\";v=\"124\", \"Brave\";v=\"124\", \"Not-A.Brand\";v=\"99\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"notes.gharnas.duckdns.org"}} duration=0.000846325 status=502 err_id=jyvyzthtj err_trace=reverseproxy.statusError (reverseproxy.go:1267)
ERR ts=1715264408.541373 logger=http.log.error msg=dial tcp 192.168.1.9:22300: connect: connection refused request={"remote_ip":"192.168.1.223","remote_port":"60292","client_ip":"192.168.1.223","proto":"HTTP/3.0","method":"GET","host":"notes.gharnas.duckdns.org","uri":"/","headers":{"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br, zstd"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Priority":["u=0, i"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\"Chromium\";v=\"124\", \"Brave\";v=\"124\", \"Not-A.Brand\";v=\"99\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"notes.gharnas.duckdns.org"}} duration=0.00038693 status=502 err_id=q3b4r5mz9 err_trace=reverseproxy.statusError (reverseproxy.go:1267)
ERR ts=1715264410.9711967 logger=http.log.error msg=dial tcp 192.168.1.9:22300: connect: connection refused request={"remote_ip":"192.168.1.223","remote_port":"60292","client_ip":"192.168.1.223","proto":"HTTP/3.0","method":"GET","host":"notes.gharnas.duckdns.org","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Priority":["u=0, i"],"Cache-Control":["no-cache"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua":["\"Chromium\";v=\"124\", \"Brave\";v=\"124\", \"Not-A.Brand\";v=\"99\""],"Sec-Gpc":["1"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["document"],"Pragma":["no-cache"],"Sec-Ch-Ua-Platform":["\"Windows\""],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"notes.gharnas.duckdns.org"}} duration=0.000973919 status=502 err_id=dsn20u1hd err_trace=reverseproxy.statusError (reverseproxy.go:1267)
2

It seems like your upstream application goes down then back up. Are they being killed by the operating system? This may happen if the operating system is overwhelmed with not enough free memory or CPU overload.

2 Likes
4

Ahh! I see, it suddenly started happening since last 2 days and it goes down for Jellyfin quite often. So it’s all about RAM or CPU correct?

5

Most likely, but really depneds on your OS. Check the logs of the init system and maybe kernel logs/dmesg.

2 Likes
7

As I checked for the Application, I am able to access it via IP:port, so it doesn’t seem to go down it is only the URL that keeps turn off and on.

As I tested it trice earlier this morning with Jellyfin, and IP:port worked each time while my link stopped working for 2 to 3 minutes, and it kept happening every 10 minutes or so

8

Attaching few logs for Jellyfin. It does work flawlessly on local IP:port (192.168.1.9:8096)

ERR ts=1715272493.9494827 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=2.891943004 request={"remote_ip":"192.168.1.3","remote_port":"33680","client_ip":"192.168.1.3","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/75db5a7c-82c7-9c78-b2ff-778337e6a4bc/stream?static=true&mediaSourceId=75db5a7c82c79c78b2ff778337e6a4bc&streamOptions=%7B%7D","headers":{"Accept":["*/*"],"Range":["bytes=1721031453-"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"],"X-Forwarded-For":["192.168.1.3"],"Icy-Metadata":["1"],"User-Agent":["libmpv"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=writing: write tcp 192.168.1.197:443->192.168.1.3:33680: write: connection reset by peer
ERR ts=1715272503.6515975 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=6.902231798 request={"remote_ip":"192.168.1.3","remote_port":"55522","client_ip":"192.168.1.3","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/75db5a7c-82c7-9c78-b2ff-778337e6a4bc/stream?static=true&mediaSourceId=75db5a7c82c79c78b2ff778337e6a4bc&streamOptions=%7B%7D","headers":{"User-Agent":["libmpv"],"Accept":["*/*"],"Range":["bytes=912655233-"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"],"X-Forwarded-For":["192.168.1.3"],"Icy-Metadata":["1"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=reading: context canceled
ERR ts=1715272566.059586 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=8.692780184 request={"remote_ip":"192.168.1.3","remote_port":"58644","client_ip":"192.168.1.3","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/cc9dc0fd-b7ac-87d3-3dbf-9dd9a715020a/stream?static=true&mediaSourceId=cc9dc0fdb7ac87d33dbf9dd9a715020a&streamOptions=%7B%7D","headers":{"Range":["bytes=0-"],"X-Forwarded-For":["192.168.1.3"],"Icy-Metadata":["1"],"User-Agent":["libmpv"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=writing: write tcp 192.168.1.197:443->192.168.1.3:58644: write: connection reset by peer
ERR ts=1715272584.8801365 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=6.557281375 request={"remote_ip":"192.168.1.3","remote_port":"46342","client_ip":"192.168.1.3","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/cc9dc0fd-b7ac-87d3-3dbf-9dd9a715020a/stream?static=true&mediaSourceId=cc9dc0fdb7ac87d33dbf9dd9a715020a&streamOptions=%7B%7D","headers":{"X-Forwarded-For":["192.168.1.3"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"],"Icy-Metadata":["1"],"User-Agent":["libmpv"],"Accept":["*/*"],"Range":["bytes=1339-"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=writing: write tcp 192.168.1.197:443->192.168.1.3:46342: write: connection reset by peer
ERR ts=1715272593.554295 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=7.068422418 request={"remote_ip":"192.168.1.3","remote_port":"54682","client_ip":"192.168.1.3","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/cc9dc0fd-b7ac-87d3-3dbf-9dd9a715020a/stream?static=true&mediaSourceId=cc9dc0fdb7ac87d33dbf9dd9a715020a&streamOptions=%7B%7D","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"],"User-Agent":["libmpv"],"Accept":["*/*"],"Range":["bytes=46962026-"],"X-Forwarded-For":["192.168.1.3"],"Icy-Metadata":["1"]},"tls":{"resumed":false,"version":771,"cipher_suite":52393,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=writing: write tcp 192.168.1.197:443->192.168.1.3:54682: write: connection reset by peer
ERR ts=1715272603.5106115 logger=http.handlers.reverse_proxy msg=aborting with incomplete response upstream=192.168.1.9:8096 duration=5.85084069 request={"remote_ip":"192.168.1.223","remote_port":"47530","client_ip":"192.168.1.223","proto":"HTTP/1.1","method":"GET","host":"media.gharnas.duckdns.org","uri":"/Videos/75db5a7c82c79c78b2ff778337e6a4bc/stream.mkv?Static=true&mediaSourceId=75db5a7c82c79c78b2ff778337e6a4bc&deviceId=SmVsbHlmaW5NZWRpYVBsYXllciAxLjcuMSAod2luZG93cy14ODZfNjQgMTApfDE2Njg4Nzg3NzIzMzM1&api_key=09713071671a440e954a9d162385e92c&Tag=c66e4eab2ca7b6fc31310f338cf77194","headers":{"Range":["bytes=1879416576-"],"X-Forwarded-For":["192.168.1.223"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["media.gharnas.duckdns.org"],"Icy-Metadata":["1"],"User-Agent":["JellyfinMediaPlayer"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"media.gharnas.duckdns.org"}} error=writing: write tcp 192.168.1.197:443->192.168.1.223:47530: write: connection reset by peer
9

This definitely isn’t Caddy. Something in your network or your upstream app is doing it. A one-time access of the upstream IP/port isn’t the same as a proxy continuously pushing bytes back and forth.

To have a better idea, can you share the Caddyfile? Also how are you running Caddy and the upstream apps?

2 Likes
10

Sure I do understand that accessing the IP:port doesn’t mean it is same as proxy continuously pinging it. I just shared that to confirm that, the services weren’t getting killed by OS in the backend. As the services were still up on IP:port

Sure here is my caddyfile

{
    acme_ca https://acme-v02.api.letsencrypt.org/directory
    acme_dns duckdns random-lola-toll-boot-qwsadsadasd
}

ducknas.duckdns.org:443 {
    tls {
        dns duckdns random-lola-toll-boot-qwsadsadasd
    }

    # This setting may have compatibility issues with some browsers
    # (e.g., attachment downloading on Firefox). Try disabling this
    # if you encounter issues.
    encode gzip
    
    reverse_proxy 192.168.1.9:80
 
}

*.ducknas.duckdns.org:443 {
    tls {
        dns duckdns random-lola-toll-boot-qwsadsadasd
    }

    # This setting may have compatibility issues with some browsers
    # (e.g., attachment downloading on Firefox). Try disabling this
    # if you encounter issues.
    encode gzip
    
    @files host files.ducknas.duckdns.org
        handle @files {
                reverse_proxy 192.168.1.9:8081
        }

    @tv host tv.ducknas.duckdns.org
        handle @tv {
                reverse_proxy 192.168.1.9:8989
        } 

    @dash host dash.ducknas.duckdns.org
        handle @dash {
                reverse_proxy 192.168.1.9:8090
        } 

    @ssh host ssh.ducknas.duckdns.org
        handle @ssh {
            reverse_proxy 192.168.1.9:3000
        }

    @vault host vault.ducknas.duckdns.org
        handle @vault {
            reverse_proxy vaultwarden:80
            reverse_proxy /notifications/hub vaultwarden:3012
        }
        
    @notes host notes.ducknas.duckdns.org
        handle @notes {
            reverse_proxy 192.168.1.9:22300
        }

    @movies host movies.ducknas.duckdns.org
        handle @movies {
                reverse_proxy 192.168.1.9:7878
        }

    @subs host subs.ducknas.duckdns.org
        handle @subs {
                reverse_proxy 192.168.1.9:6767
        }

    @docks host docks.ducknas.duckdns.org
        handle @docks {
                reverse_proxy 192.168.1.9:9000
        }
    
    # @speedtest host speedtest.ducknas.duckdns.org
    #    handle @speedtest {
    #            reverse_proxy 192.168.1.9:49159
    #    }

    @flare host flare.ducknas.duckdns.org
        handle @flare {
                reverse_proxy 192.168.1.9:8191
        } 

    @torrents host torrents.ducknas.duckdns.org
        handle @torrents {
                reverse_proxy 192.168.1.9:8080
        } 

    @photos host photos.ducknas.duckdns.org
        handle @photos {
                reverse_proxy 192.168.1.9:8089
        } 

    @sync host sync.ducknas.duckdns.org
        handle @sync {
                reverse_proxy 192.168.1.9:8384
        } 

    @backup host backup.ducknas.duckdns.org
        handle @backup {
                reverse_proxy 192.168.1.9:8200
        } 
    
    @indexer host indexer.ducknas.duckdns.org
        handle @indexer {
                reverse_proxy 192.168.1.9:9117
        } 
    
    @monitor host monitor.ducknas.duckdns.org
        handle @monitor {
                reverse_proxy 192.168.1.9:19999
        } 

    @media host media.ducknas.duckdns.org
        handle @media {
                reverse_proxy 192.168.1.9:8096
                reverse_proxy 192.168.1.9:8920
                reverse_proxy 192.168.1.9:7359
                reverse_proxy 192.168.1.9:1900
        } 

    @adguard host adguard.ducknas.duckdns.org
        handle @adguard {
                reverse_proxy adguardhome:80
                reverse_proxy adguardhome:53
                reverse_proxy adguardhome:67
                reverse_proxy adguardhome:68
                reverse_proxy adguardhome:443
                reverse_proxy adguardhome:3000
                reverse_proxy adguardhome:853
                reverse_proxy adguardhome:784
                reverse_proxy adguardhome:8853
                reverse_proxy adguardhome:5443
        } 


}
11

These don’t make sense. You can’t proxy to multiple different ports at the same time like that (unless you’re trying to load balance between them, but that’s not what you’re trying to do, I’m sure). Each port has a different purpose. You need to tell Caddy how to split the traffic if at all.

Remove all those reverse_proxy lines except the first in each handle. You should proxy to the app’s HTTP port. For Jellyfin, that’s typically 8096, and for adguard, as far as I know, it’s port 80. It certainly doesn’t make sense to proxy to adguard’s port 53 for example, because that’s the DNS port. Caddy is not a DNS server, it’s an HTTP server.

2 Likes
12

Ahh! Thank you so much, I thought so the same at first when I first made this caddyfile (coping/editing it from some youtube guide). But as everything worked great for almost a year before it gave me some trouble, so I didn’t give it much thought back then. I would definitely make the changes and see how it goes, thank you so much.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.