Suddenly stopped working - timing out without errors

1. Output of caddy version:

2. How I run Caddy:

a. System environment:

docker

b. Command:

docker compose up -d

c. Service/unit/compose file:

version: "2.1"

networks:
  caddy:

services:    
  caddy:
    image: caddy:2.3.0
    restart: unless-stopped
    container_name: caddy
    ports:
      - 84:80
      - 444:443
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
      - ./caddy/site:/srv
      - ./caddy/caddy_data:/data
      - ./caddy/caddy_config:/config
    networks:
      - caddy

  portainer:
    image: portainer/portainer-ce
    container_name: portainer_ce
    ports:
      - 9000:9000
    volumes:
      - ./portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
    networks:
      - caddy

  homer:
    image: b4bz/homer:latest
    container_name: homer
    environment:
      - PUID=1000
      - PGID=1000    
    volumes:
      - ./homer_assets/:/www/assets
    ports:
      - 8095:8080
    restart: always

volumes:
  caddy_data:
    external: true
  caddy_config:

d. My complete Caddy config:

{
	email email@address.com
}

jellyfin.domain {
	reverse_proxy 192.168.86.60:8096
}

portainer.domain {
	reverse_proxy portainer:9000
}

photoprism.domain {
	reverse_proxy 192.168.86.60:2342
}

audioshelf.domain{
	reverse_proxy 192.168.86.60:13378
}


md.domain, element.md.domain, matrix.md.domain {

      # creates letsencrypt certificate
      # tls your@email.com

      header {
                # Enable HTTP Strict Transport Security (HSTS) to force clients to always connect via HTTPS
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
                # Enable cross-site filter (XSS) and tell browser to block detected attacks
                X-XSS-Protection "1; mode=block"
                # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
                X-Content-Type-Options "nosniff"
                # Disallow the site to be rendered within a frame (clickjacking protection)
                X-Frame-Options "DENY"
                # X-Robots-Tag
                X-Robots-Tag "noindex, noarchive, nofollow"
        }

        handle {
              encode zstd gzip

              reverse_proxy http://192.168.86.72:81 {
                     header_up X-Forwarded-Port {http.request.port}
                     header_up X-Forwarded-Proto {http.request.scheme}
        }
}

3. The problem I’m having:

suddenly stopped working. i was just restarting my server and after that i noticed i couldn’t access jellyfin. direct ip was working so i realized it was a caddy issue. unable to go through caddy to any of my services. if i try to access them, the request times out. it’ll keep loading for awhile until it says timed out. i’ll put a snippet below the dockers logs.

4. Error messages and/or full log output:

INF ts=1670382965.1011183 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile

INF ts=1670382965.1080086 logger=admin msg=admin endpoint started address=tcp/localhost:2019 enforce_origin=false origins=["localhost:2019","[::1]:2019","127.0.0.1:2019"]

INF ts=1670382965.1099286 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc0003ca000

INF ts=1670382965.1108608 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443

INF ts=1670382965.1108844 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0

INF ts=1670382965.112319 logger=http msg=enabling automatic TLS certificate management domains=["element.md.domain","matrix.md.domain","portainer.domain","wireguard.domain","jellyfin.domain","photoprism.domain","audioshelf.domain","md.domain"]

WRN ts=1670382985.1328013 logger=tls msg=stapling OCSP error=no OCSP stapling for [element.md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:52056->127.0.0.11:53: i/o timeout

WRN ts=1670383005.145432 logger=tls msg=stapling OCSP error=no OCSP stapling for [matrix.md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:37205->127.0.0.11:53: i/o timeout

WRN ts=1670383025.1496916 logger=tls msg=stapling OCSP error=no OCSP stapling for [portainer.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:43486->127.0.0.11:53: i/o timeout

WRN ts=1670383045.1563635 logger=tls msg=stapling OCSP error=no OCSP stapling for [wireguard.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:35658->127.0.0.11:53: i/o timeout

WRN ts=1670383065.1666393 logger=tls msg=stapling OCSP error=no OCSP stapling for [jellyfin.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:37986->127.0.0.11:53: i/o timeout

WRN ts=1670383085.1866305 logger=tls msg=stapling OCSP error=no OCSP stapling for [photoprism.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:36015->127.0.0.11:53: i/o timeout

WRN ts=1670383105.1930833 logger=tls msg=stapling OCSP error=no OCSP stapling for [audioshelf.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:37515->127.0.0.11:53: i/o timeout

WRN ts=1670383125.2059655 logger=tls msg=stapling OCSP error=no OCSP stapling for [md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:36209->127.0.0.11:53: i/o timeout

INF ts=1670383125.212933 msg=autosaved config file=/config/caddy/autosave.json

INF ts=1670383125.2129765 msg=serving initial configuration

INF ts=1670383125.2275429 logger=tls msg=cleaned up storage units

INF ts=1670383438.923854 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile

INF ts=1670383438.9287724 logger=admin msg=admin endpoint started address=tcp/localhost:2019 enforce_origin=false origins=["localhost:2019","[::1]:2019","127.0.0.1:2019"]

INF ts=1670383438.9300244 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc000436e00

INF ts=1670383438.9313545 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443

INF ts=1670383438.9313731 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0

INF ts=1670383438.9334924 logger=http msg=enabling automatic TLS certificate management domains=["portainer.domain","wireguard.domain","jellyfin.domain","photoprism.domain","audioshelf.domain","md.domain","element.md.domain","matrix.md.domain"]

INF ts=1670383438.9750254 logger=tls msg=cleaned up storage units

WRN ts=1670383458.9485354 logger=tls msg=stapling OCSP error=no OCSP stapling for [portainer.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:41198->127.0.0.11:53: i/o timeout

WRN ts=1670383478.953822 logger=tls msg=stapling OCSP error=no OCSP stapling for [wireguard.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:43660->127.0.0.11:53: i/o timeout

WRN ts=1670383498.957152 logger=tls msg=stapling OCSP error=no OCSP stapling for [jellyfin.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:40503->127.0.0.11:53: i/o timeout

WRN ts=1670383518.9601293 logger=tls msg=stapling OCSP error=no OCSP stapling for [photoprism.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:36090->127.0.0.11:53: i/o timeout

WRN ts=1670383538.9715683 logger=tls msg=stapling OCSP error=no OCSP stapling for [audioshelf.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:51031->127.0.0.11:53: i/o timeout

WRN ts=1670383558.981097 logger=tls msg=stapling OCSP error=no OCSP stapling for [md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:46498->127.0.0.11:53: i/o timeout

WRN ts=1670383578.9904218 logger=tls msg=stapling OCSP error=no OCSP stapling for [element.md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:59288->127.0.0.11:53: i/o timeout

WRN ts=1670383599.0029407 logger=tls msg=stapling OCSP error=no OCSP stapling for [matrix.md.domain]: making OCSP request: Post "http://r3.o.lencr.org": dial tcp: lookup r3.o.lencr.org on 127.0.0.11:53: read udp 127.0.0.1:55137->127.0.0.11:53: i/o timeout

INF ts=1670383599.0101924 msg=autosaved config file=/config/caddy/autosave.json

INF ts=1670383599.0102413 msg=serving initial configuration

The connection has timed out

An error occurred during a connection to jellyfin.domain

    The site could be temporarily unavailable or too busy. Try again in a few moments.
    If you are unable to load any pages, check your computer’s network connection.
    If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the web.

5. What I already tried:

ports are at 84/444 since that’s what i had working before and that’s what my router has. tried 443 but that didn’t work. tried to go back to caddy:2.3.0, that didn’t work either. tried redownloading latest images as well. having a hard time figure out what the issue is.

6. Links to relevant resources:

That’s a very old version. Please upgrade to v2.6.2. We can’t reasonably support old versions of Caddy.

Docker Compose file versions are deprecated now – Docker Compose v2 (v2 of the program, not v2 of the file format) has done away with file versions. File v2 is super old anyways, v3 of the file format has been around for many years now.

You probably don’t need to publish these ports, since you’re proxying to it from Caddy, which happens via the Docker network and not via the host machine.

Same here, probably. You can proxy to it with Caddy.

I don’t understand why your logs are in that format. That’s not a log format we support (or not anymore, anyways).

Seems like your machine’s DNS resolver is having some issues. You’ll need to look into that.

I don’t understand. What do you mean by “that’s what my router has”?

For public ACME certificates, you must use ports 80 and 443. That’s a requirement of the ACME protocol.

Clearly there’s a networking issue between your browser and your server. There’s no really enough info here for us to know what the problem is.

Try making a request with curl -v, it’ll give us more information about what’s going on.

It’s possible that your router doesn’t support NAT hairpinning, but it depends on what IP address jellyfin.domain resolves to. We need to see what that looks like (which curl -v would show).

2 Likes

hey thanks for the reply! here’s some more context, hopefully it helps

so i can just remove the external port? would the syntax just be

ports:
- 9000

these are the logs from docker but i pulled it from portainer, maybe that’s why it was a bit strange

sorry, i meant that i had the ports in docker compose as 84 and 444, and i set up my port forwarding rules accordingly, and it was working all this time. i think externally it was 80/443, but internally different, but i could be confusing it.

here’s a curl v of jellyfin.domain:

:~/system$ curl -v jellyfin.domain
*   Trying external_ip:80...
* connect to external_ip port 80 failed: Connection timed out
* Failed to connect to jellyfin.domain port 80 after 130709 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to jellyfin.domain port 80 after 130709 ms: Connection timed out

The syntax would be:

expose:
  - 9000

But you don’t even need that; all ports are implicitly “exposed” to other containers in the same network. The expose option mostly acts as documentation, really.

Try making a request from outside your network, e.g. from your cell phone (not on wifi, on cellular networks).

If that works, then the issue is that your router is doesn’t support NAT hairpinning, which means it doesn’t know how to route packets that have your WAN IP on it back into your LAN to reach your server.

The fix for this is either to get a new router that does support this (:man_shrugging:), or run a DNS server inside your local network and configure your router to use that as your DNS server, and configure the DNS server to resolve your domain to your LAN IP, which would make devices inside your network directly connect to your local server, but every device outside your network would still use the WAN IP to connect.

good to know, thank you.

if i try jellyfin.domain, it says it can’t be reached. i had JUST tried this a few minutes ago, it said that it was not secure. i hit continue, but then it just times out.

edit:
i can ping jellyfin.domain without issues

was getting some docker error, re-did docker.
then was getting some certificate error from caddy (was saying firewall), so i nuked caddy and re-did that.
then did a couple system reboots and now its working. wish i had more to offer if someone comes across this in the future, but that’s all i can say. best of luck!

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.