Reverse proxy not working as expected after update from 2.7.6 > 2.8.4

caddy_appreciator · July 22, 2024, 5:59pm

1. The problem I’m having:

Caddy as a reverse proxy is not working as expected since an update from 2.7.6 > 2.8.4. I changed nothing aside from updating, and I’m having trouble figuring out what is wrong.

I checked the 2.8 breaking changes on github and noticed mention of ZeroSSL now requiring an email, which I previously did not have in my Caddyfile. However, I tried adding my email and still couldn’t get things working, so either it’s unrelated or I added the email wrong.

2. Error messages and/or full log output:

Here is the output of the docker logs:

docker logs --tail 50 --follow --timestamps caddy-caddy-1

2024-07-22T17:43:51.334969727Z {"level":"info","ts":1721670231.3348753,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
2024-07-22T17:43:51.335775857Z {"level":"info","ts":1721670231.3356664,"msg":"adapted config to JSON","adapter":"caddyfile"}
2024-07-22T17:43:51.336441047Z {"level":"info","ts":1721670231.3363712,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
2024-07-22T17:43:51.336582644Z {"level":"info","ts":1721670231.3365228,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
2024-07-22T17:43:51.336592050Z {"level":"info","ts":1721670231.3365357,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
2024-07-22T17:43:51.336635221Z {"level":"info","ts":1721670231.336594,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000053300"}
2024-07-22T17:43:51.336853709Z {"level":"info","ts":1721670231.3368046,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
2024-07-22T17:43:51.336896507Z {"level":"info","ts":1721670231.336856,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
2024-07-22T17:43:51.336997286Z {"level":"info","ts":1721670231.3369484,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
2024-07-22T17:43:51.337023984Z {"level":"info","ts":1721670231.3369994,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
2024-07-22T17:43:51.337035359Z {"level":"info","ts":1721670231.3370214,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jellyseerr.kalen.io","jellyfin.kalen.io"]}
2024-07-22T17:43:51.346763472Z {"level":"info","ts":1721670231.346683,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
2024-07-22T17:43:51.346832500Z {"level":"info","ts":1721670231.3468037,"msg":"serving initial configuration"}
2024-07-22T17:43:51.363992001Z {"level":"info","ts":1721670231.363879,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"62f5e115-7928-43da-8192-ce377c853b1d","try_again":1721756631.3638766,"try_again_in":86399.999999723}
2024-07-22T17:43:51.364017971Z {"level":"info","ts":1721670231.3639472,"logger":"tls","msg":"finished cleaning storage units"}

When adding ‘debug’ mode to the CaddyFile, no further useful information is given because I get the below error. I’m assuming due to bad syntax? Please advise how to properly enable debug in the global options.

{
        debug
}

{
        acme_dns cloudflare API_TOKEN_HERE
}

jellyfin.kalen.io {
        reverse_proxy 192.168.1.101:8096
}

jellyseerr.kalen.io {
        reverse_proxy 192.168.1.101:5055
}

2024-07-22T17:47:50.226760822Z {"level":"info","ts":1721670470.2266097,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
2024-07-22T17:47:50.226785414Z Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first
2024-07-22T17:47:51.153377011Z {"level":"info","ts":1721670471.1532276,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
2024-07-22T17:47:51.153400704Z Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first
2024-07-22T17:47:52.230445426Z {"level":"info","ts":1721670472.2303288,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
2024-07-22T17:47:52.230521176Z Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first
2024-07-22T17:47:53.346945691Z {"level":"info","ts":1721670473.3468075,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
2024-07-22T17:47:53.346978813Z Error: adapting config using caddyfile: server block without any key is global configuration, and if used, it must be first

3. Caddy version:

2.7.6-alpine (Docker) upgraded to 2.8.4-alpine (Docker)

4. How I installed and ran Caddy:

a. System environment:

OS: Debian 12
Architecture: x86_64
Kernel: 6.1.0-23
Docker version 27.0.3, build 7d4bcd8

b. Command:

I created the docker container using an image built from a Dockerfile. Here is the content of my Dockerfile:

FROM caddy:2.8.4-builder-alpine AS builder

RUN xcaddy build \
	--with github.com/caddy-dns/cloudflare

FROM caddy:2.8.4

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

c. Service/unit/compose file:

This is the docker-compose.yml I ran after building the Dockerfile image:

services:
  caddy:
    image: caddy:2.8.4-alpine
    build: .
    security_opt:
      - label:disable
      - no-new-privileges:true
    restart: unless-stopped
    environment:
      - PUID=3000
      - PGID=3000
    volumes:
      - /home/kalen/caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy-data:/data
      - caddy-config:/config
    ports:
      - 80:80
      - 443:443

volumes:
  caddy-data:
  caddy-config:

d. My complete Caddy config:

{
        acme_dns cloudflare API_TOKEN_HERE
}

jellyfin.kalen.io {
        reverse_proxy 192.168.1.101:8096
}

jellyseerr.kalen.io {
        reverse_proxy 192.168.1.101:5055
}

Thanks!

francislavoie · July 22, 2024, 11:52pm

caddy_appreciator:

When adding ‘debug’ mode to the CaddyFile, no further useful information is given because I get the below error. I’m assuming due to bad syntax? Please advise how to properly enable debug in the global options.
{
        debug
}

{
        acme_dns cloudflare API_TOKEN_HERE
}

That’s invalid. You can only have a single global options block. Merge those together.

Aside from that, I don’t see any evidence of a problem at all. There’s no errors in your logs, you haven’t shown an example with curl -v which shows the problem, etc. In what way does it not work? Show more detail.

caddy_appreciator · July 23, 2024, 4:45pm

Thanks for the reply. I’m not sure what the problem is beyond “I can’t get requests to Caddy to resolve, and updating Caddy is the only change I (knowingly) made”.

I was hoping I’m missing something obvious, but it sounds like the problem may lie somewhere else. I’ll keep troubleshooting.

How would I go about testing the connection with curl? -v? I’ve only used curl in the context of downloading files.

francislavoie · July 24, 2024, 3:03am

Just make a request to a URL you would normally hit in your browser that is exhibiting the issue.

The problem here is you haven’t actually described any kind of problem. I don’t know what you expect to get out of this.

caddy_appreciator · July 24, 2024, 2:49pm

The output of curl -verbose jellyfin.kalen.io is simply:

* Could not resolve host: jellyfin.kalen.io
* Closing connection 0
curl: (6) Could not resolve host: jellyfin.kalen.io

The real problem is I have no idea what I’m doing . But! I did learn how to debug and a new way to test with curl, so thanks for that.

In any case, it sounds like the issue is occurring at some point before the request gets to Caddy. Please mark this request as resolved, and I’ll keep smashing my head against other walls.

francislavoie · July 24, 2024, 3:07pm

-v and -verbose are not the same thing. Doing -verbose is the same as doing -v -e rbose (i.e. -v and -e which sets the Referer to rbose). You can do --verbose if you want.

This means your machine’s DNS resolver isn’t resolving that domain to an IP address.

From my machine:

$ host jellyfin.kalen.io                                                             
jellyfin.kalen.io has address 100.97.98.85

$ curl -v jellyfin.kalen.io                                                          
*   Trying 100.97.98.85:80...
* connect to 100.97.98.85 port 80 failed: Network is unreachable
* Failed to connect to jellyfin.kalen.io port 80 after 35 ms: Network is unreachable
* Closing connection 0
curl: (7) Failed to connect to jellyfin.kalen.io port 80 after 35 ms: Network is unreachable

$ curl -v https://jellyfin.kalen.io                                                  
*   Trying 100.97.98.85:443...
* connect to 100.97.98.85 port 443 failed: Network is unreachable
* Failed to connect to jellyfin.kalen.io port 443 after 6 ms: Network is unreachable
* Closing connection 0
curl: (7) Failed to connect to jellyfin.kalen.io port 443 after 6 ms: Network is unreachable

In other words, I do see that your domain resolves to an IP address from my perspective (which means your system’s DNS resolver must be messed up for some reason), but your port 80 and 443 aren’t open and accessible.

system · August 23, 2024, 3:08pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.