RESOLVED: Reverse proxy not working after apt upgrade

Help
1

1. Output of caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

sudo systemctl start caddy.service

sudo systemctl.status.caddy.service

● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-11-05 13:26:17 UTC; 5s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 2955 (caddy)
      Tasks: 9 (limit: 4433)
     CGroup: /system.slice/caddy.service
             └─2955 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0024934,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0026119,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0029204,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UD>
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.003377,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.003602,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0036354,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.userdomain.co.uk"]}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0052269,"logger":"tls","msg":"finished cleaning storage units"}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0060437,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 05 13:26:17 userdomain caddy[2955]: {"level":"info","ts":1667654777.0062468,"msg":"serving initial configuration"}
Nov 05 13:26:17 userdomain systemd[1]: Started Caddy.

a. System environment:

Ubuntu 20.04LTS on rPi 4 8GB

b. Command:

sudo systemctl start caddy.service

d. My complete Caddy config:

sub.userdomain.com {
  log {
    level INFO
    output file /var/log/caddy/caddy.log {
      roll_size 10MB
      roll_keep 10
    }
  }
  tls user@userdomain.com
  encode gzip
  reverse_proxy localhost:8080
}

3. The problem I’m having:

Did an apt upgrade, rebooted & caddy is no longer working as the reverse proxy that had been running for over a year with no problems

4. Error messages and/or full log output:

Nov 05 13:11:02 userdomain caddy[2384]: HOME=/var/lib/caddy
Nov 05 13:11:02 userdomain caddy[2384]: LOGNAME=caddy
Nov 05 13:11:02 userdomain caddy[2384]: USER=caddy
Nov 05 13:11:02 userdomain caddy[2384]: INVOCATION_ID=e7f2e3679dab44babad0b7d832e55392
Nov 05 13:11:02 userdomain caddy[2384]: JOURNAL_STREAM=9:32697
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.572663,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"warn","ts":1667653862.5763557,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5788927,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5794866,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5795627,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5795789,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400058ad90"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5804887,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5806038,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.58086,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5811415,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5813503,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5822768,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.userdomain.co.uk"]}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5856338,"logger":"tls","msg":"finished cleaning storage units"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.588049,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 05 13:11:02 userdomain caddy[2384]: {"level":"info","ts":1667653862.5882688,"msg":"serving initial configuration"}
Nov 05 13:11:02 userdomain systemd[1]: Started Caddy.
Nov 05 13:12:57 userdomain caddy[2384]: {"level":"info","ts":1667653977.1751099,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Nov 05 13:12:57 userdomain caddy[2384]: {"level":"warn","ts":1667653977.1754177,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Nov 05 13:12:57 userdomain caddy[2384]: {"level":"info","ts":1667653977.1760979,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x400058ad90"}
Nov 05 13:12:57 userdomain caddy[2384]: {"level":"info","ts":1667653977.1776397,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Nov 05 13:12:57 userdomain caddy[2384]: {"level":"info","ts":1667653977.1777112,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
Nov 05 13:12:57 userdomain systemd[1]: Stopping Caddy...
Nov 05 13:12:57 userdomain systemd[1]: caddy.service: Succeeded.
Nov 05 13:12:57 userdomain systemd[1]: Stopped Caddy.
Nov 05 13:13:04 userdomain systemd[1]: Starting Caddy...
Nov 05 13:13:04 userdomain caddy[2477]: caddy.HomeDir=/var/lib/caddy
Nov 05 13:13:04 userdomain caddy[2477]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Nov 05 13:13:04 userdomain caddy[2477]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Nov 05 13:13:04 userdomain caddy[2477]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Nov 05 13:13:04 userdomain caddy[2477]: caddy.Version=v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
Nov 05 13:13:04 userdomain caddy[2477]: runtime.GOOS=linux
Nov 05 13:13:04 userdomain caddy[2477]: runtime.GOARCH=arm64
Nov 05 13:13:04 userdomain caddy[2477]: runtime.Compiler=gc
Nov 05 13:13:04 userdomain caddy[2477]: runtime.NumCPU=4
Nov 05 13:13:04 userdomain caddy[2477]: runtime.GOMAXPROCS=4
Nov 05 13:13:04 userdomain caddy[2477]: runtime.Version=go1.19.2
Nov 05 13:13:04 userdomain caddy[2477]: os.Getwd=/
Nov 05 13:13:04 userdomain caddy[2477]: LANG=C.UTF-8
Nov 05 13:13:04 userdomain caddy[2477]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Nov 05 13:13:04 userdomain caddy[2477]: NOTIFY_SOCKET=/run/systemd/notify
Nov 05 13:13:04 userdomain caddy[2477]: HOME=/var/lib/caddy
Nov 05 13:13:04 userdomain caddy[2477]: LOGNAME=caddy
Nov 05 13:13:04 userdomain caddy[2477]: USER=caddy
Nov 05 13:13:04 userdomain caddy[2477]: INVOCATION_ID=f4bf885304704217b6f6c7fdc550c18a
Nov 05 13:13:04 userdomain caddy[2477]: JOURNAL_STREAM=9:36875
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.296173,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"warn","ts":1667653984.3000758,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3025498,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3030994,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000696e00"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.303185,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3032284,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.304105,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.304253,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3045518,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3048193,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3050392,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3050745,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.userdomain.co.uk"]}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.308436,"logger":"tls","msg":"finished cleaning storage units"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3114586,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 05 13:13:04 userdomain caddy[2477]: {"level":"info","ts":1667653984.3117049,"msg":"serving initial configuration"}
Nov 05 13:13:04 userdomain systemd[1]: Started Caddy.

5. What I already tried:

I recall having a problem when I first installed Caddy, & after the usual searching found a stack exchange article that suggested running:

sudo setcap 'cap_net_bind_service=+ep' /usr/bin/caddy

which got the service running - I must admit I blidnly followed the suggestion as I was despearate to get it to work, & this worked. I also admit not knowing exactly what this does, but it got Caddy working back then.

I’ve tried stopping caddy.service:

sudo systemctl stop caddy

& then running from command line (instead of as a service):

caddy reverse-proxy --from :443 --to :8080 &

which returned:

[1] 2846
ubuntu@userdomain:~$ 2022/11/05 13:23:49.546        WARN    admin   admin endpoint disabled
2022/11/05 13:23:49.546 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "proxy", "https_port": 443}
2022/11/05 13:23:49.546 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "proxy"}
2022/11/05 13:23:49.546 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0x40004479d0"}
2022/11/05 13:23:49.547 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2022/11/05 13:23:49.548 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022/11/05 13:23:49.548 INFO    http.log        server running  {"name": "proxy", "protocols": ["h1", "h2", "h3"]}
2022/11/05 13:23:49.548 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
Caddy proxying http://:443 -> :8080
2022/11/05 13:23:49.548 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/ubuntu/.local/share/caddy"}
2022/11/05 13:23:49.549 INFO    tls     finished cleaning storage units

Not sure where to go from here.

Running vaultWarden in docker, so might try to go down that route i.e. install caddy in a docker container

Would appreciate any help.

2

Just did a fresh install &… same problem :-\

However, I found this but can’t work out how to “just configure your OS/browser to trust Caddy’s internal CA”.

Would appreciate some simple steps, for FF :smile:

3

That’s not necessary if you’re running Caddy as a systemd service. The service config sets up appropriate permissions for binding to low ports.

That setcap line is to allow a program to bind to low ports (i.e. port numbers under 1024) when run as users that don’t have elevated privileges.

I’m not really seeing any issues in your logs. Only this one thing that you should take care of though, to make HTTP/3 perform optimally:

That’s not equivalent. If you run Caddy as a different user than caddy (which is what the systemd service does) then it’ll use a different storage location for certs. That’s not ideal, because that means you might cause Caddy to issue certs again since it couldn’t find them in the current user’s storage location.

Please be more specific by “no longer working”. What’s your evidence of a problem? You haven’t actually shown it not working. You haven’t shown any symptoms. There’s no way to help unless you provide more details.

4

Thanks @francislavoie for your detailed reply.

I found this link in ‘systemctl status caddy’, which provides this command to increase the maximum buffer size:

sysctl -w net.core.rmem_max=2500000

Running ‘systemctl status caddy’ again & that issue is gone:

user@userdomain:~$ sudo systemctl status caddy
● caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-11-06 07:17:21 UTC; 44min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 2028 (caddy)
      Tasks: 10 (limit: 4433)
     CGroup: /system.slice/caddy.service
             └─2028 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Nov 06 07:17:21 userdomain caddy[2028]: {"level":"info","ts":1667719041.8913496,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Nov 06 07:17:21 userdomain caddy[2028]: {"level":"info","ts":1667719041.8916125,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.userdomain.co.uk"]}
Nov 06 07:17:21 userdomain caddy[2028]: {"level":"info","ts":1667719041.893643,"logger":"tls","msg":"finished cleaning storage units"}
Nov 06 07:17:21 userdomain caddy[2028]: {"level":"info","ts":1667719041.8938406,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Nov 06 07:17:21 userdomain caddy[2028]: {"level":"info","ts":1667719041.8940282,"msg":"serving initial configuration"}
Nov 06 07:17:21 userdomain systemd[1]: Started Caddy.
Nov 06 07:18:08 userdomain caddy[2028]: {"level":"error","ts":1667719088.2844994,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"sub.userdomain.co.uk","error":"no information found to solve challenge for identi>
Nov 06 07:18:08 userdomain caddy[2028]: {"level":"error","ts":1667719088.2849765,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"sub.userdomain.co.uk","error":"no information found to solve challenge for identi>
Nov 06 07:18:08 userdomain caddy[2028]: {"level":"error","ts":1667719088.6221027,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"sub.userdomain.co.uk","error":"no information found to solve challenge for identi>
Nov 06 07:18:08 userdomain caddy[2028]: {"level":"error","ts":1667719088.62269,"logger":"tls.issuance.acme","msg":"looking up info for HTTP challenge","host":"sub.userdomain.co.uk","error":"no information found to solve challenge for identifi>

I point one of my domain names to this instance, which normally serves the required service, but am now seeing this:

Screenshot 2022-11-06 082215
Screenshot 2022-11-06 082215670×612 21.8 KB

If I try just using my local network, I get:

Screenshot 2022-11-06 081049
Screenshot 2022-11-06 081049825×629 51.8 KB

Previously, this worked okay & I could see the served page. Now, the only way I can see that page is by using a private browser window:

Screenshot 2022-11-06 083354
Screenshot 2022-11-06 083354996×669 50.3 KB

Hope that provides more details, but happy to run any command that may help.

Cheers!

5

Try connecting from outside your network (via your cell phone or whatever). Does it work in that case? If so then it’s an issue with your router not supporting NAT hairpinning.

1 Like
6

OMG - Genius :star_struck:

Yep that worked. I forgot about the new (ISP supplied) router sent last week - everything else has been working fine, so didn’t even think of that - d’oh!

I haven’t heard of this term, so will do some research, but do you know if there’s any way to get around the router not supporting NAT hairpinning?

Cheers!

1 Like
7

The best option is to set up a DNS server in your LAN that resolves your domain to your LAN IP address instead of your WAN IP.

8

Thanks @francislavoie more fun projects to try :wink:

Really grateful for your help :pray:

9

This topic was automatically closed after 30 days. New replies are no longer allowed.