Caddy propagates sends only some requests to local backend

1. The problem I’m having:

I’m trying to setup a simple reverse proxy with already existing certificates.
On the localhost I have a dummyhttp running. Server is pretty much idling.

The problem is that whenever I call sub.mysever.com via http or curl, only some requests are being directed to the backend. The order/number of passing request through looks random.
Here is request-console:

user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 0
Date: Fri, 02 Feb 2024 17:09:27 GMT
Server: Caddy



user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Fri, 2 Feb 2024 17:09:28 +0000
Server: Caddy

dummyhttp


user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Fri, 2 Feb 2024 17:09:30 +0000
Server: Caddy

dummyhttp


user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Fri, 2 Feb 2024 17:09:31 +0000
Server: Caddy

dummyhttp


user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 0
Date: Fri, 02 Feb 2024 17:09:32 GMT
Server: Caddy



user@centracorn ~> http get "https://sub.myserver.com"
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Fri, 2 Feb 2024 17:09:33 +0000
Server: Caddy

dummyhttp


user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
dummyhttp                                                                     
user@centracorn ~> curl -XGET "https://sub.myserver.com"
user@centracorn ~> curl -XGET "https://sub.myserver.com"
dummyhttp               

2. Error messages and/or full log output:

No errors or logs from caddy.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

OS: Ubuntu 23.10
System:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         40 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        QEMU
  Model name:            AMD EPYC-Milan Processor
    BIOS Model name:     NotSpecified  CPU @ 2.0GHz
    BIOS CPU family:     1
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  2
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            4792.80
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat ps
                         e36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb
                         rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulq
                         dq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave a
                         vx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a mi
                         salignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ssb
                         d ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid
                         rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsave
                         s clzero xsaveerptr wbnoinvd arat umip pku ospke rdpid fsrm
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   64 KiB (2 instances)
  L1i:                   64 KiB (2 instances)
  L2:                    1 MiB (2 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-3
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Mitigation; safe RET
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional
                         , RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

b. Command:

I followed official guidelines:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

and

caddy start -c Caddyfile -w

d. My complete Caddy config:

sub.myserver.com {
	reverse_proxy localhost:3000
	tls /etc/letsencrypt/live/myserver.com/fullchain.pem /etc/letsencrypt/live/myserver.com/privkey.pem
}

I strongly recommend running Caddy as a systemd service, avoid using caddy start. Keep Caddy Running — Caddy Documentation

Add the debug global option, then show Caddy’s logs. We need to see what the reverse_proxy is doing.

Using caddy start throws away your logs by default, so that’s one of the many reasons why running as a systemd service is better.

Why don’t you just let Caddy issue its own certificates?

1 Like

Two suggestions did the trick.
I was using nginx with certbot/letsencrypt before, so decided to re-use the certificates.
Through enabled logging and running Caddy as a service I saw that Caddy did not have permissions for the certificates (I’m not sure how it even worked before…)

Solution:

  • Run Caddy as a service
  • Let Caddy use own certificates (or make sure that Caddy has sufficient permissions)
1 Like