`reverse_proxy` and http/3

1. Caddy version (caddy version):

v2.5.0 h1:eRHzZ4l3X6Ag3kUt8nj5IxATprhqKq/wToP7OHlXWA0=

2. How I run Caddy:

systemd unit

a. System environment:

ubuntu 22.04, systemd 249 (249.11-0ubuntu3.1)

b. Command:

c. Service/unit/compose file:

stock

d. My complete Caddyfile or JSON config:

{
  servers :443 {
    protocol {
      experimental_http3
    }
  }
  # ...
}

(proxy) {
  header_down Location "http://" "https://"
  header_down -Strict-Transport-Security
  header_down -Server
}

(common) {
  tls {
    key_type p256
    issuer acme {
      dir https://acme-v02.api.letsencrypt.org/directory
      test_dir https://acme-staging-v02.api.letsencrypt.org/directory
      email ...
    }
    issuer acme {
      dir https://dv.acme-v02.api.pki.goog/directory
      test_dir https://dv.acme-v02.test-api.pki.goog/directory
      email ...
      eab ...
    }
    issuer acme {
      dir https://api.buypass.com/acme/directory
      test_dir https://api.test4.buypass.no/acme/directory
      email ...
    }
    issuer acme {
      dir https://acme.zerossl.com/v2/DV90
      eab ...
      email ...
    }
  }
  header Strict-Transport-Security "max-age=31536000; includeSubDomains"
  header Expect-CT "max-age=31536000, enforce"
  header ?Content-Security-Policy "upgrade-insecure-requests; default-src 'self'; img-src https:; style-src 'self' 'unsafe-inline'"
  header ?X-Content-Type-Options "nosniff"
  header ?X-Frame-Options "DENY"
  header ?Referrer-Policy "same-origin"
  header ?Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-captur\
e=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyr\
oscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials\
-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"
  header ?X-XSS-Protection "0"
}

qualcuno.xyz {
  import common
  respond "Nothing to see here ;)"
}

nc.qualcuno.xyz {
  reverse_proxy localhost:8081 {
    import proxy
    max_buffer_size 512000
    header_down Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(self), battery=(), camera=(self), cross-origin-isolated=(), \
display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(self), g\
eolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(self), midi=(), navigation-override=(), payment=(), picture-in-picture=(),\
 publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"
  }
  import common
}

wallabag.qualcuno.xyz {
  reverse_proxy localhost:8082 {
    import proxy
    header_down Set-Cookie "$" "; secure; SameSite=lax"
  }
  header Content-Security-Policy "upgrade-insecure-requests; default-src 'none'; img-src https:; style-src 'self' 'unsafe-inline'; script-src 'self' \
'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-ancestors 'none'; font-src 'self'"
  import common
}

reader.qualcuno.xyz {
  redir /index.php /
  reverse_proxy localhost:8084 {
    import proxy
  }
  import common
  header Content-Security-Policy ""
}

quake.qualcuno.xyz {
  reverse_proxy localhost:8083 {
    import proxy
  }
  header Content-Security-Policy ""
  import common
}

lampone.qualcuno.xyz {
  file_server {
    root /var/www/html
  }
  import common
}

# and more site blocks...

3. The problem I’m having:

Apex, quake and lampone respond over http/3, reader, wallabag, and nc do not, and stay on http 2.

Apex and lampone are served directly by caddy, quake proxies a node.js app, reader, wallabag and nc should be nginx-in-docker upstreams (wallabag might be apache).

I am using Firefox 100.0

4. Error messages and/or full log output:

5. What I already tried:

I tried reading documentation and other site blocks, it’s how I discovered that reverse_proxy works with http/3 on quake.qualcuno.xyz.

6. Links to relevant resources:

You missed section 4 (logs) – we’ll need that in order to be able to provide any help, since it’ll be hard to diagnose this blindly. Please fill that out, thanks!

Ok, just tell me what to do.

root@Quake:~# journalctl --no-pager -xeu caddy > caddylog
root@Quake:~# du -h caddylog
228K    caddylog
root@Quake:~# grep -i 'http/3' < caddylog
May 02 23:04:45 Quake caddy[2356978]: {"level":"info","ts":1651532685.1916838,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 02 23:14:53 Quake caddy[2356978]: {"level":"info","ts":1651533293.4083898,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 02 23:38:32 Quake caddy[2356978]: {"level":"info","ts":1651534712.6590314,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 02 23:46:19 Quake caddy[2356978]: {"level":"info","ts":1651535179.0140927,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 02 23:47:59 Quake caddy[2356978]: {"level":"info","ts":1651535279.262374,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 00:24:55 Quake caddy[2356978]: {"level":"info","ts":1651537495.8846557,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 10:12:49 Quake caddy[2356978]: {"level":"info","ts":1651572769.2626338,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 10:30:35 Quake caddy[2356978]: {"level":"info","ts":1651573835.01669,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 11:13:06 Quake caddy[2356978]: {"level":"info","ts":1651576386.8775527,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 11:16:58 Quake caddy[2356978]: {"level":"info","ts":1651576618.9654331,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:26:36 Quake caddy[2356978]: {"level":"info","ts":1651584396.525819,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:28:42 Quake caddy[2356978]: {"level":"info","ts":1651584522.7862935,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:31:47 Quake caddy[2356978]: {"level":"info","ts":1651584707.3117476,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:33:45 Quake caddy[2356978]: {"level":"info","ts":1651584825.4299178,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:41:02 Quake caddy[2356978]: {"level":"info","ts":1651585262.3646557,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 03 13:47:59 Quake caddy[2356978]: {"level":"info","ts":1651585679.7103815,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 04 22:29:13 Quake caddy[1049]: {"level":"info","ts":1651703353.8885748,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 06 10:39:45 Quake caddy[1049]: {"level":"info","ts":1651833585.2512622,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 06 13:12:18 Quake caddy[1049]: {"level":"info","ts":1651842738.297053,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}

May 06 15:06:20 Quake caddy[1049]: {"level":"error","ts":1651849580.5492444,"logger":"http.log.error","msg":"read tcp 127.0.0.1:60438->127.0.0.1:8081: read: connection reset by peer","request":{"remote_ip":"185.178.95.254","remote_port":"56904","proto":"HTTP/3","method":"GET","host":"nc.qualcuno.xyz","uri":"/ocs/v2.php/apps/notifications/api/v2/notifications","headers":{"Accept":["application/json, text/plain, */*"],"Accept-Language":["it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3"],"Sec-Fetch-Mode":["cors"],"Dnt":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0"],"Accept-Encoding":["gzip, deflate, br"],"Requesttoken":["WhStGF0r9U/ybbp23K8Qc+/knphGtLQjePD+9Flt98I=:L3j1bzcesz2rIP4BkMh+Eb/XxMs/4fdZHZbVvD8UtJA="],"If-None-Match":["\"dab6c9e12d79082afcba53a857a84e6d\""],"Alt-Used":["nc.qualcuno.xyz"],"Cookie":[],"Sec-Fetch-Dest":["empty"],"Sec-Fetch-Site":["same-origin"],"Sec-Gpc":["1"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.002173592,"status":502,"err_id":"hakfut432","err_trace":"reverseproxy.statusError (reverseproxy.go:1166)"}

May 06 15:46:28 Quake caddy[1008]: {"level":"info","ts":1651851988.5803185,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
May 06 16:01:42 Quake caddy[1125]: {"level":"info","ts":1651852902.3165686,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}

Thanks. It looks like the backend for nc is dropping/resetting the connection. Otherwise it looks like things are working fine in Caddy.

That could’ve been me rebooting the machine, eh.

I just realized that reader is responding on http/3 now, nc insists on using http/2. — I spoke too soon. It looks like requests for / get http/3, other paths get http/2 :confused: