UDP-Receive-Buffer-Size (QUIC)

1. Caddy version (caddy version):

v2.3.0

2. How I run Caddy:

service caddy start

a. System environment:

Debian 10.7 : Linux zzori 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 GNU/Linux
CentOS 8.3.2011 : Linux li1698-224.members.linode.com 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19 17:20:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 20.04
The same log was checked for all three OS.
systemd

b. Command:

service caddy start

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        servers {
                protocol {
                        experimental_http3
                }
        }
}

zzori.com www.zzori.com {
        header {
                # enable HSTS
                Strict-Transport-Security max-age=31536000; includeSubdomains; preload;

                # disable clients from sniffing the media type
                X-Content-Type-Options nosniff

                # clickjacking protection
                X-Frame-Options DENY

                # keep referrer data off of HTTP connections
                Referrer-Policy no-referrer-when-downgrade
        }

        root * /home/zzori/zzori.com
        file_server
        encode zstd gzip
        php_fastcgi unix//run/php/php8.0-fpm.sock

        log {
                format single_field common_log
                output file /var/log/caddy/zzori.com.log {
                        roll_size 50mb
                        roll_keep 8
                        roll_keep_for 720h
                }
        }
}

3. The problem I’m having:

Jan 24 17:38:09 zzori caddy[1134]: 2021/01/24 17:38:09 connection doesn’t allow setting of receive buffer size. See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.

root@zzori:/etc/caddy# sysctl -a | grep rmem_max
net.core.rmem_max = 26214400

4. Error messages and/or full log output:

Jan 24 17:38:09 zzori systemd[1]: Started Caddy.
Jan 24 17:38:09 zzori caddy[1134]: caddy.HomeDir=/var/lib/caddy
Jan 24 17:38:09 zzori caddy[1134]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 24 17:38:09 zzori caddy[1134]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 24 17:38:09 zzori caddy[1134]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 24 17:38:09 zzori caddy[1134]: caddy.Version=v2.3.0
Jan 24 17:38:09 zzori caddy[1134]: runtime.GOOS=linux
Jan 24 17:38:09 zzori caddy[1134]: runtime.GOARCH=amd64
Jan 24 17:38:09 zzori caddy[1134]: runtime.Compiler=gc
Jan 24 17:38:09 zzori caddy[1134]: runtime.NumCPU=6
Jan 24 17:38:09 zzori caddy[1134]: runtime.GOMAXPROCS=6
Jan 24 17:38:09 zzori caddy[1134]: runtime.Version=go1.15.6
Jan 24 17:38:09 zzori caddy[1134]: os.Getwd=/
Jan 24 17:38:09 zzori caddy[1134]: LANG=en_US.UTF-8
Jan 24 17:38:09 zzori caddy[1134]: LANGUAGE=en_US:en
Jan 24 17:38:09 zzori caddy[1134]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Jan 24 17:38:09 zzori caddy[1134]: HOME=/var/lib/caddy
Jan 24 17:38:09 zzori caddy[1134]: LOGNAME=caddy
Jan 24 17:38:09 zzori caddy[1134]: USER=caddy
Jan 24 17:38:09 zzori caddy[1134]: INVOCATION_ID=79ea134952ec43a1827b02250278ff97
Jan 24 17:38:09 zzori caddy[1134]: JOURNAL_STREAM=9:23832
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.134503,“msg”:“using provided configuration”,“config_file”:"/etc/caddy/Caddyfile",“config_adapter”:""}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1367881,“logger”:“admin”,“msg”:“admin endpoint started”,“address”:“tcp/localhost:2019”,“enforce_origin”:false,“origins”:[“127.0.0.1:2019”,“loc
alhost:2019”,"[::1]:2019"]}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1371374,“logger”:“http”,“msg”:“server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS”,“serve
r_name”:“srv0”,“https_port”:443}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1371698,“logger”:“http”,“msg”:“enabling automatic HTTP->HTTPS redirects”,“server_name”:“srv0”}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1372077,“logger”:“tls.cache.maintenance”,“msg”:“started background certificate maintenance”,“cache”:“0xc00045a850”}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1379573,“logger”:“http”,“msg”:“enabling experimental HTTP/3 listener”,“addr”:":443"}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1380258,“logger”:“http”,“msg”:“enabling automatic TLS certificate management”,“domains”:[“db.zzori.com”,“zzori.com”,“www.zzori.com”]}
Jan 24 17:38:09 zzori caddy[1134]: 2021/01/24 17:38:09 connection doesn’t allow setting of receive buffer size. See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1456811,“msg”:“autosaved config”,“file”:"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.1457145,“msg”:“serving initial configuration”}
Jan 24 17:38:09 zzori caddy[1134]: {“level”:“info”,“ts”:1611477489.146143,“logger”:“tls”,“msg”:“cleaned up storage units”}
Jan 24 17:38:33 zzori systemd[1]: Starting Cleanup of Temporary Directories…

5. What I already tried:

root@zzori:/etc/caddy# sysctl -a | grep rmem_max
net.core.rmem_max = 26214400

Looking at the guide UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub and increasing the net.core.rmem_max value, but still in syslog the connection doesn’t allow setting of receive buffer size The message is confirmed.

6. Links to relevant resources:

The formatting of your post seems broken, and I don’t see any question. Please fix your post!

First, thank you for your interest. It’s my first time, so it’s broken because the format is insufficient. Correctly corrected.

1 Like

/cc @marten-seemann If you have any ideas? Thanks!

FYI, you’ll need to wrap the header value here with " quotes. The Caddyfile parses what you have as a header replacement operation instead of a set operation, so you end up with the wrong result.

See the adapted JSON (which you can get by running caddy adapt --pretty on your Caddyfile):

{
    "handler": "headers",
    "response": {
        "replace": {
            "Strict-Transport-Security": [
                {
                    "replace": "includeSubdomains;",
                    "search_regexp": "max-age=31536000;"
                }
            ]
        },
        "set": {
            "Referrer-Policy": [
                "no-referrer-when-downgrade"
            ],
            "X-Content-Type-Options": [
                "nosniff"
            ],
            "X-Frame-Options": [
                "DENY"
            ],
            "preload;": [
                ""
            ]
        }
    }
},

So change that line to:

Strict-Transport-Security "max-age=31536000; includeSubdomains; preload;"
1 Like

Changed

{
        servers {
                protocol {
                        experimental_http3
                }
        }
        email jydjchl5v2@knou.ac.kr
}

zzori.com www.zzori.com {
        header {
                Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
                X-Content-Type-Options "nosniff"
                X-Frame-Options "DENY"
                Referrer-Policy "no-referrer-when-downgrade"
        }

        root * /home/zzori/zzori.com
        file_server
        encode zstd gzip
        php_fastcgi unix//run/php/php8.0-fpm.sock


        log {
                format single_field common_log
                output file /var/log/caddy/zzori.com.log {
                        roll_size 50mb
                        roll_keep 8
                        roll_keep_for 720h
                }
        }
}

As you told me, we have completed the modification of the HSTS part.
caddy adapt --pretty result!!

                                                                                                        "handler": "headers",
                                                                                                        "response": {
                                                                                                                "set": {
                                                                                                                        "Referrer-Policy": [
                                                                                                                                "no-referrer-when-downgrade"
                                                                                                                        ],
                                                                                                                        "Strict-Transport-Security": [
                                                                                                                                "max-age=31536000; includeSubdomains; preload"
                                                                                                                        ],
                                                                                                                        "X-Content-Type-Options": [
                                                                                                                                "nosniff"
                                                                                                                        ],
                                                                                                                        "X-Frame-Options": [
                                                                                                                                "DENY"
                                                                                                                        ]
                                                                                                                }

I was setting up the HSTS wrong. Thank you for your help.
However, QUIC-related messages are still occurring.

Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.278393,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2804413,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::
1]:2019","127.0.0.1:2019"]}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2807424,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00037e620"}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2807496,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","serve
r_name":"srv0","https_port":443}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2807949,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2814686,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2815573,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["zzori.com","www.zzori.com","db.zzori.com"]}
**Jan 25 04:18:29 zzori caddy[5991]: 2021/01/25 04:18:29 connection doesn't allow setting of receive buffer size. See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.**
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2891095,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2891338,"msg":"serving initial configuration"}
Jan 25 04:18:29 zzori caddy[5991]: {"level":"info","ts":1611515909.2896502,"logger":"tls","msg":"cleaned up storage units"}

Jan 25 04:18:29 zzori caddy[5991]: 2021/01/25 04:18:29 connection doesn’t allow setting of receive buffer size. See UDP Receive Buffer Size · lucas-clemente/quic-go Wiki · GitHub for details.

I’m not sure because I’m not a developer.
Below is what I inquired about quic-go Please note.

https://github.com/lucas-clemente/quic-go/issues/3024#issuecomment-767251199

I have also been receiving buffer size errors in my logs on both Arch Linux and Ubuntu, while using caddy v2.3.0 with experimental_http3 enabled, like this:

Jan 29 15:15:17 america caddy[62691]: 2021/01/29 15:15:17 connection doesn't allow setting of receive buffer size. See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.

@skyfaller it’s being tracked in this issue, and possibly fixed in v2.4.0

https://github.com/caddyserver/caddy/issues/3998#issuecomment-769586616

There’s an open PR if you’d like to try a build to confirm if it fixes it. See the Github Actions job, build artifacts are available there.

2 Likes

First of all, thank you so much for resolving this issue. I will write my review here.

https://github.com/francislavoie/caddy/tree/udp-conn

I took the source from the link above and built it. It was built normally, changed the /usr/bin/caddy file and restarted the caddy service.

When the net.core.rmem_max=2500000 option is not set in sysctl, the log message is checked as below. It is more specific than the existing message, so it looks good.

Blockquote
Jan 30 00:55:20 localhost caddy[6039]: 2021/01/30 00:55:20 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size ¡ quic-go/quic-go Wiki ¡ GitHub for details.
Blockquote

sysctl on
Setting net.core.rmem_max=2500000 now starts normally without the above message, unlike before.

Blockquote
Jan 30 01:16:25 localhost systemd[1]: Started Caddy.
Jan 30 01:16:25 localhost caddy[6635]: caddy.HomeDir=/var/lib/caddy
Jan 30 01:16:25 localhost caddy[6635]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jan 30 01:16:25 localhost caddy[6635]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jan 30 01:16:25 localhost caddy[6635]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jan 30 01:16:25 localhost caddy[6635]: caddy.Version=(devel)
Jan 30 01:16:25 localhost caddy[6635]: runtime.GOOS=linux
Jan 30 01:16:25 localhost caddy[6635]: runtime.GOARCH=amd64
Jan 30 01:16:25 localhost caddy[6635]: runtime.Compiler=gc
Jan 30 01:16:25 localhost caddy[6635]: runtime.NumCPU=1
Jan 30 01:16:25 localhost caddy[6635]: runtime.GOMAXPROCS=1
Jan 30 01:16:25 localhost caddy[6635]: runtime.Version=go1.15.7
Jan 30 01:16:25 localhost caddy[6635]: os.Getwd=/
Jan 30 01:16:25 localhost caddy[6635]: LANG=en_US.UTF-8
Jan 30 01:16:25 localhost caddy[6635]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Jan 30 01:16:25 localhost caddy[6635]: HOME=/var/lib/caddy
Jan 30 01:16:25 localhost caddy[6635]: LOGNAME=caddy
Jan 30 01:16:25 localhost caddy[6635]: USER=caddy
Jan 30 01:16:25 localhost caddy[6635]: INVOCATION_ID=8a2bb8795b7e467ca7584636d0912fe5
Jan 30 01:16:25 localhost caddy[6635]: JOURNAL_STREAM=9:12906114
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8823743,“msg”:“using provided configuration”,“config_file”:“/etc/caddy/Caddyfile”,“config_adapter”:“”}
Jan 30 01:16:25 localhost caddy[6635]: [WARNING][caddyfile] /etc/caddy/Caddyfile:11: input is not formatted with ‘caddy fmt’
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8916087,“logger”:“admin”,“msg”:“admin endpoint started”,“address”:“tcp/localhost:2019”,“enforce_origin”:false,“origins”:[“localhost:2019”
,“[::1]:2019”,“127.0.0.1:2019”]}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8934371,“logger”:“http”,“msg”:“server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS”,"
server_name":“srv0”,“https_port”:443}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8937387,“logger”:“http”,“msg”:“enabling automatic HTTP->HTTPS redirects”,“server_name”:“srv0”}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8960564,“logger”:“http”,“msg”:“enabling experimental HTTP/3 listener”,“addr”:“:443”}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.8965054,“logger”:“http”,“msg”:“enabling automatic TLS certificate management”,“domains”:[“db1.zzori.com”,“jihye.pw”,“www.jihye.pw”]}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.9032834,“logger”:“tls.cache.maintenance”,“msg”:“started background certificate maintenance”,“cache”:“0xc000239ea0”}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.922352,“msg”:“autosaved config”,“file”:“/var/lib/caddy/.config/caddy/autosave.json”}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.922966,“msg”:“serving initial configuration”}
Jan 30 01:16:25 localhost caddy[6635]: {“level”:“info”,“ts”:1611969385.9300609,“logger”:“tls”,“msg”:“cleaned up storage units”}
Blockquote

I have one more question.
Resolving the above issue, I expected QUIC to be supported in the URL below, but it still says it doesn’t. It still looks like the image below.

In case you didn’t know, I set it up as below, the same as Google’s Alt-Svc header.

Blockquote
header {
Alt-Svc “h3-32=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"
; ma=2592000; v="46,43"”
Strict-Transport-Security “max-age=31536000; includeSubdomains; preload”
X-Content-Type-Options “nosniff”
X-Frame-Options “DENY”
Referrer-Policy “no-referrer-when-downgrade”
}

Blockquote

The result of this setting is the same as below. Still, it turns out that QUIC doesn’t work.
Of course, I am not sure if the test on the site is truly accurate.

Please check the above issues. Thank you again.

Instead of escaping the quotes, you may use ` as a string/token delimiter at the start and end of your header value.

I don’t know much about actual QUIC/HTTP3 behaviour so @marten-seemann would need to chime in if he can.

I’m pretty sure we don’t enable Google’s QUIC – you want the standardized HTTP/3 instead, which seems to be working.

This topic was automatically closed after 30 days. New replies are no longer allowed.