Caddy 2 equivalent for HTTP header and body limits

mvance · September 29, 2020, 2:56am

1. Caddy version (`caddy version`):

Working to switch to:
v2.2.0 h1:sMUFqTbVIRlmA8NkFnNt9l7s0e+0gw+7GPIrhty905A=

Currently on:
Caddy v1.0.0 (h1:KI6RPGih2GFzWRPG8s9clKK28Ns4ZlVMKR/v7mxq6+c=)

2. How I run Caddy:

a. System environment:

Ubuntu 20.04
systemd
local network only

b. Command:

Currently:

ExecStart=/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/www/caddy/error.lan

Plan to switch to:

ExecStart=/usr/local/bin/caddy run -config=/etc/caddy/Caddyfile

c. Service/unit/compose file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/caddy/ssl/

; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/www/caddy/error.lan
ExecReload=/bin/kill -USR1 $MAINPID

; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=512

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/caddy/ssl, because we want Letsencrypt-certificates there.
; except log directory because want to write logs there
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/caddy/ssl/
ReadWriteDirectories=/var/log/caddy/rclone-restic-rest/

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

but plan to revise to:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
Restart=on-abnormal

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/caddy/ssl/

; Your service file will vary depending on if you use caddy run or caddy start. 
; Probably best to use run.
ExecStart=/usr/local/bin/caddy run -config=/etc/caddy/Caddyfile
; v2 doesn’t implement USR1.
; Use the caddy reload command or the API instead to load new configuration.
ExecReload=/usr/local/bin/caddy reload -config=/etc/caddy/Caddyfile

; Use graceful shutdown with a reasonable timeout
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=512

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/caddy/ssl, because we want Letsencrypt-certificates there.
; except log directory because want to write logs there
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/caddy/ssl/
ReadWriteDirectories=/var/log/caddy/rclone-restic-rest/

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

This is my old Caddyfile. My issue is finding suitable adaptions in v2 for for Caddy v1’s limits and timeouts:

192.168.1.106:8889 {
    tls /etc/caddy/ssl/server-bundle.pem /etc/caddy/ssl/end_device-key.pem {
        protocols tls1.2 tls1.3
    }

    # Reverse proxy to rclone restic rest service
    proxy / localhost:8080 {
        transparent
        max_conns 1024
    }

    root /var/www/caddy/rclone-restic-rest/

    header / {
        # Enable HTTP Strict Transport Security (HSTS) to force clients to always
        # connect via HTTPS (do not use if only testing)
        # Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
        X-Content-Type-Options "nosniff"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Hide server header field
        -Server
        # Disable caching since https
        # The cache should not store anything about the client request or server response.
        Cache-Control "no-store"
        Pragma "no-cache"
    }
    
    # Better manage server connectivity in the face of buggy or malicious clients.
    timeouts {
        read   10s # Maximum duration for reading the entire request, including the body.
        header 10s # The amount of time allowed to read request headers.
        write  10s # The maximum duration before timing out writes of the response.
        idle   2m # The maximum time to wait for the next request (when using keep-alive).
    }
    
    # Limit the HTTP header and body sizes to reasonable values
    # Increase if legitmate traffic results in HTTP 413 errors
    limits {
        header 4kb
        #body   / 10mb
    }

    # use gzip compression if the client supports it.
    gzip
    
    log / /var/log/caddy/rclone-restic-rest/access.log {
        rotate_size     5       # Rotate after the log file reaches this size (in megabytes)
        rotate_age      90      # Keep rotated files for this many days
        rotate_keep     40      # Keep at most this many log files (older files get pruned)
        rotate_compress         # compress rotated log files. gzip is the only format supported.
    }
    
    errors /var/log/caddy/rclone-restic-rest/error.log {
        rotate_size     5      # Rotate after the log file reaches this size (in megabytes)
        rotate_age      90     # Keep rotated files for this many days
        rotate_keep     40     # Keep at most this many log files (older files get pruned)
        rotate_compress        # compress rotated log files. gzip is the only format supported.
    }
}

192.168.1.106:9089 {
    tls /etc/caddy/ssl/prometheus-bundle.pem /etc/caddy/ssl/prometheus-key.pem {
        protocols tls1.3
    }

    # Reverse proxy to Prometheus
    proxy / localhost:9090 {
        transparent
        max_conns 1024
    }

    root /var/www/caddy/prometheus/

    basicauth / name password

    header / {
        # Enable HTTP Strict Transport Security (HSTS) to force clients to always
        # connect via HTTPS (do not use if only testing)
        # Strict-Transport-Security "max-age=31536000;"
        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"
        # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
        X-Content-Type-Options "nosniff"
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"
        # Hide server header field
        -Server
        # Disable caching since https
        # The cache should not store anything about the client request or server response.
        Cache-Control "no-store"
        Pragma "no-cache"
    }
    
    # Better manage server connectivity in the face of buggy or malicious clients.
    timeouts {
        read   10s # Maximum duration for reading the entire request, including the body.
        header 10s # The amount of time allowed to read request headers.
        write  10s # The maximum duration before timing out writes of the response.
        idle   2m # The maximum time to wait for the next request (when using keep-alive).
    }
    
    # Limit the HTTP header and body sizes to reasonable values
    # Increase if legitmate traffic results in HTTP 413 errors
    limits {
        header 4kb
        body   / 10mb
    }

    # use gzip compression if the client supports it.
    gzip
    
    log / /var/log/caddy/prometheus/access.log {
        rotate_size     5       # Rotate after the log file reaches this size (in megabytes)
        rotate_age      90      # Keep rotated files for this many days
        rotate_keep     40      # Keep at most this many log files (older files get pruned)
        rotate_compress         # compress rotated log files. gzip is the only format supported.
    }
    
    errors /var/log/caddy/prometheus/error.log {
        rotate_size     5      # Rotate after the log file reaches this size (in megabytes)
        rotate_age      90     # Keep rotated files for this many days
        rotate_keep     40     # Keep at most this many log files (older files get pruned)
        rotate_compress        # compress rotated log files. gzip is the only format supported.
    }
}

For completeness, here’s what I have in mind for v2. However, I realize I’ll have to adapt to JSON and make some additional edits/additions to achieve parity.

192.168.1.106:8889 {
    # https://caddyserver.com/docs/caddyfile/directives/tls
    tls /etc/caddy/ssl/server-bundle.pem /etc/caddy/ssl/end_device-key.pem {
        protocols tls1.2 tls1.3
    }

    # https://caddyserver.com/docs/caddyfile/directives/reverse_proxy
    # Reverse proxy to rclone restic rest service
    reverse_proxy localhost:8080 {
        #### max_conns 1024
        #### max_conns isn't in Caddyfile anymore but is keep_alive.max_idle_conns_per_host in JSON
    }

    root * /var/www/caddy/rclone-restic-rest/

    # https://caddyserver.com/docs/caddyfile/directives/header
    header {
        # enable HSTS
        # Enable HTTP Strict Transport Security (HSTS) to force clients to always
        # connect via HTTPS (do not use if only testing)
        # Strict-Transport-Security "max-age=31536000;"

        # disable clients from sniffing the media type
        # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
        X-Content-Type-Options "nosniff"

        # clickjacking protection
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"

        # keep referrer data off of HTTP connections
        Referrer-Policy no-referrer-when-downgrade

        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"

        # Hide server header field
        -Server
        
        # Disable caching since https
        # The cache should not store anything about the client request or server response.
        Cache-Control "no-store"
        Pragma "no-cache"
    }
    
    # Better manage server connectivity in the face of buggy or malicious clients.
    ###timeouts {
    ###    read   10s # Maximum duration for reading the entire request, including the body.
    ###    header 10s # The amount of time allowed to read request headers.
    ###    write  10s # The maximum duration before timing out writes of the response.
    ###    idle   2m # The maximum time to wait for the next request (when using keep-alive).
    ###}
    ### In JSON
    # read_timeout
    # read_header_timeout
    # write_timeout
    # idle_timeout 
    
    #### Limit the HTTP header and body sizes to reasonable values
    #### Increase if legitmate traffic results in HTTP 413 errors
    ###limits {
    ###    header 4kb
    ###    #body   / 10mb
    ###}
    # think it is max_header_bytes and request_body with max_size.
    # https://caddyserver.com/docs/json/apps/http/servers/max_header_bytes/
    # https://caddyserver.com/docs/json/apps/http/servers/routes/handle/request_body/

    # https://caddyserver.com/docs/caddyfile/directives/encode
    # use gzip compression and Zstandard if the client supports it.
    encode zstd gzip
    
    # https://caddyserver.com/docs/caddyfile/directives/log
    log {
        output file /var/log/caddy/rclone-restic-rest/caddy.log {
            roll_size     5mb       # Rotate after the log file reaches this size (in megabytes)
            roll_keep     40        # Keep at most this many log files (older files get pruned)
            roll_keep_for 90d       # Keep rotated files for this many days
        }
    }

}

192.168.1.106:9089 {
    tls /etc/caddy/ssl/prometheus-bundle.pem /etc/caddy/ssl/prometheus-key.pem {
        protocols tls1.3
    }

    # Reverse proxy to Prometheus
    proxy localhost:9090

    root * /var/www/caddy/prometheus/

    # Use https://caddyserver.com/docs/command-line#caddy-hash-password
    basicauth / Bob JDJhJDEwJEVCNmdaNEg2Ti5iejRMYkF3MFZhZ3VtV3E1SzBWZEZ5Q3VWc0tzOEJwZE9TaFlZdEVkZDhX

    # https://caddyserver.com/docs/caddyfile/directives/header
    header {
        # enable HSTS
        # Enable HTTP Strict Transport Security (HSTS) to force clients to always
        # connect via HTTPS (do not use if only testing)
        # Strict-Transport-Security "max-age=31536000;"

        # disable clients from sniffing the media type
        # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
        X-Content-Type-Options "nosniff"

        # clickjacking protection
        # Disallow the site to be rendered within a frame (clickjacking protection)
        X-Frame-Options "DENY"

        # keep referrer data off of HTTP connections
        Referrer-Policy no-referrer-when-downgrade

        # Enable cross-site filter (XSS) and tell browser to block detected attacks
        X-XSS-Protection "1; mode=block"

        # Hide server header field
        -Server
        
        # Disable caching since https
        # The cache should not store anything about the client request or server response.
        Cache-Control "no-store"
        Pragma "no-cache"
    }
    
    #### Better manage server connectivity in the face of buggy or malicious clients.
    ###timeouts {
    ###    read   10s # Maximum duration for reading the entire request, including the body.
    ###    header 10s # The amount of time allowed to read request headers.
    ###    write  10s # The maximum duration before timing out writes of the response.
    ###    idle   2m # The maximum time to wait for the next request (when using keep-alive).
    ###}
    
    #### Limit the HTTP header and body sizes to reasonable values
    #### Increase if legitmate traffic results in HTTP 413 errors
    ###limits {
    ###    header 4kb
    ###    body   / 10mb
    ###}

    # https://caddyserver.com/docs/caddyfile/directives/encode
    # use gzip compression and Zstandard if the client supports it.
    encode zstd gzip
    
    # https://caddyserver.com/docs/caddyfile/directives/log
    log {
        output file /var/log/caddy/rclone-restic-rest/caddy.log {
            roll_size     5mb      # Rotate after the log file reaches this size (in megabytes)
            roll_keep     40       # Keep at most this many log files (older files get pruned)
            roll_keep_for 90d      # Keep rotated files for this many days
        }
    }
}

3. The problem I’m having:

Find a way to get a similarly secure Caddy 2 config. Namely, I’m looking to find suitable replacements for Caddy v1’s limits and timeouts:

    # Limit the HTTP header and body sizes to reasonable values
    # Increase if legitimate traffic results in HTTP 413 errors
    limits {
        header 4kb
        body   / 10mb
    }

and

    # Better manage server connectivity in the face of buggy or malicious clients.
    timeouts {
        read   10s # Maximum duration for reading the entire request, including the body.
        header 10s # The amount of time allowed to read request headers.
        write  10s # The maximum duration before timing out writes of the response.
        idle   2m # The maximum time to wait for the next request (when using keep-alive).
    }

For limits header, I think max_header_bytes may work. However, I’m not sure what to use the body portion. Maybe request_body? Am I on the right track?

For timeouts, I’m fairly certain the following are suitable alternatives:

Please let me know if that’s incorrect.

4. Error messages and/or full log output:

N/A

5. What I already tried:

Reading through the tutorials, upgrading to Caddy 2 article, Caddyfile docs, and JSON config structure docs. I’ve also searched the forum and Github.

6. Links to relevant resources:

These were seem related:

These links were also above. Including here for completeness:

francislavoie · September 29, 2020, 4:22am

Yeah, you’re on the right track re header/body size and timeouts.

The problem is that all of these options are specifically on the server (aside from max request body size) but in the Caddyfile, a site block does not map 1:1 to a server in JSON. This is because a server has listener addresses, and they must be unique, i.e. you can’t have two servers listen on the same port.

To deal with this, for UX, the Caddyfile will map site onto their appropriate servers based on their listener addresses, then scope each domain using a subroute and a host matcher.

That means that it’s not possible to configure timeouts for one site that is served on port 443 but not another, because timeouts are at the server level (timeouts are handled in the Golang stdlib, on http.Server). Basically, the timeouts and header byte limits apply before Caddy even knows what the Host header is, so we can’t scope it to a specific domain even if we tried to override the stdlib behaviour.

There’s an open issue to discuss how we can deal with this, but it hasn’t gone too far yet, for the reasons described.

https://github.com/caddyserver/caddy/issues/3669

So yeah, for now, if you need control over those things, you’ll need to use JSON. You may not even need to set those timeouts though, they’re usually not necessary. The default behaviour is typically good enough.

mvance · September 30, 2020, 3:18am

Thank you for the info and reply. This helps.

system · October 29, 2020, 2:56am

This topic was automatically closed after 30 days. New replies are no longer allowed.