Way to achieve zero downtime for existing connections when dynamic `PATCH` maps

1. The problem I’m having:

I am trying to forwarder requests with different X-ID headers to different upstream IPs, as shown in Caddyfile below.

My idea is, when I get a new ID - IP relation, I using PATCH api to dynamically update the map in the Caddyfile. New HTTP request with new X-ID header will be forwarded to the new IP.

As I expected, since Caddy updates config with zero-downtime, the existing HTTP connections will not be affected. But I am wrong. Every time when I using PATCH, the Caddy server will be restarted, and existing connection will be closed.

Could you please help me figure out, if I missed some important configurations for keep existing connections, or is there a better practical way to achieve my goal?

Many thanks!


The way I call PATCH

    curl -X PATCH http://localhost:2019/id/proxy-map/mappings \
    -H "Content-Type: application/json" \
    -d '[
        {
        "input": "id1",
        "outputs": ["127.0.0.2"]
        },
        {
        "input": "id2",
        "outputs": ["127.0.0.3"]
        }
        ]'

2. Error messages and/or full log output:

2026-06-02T22:29:21.011Z        INFO    admin.api       received request        {"method": "PATCH", "host": "127.0.0.1:2019", "uri": "/id/proxy-map/mappings", "remote_ip": "127.0.0.1", "remote_port": "46160", "headers": {"Accept-Encoding":["gzip"],"Content-Length":["716"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
2026-06-02T22:29:21.011Z        INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2026-06-02T22:29:21.012Z        INFO    http.auto_https automatic HTTPS is completely disabled for server       {"server_name": "srv0"}
2026-06-02T22:29:21.012Z        WARN    http    HTTP/2 skipped because it requires TLS  {"network": "tcp", "addr": ":8080"}
2026-06-02T22:29:21.012Z        WARN    http    HTTP/3 skipped because it requires TLS  {"network": "tcp", "addr": ":8080"}
2026-06-02T22:29:21.012Z        INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2026-06-02T22:29:21.012Z        INFO    http    servers shutting down with eternal grace period
2026-06-02T22:29:21.012Z        INFO    autosaved config (load with --resume flag)      {"file": "/config/caddy/autosave.json"}
2026-06-02T22:29:21.013Z        INFO    admin   stopped previous server {"address": "localhost:2019"}

3. Caddy version:

v2.11

4. How I installed and ran Caddy:

Install: by docker container
Run:

    command:
    - caddy
    args:
    - run
    - --config
    - /etc/caddy/Caddyfile.json
    - --resume

4a. System environment:

K8s environment with NET_ADMIN capabilities and hostNetwork config.

4b. Command:

    command:
    - caddy
    args:
    - run
    - --config
    - /etc/caddy/Caddyfile.json
    - --resume

4d. My complete Caddy config:

My original Caddyfile:

{
    admin localhost:2019
    auto_https off
    log {
        format console {
            time_format "2006-01-02T15:04:05.000Z07:00"
        }
    }
}

:8080 {
    map {http.request.header.X-ID} {upstream_ip} {
        # Example:
        # id1 28.0.1.1
        # id2 28.0.1.2
        __placeholder__ "" # force JSON convertor to generate mappings
        default ""
    }

    @dynamic {
        header X-ID *
        header_regexp port X-Port ^[0-9]+$
        expression {upstream_ip} != ""
    }

    handle @dynamic {
        reverse_proxy {upstream_ip}:{http.request.header.X-Port} {
            header_up -X-ID
            header_up -X-Port
        }
    }

    handle {
        header Content-Type application/json
        respond `{"error": "Invalid request", "id": "{http.request.header.X-ID}"}` 404
    }
}

I coverted it to the JSON format for adding @id: proxy-map

{
  "admin": {
    "listen": "localhost:2019"
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "console",
          "time_format": "2006-01-02T15:04:05.000Z07:00"
        }
      }
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":8080"
          ],
          "routes": [
            {
              "handle": [
                {
                  "defaults": [
                    ""
                  ],
                  "destinations": [
                    "{upstream_ip}"
                  ],
                  "handler": "map",
                  "mappings": [
                    {
                      "input": "__placeholder__",
                      "outputs": [
                        ""
                      ]
                    }
                  ],
                  "source": "{http.request.header.X-ID}",
                  "@id": "proxy-map"
                }
              ]
            },
            {
              "group": "group2",
              "match": [
                {
                  "expression": "{upstream_ip} != \"\"",
                  "header": {
                    "X-Id": [
                      "*"
                    ]
                  },
                  "header_regexp": {
                    "X-Port": {
                      "name": "port",
                      "pattern": "^[0-9]+$"
                    }
                  }
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "delete": [
                                "X-ID",
                                "X-Port"
                              ]
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "{upstream_ip}:{http.request.header.X-Port}"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "group": "group2",
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Content-Type": [
                                "application/json"
                              ]
                            }
                          }
                        },
                        {
                          "body": "{\"error\": \"Invalid request\", \"id\": \"{http.request.header.X-ID}\"}",
                          "handler": "static_response",
                          "status_code": 404
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ],
          "automatic_https": {
            "disable": true
          }
        }
      }
    }
  }
}

5. What I already tried, and links to relevant resources:

  • Deploy Caddy server

  • PATCH the map with 1 record: {ID-01: IP-01}

  • Start a websocket connection to ID-01

  • PATCH the map with 2 records: {ID-01: IP-01, ID-02: IP-02}

  • The websocket connection broken, the Caddy’s log shown the server reloaded.

Assistance disclosure

Post written by myself, but AI used for research.

For normal HTTP requests Caddy config reloads are graceful. For long-lived proxied streams such as WebSockets, the relevant option is stream_close_delay on reverse_proxy.

Try:

reverse_proxy {upstream_ip}:{http.request.header.X-Port} {
    header_up -X-ID
    header_up -X-Port
    stream_close_delay 5m
}

Without that, active streams may be closed when the old reverse_proxy handler is unloaded during a config change. Set the delay to whatever reconnection/drain window makes sense for your clients.

That said, patching Caddy config for every ID-to-IP change is probably not the best long-term design. Caddy config reloads are for configuration changes, not high-churn routing data. If this mapping changes frequently, a custom dynamic upstream module, service discovery layer or stable internal lookup service would be a better data-plane design.

JSON equivalent:

{
  "handler": "reverse_proxy",
  "stream_close_delay": "5m",
  "headers": {
    "request": {
      "delete": ["X-ID", "X-Port"]
    }
  },
  "upstreams": [
    {
      "dial": "{upstream_ip}:{http.request.header.X-Port}"
    }
  ]
}

Thank you for your replay! The stream_close_delay config does work!

I have some follow up questions:

I set this stream_close_delay to 4h,because my websocket connection is expected to last for at most 4 hours. Would this configure introduce some other issues?

The document says:

By default, WebSocket connections are forcibly closed when the config is reloaded. Each request holds a reference to the config, so closing old connections is necessary to keep memory usage in check.

Does this means that I will face OOM problems because of the 4-hour delay?

My guess is No. Because without the config reloading, these websocket connections would still be there, the same as I keeping them after reloading. :thinking:


My mapping changes frequently (about 1 change/minute). You suggest to use custom dynamic upstream module and service discovery layer. Could you provide more details?

The solution I can propose is to set up a DNS server, dynamically inserting ID to IP mappings, and using static Caddyfile like reverse_proxy: {x-id}.local-domain-name:{x-port} for proxying. It would introduce more complexity. I’m wondering if it’s possible that I’m using Caddy in the wrong context. Whether it might not be the best choice for solving my particular problem. :face_with_crossed_out_eyes: