Intermittent NOP

1. The problem I’m having:

Intermittent 200 NOP from Caddy, when proxying to downstream services on the docker bridged network. When I start to see NOP, a Caddy restart fixes the problem for a period.

My hosting situation is a kind of funky, so I’ll explain it with a diagram.

               ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ 
                             ┌──────────────┐          ┌─────────┐        │
┌──────────┐   │             │ ca.mafro.net │          │ Pi Hole │         
│ External │                 └──────────────┘       ┌─▶│         │        │
│   HTTPS  │───┼─┐                   ▲             80  └─────────┘         
└──────────┘     via               4433             │                     │
               │ tsnet              │               │                      
                 │     ┌─────────────────────────┐  │                     │
               │ └────▶│    Caddy v2             │  │      ┌───────────┐   
┌──────────┐     ┌────▶│ with caddy-docker-proxy │──┴─┐    │   Home    │  │
│ Internal │   │ 443   │ with caddy-tailscale    │   8123─▶│ Assistant │   
│   HTTPS  │─────┘     └─────────────────────────┘         └───────────┘  │
└──────────┘   │                                                           
                ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

I run a custom CA using smallstep, and use caddy-docker-proxy to automatically expose hosts via Caddy, reverse-proxied over the docker bridged network. There are many more services running than are shown in this diagram.

2. Error messages and/or full log output:

Some example NOP log entries, followed by me restarting Caddy.

caddy-1  | 2025/06/30 14:35:45.288      INFO    http.log.access NOP     {"request": {"remote_ip": "192.168.1.182", "remote_port": "52252", "client_ip": "192.168.1.182", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net", "uri": "/", "headers": {"Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Mode": ["navigate"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Priority": ["u=0, i"], "Sec-Fetch-Dest": ["empty"], "Purpose": ["prefetch"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"137\", \"Chromium\";v=\"137\", \"Not/A)Brand\";v=\"24\""], "Accept-Language": ["en-AU,en-GB;q=0.9,en;q=0.8"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"], "Sec-Purpose": ["prefetch;prerender"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Cookie": ["REDACTED"], "Sec-Fetch-Site": ["same-origin"], "Sec-Ch-Ua-Mobile": ["?0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "bytes_read": 0, "user_id": "", "duration": 0.000011282, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}                                                            
caddy-1  | 2025/06/30 14:35:45.864      INFO    http.log.access NOP     {"request": {"remote_ip": "192.168.1.182", "remote_port": "52252", "client_ip": "192.168.1.182", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net", "uri": "/favicon.ico", "headers": {"Sec-Ch-Ua": ["\"Google Chrome\";v=\"137\", \"Chromium\";v=\"137\", \"Not/A)Brand\";v=\"24\""], "Sec-Fetch-Dest": ["empty"], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Fetch-Site": ["same-origin"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Priority": ["u=1, i"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Cookie": ["REDACTED"], "Accept-Language": ["en-AU,en-GB;q=0.9,en;q=0.8"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Referer": ["https://home.mafro.net/"], "Dnt": ["1"], "Sec-Fetch-Mode": ["no-cors"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "bytes_read": 0, "user_id": "", "duration": 0.000016243, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}            
caddy-1  | 2025/06/30 14:35:47.387      INFO    http.log.access NOP     {"request": {"remote_ip": "192.168.1.182", "remote_port": "52252", "client_ip": "192.168.1.182", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net", "uri": "/sw-modern.js", "headers": {"Service-Worker": ["script"], "Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["same-origin"], "If-None-Match": ["\"1848b3cff5fc6a00-6c81\""], "Sec-Fetch-Dest": ["serviceworker"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"], "Accept-Language": ["en-AU,en-GB;q=0.9,en;q=0.8"], "If-Modified-Since": ["Fri, 13 Jun 2025 20:22:41 GMT"], "Priority": ["u=4, i"], "Cookie": ["REDACTED"], "Referer": ["https://home.mafro.net/sw-modern.js"], "Accept": ["*/*"], "Cache-Control": ["max-age=0"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "bytes_read": 0, "user_id": "", "duration": 0.000014789, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}
caddy-1  | 2025/06/30 14:35:52.400      INFO    shutting down apps, then terminating    {"signal": "SIGTERM"}
caddy-1  | 2025/06/30 14:35:52.400      WARN    exiting; byeee!! 👋     {"signal": "SIGTERM"}

3. Caddy version:

This .env is used by the docker-compose.yml below.

# https://hub.docker.com/_/caddy/tags
CADDY_DOCKER_TAG=2.10.0

# https://github.com/lucaslorentz/caddy-docker-proxy/releases
CADDY_DOCKER_PROXY_PLUGIN=v2.9.2

# https://github.com/caddy-dns/gandi/releases
CADDY_GANDI_PLUGIN=v1.0.4

4. How I installed and ran Caddy:

Custom Dockerfile:

ARG CADDY_DOCKER_TAG

FROM caddy:${CADDY_DOCKER_TAG}-builder AS builder

ARG CADDY_DOCKER_PROXY_PLUGIN
ARG CADDY_GANDI_PLUGIN

RUN xcaddy build \
        --with "github.com/lucaslorentz/caddy-docker-proxy/v2@${CADDY_DOCKER_PROXY_PLUGIN}" \
        --with "github.com/caddy-dns/gandi@${CADDY_GANDI_PLUGIN}" \
        --with "github.com/tailscale/caddy-tailscale"


FROM caddy:${CADDY_DOCKER_TAG}-alpine

ARG CADDY_DOCKER_TAG
ARG CADDY_DOCKER_PROXY_PLUGIN
ARG CADDY_GANDI_PLUGIN
ENV CADDY_DOCKER_TAG=${CADDY_DOCKER_TAG}
ENV CADDY_DOCKER_PROXY_PLUGIN=${CADDY_DOCKER_PROXY_PLUGIN}
ENV CADDY_GANDI_PLUGIN=${CADDY_GANDI_PLUGIN}

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

a. System environment:

Debian bookworm
Docker version 28.1.1, build 4eba377
Docker Compose version v2.35.1

b. Command:

docker compose up -d

c. Compose file:

---
volumes:
  caddy_data: {}

networks:
  caddy:
    name: caddy
    driver: bridge

services:
  caddy:
    image: mafrosis/caddy
    build:
      context: .
      args:
        CADDY_DOCKER_TAG: ${CADDY_DOCKER_TAG}
        CADDY_DOCKER_PROXY_PLUGIN: ${CADDY_DOCKER_PROXY_PLUGIN}
        CADDY_GANDI_PLUGIN: ${CADDY_GANDI_PLUGIN}
    restart: unless-stopped
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
      - CADDY_DOCKER_CADDYFILE_PATH=/Caddyfile
    env_file:
      - .secrets
    ports:
      - 443:443
      - 2020:2020
    networks:
      - caddy
    extra_hosts:
      ca.mafro.net: 192.168.1.198  # Caddy needs to reach internal CA
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
      - ./Caddyfile:/Caddyfile:ro  # Base Caddyfile extended by docker-proxy
      - $HOME/.step/certs/root_ca.crt:/root/step_ca.crt:ro  # Referenced by containers using the proxy
      - $HOME/.step/certs/root_ca.crt:/etc/ssl/certs/step_ca.crt:ro  # Local trust for Step CA
    labels:
      caddy.email: bob@mafro.net
      caddy.log.format: console

d. My complete Caddy config:

{
        email bob@mafro.net
}

# Static reverse proxy to non-docker endpoint
unifi.mafro.net:443 {
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
        reverse_proxy 192.168.1.1:443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
}
unifi.ts.mafro.net:443 {
        tls {
                dns gandi {env.GANDI_API_TOKEN}
        }
        reverse_proxy 192.168.1.1:443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
        bind tailscale/caddy
}

5. Links to relevant resources:

And an example docker-compose.yml which leverages caddy-docker-proxy:

volumes:
  pihole-ftl: {}

networks:
  caddy:
    external: true

services:
  pihole:
    image: pihole/pihole:${PIHOLE_VERSION}
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    networks:
      - default
      - caddy
    ports:
      - 53:53/tcp
      - 53:53/udp
    environment:
      - TZ=Australia/Melbourne
      - DNS1=1.1.1.1
      - DNS2=1.0.0.1
    volumes:
      - ./etc-pihole/:/etc/pihole/
      - ./etc-dnsmasq.d/:/etc/dnsmasq.d/
      - pihole-ftl:/etc/ftldb
    env_file:
      - .secrets
    labels:
      caddy: pihole.mafro.net
      caddy.tls.ca: https://ca.mafro.net:4433/acme/acme/directory
      caddy.tls.ca_root: /root/step_ca.crt
      caddy.reverse_proxy: "{{upstreams 80}}"

Nice diagram!

I don’t see a definition for a site called home.mafro.net, so Caddy doesn’t have any routes configured for those requests.

1 Like

home.mafro.net is the Home Assistant instance. Its definition is created on-the-fly by caddy-docker-proxy.

Every service on this host is defined with compose, and then exposed via Caddy using this labels trick from caddy-docker-proxy.

You can get the current CDP generated config from the Caddy container in order to post the actual complete configuration:

docker compose exec caddy cat /config/caddy/Caddyfile.autosave

Here’s some more logging from last night with debug enabled, when I received some NOPs and had to restart.

caddy-1  | 2025/07/02 12:33:09.821      DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.16.17.5:8123", "total_upstreams": 1}                                                                                               
caddy-1  | 2025/07/02 12:33:09.829      DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.16.17.5:8123", "duration": 0.00776079, "request": {"remote_ip": "172.16.17.1", "remote_port": "60652", "client_ip": "172.16.17.1", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net:443", "uri": "/api/prometheus", "headers": {"User-Agent": ["Prometheus/2.53.4"], "X-Forwarded-For": ["172.16.17.1"], "X-Forwarded-Proto": ["https"], "X-Forwarded-Host": ["home.mafro.net:443"], "X-Prometheus-Scrape-Timeout-Seconds": ["10"], "Accept": ["application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2"], "Accept-Encoding": ["gzip"], "Authorization": ["REDACTED"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "headers": {"Server": [""], "X-Frame-Options": ["SAMEORIGIN"], "Content-Length": ["20945"], "Date": ["Wed, 02 Jul 2025 12:33:09 GMT"], "Content-Type": ["text/plain"], "Referrer-Policy": ["no-referrer"], "X-Content-Type-Options": ["nosniff"]}, "status": 200}
caddy-1  | 2025/07/02 12:33:12.826      DEBUG   docker-proxy    Skipping swarm config caddyfiles because swarm is not available                                                                                                                          
caddy-1  | 2025/07/02 12:33:12.837      DEBUG   docker-proxy    Skipping swarm services because swarm is not available
caddy-1  | 2025/07/02 12:34:09.820      DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "172.16.17.5:8123", "total_upstreams": 1}
caddy-1  | 2025/07/02 12:34:09.828      DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "172.16.17.5:8123", "duration": 0.00786335, "request": {"remote_ip": "172.16.17.1", "remote_port": "60652", "client_ip": "172.16.17.1", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net:443", "uri": "/api/prometheus", "headers": {"X-Forwarded-Host": ["home.mafro.net:443"], "User-Agent": ["Prometheus/2.53.4"], "X-Prometheus-Scrape-Timeout-Seconds": ["10"], "Authorization": ["REDACTED"], "Accept": ["application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2"], "Accept-Encoding": ["gzip"], "X-Forwarded-For": ["172.16.17.1"], "X-Forwarded-Proto": ["https"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "headers": {"Content-Type": ["text/plain"], "Referrer-Policy": ["no-referrer"], "X-Content-Type-Options": ["nosniff"], "Server": [""], "X-Frame-Options": ["SAMEORIGIN"], "Content-Length": ["20945"], "Date": ["Wed, 02 Jul 2025 12:34:09 GMT"]}, "status": 200}                                                                        
caddy-1  | 2025/07/02 12:34:12.837      DEBUG   docker-proxy    Skipping swarm config caddyfiles because swarm is not available                                                                                                                          
caddy-1  | 2025/07/02 12:34:12.844      DEBUG   docker-proxy    Skipping swarm services because swarm is not available
caddy-1  | 2025/07/02 12:34:13.195      DEBUG   events  event   {"name": "tls_get_certificate", "id": "2aed3c0d-4b28-4549-a2d4-50a919f705de", "origin": "tls", "data": {"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"home.mafro.net","SupportedCurves":[27242,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771],"RemoteAddr":{"IP":"192.168.1.182","Port":50604,"Zone":""},"LocalAddr":{"IP":"172.16.17.2","Port":443,"Zone":""}}}}                                      
caddy-1  | 2025/07/02 12:34:13.195      DEBUG   tls.handshake   choosing certificate    {"identifier": "home.mafro.net", "num_choices": 1}                                                                                                               
caddy-1  | 2025/07/02 12:34:13.195      DEBUG   tls.handshake   default certificate selection results   {"identifier": "home.mafro.net", "subjects": ["home.mafro.net"], "managed": true, "issuer_key": "acme-v02.api.letsencrypt.org-directory", "hash": "2c76b364b026b396ace113f3f5c648ff752faa417bb8d1daf2dc8feedcbd6aae"}
caddy-1  | 2025/07/02 12:34:13.195      DEBUG   tls.handshake   matched certificate in cache    {"remote_ip": "192.168.1.182", "remote_port": "50604", "subjects": ["home.mafro.net"], "managed": true, "expiration": "2025/08/23 08:06:23.000", "hash": "2c76b364b026b396ace113f3f5c648ff752faa417bb8d1daf2dc8feedcbd6aae"}
caddy-1  | 2025/07/02 12:34:13.209      INFO    http.log.access NOP     {"request": {"remote_ip": "192.168.1.182", "remote_port": "50604", "client_ip": "192.168.1.182", "proto": "HTTP/2.0", "method": "GET", "host": "home.mafro.net", "uri": "/", "headers": {"Purpose": ["prefetch"], "Sec-Fetch-Mode": ["navigate"], "Cookie": ["REDACTED"], "Sec-Ch-Ua": ["\"Google Chrome\";v=\"137\", \"Chromium\";v=\"137\", \"Not/A)Brand\";v=\"24\""], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Dnt": ["1"], "Priority": ["u=0, i"], "Sec-Purpose": ["prefetch;prerender"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"], "Accept-Language": ["en-AU,en-GB;q=0.9,en;q=0.8"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["empty"], "Sec-Ch-Ua-Mobile": ["?0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Site": ["same-origin"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "home.mafro.net"}}, "bytes_read": 0, "user_id": "", "duration": 0.000012844, "size": 0, "status": 0, "resp_headers": {"Server": ["Caddy"], "Alt-Svc": ["h3=\":443\"; ma=2592000"]}}

And the full config from docker compose exec caddy cat /config/caddy/Caddyfile.autosave:

{
        email dev@mafro.net
        log {
                format console
                level debug
        }
}
unifi.mafro.net:443 {
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
        reverse_proxy 192.168.1.1:443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
}
unifi.ts.mafro.net:443 {
        tls {
                dns gandi {env.GANDI_API_TOKEN}
        }
        reverse_proxy 192.168.1.1:443 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
        bind tailscale/caddy
}
grafana.mafro.net {
        reverse_proxy 172.16.17.7:3000
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
}
home.mafro.net {
        reverse_proxy 172.16.17.5:8123
        tls {
                dns gandi {env.GANDI_API_TOKEN}
        }
}
home.ts.mafro.net {
        bind tailscale/caddy
        reverse_proxy 172.16.17.5:8123
        tls {
                dns gandi {env.GANDI_API_TOKEN}
        }
}
photos.mafro.net {
        reverse_proxy 172.16.17.8:2283
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
}
pihole.mafro.net {
        reverse_proxy 172.16.17.9:80
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
}
syncthing.mafro.net {
        reverse_proxy 172.16.17.10:8384
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
}
zigbee2mqtt.mafro.net {
        reverse_proxy 172.16.17.6:8080
        tls {
                ca https://ca.mafro.net:4433/acme/acme/directory
                ca_root /root/step_ca.crt
        }
}