Huge discrepancy in latency as calculated from Caddy logs and client

1. The problem I’m having:

  1. I cannot find any official documentation, that confirms if the duration field in Caddy logs refers to the whole request-response time including the time spent in the upstream. The closest thing I found is this, but official documentation on logs fields would be helpful.

  2. I am trying to debug the performance issues. In our setup, client requests directly go to Caddy, which is being used for custom SSL. For this architecture, the difference between duration (if the assumption made in point 1 is correct) in Caddy logs and the time client (chrome browser) waits for server response should be comparable. But I am seeing huge difference (~500ms in ‘Waiting for Server Response’ in chrome timings and ~80ms in Caddy logs duration). Even if we accommodate network delays, this is quite high.

  3. I am seeing a very huge ‘Initial connection’ time in the chrome browser (>900ms) but the load on Caddy machine is quite low (<2% CPU utilization and <8% memory utilisation). Any way by which we can reduce ‘Initial connection’ delays?

2. Error messages and/or full log output:

Caddy log (with duration ~80ms)
Note: Yes, this is a 301 request. We need to optimize the redirection time.

{
    "level": "info",
    "ts": 1735305388.558313,
    "logger": "http.log.access.log1",
    "msg": "handled request",
    "request": {
        "remote_ip": "146.190.9.43",
        "remote_port": "60133",
        "client_ip": "146.190.9.43",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "scnv.io",
        "uri": "/Z79J",
        "headers": {
            "Sec-Fetch-User": [
                "?1"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br, zstd"
            ],
            "Sec-Ch-Ua-Mobile": [
                "?0"
            ],
            "Upgrade-Insecure-Requests": [
                "1"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
            ],
            "Accept": [
                "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
            ],
            "Sec-Fetch-Mode": [
                "navigate"
            ],
            "Priority": [
                "u=0, i"
            ],
            "Sec-Ch-Ua": [
                "\"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\""
            ],
            "Sec-Ch-Ua-Platform": [
                "\"macOS\""
            ],
            "Sec-Fetch-Site": [
                "none"
            ],
            "Sec-Fetch-Dest": [
                "document"
            ],
            "Accept-Language": [
                "en-GB,en;q=0.9"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "scnv.io"
        }
    },
    "bytes_read": 0,
    "user_id": "",
    "duration": 0.078568326,
    "size": 0,
    "status": 301,
    "resp_headers": {
        "Content-Language": [
            "en"
        ],
        "Set-Cookie": [
            "REDACTED"
        ],
        "X-Content-Type-Options": [
            "nosniff"
        ],
        "Date": [
            "Fri, 27 Dec 2024 13:16:28 GMT"
        ],
        "Server": [
            "Caddy"
        ],
        "Alt-Svc": [
            "h3=\":443\"; ma=2592000"
        ],
        "Report-To": [
            "{\"group\":\"default\",\"max_age\":86400,\"endpoints\":[{\"url\":\"https://scanova.report-uri.com/a/d/g\"}],\"include_subdomains\":true}"
        ],
        "Strict-Transport-Security": [
            "max-age=31536000;includeSubdomains"
        ],
        "X-Frame-Options": [
            "DENY",
            "DENY"
        ],
        "Vary": [
            "Accept-Language, Cookie"
        ],
        "Content-Type": [
            "text/html; charset=utf-8"
        ],
        "Cross-Origin-Opener-Policy": [
            "same-origin"
        ],
        "Referrer-Policy": [
            "strict-origin",
            "same-origin"
        ],
        "X-Xss-Protection": [
            "1; mode=block"
        ],
        "Location": [
            "https://scanova.io"
        ],
        "Content-Security-Policy-Report-Only": [],
        "Content-Length": [
            "0"
        ]
    }
}

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Running as a service on an AWS EC2 machine

a. System environment:

ubuntu

b. Command:

No access to the production machine

c. Service/unit/compose file:

Running as a service

d. My complete Caddy config:

Since, this is a performance question, will supply if required.

Caddy cannot see the latency from your client (browser) to Caddy. If Caddy reports 80ms, then that’s the duration from when Caddy first receives the request until the request leaves Caddy (response is sent out).

The entire request/response within Caddy is 80ms. If you’re seeing ~500ms in Chrome, then the request and response outside of Caddy (from browser to Caddy server, and the trip back) took 500ms - 80ms = 420ms. The server is in India. The client is in California. 420ms to cross 8000 miles is minuscule.

Anyways, if you want to optimize this latency, you need servers around the World to be closer to your clients.

1 Like