Redirecting example.com to www.example.com doesn't issue certificate for www.example.com

1. The problem I’m having:

Hi everyone, I am running SaasCustomDomains.com.

I want to redirect from aellion.com to www.aellion.com but I’m having issues. Caddy doesn’t issue certificate for www.aellion.com.

This is my setup:

root domain matcher

This matcher matches aellion.com and redirects to www.aellion.com

{
  "match": [
    {
      "host": [
        "aellion.com"
      ]
    }
  ],
  "handle": [
    {
      "handler": "static_response",
      "headers": {
        "Location": [
          "https://www.{http.request.host}{http.request.uri}"
        ]
      },
      "status_code": 302
    }
  ],
  "terminal": true
}

www matcher

This matcher matches www.aellion.com and proxies the request to domain.com:443

{
  "match": [
    {
      "host": [
        "www.aellion.com"
      ]
    }
  ],
  "handle": [
    {
      "handler": "reverse_proxy",
      "upstreams": [
        {
          "dial": "domain.com:443"
        }
      ],
      "transport": {
        "protocol": "http",
        "tls": {}
      },
      "headers": {
        "request": {
          "set": {
            "Host": [
              "{http.reverse_proxy.upstream.host}"
            ]
          }
        }
      }
    }
  ],
  "terminal": true
}

What happens when I visit aellion.com is Caddy issues a certificate for aellion.com and redirects to www.aellion.com.

But then browsers (Chrome/Firefox/Safari) show the SSL issue for www.aellion.com:

Why doesn’t Caddy issue a cert for the www.aellion.com too?

The confusing part is, when I removed the matcher that redirects aellion.com to www.aellion.com, and visited www.aellion.com, then Caddy issued the certificate for www.aellion.com. Then I brought back the matcher-redirector and aellion.com would redirect to www.aellion.com and everything worked as expected.

Can anyone explain to me what is going on?

How can I just have a config that redirects to www and handles all the certificates properly?

3. Caddy version:

2.6.4

4. How I installed and ran Caddy:

a. System environment:

Docker (Linux)

b. Command:

caddy run --config /etc/caddy/caddy.json

d. My complete Caddy config:

{
  "apps": {
    "http": {
      "servers": {
        "proxy_status_server": {
          "listen": [
            ":8082"
          ],
          "automatic_https": {
            "disable": true,
            "disable_redirects": true
          },
          "routes": [
            {
              "match": [
                {
                  "path": [
                    "/custom-domains-proxy-status"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "static_response",
                  "status_code": 200,
                  "body": "OK"
                }
              ],
              "terminal": true
            }
          ]
        },
        "tls_terminator": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "www.aellion.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "domain.com:443"
                    }
                  ],
                  "transport": {
                    "protocol": "http",
                    "tls": {}
                  },
                  "headers": {
                    "request": {
                      "set": {
                        "Host": [
                          "{http.reverse_proxy.upstream.host}"
                        ]
                      }
                    }
                  }
                }
              ],
              "terminal": true
            },
            {
              "match": [
                {
                  "host": [
                    "aellion.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "static_response",
                  "headers": {
                    "Location": [
                      "https://www.{http.request.host}{http.request.uri}"
                    ]
                  },
                  "status_code": 302
                }
              ],
              "terminal": true
            }
          ],
          "logs": {}
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "on_demand": true
          }
        ],
        "on_demand": {
          "ask": "https://[redacted_domain]/control/caddy/ask",
          "rate_limit": {
            "interval": "10m",
            "burst": 100
          }
        }
      },
      "cache": {
        "capacity": 100000
      }
    }
  },
  "admin": {
    "identity": {
      "issuers": [
        {
          "module": "acme",
          "email": "[redacted]"
        }
      ]
    }
  },
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access"
        ],
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy/caddy.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        }
      },
      "log0": {
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy/access.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        },
        "include": [
          "http.log.access"
        ]
      }
    }
  },
  "storage": {
    "module": "s3",
    "host": "s3.amazonaws.com",
    "bucket": "[redacted_bucket_name]",
    "prefix": "[redacted]",
    "insecure": false
  }
}

Thank you all :slight_smile:

That’s quite an old version. Please upgrade to the latest, v2.7.6

Hard to say without seeing your logs.

You’ll want to remove this. It’s deprecated and slated for removal, it doesn’t work properly. Your ask endpoint should reject any unknown/unregistered domains.

Whoah, I removed the redundant rate_limit and just caused an outage.

{"level":"info","ts":1712938756.7053375,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/caddy.log"}
panic: invalid configuration: maxEvents = 0 and window != 0 would not allow any events

goroutine 2261107 [running]:
github.com/caddyserver/certmagic.(*RingBufferRateLimiter).SetMaxEvents(0x400008d540, 0x0)
	github.com/caddyserver/certmagic@v0.17.2/ratelimiter.go:154 +0x27c
github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).Provision(0x4040a28dd0, {{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x4057740fc0, 0x2, ...}})
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddytls/tls.go:184 +0x99c
github.com/caddyserver/caddy/v2.Context.LoadModuleByID({{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x4057740fc0, 0x2, 0x2}}, ...)
	github.com/caddyserver/caddy/v2@v2.6.4/context.go:347 +0x420
github.com/caddyserver/caddy/v2.Context.App({{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x4015e33fc0, 0x1, 0x1}}, ...)
	github.com/caddyserver/caddy/v2@v2.6.4/context.go:418 +0xf8
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Provision(0x4050c4a960, {{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x4015e33fc0, 0x1, ...}})
	github.com/caddyserver/caddy/v2@v2.6.4/modules/caddyhttp/app.go:159 +0x88
github.com/caddyserver/caddy/v2.Context.LoadModuleByID({{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x4015e33fc0, 0x1, 0x1}}, ...)
	github.com/caddyserver/caddy/v2@v2.6.4/context.go:347 +0x420
github.com/caddyserver/caddy/v2.Context.App({{0x19da158, 0x40534b8050}, 0x404411b920, 0x4041457ea0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}}, ...)
	github.com/caddyserver/caddy/v2@v2.6.4/context.go:418 +0xf8
github.com/caddyserver/caddy/v2.run.func3(...)
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:477
github.com/caddyserver/caddy/v2.run(0x405f63e000?, 0x1)
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:482 +0x460
github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun({0x405f5f0000, 0x3ca7c, 0x3e000}, 0x1)
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:337 +0xe4
github.com/caddyserver/caddy/v2.changeConfig({0x151d7c8, 0x4}, {0x15255b6, 0x7}, {0x406086e000, 0x8ea8e, 0xac000}, {0x0, 0x0}, 0x0)
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:228 +0x628
github.com/caddyserver/caddy/v2.finishSettingUp.func1({0x406086e000, 0x8ea8e, 0xac000})
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:558 +0x88
github.com/caddyserver/caddy/v2.finishSettingUp.func3()
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:608 +0x30
created by github.com/caddyserver/caddy/v2.finishSettingUp
	github.com/caddyserver/caddy/v2@v2.6.4/caddy.go:608 +0x58c

That was unexpected to say the least :joy:

Regarding the redirect to www.aellion.com, the only Caddy logs that mention www.aellion.com are these:

{"level":"info","ts":"2024-04-12T16:13:33.750Z","logger":"http","msg":"enabling automatic TLS certificate management","domains":["aellion.com", "www.aellion.com"]}

There are more domains in that domains array but I redacted them.

And in the access logs I see the requests are coming in for aellion.com and that the redirect happens:

{
  "level": "info",
  "ts": "2024-04-12T16:01:12.242Z",
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_ip": "XXX.XXX.XXX.XXX",
    "remote_port": "65047",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "aellion.com",
    "uri": "/",
    "headers": {
      "Accept-Language": [
        "en-US,en;q=0.9,hr;q=0.8,sr;q=0.7,bs;q=0.6"
      ],
      "Sec-Ch-Ua": [
        "\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "Sec-Fetch-Mode": [
        "navigate"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br, zstd"
      ],
      "Sec-Fetch-User": [
        "?1"
      ],
      "Sec-Fetch-Dest": [
        "document"
      ],
      "Sec-Purpose": [
        "prefetch;prerender"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
      ],
      "Sec-Fetch-Site": [
        "none"
      ],
      "Cookie": [],
      "Sec-Ch-Ua-Mobile": [
        "?0"
      ],
      "Sec-Ch-Ua-Platform": [
        "\"macOS\""
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
      ],
      "Purpose": [
        "prefetch"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "aellion.com"
    }
  },
  "user_id": "",
  "duration": 0.000227121,
  "size": 0,
  "status": 302,
  "resp_headers": {
    "Content-Type": [],
    "Server": [
      "Caddy"
    ],
    "Location": [
      "https://www.aellion.com/"
    ]
  }
}

But that was expected because I see the redirect happening in the browser:

This issue only happens when I have the aellion.com -> www.aellion.com redirect. Without the redirect Caddy issues cert for www.aellion.com, no problem.

I just added a new domain, www2.aellion.com, and Caddy issued the cert immediately.

Do you remember anyone complaining about something similar?

So weird.

To make sure this is not somehow only affecting www.aellion.com, I tested with another domain — aellion-test.site.

This is the matcher that redirects:

{
  "match": [
    {
      "host": [
        "aellion-test.site",
        "aellion.com"
      ]
    }
  ],
  "handle": [
    {
      "handler": "static_response",
      "headers": {
        "Location": [
          "https://www.{http.request.host}{http.request.uri}"
        ]
      },
      "status_code": 302
    }
  ],
  "terminal": true
}

This is the matcher for www.aellion-test.site:

{
  "match": [
    {
      "host": [
        "www2.aellion.com",
        "www.aellion-test.site",
        "www.aellion.com"
      ]
    }
  ],
  "handle": [
    {
      "handler": "reverse_proxy",
      "upstreams": [
        {
          "dial": "cal.com:443"
        }
      ],
      "transport": {
        "protocol": "http",
        "tls": {}
      },
      "headers": {
        "request": {
          "set": {
            "X-Served-For": [
              "{http.request.host}"
            ],
            "X-SaaS-Domains-IP": [
              "{http.request.remote.host}"
            ],
            "Host": [
              "{http.reverse_proxy.upstream.host}"
            ]
          }
        }
      }
    }
  ],
  "terminal": true
}

So www2.aellion.com works, but www.aellion.com and www.aellion-test.site that both have redirects are not working. Certs are not being issued, not even if I visit these www domains directly.

This is so strange.

@matt just tagging you in case maybe you saw something like this happening before?

Thank you everyone who looks into this. I appreciate it so much.

Small update @francislavoie, I am in process of upgrading Caddy to 2.7.6.

Will let you know if this fixes the issue.

Unfortunately, upgrading to v2.7.6 did not fix the issue.

I am open to any suggestions in helping me debug this.

I tried with a third domain — saasstatuspage.com.

There’s no mention of www.saasstatuspage.com in Caddy logs. Again, the certificate is issued only for the root domain saasstatuspage.com and not the www subdomain.

Caddy logs

  {
    "level": "info",
    "ts": "2024-04-12T19:15:41.283Z",
    "logger": "tls.on_demand",
    "msg": "obtaining new certificate",
    "remote_ip": "37.228.200.181",
    "remote_port": "65118",
    "server_name": "saasstatuspage.com"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:42.063Z",
    "logger": "tls.obtain",
    "msg": "acquiring lock",
    "identifier": "saasstatuspage.com"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:42.063Z",
    "logger": "tls.obtain",
    "msg": "lock acquired",
    "identifier": "saasstatuspage.com"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:42.260Z",
    "logger": "tls.obtain",
    "msg": "obtaining certificate",
    "identifier": "saasstatuspage.com"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:42.785Z",
    "logger": "http",
    "msg": "waiting on internal rate limiter",
    "identifiers": [
      "saasstatuspage.com"
    ],
    "ca": "https://acme-v02.api.letsencrypt.org/directory",
    "account": ""
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:42.785Z",
    "logger": "http",
    "msg": "done waiting on internal rate limiter",
    "identifiers": [
      "saasstatuspage.com"
    ],
    "ca": "https://acme-v02.api.letsencrypt.org/directory",
    "account": ""
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:43.808Z",
    "logger": "http.acme_client",
    "msg": "trying to solve challenge",
    "identifier": "saasstatuspage.com",
    "challenge_type": "tls-alpn-01",
    "ca": "https://acme-v02.api.letsencrypt.org/directory"
  },
  {
    "level": "error",
    "ts": "2024-04-12T19:15:44.654Z",
    "logger": "http.acme_client",
    "msg": "challenge failed",
    "identifier": "saasstatuspage.com",
    "challenge_type": "tls-alpn-01",
    "problem": {
      "type": "urn:ietf:params:acme:error:unauthorized",
      "title": "",
      "detail": "Incorrect validation certificate for tls-alpn-01 challenge. Requested saasstatuspage.com from 99.83.186.151:443. Received certificate with acmeValidationV1 extension value 7953a18d71a60b7cf17fe8df9505c501a074a1776d909fad69272b99a33ef0c2 but expected cd94c035a92a7646006b18ab58f1b09e92c549a459d26be080c72a0bb58f2711.",
      "instance": "",
      "subproblems": []
    }
  },
  {
    "level": "error",
    "ts": "2024-04-12T19:15:44.654Z",
    "logger": "http.acme_client",
    "msg": "validating authorization",
    "identifier": "saasstatuspage.com",
    "problem": {
      "type": "urn:ietf:params:acme:error:unauthorized",
      "title": "",
      "detail": "Incorrect validation certificate for tls-alpn-01 challenge. Requested saasstatuspage.com from 99.83.186.151:443. Received certificate with acmeValidationV1 extension value 7953a18d71a60b7cf17fe8df9505c501a074a1776d909fad69272b99a33ef0c2 but expected cd94c035a92a7646006b18ab58f1b09e92c549a459d26be080c72a0bb58f2711.",
      "instance": "",
      "subproblems": []
    },
    "order": "https://acme-v02.api.letsencrypt.org/acme/order/1488475046/260378425197",
    "attempt": 1,
    "max_attempts": 3
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:46.050Z",
    "logger": "http.acme_client",
    "msg": "authorization finalized",
    "identifier": "saasstatuspage.com",
    "authz_status": "valid"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:47.371Z",
    "logger": "tls.obtain",
    "msg": "certificate obtained successfully",
    "identifier": "saasstatuspage.com"
  },
  {
    "level": "info",
    "ts": "2024-04-12T19:15:47.371Z",
    "logger": "tls.obtain",
    "msg": "releasing lock",
    "identifier": "saasstatuspage.com"
  }

I have this tab open and it’s next on my list after I finish something for another sponsor – I’ll be looking at this soon!

1 Like

Hey Matt, thanks for looking into it. I appreciate it.

No pressure over the weekend though. I’m not in a huge rush :slight_smile:

This is why we (esp. Francis) ask to update to the latest version. That bug was fixed over a year ago:

Also, did you do a config reload or a complete process stop/start? A reload shouldn’t cause any downtime if a new config has an error loading.

Please verify that you are actually running the v2.7.6 binary and then I will revisit this, since I’m sure the panic does not exist in v2.7.6.

Oh sorry, I was unclear. I removed the rate_limit before upgrading to 2.7.6. and there was the error.

I used caddy reload. Not sure why Caddy process failed completely and stopped working.

Anyway, it’s ok now because I upgraded to 2.7.6. I can confirm that.

Ok, thanks for clarifying.

What is in your Caddy logs, esp. when requests to aellion.com and www.aellion.com come in? (not just access logs, but any logs) Enable debug-level logging on the default logger as well. The more log lines you post the better.

What is your current config btw? I tried running the config you posted above but it doesn’t work for v2.7.6 – and even after modifying it slightly to get it to run I wasn’t able to reproduce the behavior you’re reporting.

More complete logs (at debug level) and current config – and ideally reproducibility – would be most helpful.

I am pretty sure I’m running 2.7.6.

This is me SSH-ing to a production box that runs Caddy in a Docker container.

[root@ip-10-0-1-105 caddy]# docker exec -it 1883bcfbbcfe sh
/srv # caddy -v
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

What is your current config btw? I tried running the config you posted above but it doesn’t work for v2.7.6 – and even after modifying it slightly to get it to run I wasn’t able to reproduce the behavior you’re reporting.

I redacted the routes in the config because there are so many and I can’t share them publicly because they belong to my customers. Can I send you the full config privately via email?

But other than that the config is the same as my production config. Unless I made a mistake copy-pasting it here.

What error did you see when you tried running my config with 2.7.6.?

More complete logs (at debug level) and current config – and ideally reproducibility – would be most helpful.

Regarding the logs. I enabled debug logs for my default and log0 loggers.

But, I couldn’t find anything except these access logs. Something caught my eye (in this example I’ll use saasstatuspage.com and www.saasstatuspage.com instead of aellion.comsaasstatuspage.com is one of the domains I tested this with)

Request to www.saasstatuspage.com gets HTTP response 308 Permanent Redirect with Location: https://www.saasstatuspage.com. Is this expected?

Here are the logs:

# request to saasstatuspage.com
{
  "level": "info",
  "ts": "2024-04-14T02:12:15.502Z",
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_ip": "15.204.182.106",
    "remote_port": "47464",
    "client_ip": "15.204.182.106",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "saasstatuspage.com",
    "uri": "/",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
      ],
      "Accept": [
        "*/*"
      ],
      "Accept-Encoding": [
        "gzip"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 771,
      "cipher_suite": 49195,
      "proto": "",
      "server_name": "saasstatuspage.com"
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 0.000301961,
  "size": 0,
  "status": 302,
  "resp_headers": {
    "Location": [
      "https://www.saasstatuspage.com/"
    ],
    "Content-Type": [],
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ]
  }
}

# request to www.saasstatuspage.com that ends up with 308 Permanent Redirect and connection closed
{
  "level": "info",
  "ts": "2024-04-14T09:43:51.536Z",
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_ip": "15.204.161.7",
    "remote_port": "2526",
    "client_ip": "15.204.161.7",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "www.saasstatuspage.com",
    "uri": "/",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
      ],
      "Accept": [
        "*/*"
      ],
      "Accept-Encoding": [
        "gzip"
      ]
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 0.000056993,
  "size": 0,
  "status": 308,
  "resp_headers": {
    "Server": [
      "Caddy"
    ],
    "Connection": [
      "close"
    ],
    "Location": [
      "https://www.saasstatuspage.com/"
    ],
    "Content-Type": []
  }
}

I’ll email you my full unredacted Caddy JSON config at ****@d****m.com.

Thank you so much for looking into this :pray:

I finally figured it out.

It was that my /ask endpoint wouldn’t accept the www subdomain and returned false. It’s so obvious but somehow I completely forgot to check that. Finally, when I checked, there was a little bug there.

Huh, finally! Thank you everyone for the help.

btw, shouldn’t I’ve seen logs of /ask returning false in Caddy logs?

2 Likes

Pretty sure you would, in debug logs (not access logs).

Anyway, glad you figured it out!

2 Likes

If not in that version then I think we do in the upcoming 2.8 at least. I seem to remember adding more debug logging there recently.

1 Like

Thank you guys for being so vigilant and looking into this over the weekend. I hope I didn’t ruin the Sunday for you.

1 Like