Caddy, HTTPS, IDNA Domains?

1. Caddy version (caddy version):

v.2.3.0 (Custom build + Gandi DNS ACME)

2. How I run Caddy:

JSON config file, plus two env-vars to stipulate the config and data directories.

a. System environment:

VPS, Debian

b. Command:

# XDG_DATA_HOME=/opt/caddy/data XDG_CONFIG_HOME=/opt/caddy/config caddy run --config /opt/caddy/caddy_config.json

c. Service/unit/compose file:

Running manually at present while I debug.

d. My complete Caddyfile or JSON config:

{
  "apps": {
    "tls": {
      "automation": {
        "policies": [
          {
            "issuers": [
              {
                "email": "<redacted>",
                "module": "acme",
                "challenges": {
                  "dns": {
                    "provider": {
                      "name": "gandi",
                      "api_token": "<not today, satan>"
                    }
                  }
                }
              }
            ]
          }
        ]
      }
    },
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "example.domaĆ­n.ie"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Strict-Transport-Security": [
                                "max-age=31556925"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Content-Type-Options": [
                                "nosniff"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Download-Options": [
                                "noopen"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Frame-Options": [
                                "DENY"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Permitted-Cross-Domain-Policies": [
                                "none"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Robots-Tag": [
                                "noindex,nofollow,nosnippet,noarchive"
                              ]
                            }
                          }
                        },
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "X-Xss-Protection": [
                                "1; mode=block"
                              ]
                            }
                          }
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:7156"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

NB: Note the domain name contains an accented character ā€œĆ­ā€. The actual domain has an accented ā€œĆ³ā€, in case that matters :stuck_out_tongue:

3. The problem Iā€™m having:

Lets-Encrypt is rejecting the domain accusing it of having an ā€œinvalid characterā€. Of course Iā€™m eyerolling about the anglocentricity of the whole thing, but really I just want a cert that will work so Iā€™m open to suggestions: Should I be putting the punycode domain in the caddyfile? If so, this should probably be documented at the Caddy end of things. Or is it just that Lets-Encrypt donā€™t understand non-English domains? e.g. is this fixable at my end?

4. Error messages and/or full log output:

The ACME challenge with Lets-Encrypt fails with this error:

2021/01/21 15:08:45.186	INFO	using provided configuration	{"config_file": "/opt/caddy/caddy_config.json", "config_adapter": ""}
2021/01/21 15:08:45.187	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/01/21 15:08:45.188	INFO	http	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2021/01/21 15:08:45.188	INFO	http	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2021/01/21 15:08:45.189	INFO	http	enabling automatic TLS certificate management	{"domains": ["example.domaĆ­n.ie"]}
2021/01/21 15:08:45.190	INFO	autosaved config	{"file": "/opt/caddy/config/caddy/autosave.json"}
2021/01/21 15:08:45.190	INFO	serving initial configuration
2021/01/21 15:08:45.190	INFO	tls.obtain	acquiring lock	{"identifier": "example.domaĆ­n.ie"}
2021/01/21 15:08:45.190	INFO	tls.obtain	lock acquired	{"identifier": "example.domaĆ­n.ie"}
2021/01/21 15:08:45.200	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["example.domaĆ­n.ie"]}
2021/01/21 15:08:45.201	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["example.domaĆ­n.ie"]}
2021/01/21 15:08:45.201	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "<redacted>"}
2021/01/21 15:08:45.202	INFO	tls	cleaned up storage units
2021/01/21 15:08:46.471	INFO	tls.issuance.acme.acme_client	validations succeeded; finalizing order	{"order": "https://acme-v02.api.letsencrypt.org/acme/order/<redacted>/<redacted>"}
2021/01/21 15:08:46.660	ERROR	tls.obtain	will retry	{"error": "[example.domaĆ­n.ie] Obtain: [example.domaĆ­n.ie] finalizing order https://acme-v02.api.letsencrypt.org/acme/order/<redacted>/<redacted>: request to https://acme-v02.api.letsencrypt.org/acme/finalize/<redacted>/<redacted> failed after 1 attempts: HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error finalizing order :: Cannot issue for \"example.domaĆ­n.ie\": Domain name contains an invalid character (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.469109322, "max_duration": 2592000}

5. What I already tried:

TBH, I donā€™t want to experiment too much in case I get blacklisted by the ACME providers or end up rate limited for a week.

I had been having HTTP-code-400 errors with zeroSSL when I was using the HTTP ACME challenge mode, and it was telling me the CSRs didnā€™t matchā€¦ I didnā€™t get far in resolving this issue, so I just switched to Gandi DNS and hoped that would fix things, or at least give a more informative error.

I disabled ZeroSSL because I felt the fallback was just risking me getting banned from both services while I debugged, and because oddly I was not getting an error with Lets-Encrypt at firstā€¦ it would seem to succeed, and then there would be no log-line indicating failure, and it would proceed to ZeroSSL. So I was hoping to shake this bug out, too.

Perhaps the unspecified CSR mismatch with ZeroSSL was due to this issue also? It isnā€™t clear to me why I only started getting this clearly-worded error when I enabled Gandi-based DNS ACME, and not when I was attempting the HTTP ACME challenge.

6. Links to relevant resources:

None found.

My first search for ā€œIDNA Lets Encryptā€ didnā€™t find this, apparently because the keyword is ā€œIDNā€, but hereā€™s Lets-Encrypt saying that they only support punycode: IDN Support enabled - Issuance Policy - Let's Encrypt Community Support

I may then try this again with the punycode domain in Caddy, and hope that this still permits the browser to interact normally.

If this is the problem, then I would consider it a missing feature in Caddy that it doesnā€™t punycode-encode requests to Lets-Encrypt, or doesnā€™t detect and document the error before it arises (e.g., validating that domain name requests are valid candidates for the cert issuerā€™s ACME process)ā€¦

Iā€™ll return here after attempting this with punycode

This was already fixed over two weeks ago, but hasnā€™t been released yet ā€“ just build from master: Build from source ā€” Caddy Documentation

Always always always use Letā€™s Encryptā€™s staging endpoint when experimenting! :slight_smile:

Hey, thanks for the really prompt reply! Glad to hear itā€™s fixed - I think Iā€™ve been able to manually fix things by putting in the domain in punycode.

However, I now have a new problem, which is that the reverse-proxy is proxying the punycode domain back to the server behind Caddy, and itā€™s rejecting it :man_facepalming: - so now Iā€™m digging in the docs for a way to rewrite the Host for proxied traffic. Is this already covered transparently by the fix in the latest version?

What are your configs? You shouldnā€™t have to specify the domain in punycodeā€¦

To be clear, I have not yet attempted a custom build from the main repo, because I donā€™t have a build system installed yet and I donā€™t know how to pull in the Gandi DNS-ACME plugin.

So when I said I was specifying the domain in punycode, that was in Caddy v.2.3.0.

As I now have a working certificate on a stable version of Caddy I was going to attempt to use it as-is, for now, but Iā€™m having the above-mentioned issue where the server behind caddy wonā€™t accept the Punycode domain name, and for various reasons I donā€™t want to configure that server to use Punycode - the authenticity of the domain name in the configuration matters more to the API (Itā€™s a social media server, ā€œepicyonā€, so other Fediverse users need to see the correct domainā€¦)

I may look into building the version in HEAD if I canā€™t figure a workaround, and hope that it is able to give the proxied server the authentic domain name.
I appreciate the help, thank you!

To build from the latest on the git repo, you can just do xcaddy build master --with <your plugins>

Or instead of master, specify a commit hash, if you prefer.

1 Like

In that case, please use the latest HEAD so that you can specify your domains with unicode. It should work. Please let me know if it doesnā€™t. Thanks!

Iā€™m now using a build made using xcaddy and ā€œheadā€ as the target, with the Gandi plugin. Thanks for the advice on how to use.

Unfortunately, the nature of the problem has changed but not disappearedā€¦

TLS Cert Behaviour

Given just the directive of the unicode-style domain name, e.g. example.domaĆ­n.ie, rather than the punycode equivalent, the server simply started up without requesting a new domain: evidently it recognised that the existing cert (which was requesting by putting the punycode into the caddyfile) would cover the unicode form also.

In the interests of science I added a new subdomain, epicyon.domaĆ­n.ie, to see if that would work with ACME: eureka, indeed it did! A certificate for that subdomain was collected successfully.

However, if I actually went to the target subdomain in a browser, it would hit me with a certificate error. In order to actually successfully host the IDN domain from caddy, I had to go back to the caddyfile and add the punycode domain. Note that this is inconsistent with the above block for the original domain which had a certificate deliberately issued for punycode. The original domainā€™s cert was requested by a caddy block that had as its address a punycode domain name, and when I upgraded to the HEAD version of Caddy, it continued working if I changed the caddyfile to the unicode domain without Punycode (although, as Iā€™ll discuss below, the proxying behaviour broke in this instance).

The new subdomain on the other hand had a certificate requested by Caddy while using the unicode version of the address, NOT the punycode. And the cert issued successfully, but queries for the punycode version (which Firefox issues) failed to retrieve a valid certificate. It would get as far as a handshake and then the browser would abort.

However, if I reverted and added the punycode version of the new subdomain, caddy began serving it successfully once again. But the proxying behaviour was the same as the original.

Inconsistent Proxy Behaviour

When I had used punycode and the stable version of caddy, the browserā€™s requests to caddy would work at Caddyā€™s layer, and the requests were being passed-back to the proxied server with the punycode domain name. It would ā€œworkā€ from Caddyā€™s POV but the backend was getting the punycode domain, not the unicode domain.

When I switched to the new version of Caddy (git HEAD) it was able to negotiate certs for unicode, which was good, but if I only included the unicode domain then it would resolve and deliver an empty response over TLS, but the request would not be proxied to the backend. Iā€™d get response headers from Caddy as if it were serving an empty document.

To get Caddy to start hitting the proxied back-end, I had to include the punycode domain in the address list, again. Very odd: the unicode domain was matching on requests for punycode (recall that Firefox sends all IDN requests using punycode encoding, apparently), and responds with a TLS session for same, but it wasnā€™t actually following through on the directives unless I included punycode in the address list alongside the unicode domain.

Failure to set headers?

This one might be just me, but: the backend Iā€™m using is very unhappy with the whole punycode thing, but it also rejects the unicode domain if I specifically send it the unicode domain from curl; the server doesnā€™t handle the encoding well, and it breaks. But strangely, Pythonā€™s requests module sends the host in a MIME format, as a string that starts like this:
"=?utf-8?...
ā€¦and this was being accepted by the backend.
Unfortunately, if I pass this header to Caddy, it rejects the request as being malformed. That might be fodder for another bug report, I guessā€¦ But, I figured, letā€™s just play along and set a Header directive to edit the host header in Caddy, so that if I receive the IDN or Unicode header (which the backend rejects in both cases) I can edit it to the encoded version that the backend does accept.

Unfortunately this does not work. The header simply does not get set or overridden, as far as I can see. From the caddyfile Iā€™m using to generate the JSON config (because I find the docs for the JSON syntax very difficult to read, and there are no clear examples to adaptā€¦):

social.<domain_unicode>.ie, social.<domain_punycode>.ie, epicyon.<domain_unicode>.ie, epicyon.<domain_punycode>.ie {
  # Really struggling to get Caddy to pass a host that the backend Python server will accept..
  header Host "=?utf-8?b?<redacted>?="
  reverse_proxy http://localhost:7156
}

I have verified that this ends up in a ā€œsetā€ type header block in the JSON rather than an Add, so it should replace even when present, right?

Current Status

If I include the punycode versions of all the domains in the address list, I can reach the back-end. And I think that if I can get caddy to set the Host header to the MIME encoded form, then I may be good-to-go. Iā€™ll still apparently have a problem where Pythonā€™s requests module will get rejected by Caddy because Caddy itself rejects MIME encoded Host headers, but thatā€™s a problem for another day, and something I can override deliberately if it is an issue.

Any suggestions for the Header issue?

Can you be more specific? What error?

Please show the full output of curl -v.

Please also show your latest Caddyfile or JSON config, with unredacted domain names, as well as sample requests that are failing.

It will also be prudent to enable debug logging and post those logs. We need more information. There seems to be a lot here and it will take a lot of time. Please break this down into one issue per topicā€¦ Thanks.

Sorry for the forking nature of the issue, Iā€™m documenting this as it comes up - it started as one thing, now itā€™s threeā€¦ Although Iā€™ve restructured the caddyfile and I think I may have been misusing the header directive to accidentally filter out requests rather than setting a header, or something? In any case I seem to now be able to set headers to the backend and it hasnā€™t solved my current problem. Iā€™ll keep hacking.

To help document the IDN issue though, Iā€™ll try to get more details if I can. Iā€™m afraid I donā€™t know how to improve the logging of caddy to help debug this situation, or I would have done so already: can you suggest a method that would generate useful logs? Iā€™m assuming the logs will go to journalctl, then?

Enable debug-level logging: JSON Config Structure - Caddy Documentation

Then please post your current config, and curl -v input and outputs.

I find the structure of the JSON config docs very confusing, but I found an example of the Caddyfile log directive and edited the JSON afterwards to add "level": "DEBUG" to the log0 entry of the logs configā€¦ unfortunately, itā€™s still only giving me errors in the logfile, not ā€œsuccessfulā€ requests, so Iā€™m not getting very useful data. How might I get it to output full debug-level logs including successful (e.g. status 200) requests?

What specifically is confusing about them? They mirror your existing JSON config ā€“ youā€™ve already explored these to make your JSON config, right?

Again, please post your config. Without it, your guess is better than mine. This is now the third time Iā€™m asking and helping in this thread is starting to waste my time and yours.

Hi, hereā€™s the config - sorry for missing the first request, I was racing the clock in my time-zone before end-of-day and Iā€™ve been trying to make progress on this all day. Frustrating times but I do appreciate the help.

Note the log-level in log0 - since changing this to ā€œDEBUGā€ I have not noticed any difference in the actual output, Iā€™m still only seeing errors. Perhaps it belongs elsewhere in the logs object?

{
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access.log0"
        ]
      },
      "log0": {
        "level": "DEBUG",
        "writer": {
          "filename": "/opt/caddy/access.log",
          "output": "file"
        },
        "include": [
          "http.log.access.log0"
        ]
      }
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "social.domaĆ­n.ie",
                    "social.xn--doman-2sa.ie",
                    "epicyon.domaĆ­n.ie",
                    "epicyon.xn--doman-2sa.ie"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": [
                                  "=?utf-8?b?<redacted>?="
                                ]
                              }
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "localhost:7156"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "logs": {
            "logger_names": {
              "epicyon.domaĆ­n.ie": "log0",
              "epicyon.xn--doman-2sa.ie": "log0",
              "social.domaĆ­n.ie": "log0",
              "social.xn--doman-2sa.ie": "log0"
            }
          }
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [
              "epicyon.domaĆ­n.ie",
              "epicyon.xn--doman-2sa.ie",
              "social.domaĆ­n.ie",
              "social.xn--doman-2sa.ie"
            ],
            "issuers": [
              {
                "email": "<redacted>",
                "module": "acme"
              },
              {
                "email": "<redacted>",
                "module": "zerossl"
              }
            ]
          }
        ]
      }
    }
  }
}

Aside on Docs Comprehensibility

Take this with ā€œa pinch of saltā€ because it could just be me and my brain.

As far as the issues with the JSON docs: with time Iā€™m sure I could get used to the layout and find it elegant, but when Iā€™m just looking up ā€œwhere do I put my email for the acme providersā€, it can be hard to get that answer. The email field might be presented as a field in an object which is documented on the page I foundā€¦ but where does that object go? The ā€œat a glanceā€ view of config nesting doesnā€™t provide as much info as it couldā€¦ a well-documented example containing all the supported fields, perhaps with the defaults, might help. And where sub-objects are being documented, a clearer impression of their ā€˜pathā€™ in the root JSON object might help, too.

I think thereā€™s also something fundamentally harder to document about JSON: itā€™s far more verbose, especially if pretty-printed, and it doesnā€™t support any comment, so examples have to be presented in fragmentary form to be brief, but then they lack the useful context. With Caddyfiles it was possible to document things by having examples that were documented in-line using comments, and the brevity of the caddyfile meant you could usually see the prelude and maybe the postmatter of the documented example all on one screen.

My workflow so far has been to try and get the fields I need in the caddyfile, convert them, and note where they end up in the resulting JSON. Then I revert to the docs for further guidance once I know where to put things. Sometimes when not many examples exist to copy, like setting ā€œDEBUGā€ loglevel, I try to figure out where to put the line from the JSON-only docs and it crashes quickly if the config is wrongā€¦

You should set the log level of the default logger to DEBUG. Your log0 logger is only access logs. The default logger is everything else.

Re JSON docs, I think you should consider using the caddy-json-schema plugin, itā€™ll help you figure out what goes where, when editing in your IDE.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.