Cert_Issuer defaults to LetsEncrypt

Help
1

1. Caddy version (caddy version):

v2.4.0-beta.1.0.20210227022758-ec309c6d52fd h1:Fvxh1kW7soG+k+0oG17Tn1+LYsYowXMHwtTIGUuDDc8=

2. How I run Caddy:

Built using Dockerfile

FROM caddy:builder AS builder

RUN xcaddy build ec309c6d52fdfce0431a1303a49f28c3f546176a \
    --with github.com/mholt/caddy-dynamicdns \
    --with github.com/caddy-dns/cloudflare

FROM caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

Running in Docker, built from Dockerfile

b. Command:

... Docker-compose.yml

    command: ["caddy", "run", "--config", "/etc/caddy/Caddyfile"]

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

{
        dynamic_dns {
                 provider cloudflare APIKEY
                domains {
                        site.com www @
                }
                check_interval 5m
        }
        acme_dns cloudflare APIKEY
        cert_issuer zerossl
        cert_issuer acme
        email webmaster@site.com
}

# Add gzip compression to requests
(encoding) {
        encode gzip zstd
}

# Add Security headers
(SecurityHeaders) {
        header {
                # Server name removing
                -Server
                ## Server site.com
                X-Content-Type-Options "nosniff"
                # Disallow the site to be rendered within a frame (clickjacking protection)
                X-Frame-Options "SAMEORIGIN"
                Referrer-Policy "no-referrer-when-downgrade"
                # Enable HTTP Strict Transport Security (HSTS)
                Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
                # Enable cross-site filter (XSS) and tell browser to block detected attacks
                X-Xss-Protection "1; mode=block"
                # Prevent search engines from indexing (optional)
                X-Robots-Tag "none, noimageindex, noarchive, nocache, nosnippet"
                Cache-Control "public, max-age=15, must-revalidate"
                Feature-Policy "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'"
                Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()"
                Content-Security-Policy "upgrade-insecure-requests"
                X-Permitted-Cross-Domain-Policies "none"
                X-Download-Options "noopen"
        }
}

(logging) {
        log {
                output file /var/log/caddy-{args.0}-access.log {
                        roll true
                        roll_size 1Mib
                        roll_local_time true
                        roll_keep_for 7d
                }
        }
}

(errors) {
        handle_errors {
                rewrite * /{http.error.status_code}
                reverse_proxy https://httpstatusdogs.com {
                        header_up Host httpstatusdogs.com
                }
        }
}

(common) {
        import SecurityHeaders
        import encoding
        import errors
}


(tls) {
        import common
        tls {
                resolvers 8.8.8.8:53
                curves secp521r1 secp384r1
        }
}

:2018 {
        metrics
}

ssl.site.com {
        import tls
        import logging ssl
        respond "Site to check SSL"
}

proxy.site.com {
        import tls
        import logging proxy
        respond "Caddy Proxy Up!"
}

sabnzbd.site.com {
        import tls
        import logging sabnzbd
        reverse_proxy 192.168.XX.XXX:8680
}

# This site uses it's on internal cert, I would to ensure that I've configured this correctly
pf.site.com {
        import tls
        import logging pf
        reverse_proxy https://192.168.X.X:8443 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

etc....

3. The problem I’m having:

I have an issue and a few questions…
I assembled the above config from examples I’ve found on the Caddy Server forums, and for the most part it works without issue. I’m able to get my Certs, Update DNS (changed to CloudFlare this weekend), serve my websites…

Though, I noticed that even if I have ZeroSSL first, if I force the reissuance of a cert by deleting the directory proxy.site.com in caddy/certificates/acme.zerossl…, it requests a new cert from LE. When I stop/up -d the container, it checks for a cert for proxy.site.com and gets a cert from LE.

If I run the validate command, it completes successfully.

docker-compose exec caddy2 caddy validate --config /etc/caddy/Caddyfile

Also, I want to make sure that it is using the DNS-01 challenge, I seem to have the correct command acme_dns, but I’m not sure if that is the correct usage. I’m unable to tell from the caddy2 container logs whether it is performing a DNS-01 challenge.

I also would need your guidance to optimize my config.
I have a site that has it’s own/self-signed cert and I would like to keep the communication secure between the site and Caddy config/reverse proxy… The only way I was able to make it work was to tls_insecure_skip_verify…Is there a better way to achieve this?

I’ve also configured my sites to request individual certs to make it more secure. Yes, I could do this with a Wildcard, but it was my choice to go with individual certs. But, I would like that if a site is not valid, that it display a custom error site. (errors).

For example…

q.site.com would display… 404 not found error from the custom site, but still display https://q.site.com

4. Error messages and/or full log output:

caddy2         | {"level":"info","ts":1615246479.8447576,"logger":"tls.obtain","msg":"acquiring lock","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615246479.851522,"logger":"tls.obtain","msg":"lock acquired","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615246479.852667,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"info","ts":1615246479.852694,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"debug","ts":1615246480.059418,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"debug","ts":1615246480.1085842,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103_idO1wQzX0xoFvD1i5d6QoF-CL-yvcV-OJUE2mYPLv4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"debug","ts":1615246480.357558,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["xxxxxx"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["334"],"Content-Type":["application/json"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/xxxxxx/xxxxxx"],"Replay-Nonce":["0103inwG7Vomfh-v5xtV8y_xtmodN6tSUZ6FkSvTGCLYXvg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"debug","ts":1615246480.4378402,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/11129044150","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["xxxxxx"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["605"],"Content-Type":["application/json"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["010414EufhRANNUaOu3bUwvbpKMJ5rnvbmHhp8wg6kEUUi0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"info","ts":1615246480.4384246,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/xxxxxx/xxxxxx"}
caddy2         | {"level":"debug","ts":1615246480.921991,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/finalize/xxxxxx/xxxxxx","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["xxxxxx"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["438"],"Content-Type":["application/json"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/xxxxxx/xxxxxx"],"Replay-Nonce":["0104lpmKjIqiiLOi_kQmuM4oVzaLnM9-rJeypmp0jSadt3o"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"debug","ts":1615246480.9997852,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3157"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 08 Mar 2021 23:34:40 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/1>;rel=\"alternate\""],"Replay-Nonce":["0103-M2Q63fvdag3YZnB9JDybJSL2u5GYmbDPCPu927BIIA"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"debug","ts":1615246481.0776393,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/cert/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/1","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-beta.1.0.20210227022758-ec309c6d52fd CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["3397"],"Content-Type":["application/pem-certificate-chain"],"Date":["Mon, 08 Mar 2021 23:34:41 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/cert/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/0>;rel=\"alternate\""],"Replay-Nonce":["0103jPvKSFsMLCKpC2BHPO56yrvfHlEH0afOeozT6yQpwes"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
caddy2         | {"level":"info","ts":1615246481.0778122,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}
caddy2         | {"level":"info","ts":1615246481.0785155,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615246481.0785413,"logger":"tls.obtain","msg":"releasing lock","identifier":"proxy.site.com"}
caddy2         | {"level":"debug","ts":1615246481.0796878,"logger":"tls","msg":"loading managed certificate","domain":"proxy.site.com","expiration":1623018880,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}

5. What I already tried:

I’ve tried various configs from different forum posts…

6. Links to relevant resources:

2

It’s because you have these lines:

This sets different configuration for your each site, and those will inherit the default order because it’s not specified there.

If you run caddy adapt --pretty on your config and look at the tls section, you’ll probably see two automation policies (you omitted the rest of your sites so I’m making an assumption here). The first one will have a subjects “matcher” which makes the first apply – this one will have it in the order of acme then zerossl. But there will be a second automation policy without a subjects matcher which has the order of zerossl then acme.

The solution in your case would be to remove the cert_issuer globals and move it to your tls snippet with the issuer subdirective. This will make sure that the order properly applies to all your sites.

That’s because you’re still using the same account (email) so Let’s Encrypt “remembers” you, and says “hey, you already have an issued cert for that account, so here you go” and has you download it again, without issuing a new one.

You should probably switch to the Let’s Encrypt staging endpoint if you want to test issuance.

Yeah, with the tls_trusted_ca_certs transport option. You may also need to configure tls_server_name so that the SNI matches what domain is actually in the certificate even though you’re connecting with an IP address. If the self-signed cert doesn’t have a domain in it, you probably should make sure it has one (or that IP address).

Alternatively, you could run Caddy on that server too and do mTLS between the two Caddy instances - this means the publicly accessible Caddy instance would act as an ACME server (with the acme_server directive) and your private one would point to the public one with the acme_ca global option.

Here’s a nice wiki explaining how that works:

3

Yup… That is what I see… Why doesn’t the global cert_issuer take priority.
In my use case, the cert_issuer applies to all sites, so I would consider it a global option.

Then, to correctly code it then, I would need to do something like this? Is it necessary to disable HTTP and ALPN to force DNS challenge, or will it automagically request new certs via DNS during next renewal cycle?

Since this is applied to all sites, can it be simplified further and “globalized”?

(dns-conf) {
        dns cloudflare <APIKEY>
        disable_http_challenge
        disable_tlsalpn_challenge
        resolvers 8.8.8.8:53
}                                                                                  
(tls) {
        import common
        tls {
                curves secp521r1 secp384r1
                issuer zerossl {
                        import dns-conf
                }
                issuer acme {
                        import dns-conf
                }
        }
}

So, this is the logs after above changes… It seems to work initially, but then LE finalizes the cert?

caddy2         | {"level":"info","ts":1615347029.0675642,"logger":"tls.obtain","msg":"acquiring lock","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615347029.0742478,"logger":"tls.obtain","msg":"lock acquired","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615347029.0750666,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"info","ts":1615347029.075088,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"info","ts":1615347029.075129,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy2         | {"level":"info","ts":1615347029.0751493,"msg":"serving initial configuration"}
caddy2         | {"level":"info","ts":1615347029.485923,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"proxy.site.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
caddy2         | {"level":"warn","ts":1615347029.6494155,"logger":"dynamic_dns.ip_sources.simple_http","msg":"IPv6 lookup failed","endpoint":"https://api.ipify.org","error":"Get \"https://api.ipify.org\": dial tcp6: address api.ipify.org: no suitable address found"}
caddy2         | {"level":"info","ts":1615347052.275148,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"info","ts":1615347052.2752306,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["proxy.site.com"]}
caddy2         | {"level":"info","ts":1615347052.7675047,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxxxxxx/xxxxxxxxxxxxx"}
caddy2         | {"level":"info","ts":1615347053.453549,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/03ff0064afd6c62e47cf8c60d735f03523f7"}
caddy2         | {"level":"info","ts":1615347053.454615,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"proxy.site.com"}
caddy2         | {"level":"info","ts":1615347053.4546554,"logger":"tls.obtain","msg":"releasing lock","identifier":"proxy.site.com"}

The code below is for a server that behaves more like a closed appliance and it generated its own self-signed cert, I was able to download the pem file and then with your suggestion adapted the config as follows. Is this correct or is there another, simpler way?

pf.site.com {
        import tls
        import logging pf
        reverse_proxy https://192.168.X.X:8443 {
                transport http {
                        tls_server_name <servername>
                        tls_trusted_ca_certs <path/to/pem/file>
                }
        }
}

Lastly, what would be the best way to handle errors? This is what I have as the last sites in my config… Basically it’s my default www site redirecting to site.com, but if a site doesn’t exist, then it should redirect to an external error code site.

www.site.com {
        import tls
        import logging www
        redir https://site.com{uri} permanent
}

site.com {
        import tls
        import logging root
        root * /srv/www.site.com
        file_server
}

Is it possible to redirect a site like, q.site.com , which doesn’t exist and sent it to an external error code site?

https:// {
        import errors
        import logging nosite
}
4

No particular reason, just an oversight.

No, if the DNS challenge is enabled, the other challenges are disabled automatically. Those options are there primarily for switching off just one at a time if you need to, e.g. you’re not opening port 80 so the HTTP challenge doesn’t work for you.

This should work:

Looks right to me.

It’s not possible to respond to a request if the TLS handshake cannot be completed, because the TLS handshake is one of the first things that happens in the request lifecycle, before any handlers are run. If Caddy doesn’t have a certificate it can use that the browsers will trust for that domain, then the browser will always present an error.

If you actually do have a certificate that matches though, like a wildcard certificate like *.site.com then you could handle the site that way and handle it however you like.

5

I’ve seen this structure quite often and I have wondered why it’s preferred to the structure below, which appears to achieve the same end result, but in fewer lines, unless it’s to separate out the logging?

site.com www.site.com {
        import tls
        import logging root
        root * /srv/www.site.com
        file_server
}

Let me guess…

*.site.com {
  @unknown host *.site.com
  handle @unknown {
	# do whatever
  }
}
6

Many people prefer to have users see the site without www. in their browser. It’s also better for SEO and whatnot to have just a single domain with the content rather than two with the same content.

1 Like
7

Hi francis,

When I make this change, I get the below error.

validate: adapting config using caddyfile: parsing caddyfile tokens for 'tls': /etc/caddy/Caddyfile:201 - Error during parsing: cannot mix issuer subdirective (explicit issuers) with other issuer-specific subdirectives (implicit issuers)
8

Hmm. Well, that’s unfortunate. The code for TLS config in the Caddyfile adapter is very complex already. Making what you want work looks non-trivial.

I think you’ll just need to go with this:

9

Thanks @basil . I’ve taken and adapted your code to display a custom 404 page.

So… mine looks like this now.

*.site.com {
       import logging nosite
       @unknown host *.site.com
       handle @unknown {
                rewrite * /404
                reverse_proxy https://httpstatusdogs.com {
                       header_up Host {http.reverse_proxy.upstream.hostport}
                }
       }
}

This works, but it sends me directly to : https://httpstatusdogs.com/404-not-found
I would like it to show that page, but still be on q.site.com

10

That’s because that site serves a redirect from /404 to /404-not-found. There’s nothing you can do about that. Probably best if you serve your own 404 page from file. Save a 404.html somewhere, then use a root + rewrite + file_server inside of your handle to serve it.

11

Francis, that works now and it’s unfortunate that it cannot be further streamlined.
I understand the complexities and the compatibility needed with the existing sites. So deviating from what is currently coded could potentially break 1000s of sites.

All I ask is if the Global options exists that it should be applicable to all sites… So cert_issuer(s) and the priority, resolvers, acme_dns should be configurable in the global options.If these are present, then it would mean that all sites should follow the same CA, dns, etc… and you don’t need individual site specific tls settings. Then if you go to the site and configure tls, then these are the exceptions and you want to override the defaults. (Different dns provider, keys, curves, issuer order.) Again, this could be a limitation of the underlying JSON format, but… this would be ideal.

And… I want to thank you and basil for your help. I’m no expert, but searching and repurposing code that I’ve found here has helped me understand and build my caddy config, coming from NGINX. Yes, I may not always use the code correctly, but that is when it becomes fun to search and find the correct way to do it.

Thanks.

12

This topic was automatically closed after 30 days. New replies are no longer allowed.