Caddy inside private corporation network

I am trying to run Caddy inside a corporate network which uses a firewall with self-signed certificates.

I am used to put these certificates in my VMs. For example, for my Arch VMs I do

sudo trust anchor cert-1.crt
sudo trust anchor cert-2.crt

Today, when building, running and trying to get some staging certificates from LetsEncrypt I stumble upon a problem with Go inside Caddy’s docker container.

This is how I build/run my Caddy

Dockerfile

FROM caddy:2.5.1-builder-alpine AS builder
RUN xcaddy build --with github.com/caddy-dns/cloudflare
FROM caddy:2.5.1-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

docker-compose.yml

version: "3.7"
services:

  caddy:
    build: ./tmp
    hostname: caddy
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./sites:/srv
      - ./Caddyfile:/etc/caddy/Caddyfile
      - data:/data
      - config:/config

volumes:
  data:
  config:

networks:
  default:
    name: caddy_net
    external: true

Only after copy and activate cert-1.crt and cert-2.crt inside Caddy’s docker container I was able to build and run it. But it was need to change Dockerfile to this

Dockerfile

FROM caddy:2.5.1-builder-alpine AS builder
COPY cert-1.crt /usr/local/share/ca-certificates/cert-1.crt
COPY cert-2.crt /usr/local/share/ca-certificates/cert-2.crt
RUN update-ca-certificates
RUN xcaddy build --with github.com/caddy-dns/cloudflare
FROM caddy:2.5.1-alpine
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

The container goes up but still shows some errors and do not get the certificates.

Anyone already had this problem and solve it?

Some of the error messages:

{"level":"info","ts":1653412048.66475,"msg":"serving initial configuration"}
{"level":"info","ts":1653412048.6669748,"logger":"tls.obtain","msg":"lock acquired","identifier":"ldap.private.corp.network.domain"}
{"level":"debug","ts":1653412048.6678371,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme-staging-v02.api.letsencrypt.org-directory"}
{"level":"warn","ts":1653412049.8716073,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1653412050.709101,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"warn","ts":1653412051.5481153,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-staging-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1653412051.54819,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"ldap.private.corp.network.domain","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"registering account [mailto:leandro.peracchi@gmail.com] with server: provisioning client: performing request: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": x509: certificate signed by unknown authority"}
{"level":"debug","ts":1653412051.5482035,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme-staging-v02.api.letsencrypt.org-directory"}
{"level":"error","ts":1653412052.9359179,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"ldap.private.corp.network.domain","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority"}
{"level":"error","ts":1653412052.9359574,"logger":"tls.obtain","msg":"will retry","error":"[ldap.private.corp.network.domain] Obtain: account pre-registration callback: performing EAB credentials request: Post \"https://api.zerossl.com/acme/eab-credentials-email\": x509: certificate signed by unknown authority","attempt":1,"retrying_in":60,"elapsed":4.268951206,"max_duration":2592000}

I think you did this in the wrong order. You want to copy the certs in your final image then run update-ca-certificates in the final image (i.e. after the FROM caddy:2.5.1-alpine):

FROM caddy:2.5.1-builder-alpine AS builder
RUN xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:2.5.1-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

COPY cert-1.crt /usr/local/share/ca-certificates/cert-1.crt
COPY cert-2.crt /usr/local/share/ca-certificates/cert-2.crt
RUN update-ca-certificates

I think the difference is that the builder image doesn’t have the ca-certificates package, but the runtime image does. So when you copy it from the builder to the runtime image, it loses all the other certs that are needed to connect to Let’s Encrypt and such.

1 Like

Tried exactly as above, did not work.

Then tried to put exact path to certificates, still didn´t work.

Output of

docker exec caddy /bin/ls -lh /usr/local/share/ca-certificates/

shows nothing.

The certificates were not supposed there? :thinking:

Are you sure you’re actually running your rebuilt image? Don’t stop + start, you need to use down to make sure the container is removed then you can up -d again.

1 Like

I was using docker-compose down -v and apparently docker was rebuilding.

But now I made a docker image prune -a and when tried to rebuild, get an error path related. Will investigate.

1 Like

This is how it almost worked.

FROM caddy:2.5.1-builder-alpine AS builder
COPY cert-1.crt /usr/local/share/ca-certificates/
COPY cert-2.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates
RUN xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:2.5.1-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY cert-1.crt /usr/local/share/ca-certificates/
COPY cert-2.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Self-signed certificates are needed in both stages, because both access internet (during build and later running Caddy).

Now I got another problem that appears to be some sort of blocking in port 53

{"level":"debug","ts":1653417364.9012,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/2529082684","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.5.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["55155944"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["826"],"Content-Type":["application/json"],"Date":["Tue, 24 May 2022 18:36:04 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0001cwhRTs6e_2PC34b8IWYgeGJaDv5m0l8JNeba1vWVKy8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
{"level":"error","ts":1653417364.9013665,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"ldap.private.corp.network.domain","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[ldap.private.corp.network.domain] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.ldap.private.corp.network.domain: dial tcp 173.245.59.138:53: connect: connection refused (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/55155944/2664914784) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
{"level":"debug","ts":1653417364.901389,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme-staging-v02.api.letsencrypt.org-directory"}

Time to talk with Dark Elves of IT Security! :rofl:

1 Like

Dark Elves of IT Security aren’t much friendly…

Is there any option in Caddy that allows validating DNS on Cloudflare, using email and API key and going over HTTPS?

You can configure resolvers alongside your DNS config, which will tell Caddy to use a different DNS resolver for the propagation checks etc. See the resolvers option:

Huh? Is your corporate network man-in-the-middle-ing all your outgoing connections? That’s horrible.

1 Like

Yes, that sucks a lot…

I must be doing something wrong, Caddy stills want to use Cloudflares DNS:port to solve challenges.

I think that order must do not influence but I tried to change order anyway.

{
        #debug
        #acme_ca https://acme-v02.api.letsencrypt.org/directory
        #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        local_certs
}

(cloudflare) {
        tls user@domain {
                #ca https://acme-v02.api.letsencrypt.org/directory
                ca https://acme-staging-v02.api.letsencrypt.org/directory
                dns cloudflare <API key>
                resolvers 192.168.2.70:53
        }
}

ldap.private.corp.network.domain {
        import cloudflare
        #reverse_proxy http://phpldapadmin:80
}

See, in log Caddy still wants to use 108.162.192.77:53 = brenda.ns.cloudflare.com

{"level":"error","ts":1653424064.7036276,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"ldap.private.corp.network.domain","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[ldap.private.corp.network.domain] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.ldap.private.corp.network.domain: dial tcp 108.162.192.77:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/55165394/2665495304) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
{"level":"error","ts":1653424064.7036748,"logger":"tls.obtain","msg":"will retry","error":"[ldap.private.corp.network.domain] Obtain: [ldap.private.corp.network.domain] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of _acme-challenge.ldap.private.corp.network.domain: dial tcp 108.162.192.77:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/55165394/2665495304) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":21.055531153,"max_duration":2592000}

Try resolvers 1.1.1.1 which is Cloudflare’s DNS resolver.

The problem is the corporate firewall. It not allow any outgoing communication on port 53.

I would need to use one of the internal DNS resolvers…

Geez. I hate corporate networks :man_facepalming:

You can turn off propagation checks by setting it to -1, might just work as-is. You might need to turn on propagation delay if it doesn’t immediately work (try like 30s to start)

I messed up with the documentation and couldn’t compose the correct Caddyfile to use the “propagation_timeout” and “propagation_delay” options. I saw that the “resolvers” option is also in the Issuers section.

Could you please show me how to assemble this setup?

1 Like
	tls user@domain {
		issuer acme {
			dir https://acme-staging-v02.api.letsencrypt.org/directory
			dns cloudflare <API key>
			propagation_timeout -1
		}
	}
{"level":"info","ts":1653426578.8046603,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: parsing caddyfile tokens for 'tls': /etc/caddy/Caddyfile:15 - Error during parsing: cannot mix issuer subdirective (explicit issuers) with other issuer-specific subdirectives (implicit issuers)

Ah right, need to put the email inside as well:

	tls {
		issuer acme {
			dir https://acme-staging-v02.api.letsencrypt.org/directory
			email user@domain
			dns cloudflare <API key>
			propagation_timeout -1
		}
	}
1 Like

@francislavoie , you’re a genius, man!

Defeated the Dark Elves of IT Security!

It worked first time! And if works with “staging” will work with “the real thing”.

(cloudflare) {
        tls {
                issuer acme {
                        dir https://acme-staging-v02.api.letsencrypt.org/directory
                        email <user@domain>
                        dns cloudflare <API key>
                        propagation_timeout -1
                        propagation_delay 30s
                }
        }
}

Now we have the complete solution for those who are being “man-in-the-middle-ing” and behind a very restrictive firewall.

FROM caddy:2.5.1-builder-alpine AS builder
COPY cert-1.crt /usr/local/share/ca-certificates/
COPY cert-2.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates
RUN xcaddy build --with github.com/caddy-dns/cloudflare

FROM caddy:2.5.1-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY cert-1.crt /usr/local/share/ca-certificates/
COPY cert-2.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Caddy Wins! Flawless Victory!!! :sunglasses:

2 Likes

I don’t think 30sec is correct btw, I think it needs to be 30s:

1 Like

Yes, you are right.

I corrected the solution post.

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.