Failing acme challange Caddy2 after adding Cloudflare plugin

So does the official container. See the docs, in particular the builder image variant.

I don’t understand the question. Yes, you need to persist Caddy’s data storage.

What’s unclear? The docs very clearly outline the syntax. Compare with what you posted above.

1 Like

It does but not as simple as this image. IMHO.

Question… well I point a volumes to ./appdata/caddy will not Caddy look into the folder and automatically find the data folder, if present? Do I really need to point it out specifically?

For me the docs isn’t clear enough, I would really love an example.
Is it in no way or form possible to get you to provide an example based on my issue?

Well, we can’t support unofficial Docker images. Our official image is what we recommend, for a variety of reasons.

This is why we recommend using the official Docker image. It’s complicated. I just took a look at the hotio image, and they do not wire things up correctly for Caddy’s data storage to be persisted.

The example is right here:

First off your the best! Thank you for all the help and time you give me! :heart_eyes:

The docker-compose is not the issue at hand, but this is how it looks now:

        volumes:
          - ./appdata/caddy:/config
          - ./appdata/caddy/data:/data

So we can leave that behind for the moment.
As far as I understand this is about how to write the Caddyfile and that is done in the same way hotio or not.
And according to the link you gave I tried to write the Caddyfile like this:

tls {
  dns cloudflare {env.CF_TOKEN}
}

grillgeek.se {  
    reverse_proxy grillgeekse
}

Which makes Caddy crash.

{"level":"info","ts":1629437558.8157573,"msg":"using provided configuration","config_file":"/config/Caddyfile","config_adapter":""}
run: adapting config using caddyfile: /config/Caddyfile:2: unrecognized directive: dns
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

Help me @francislavoie you’re my only hope!

I should also add that I feel stupid not getting this.

Please review the docs, in particular the structure of the Caddyfile.

Directives must go inside site blocks.

That still won’t work, because the hotio container doesn’t actually set Caddy’s data storage path to /data. It’s incorrectly set up for persisting data storage. I seriously cannot recommend using it.

Please read the docs on Docker Hub, in particular the section “Adding custom Caddy modules”. That’s what you should use.

I put no pride in using whatever image, I just thought that hotios way of adding plugins was neater than the official way. As far as I understand adding plugins to the official way means building my own image?
How do I keep that updated? Rebuild it from time to time?

So to the real question at hand, is there any reason for me to keep trying to use Cloudflares SSL and just give up and use Letsencrypt?
With that, back to the officail image and go from there.

Yeah, you just run docker-compose build after having changed the version of Caddy in the Dockerfile.

Well, using the DNS challenge, you are using Let’s Encrypt/ZeroSSL. You’re just configuring Caddy so that it can set DNS TXT records on your Cloudflare account, that Let’s Encrypt or ZeroSSL will verify to prove that you control that domain.

Please just show me how I should set this up. I have read the docs the examples the forums posts, still I run into a wall.

Make a directory called caddy next to your docker-compose.yml.

Put your Caddyfile in that directory.

In that directory, make a file named Dockerfile with these contents:

ARG CADDY_VERSION=2.4.3

FROM caddy:${CADDY_VERSION}-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:${CADDY_VERSION}

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

In your docker-compose.yml, add:

services:
  caddy:
    build: ./caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - $PWD/caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config

volumes:
  caddy_data:
  caddy_config:

(Replace your existing caddy service with the above, and make sure to add the volumes at the bottom of the file)

Run docker-compose build which will build the Caddy v2.4.3 with the plugins listed in the Dockerfile.

Then run docker-compose up -d as normal to boot it up.

I followed your instructions and now I have my own caddy container up and running.
Time to add stuff to the Caddyfile.

I added this:

grillgeekse {
	reverse_proxy grillgeekse
}

The logs that was put out… oh wow.

{"level":"info","ts":1629442241.5226223,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442242.7436438,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442243.8884454,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442245.1327083,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442246.7178838,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442249.1284099,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442253.105347,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442260.2996476,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442273.8552847,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
run: adapting config using caddyfile: EOF
{"level":"info","ts":1629442300.2261066,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1629442300.2314253,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1629442300.2318184,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1629442300.2318616,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1629442300.2321897,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00029d110"}
{"level":"info","ts":1629442300.234321,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1629442300.2345786,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1629442300.234838,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["grillgeekse"]}
{"level":"info","ts":1629442300.235398,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1629442300.2354624,"msg":"serving initial configuration"}
{"level":"info","ts":1629442300.2360005,"logger":"tls.obtain","msg":"acquiring lock","identifier":"grillgeekse"}
{"level":"info","ts":1629442300.25733,"logger":"tls.obtain","msg":"lock acquired","identifier":"grillgeekse"}
{"level":"info","ts":1629442301.3285463,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1629442301.3287232,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"error","ts":1629442301.483864,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grillgeekse","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"grillgeekse\": Domain name needs at least one dot"}
{"level":"warn","ts":1629442301.4848719,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1629442302.186465,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"me7eG2HK568H7pJB2EUrFg"}
{"level":"info","ts":1629442303.7822263,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1629442303.7822804,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"error","ts":1629442304.236745,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grillgeekse","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Invalid DNS identifier [grillgeekse]"}
{"level":"error","ts":1629442304.2368124,"logger":"tls.obtain","msg":"will retry","error":"[grillgeekse] Obtain: [grillgeekse] creating new order: request to https://acme.zerossl.com/v2/DV90/newOrder failed after 1 attempts: HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Invalid DNS identifier [grillgeekse] (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":3.979451989,"max_duration":2592000}
{"level":"info","ts":1629442308.6105514,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1629442308.610605,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":1629442308.6132107,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00029d110"}
{"level":"info","ts":1629442308.6133301,"logger":"tls.obtain","msg":"releasing lock","identifier":"grillgeekse"}
{"level":"error","ts":1629442308.6137311,"logger":"tls","msg":"job failed","error":"grillgeekse: obtaining certificate: context canceled"}
{"level":"info","ts":1629442308.614947,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":1629442308.6149962,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1629442309.7130032,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1629442309.7158709,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1629442309.7161958,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1629442309.7162435,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1629442309.7172883,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00021d5e0"}
{"level":"info","ts":1629442309.7175262,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1629442309.7175865,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1629442309.7176592,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["grillgeekse"]}
{"level":"info","ts":1629442309.725937,"logger":"tls.obtain","msg":"acquiring lock","identifier":"grillgeekse"}
{"level":"info","ts":1629442309.7260933,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1629442309.726113,"msg":"serving initial configuration"}
{"level":"info","ts":1629442309.7531338,"logger":"tls.obtain","msg":"lock acquired","identifier":"grillgeekse"}
{"level":"info","ts":1629442309.7680786,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1629442309.768108,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["grillgeekse"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"error","ts":1629442310.645444,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"grillgeekse","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:rejectedIdentifier - Error creating new order :: Cannot issue for \"grillgeekse\": Domain name needs at least one dot"}
{"level":"warn","ts":1629442310.646039,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
{"level":"info","ts":1629442311.2585263,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"0s6P9msdL2Bvfb8flBmVfA"}

Correct the data storage path is set to /config and data is persisted just fine. Thanks for the kind words…Shows me for trying to get people adopted to Caddy.

Alright, I finally managed to dig deeper to find it where you set the XDG paths. base/linux-amd64.Dockerfile at 971ade1a10db8691cb155cede48e8a58dacc980f · hotio/base · GitHub

The hotio github repos are very confusing to navigate because their default branches are set to something essentially useless, obfuscating where the actual Dockerfile sources are, i.e. in a non-default branch. On top of that, it’s not documented anywhere how the paths are configured (for Caddy anyways), it’s hidden beneath layers of Dockerfiles.

Also. using /config for all the storage is not consistent with our official images, so it complicates our job for providing support.

Running the container under a non-root user needlessly complicates setup, for little benefit. It means that the Caddyfile needs to override the HTTP and HTTPS ports to not use low ports, unless setcap is set allow low ports. This is just UX hurdles that aren’t really helpful.

I’m not intending to offend, but fragmentation of the community and installation methods is counterproductive, because it just adds confusion by having too many options to choose from. We’re trying to narrow things down, and we’re continually fighting misinformation from poorly written guides out in the wild which have bad recommendations (like invalid configs, using / as path matchers which only makes things apply to exactly /, setting upstream headers that may break Caddy’s default behaviour, etc).

There are reasons we decided to design our official deployment methods this way. We recommend them because they’re a known-quantity, and will work as intended.

some would agree with you here, I find that it makes perfect sense with the way it’s integrated into the CI, I try to steer people to the website though and there they can find links to the exact used source at the time for the respective docker tags, the images also contain labels with a link to the source. But in the end we can pick apart just about anything and focus on its flaws.

it’s consistent with my images though, what was the main goal and /data is a folder that’s mainly used for other things if you start going beyond the Caddy community

on clean installs by my image, it’s a non issue and many people prefer to not run as root, I see no point in doing it when it’s easily avoided

couldn’t agree more, most guides don’t even serve the purpose of helping, but merely to generate ad revenue

I respect that and if it wasn’t for the horrible UX when wanting to use plugins/modules I probably wouldn’t even have bothered making an image and used the official one.

The documentation gives both cli and compose examples with the paths to use, always /config…Not sure what “layers of Dockerfiles” is supposed to mean? Every architecture has its own Dockerfile, that’s it.

But lets get back on track here and focus on the real problem the user was experiencing, a Caddyfile config issue…doesn’t really matter how it runs, because if this was a bare metal install he could run it just the same.

I clicked every relevant link I could find on your site, and none of them led me to Dockerfiles. It was only once I clicked on the branch dropdown that I found them. I’d call that heavily obfuscated.

I strongly disagree here. The UX is as good as it can be. xcaddy is our official tool for building Caddy with plugins, and it’s what the Download page uses under the hood anyways. I don’t think it’s better to have users fetch a binary on their own out-of-band.

click the commit

It’s an opinion, I by default go looking at all the branches if I don’t find something in the deault one…calling it obfuscated is just another dig imo.

just another opinion, ME (your user), who builds docker images for fun, found it to be such a bad UX that he rather just build his own image…as do the other 17600 other docker pulls confirm…yes I know, very low numbers, but still I’m not alone in this. Not everybody has the technical know-how to self-compile. I think Unraid/Synology users have an easier time manually downloading a binary than to self build the image…but again opinions are like assholes, everyone has one

I don’t know where to begin, but to say that building your own image is good UX, I could not disagree more.
All this made me drop Caddy and I went back to linuxserver/Swag, it took me under 10 mins to set up all my subdomains with Cloudflare support. WIthout building my own image.

The caddy docs aren’t clear enough, it is written for you not for beginners like me.
And during all the help you have given most of them has been RTFM.
I am sorry to say that is not good enough.

The way Hotio has built his image and how it works should have been how the offical should work.

@francislavoie Hours later, I sat down at wrote about this experience.
In the end I added a note to Caddy and what you can do better.