Problems upgraded from 0.11 to current using Cloudflare docker

1. Caddy version (caddy version):

Using Docker image created by @SlothCroissant slothcroissant/caddy-cloudflaredns:latest

2. How I run Caddy:

Docker via this docker compose.

a. System environment:

Docker

b. Command:

c. Service/unit/compose file:

  caddy:
    container_name: Caddy
    image: slothcroissant/caddy-cloudflaredns:latest
    restart: unless-stopped
    ports:
        - '80:80'
        - '443:443'
    volumes:
        - '/sharedfolders/appdata/Caddy-docker/data:/data'
        - '/sharedfolders/appdata/Caddy-docker/config/:/config'
        - '/sharedfolders/appdata/Caddy-docker/Caddyfile:/etc/caddy/Caddyfile'
    environment:
        - CLOUDFLARE_EMAIL=myemail@myhost.com
        - CLOUDFLARE_ API_TOKEN=
        - ACME_AGREE=true

I do have the CF Token set but forum will not let me post it with a bunch of X’s

d. My complete Caddyfile or JSON config:

sub.duckdns.com:443 {
	tls myemail@myhost.com { 
  			dns cloudflare {env.CLOUDFLARE_API_TOKEN}
		}
        reverse_proxy /  localhost:8123 {
        }

3. The problem I’m having:

I’ve been running Caddy since 2018 with the same install. I’m moving all my services to Docker and am trying to upgrade from my 0.11 install to latest. It was close to miracle that I got this working years ago due to my feeble brain and am asking for help in sorting this again.

4. Error messages and/or full log output:

2022/03/19 23:42:44.127	INFO	using provided configuration	{"config_file": "/etc/caddy/Caddyfile", "config_adapter": "caddyfile"}
,2022/03/19 23:42:44.128	WARN	input is not formatted with 'caddy fmt'	{"adapter": "caddyfile", "file": "/etc/caddy/Caddyfile", "line": 2}
,2022/03/19 23:42:44.130	INFO	admin	admin endpoint started	{"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
,2022/03/19 23:42:44.130	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc0001ae0e0"}
,2022/03/19 23:42:44.131	INFO	tls.cache.maintenance	stopped background certificate maintenance	{"cache": "0xc0001ae0e0"}
,run: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: loading certificates: open /sharedfolders/appdata/Caddy/cert.pem: no such file or directory

To be honest, I did run LetsEncrypt for a short time and I’m not certain if the cert.pem is from that attempt or not.

5. What I already tried:

The error claims it cannot open the cert.pem file but I can do a listing and it appears to be there:

root@omv:~# l /sharedfolders/appdata/Caddy/cert.pem
-rw-rw-rw- 1 root users 2508 Sep 12  2018 /sharedfolders/appdata/Caddy/cert.pem

My previous Caddyfile had an entry like this:

sub.duckdns.com:443 {
	tls  /sharedfolders/appdata/Caddy/cert.pem /sharedfolders/appdata/Caddy/privkey.pem
	proxy /  localhost:8123 {
		websocket
		transparent
	}
}

6. Links to relevant resources:

Using this guide:

I’ve been wanting to update Caddy for a long time but I’ve found myself struggling to figure this out after a few attempts. Hopefully I’ve provided the information needed but will provide any and all requested. Any guidance is most appreciated.

Are you sure you loaded the config file you think you did? Did you accidentally mount your Caddy v1 Caddyfile instead?

Caddy is looking for the cert inside the container, not on your host machine (because it’s running in Docker, it’s isolated)

There’s an extra space in the key name here, just in case this was an oversight

You don’t need this in Caddy v2. The ACME terms are implicitly agreed to simply by using Caddy.

Path matching in Caddy v2 is exact; you used a path matcher of / here, which only matches exactly / and nothing else. Remove the / to match all requests.

1 Like

Looks like you’re right. I had nothing in my config dir except for an autosave file. Got rid of that sub and it appears to be loading now.

That was a goof in the posting. Forum didn’t like that variable being used so I tried a few ways to bypass. It is correct in my config file.

Removed

Looks like this now (forum doesn’t like my fake email):

sub.duckdns.org:443 {
	tls email here { 
  			dns cloudflare {env.CLOUDFLARE_API_TOKEN}
		}
        reverse_proxy localhost:8123 {
        }
}

and here’s the logs with the changes made:

{"level":"info","ts":1649764195.3852715,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"},
{"level":"warn","ts":1649764195.3862064,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2},
{"level":"info","ts":1649764195.3891864,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]},
{"level":"info","ts":1649764195.389794,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443},
{"level":"info","ts":1649764195.3898184,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"},
{"level":"info","ts":1649764195.3901834,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sub.duckdns.org"]},
{"level":"info","ts":1649764195.3904192,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"},
{"level":"info","ts":1649764195.390428,"msg":"serving initial configuration"},
{"level":"info","ts":1649764195.3904603,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"},
{"level":"info","ts":1649764195.3904796,"logger":"tls","msg":"finished cleaning storage units"},
{"level":"info","ts":1649764195.3909945,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000424000"},
{"level":"info","ts":1649764195.391946,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sub.duckdns.org"},
{"level":"info","ts":1649764195.4304008,"logger":"tls.obtain","msg":"lock acquired","identifier":"sub.duckdns.org"},
"msg":"trying to solve challenge","identifier":"sub.duckdns.org","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"},
"error":"no memory of presenting a DNS record for sub.duckdns.org (probably OK if presenting failed)"},
"error":"[sub.duckdns.org] solving challenges: presenting for challenge: adding temporary record for zone duckdns.org.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] 
{"level":"info","ts":1649764196.4005592,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["sub.duckdns.org"],"ca":
{"level":"info","ts":1649764196.4005885,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["sub.duckdns.org"],"ca":
"msg":"trying to solve challenge","identifier":"sub.duckdns.org","challenge_type":"dns-01",
"msg":"cleaning up solver","identifier":"sub.duckdns.org","challenge_type":"dns-01","error":"no memory of presenting a DNS record for sub.duckdns.org (probably OK if presenting failed)"},
{"level":"error","ts":1649764216.720962,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sub.duckdns.org","issuer":,"error":"[sub.duckdns.org] solving challenges: presenting for challenge: adding temporary record for zone duckdns.org.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] 
{"level":"error","ts":1649764216.7210455,"logger":"tls.obtain","msg":"will retry","error":"[sub.duckdns.org] Obtain: [sub.duckdns.org] solving challenges: presenting for challenge: adding temporary record for zone duckdns.org.: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] ","attempt":1,"retrying_in":60,"elapsed":21.290611261,"max_duration":2592000},

Note that I used to use Let’s Encrypt early on but switched to CloudFlare and that’s what I use now. I do see LE comments in the logs. Forum forced me to remove references to Let’s Encrypt and Acme. I sanitized the best I could while keeping as much of the errors as I could. Is there a more automated way of sanitizing these logs? Hopefully you can see what is needed.

Thank you for your detailed troubleshooting and your patience.

Wait, is CloudFlare your DNS provider, or is DuckDNS?

The DNS plugin is for writing DNS TXT records at your DNS provider. If it’s DuckDNS that controls it, then you need the duckdns plugin, not the cloudflare plugin.

I can see where that can be confusing. I am only using DuckDNS because I do not have a static IP. I am using CloudFlare for my redirects. I am using the SSL certificate from Cloudflare and my own domain for those reverse proxys. I have several but only listed the first which happened to be the duckdns one. I probably should not even be using that one at all.

CNAME myservice.mydomain. com and these are sent to the target which is my duckdns.org subdomain. Hopefully that makes sense.

I am certainly running my current install with a version specifically for CloudFlare.

Right, but the cloudflare plugin cannot write DNS records to a DuckDNS domain. The DuckDNS domain isn’t part of your CloudFlare account.

If you’re trying to get a cert for your DuckDNS domain, you need to use the duckdns plugin.

1 Like

@francislavoie

Okay. I see exactly what you’re saying. Here’s my original:

#duckdns site
name {
	tls  /sharedfolders/appdata/Caddy/cert.pem /sharedfolders/appdata/Caddy/privkey.pem
	proxy /  localhost:8123 {
		websocket
		transparent
	}
}

#My domain site
name {
        tls {
                dns cloudflare
        }
        proxy /  localhost:8123 {
                websocket
                transparent
        }
}

I was mixing the two. Really all I care about is the second config (my domain). I don’t need the duckdns at all. That’s a holdover that can go away.

I made a change to the localhost to point to the correct IP address (it’s not localhost anymore now that it’s in docker) and I’m getting a message saying that " This site can’t provide a secure connection" …

This is what my v2 looks like now:

#My domain site
name {
        tls email { 
  			dns cloudflare {env.CLOUDFLARE_API_TOKEN}
		}
       reverse_proxy 192.168.0.25:8123 {
       }
}

Sure, that’s probably correct.

But I strongly suggest formatting your Caddyfile, it looks very messy with inconsistent whitespace right now.

You can run caddy fmt --overwrite /path/to/your/Caddyfile

And you can remove the { } after reverse_proxy because you’re not using any options.

I took a step back and made an empty directory and copied my Caddyfile in it with the braces removed. My hopes is that this clean start will have a positive impact on getting this resolved. I’ve also removed the duckdns entry.

I see Caddy complaining about using fmt but I can’t figure out how to do this in my docker container. I tried to exec in with bash but I cannot get shell access. Can you tell me how I might do this with this container? Is this a show stopper?

Here’s the logs I have now:

{"level":"info","ts":1649773664.5720608,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"},
{"level":"warn","ts":1649773664.5731375,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":3},
{"level":"info","ts":1649773664.5756335,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:2019","localhost:2019"]},
{"level":"info","ts":1649773664.5759187,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443},
{"level":"info","ts":1649773664.5759323,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"},
{"level":"info","ts":1649773664.5759308,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ac000"},
{"level":"info","ts":1649773664.5763106,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"},
{"level":"info","ts":1649773664.5763538,"logger":"tls","msg":"finished cleaning storage units"},
{"level":"info","ts":1649773664.5764377,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["MYSITE"]},
{"level":"info","ts":1649773664.5767787,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"},
{"level":"info","ts":1649773664.5767887,"msg":"serving initial configuration"},
{"level":"info","ts":1649773664.5769625,"logger":"tls.obtain","msg":"acquiring lock","identifier":"MYSITE"},
{"level":"info","ts":1649773664.60984,"logger":"tls.obtain","msg":"lock acquired","identifier":"MYSITE"},
{"level":"info","ts":1649773664.9526122,"logger":"ACME","msg":"waiting on internal rate limiter","identifiers":["MYSITE"],"ca":"https: LETSENCRYPT directory","account":"MYEMAIL"},
{"level":"info","ts":1649773664.9526498,"logger":"ACME","msg":"done waiting on internal rate limiter","identifiers":["MYSITE"],"ca":"LETSENCRYPT directory","account":"MYEMAIL"},
{"level":"info","ts":1649773665.0941625,"logger":"ACME","msg":"trying to solve challenge","identifier":"MYSITE","challenge_type":"dns-01","ca":"LETSENCRYPT"},
{"level":"error","ts":1649773670.3734777,"logger":"ACME","msg":"cleaning up solver","identifier":"MYSITE","challenge_type":"dns-01","error":"no memory of presenting a DNS record for MYSITE (probably OK if presenting failed)"},
{"level":"error","ts":1649773670.4363408,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"MYSITE","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[MYSITE] solving challenges: presenting for challenge: adding temporary record for zone .: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=https: LETSENCRYPT) (ca=https: LETSENCRYPT)"},
{"level":"info","ts":1649773670.8003316,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":""},
{"level":"error","ts":1649773671.1172752,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"MYSITE","issuer":"ACME","error":"provisioning client: HTTP 500: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n [no address given] to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache Server at acme.zerossl.com Port 443</address>\n</body></html>\n"},
{"level":"error","ts":1649773671.1173236,"logger":"tls.obtain","msg":"will retry","error":"[MYSITE] Obtain: provisioning client: HTTP 500: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n [no address given] to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n<hr>\n<address>Apache Server at acme.zerossl.com Port 443</address>\n</body></html>\n","attempt":1,"retrying_in":60,"elapsed":6.507454319,"max_duration":2592000},

I’ve tried to sanitize as much as I can without losing information that may be of use in diagnosing.

Thank you for your continued patience.

I logged into Cloudflare . You can see my CNAME records I created long ago. You can also see that Caddy did something that caused these TXT records to show up. I’m not sure what that means.

Not a show stopper. Just reads nicer, quiets the warning.

The Caddy docker image is based on Alpine Linux, and doesn’t come with bash. It does have /bin/sh though. So you can do this:

docker-compose exec caddy /bin/sh

Which gives you a shell in the container. Then you can run:

caddy fmt --overwrite /etc/caddy/Caddyfile

Which should clean it up. Since the file is mounted from the host, it’ll fix the file “on both sides”.

Then obviously exit the container shell with exit.

Looks like ZeroSSL is having some problems :grimacing:

I think this is an authentication issue with CloudFlare for some reason. Are you sure you used the right credentials? Make sure to read the instructions here:

I’ve confirmed that the token I’m using is the same as I previously used. Is there any harm in creating another token? I do not want to break my current install though.

Also, I watched a video on Youtube a bit ago and they are talking about setting up a zone in CF. I do not remember doing that previously but perhaps I did. Is there a walkthru available that details the CF settings needed?

Again, thank you for your time and patience.

The README that I linked explains what you need to know

Thank you. You were kind to link the relevant part before and I had not looked at the top. My apologies. I’ve checked my CF and it appears that I have the settings correct:


I went ahead and created a new token using these settings:

I checked with the supplied curl command and I did get:

"message":"This API Token is valid and active"

I then updated my token in my Caddyfile as well as my env for the docker:

I supplied the key instead of the environment variable to ensure that the value would be used, but still not working:


{"level":"error","ts":1649804771.4206605,"logger":"tls.obtain","msg":"will retry","error":"[site] Obtain: [site] solving challenges: presenting for challenge: adding temporary record for zone mysite .: got error status: HTTP 400: [{Code:6003 Message:Invalid request headers}] (order=) (ca=)","attempt":6,"retrying_in":1200,"elapsed":1219.47136643,"max_duration":2592000}

Could you be so kind as to look over what I have here and tell me if I’m overlooking something?

@francislavoie I hope I have not caused you a lot of grief with these problems. I have been trying many different combinations to get this working and have not met with any success. I went through every step in the readme and have had no luck at getting this to work.

I attempted to configure things using JSON but I could not figure out where this configuration file should go. I did find the autosave.json and this is what it contains:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [":443"],
                    "routes": [
                        {
                            "handle": [{ "handler": "subroute", "routes": [{ "handle": [{ "handler": "reverse_proxy", "upstreams": [{ "dial": "192.168.0.25:8123" }] }] }] }],
                            "match": [{ "host": ["sub mysite com"] }],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuers": [
                            { "challenges": { "dns": { "provider": { "api_token": "{env.CF_API_TOKEN}", "name": "cloudflare" } } }, "email": "my email", "module": "acme" },
                            { "challenges": { "dns": { "provider": { "api_token": "{env.CF_API_TOKEN}", "name": "cloudflare" } } }, "email": "my email", "module": "zerossl" }
                        ],
                        "subjects": ["sub mysite com"]
                    }
                ]
            }
        }
    }
}

I’ve updated using the confirmed working token as an environment variable and have tried referencing the variable in the Caddyfile and the token itself in the Caddyfile and neither work.

It does not seem like there is a whole lot of variables in this but still I cannot get it working. This is the very last piece for modernizing my docker deployments and it is one of the most critical ones.

Any help is most appreciated from you or anyone else who can help.

Just a quick thing to double-check:
Is your API token, as shared in the cloudflare dashboard screenshots, read-only? You need DNS write access for this cloudflare dns module to work. Actually, this applies to all DNS modules.

You should be good using the provided template in Create TokenAPI token templatesEdit zone DNS, after changing the Zone ResourcesSelect zones to include or exclude , all within https://dash.cloudflare.com/profile/api-tokens.

@IndeedNotJames Thank you for taking the time to write this up. I issued a new token using these settings and updated my docker-compose to reflect the new token but I am still getting the same error mentioning invalid headers.

EDIT I just hovered over the ‘ACTIVE’ tag on the tokens page and for the new one it says ‘Never Used’. Is there a delay in when these are issued to when they start working? Can I have multiple tokens that affect the same account? Right now I have three active keys; my original, and the two I’ve created trying to get this docker up and running.

This is what it looks like in CF:

Is there a way to get more information in the logs? Is there a debug option perhaps?

Hmm… okay.

Two additional things I noticed:

  1. Your docker-compose.yml (both text and screenshot) reads CLOUDFLARE_API_TOKEN, while your latest json config (autosave.json or alternatively curl https://localhost:2019/config/) states CF_API_TOKEN

    They need to match. Might be worth double-checking :innocent:

  2. The Caddyfile you shared via screenshot right before you wrote

    shows basically the following, partially blacked out line:

    dns cloudflare {'some_value'}
    

    If you are actually trying to pass your token directly there, you need to write them without those curly brackets:

    dns cloudflare 'some_value'
    # or
    dns cloudflare some_value
    

    The curly brackets are a hint/language feature to substitute a placeholder. In your case env.*
    See Conventions — Caddy Documentation

1 Like

Yes Yes Yes! That did it. I got rid of the curly braces and it worked as expected. Thank you for you help on this!

3 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.