Need help preventing letsencrypt rate limits and not needing tls insecure skip verify

1. The problem I’m having:

I am new to Caddy, well I have been playing with it for a few weeks now. I have yet to succesfully setup https for applications like nextcloud or authentik. I have been getting rate limited and having to start over multiple times.

I did learn about the staging environment recently so was using that, but still managed to get that rate limited too.

So now I am back to using regular caddy but with no tls verify on nextcloud and authentik, in order to keep my apps running. In 24 hours I will try caddy-staging again without no tls verify and after I have done so more research before I make the same mistakes and get rate limited again (hopefully with help here)

If there is something I should be doing while I do no_tls_verify to ensure it doesn’t keep trying and get me rate limited as soon as it’s lifted.

Note: The current config with no_tls_verify is working fine but obviously I would like to use a proper TLS certificate and remove that option.

2. Error messages and/or full log output:

I am using docker so do not have record of all my failed attempts. I am not sure if the current logs are helpful as they mention a lot of rate limit errors but they are included. Please let me know if I need to update this post with fresh logs after my rate limit expires.

Due to the limit of characters allowed in post, my error logs are accessible via pastebin

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

a. System environment:

Headless Ubuntu Server Machine running Linux homebase 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Running Docker Compose version v2.17.2

Docker Version Info:

Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.20.4
 Git commit:        297e128
 Built:             Wed May 24 23:30:52 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.24
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       5d6db84
  Built:            Wed May 24 23:31:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

b. Command:

sudo docker compose up -d

c. Service/unit/compose file:

version: '3.3'
services:
    caddy:
        container_name: caddy
        ports:
            - '80:80'
            - '443:443'
        volumes:
            - caddy_data:/data
            - caddy_config:/config
            - ./Caddyfile:/etc/caddy/Caddyfile
        image: 'caddy:latest'
volumes:
    caddy_data:
    caddy_config:

d. My complete Caddy config:

notes.alfredoautomation.com {
	reverse_proxy	10.0.0.96:5230
}

cloud.alfredoautomation.com {
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/carddav /remote.php/dav 301
	reverse_proxy	https://10.0.0.96:444 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

rss.alfredoautomation.com {
	reverse_proxy	10.0.0.96:8085
	redir / https://rss.alfredoautomation.com/oauth2/oidc/redirect
}

sso.alfredoautomation.com {
	reverse_proxy	https://10.0.0.96:9444 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

photos.alfredoautomation.com {
	reverse_proxy	http://10.0.0.96:2283
}

food.alfredoautomation.com {
	reverse_proxy	10.0.0.96:4580
}

5. Links to relevant resources:

Some things I have tried include:

  • Switching my Caddyfile config for sso and cloud to http instead of https (while also setting nextcloud and sso to using non ssl ports in their own docker-composes)
  • At one point my DNS was hosted via cloudflare and I tried the cloudflare module of caddy.
  • For unrelated reasons I am no longer using cloudflare, my DNS is currently with fastmail until I setup my own dns.
  • I have switched to the regular caddy instead of caddy-cloudflare
  • as mentioned I have tried both staging environment and regular.
  • I believe at one point my staging environment was all successful but when I tried again recently it all failed.

If my research has actually made it through my head, I THINK this is what I should do when the rate limit drops:

  1. Change my Caddyfile (removing staging and debug once confirmed) to:
{
  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
  debug
}

notes.alfredoautomation.com {
	reverse_proxy	10.0.0.96:5230
}

cloud.alfredoautomation.com {
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/carddav /remote.php/dav 301
	reverse_proxy	10.0.0.96:81 
}

rss.alfredoautomation.com {
	reverse_proxy	10.0.0.96:8085
}

sso.alfredoautomation.com {
	reverse_proxy	10.0.0.96:82
}

photos.alfredoautomation.com {
	reverse_proxy	10.0.0.96:2283
}

food.alfredoautomation.com {
	reverse_proxy	10.0.0.96:4580
}
  1. Update my docker-compose files for next cloud and authentik to use ports 81 and 82 respectively.
  2. Make the other suggested changes for authentik as per the linked authentik guide in my original post.
  3. Profit?

Edit: Tried again today with my proposed changes and I think everything is working! yay!

1 Like

This should have prevented your ACME accounts and certificates from being lost when you tear down/restart your containers. What were you doing? Was this missing from your Docker Compose config initially?

Caddy would not have tried again to get certs if storage was properly persisted, so something else must have gone wrong. Caddy has all kinds of rate limit avoidance, including switching to the staging endpoint automatically when it fails the first time. See Automatic HTTPS — Caddy Documentation

Wow that is cool and I did not know that. Clearly missed it when reading through the article.

I definitely had removed the persistent volumes under the impression my failures were storing data in them and needed a fresh start.

I see now that was a not so smart idea and keeping them would have benefited me.

As I learn more about caddy and enhance my Caddyfile to do more than reverse proxies I am glad you told me that so I don’t make that mistake again!!

1 Like