Caddy working only with tls self_signed directive

Help
1

I’m new to Caddy - am using it bundled with Rocket.chat (opensource Slack-competing chat platform) as part of their Ubuntu Snap deployment model.

Its been working fine until recently updated ubuntu server and Rocket.chat, and my app could no longer be accessed over HTTPS.

I got access to my Rocket.chat app back by applying the tls self-signed directive in Caddy. My Caddyfile looks like this:

http://my-domain.com {
    redir 301 {
        / https://{host}{uri}
    }
}

https://my-domain.com {
tls self_signed
  proxy / localhost:3000 {
    websocket
    transparent
  }
}

This is working, but creates a security error in the browser with each access.

I have checked my firewall and have these configs:

To                         Action      From
--                         ------      ----
80                         ALLOW       Anywhere                  
443                        ALLOW       Anywhere                  
22                         ALLOW       Anywhere                  
3000                       ALLOW       Anywhere                  
80 (v6)                    ALLOW       Anywhere (v6)             
443 (v6)                   ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)             
3000 (v6)                  ALLOW       Anywhere (v6)

How can I diagnose what is going wrong?

2

Hey Richard, welcome.

First step would be to expound on this:

my app could no longer be accessed over HTTPS.

What do you mean by that?

And this:

recently updated ubuntu server and Rocket.chat,

Verify that this is the only thing that changed, and describe exactly what changed.

3

Hi Matt, thanks for your reply. I hope this additional information can help:

my app could no longer be accessed over HTTPS.

I would get a ‘server not available’ error - ie could not connect to my app at all over https.

recently updated ubuntu server and Rocket.chat

I ran a standard Ubuntu update:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

Also at about the same time Rocketchat did a new release via Snap which auto-updated to v 54 - some info on the Snap deployment model for the Rocket.chat app here: Deploy with Ubuntu - Rocket.Chat Docs.

I could still SSH to the server and found that my app and db were alive and well, and the update to Rocket.chat v54 had succeeded.

The problem seemed to be that https was being blocked for some reason. So I confirmed firewall was open, which it does appear to be, and then played with Caddy, and found that when using the tls self-signed directive I could connect successfully to my app - except for my browser complaining about security.

So I think I have a problem with caddy retrieving and using a Lets Encrypt cert - but I am not sure how to diagnose what is going wrong there.

Any help much appreciated.

4

Thanks, that’s helpful. What is in caddy’s log? (-log)

5

This probably a bit more useful…

Mar 27 23:23:18 Rocketchat systemd[1]: Started Service for snap application rocketchat-server.rocketchat-caddy.
Mar 27 23:23:21 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE
Mar 27 23:23:21 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Unit entered failed state.
Mar 27 23:23:21 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'.
Mar 27 23:23:21 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Service hold-off time over, scheduling restart.
Mar 27 23:23:21 Rocketchat systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy.
6

That looks like a systemd log - what about Caddy’s log?

7

Here it is, thanks Matt

Mar 29 02:26:38 Rocketchat systemd[1]: Started Service for snap application rocketchat-server.rocketchat-caddy.
Mar 29 02:26:38 Rocketchat /usr/bin/snap[33501]: cmd.go:111: DEBUG: restarting into "/snap/core/current/usr/bin/snap"
Mar 29 02:26:40 Rocketchat snap[33501]: Activating privacy features...2017/03/29 02:26:40 [<myurl.com>] failed to get certificate: acme: Error 429 - urn:acme:error:rateLim
Mar 29 02:26:40 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE
Mar 29 02:26:40 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Unit entered failed state.
Mar 29 02:26:40 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'.
Mar 29 02:26:41 Rocketchat systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Service hold-off time over, scheduling restart.
Mar 29 02:26:41 Rocketchat systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy.
8

I don’t know why but your log lines are being truncated. Fortunately, they did include “rateLim” which tells us that you’ve been rate limited by Let’s Encrypt. You’ll have to wait until the limits are expired and then try again.

9

Thanks Matt. how long is the wait? And does each attempt to connect wind up extending the wait period since I’ve triggered ratelimit again?

10

Here’s the relevant resource.

You should be able to use their staging endpoint regardless of rate limits on the production endpoint.

1 Like
11

Thanks @matt @Whitestrake for your help - I quickly looked at that Lets Encrypt ratelimit info and cant imagine how I over ran it - I’m using it in a test instance of a chat system that has extremely low usage. Anyway, I’ll wait out my week and see if it comes back to life for me.

12

Does Caddy have access to persistent storage to keep its certificates? I’m totally unfamiliar with Snaps, but they seem to share some similarities to containers. I know people running Caddy in Docker hit rate limits very quickly when they aren’t mindful of certificate storage. It only takes five startup cycles for a Dockerized Caddy to be rate limited for the same set of certificates (duplicate cert limit 5/week).

13

What @Whitestrake said, and also remember that subdomains apply to the 20/week limit.

14

Right, thanks for that. I’ll check with the Rocket.chat team who look after the Snap builds. Sounds like I could be hitting that same problem as the Docker folks.

15

Sorry to bump an old thread. @Whitestrake reached out via email and mentioned a couple of doc changes to improve our caddy documentation. Just wanted to follow up on this. Where would caddy stick the certificates in ubuntu and would it throw any errors if it were unable to write it? Or would it carry on? Snaps only have certain places files can be stored, so this could definitely be an issue much like if you didn’t use a docker volume at the location of the certs.

16

Caddy puts certificates and OCSP staples etc. in the $HOME/.caddy folder by default (created if it doesn’t exist) or, if $CADDYPATH is set, it puts them in there. Errors are logged to the Caddy log, -log flag configures that. See https://caddyserver.com/docs/cli.

Hope that is helpful!

2 Likes
17

Hi Aaron!

Caddy makes a .caddy folder in the home directory of the user that runs it, i.e./root/.caddy or /home/whitestrake/.caddy.

2 Likes
18

ah perfect! We can set the directory in a write safe place as well as maybe take a look at logs. Stdout and stderr should be hitting the journal.

Also going to push another release here in the next couple of days to get everyone on caddy 0.10 so you shouldn’t get people coming with the timeout error :slight_smile:

If you guys get any other rocket.chat related please feel free to drag me in with a mention. We love caddy :heart:

2 Likes
19

Cool! And I’ll probably push Caddy 0.10.1 out the door tonight or tomorrow morning. It’s even better than 0.10. :wink:

20

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.