Can't make secure connection to pi but can to Mac on same network

Help
1

1. The problem I’m having:

I have two machines on my local network, a mac running caddy and a pi that i’m trying to reach. The mac is working great. I can ssh in and connect to https node apps. The pi is unreachable via the domain name. ssh will not connect, nor will https. http WILL connect though. Have tried everything. this used to work with an old Pi 4, but when I upgraded to a 5, it went belly up.

2. Error messages and/or full log output:

When I try to `curl -vL https://pi.azthir-terra.com:50000, i get the following error:

* Host pi.azthir-terra.com:50000 was resolved.
* IPv6: (none)
* IPv4: 108.51.174.138
*   Trying 108.51.174.138:50000...
* Connected to pi.azthir-terra.com (108.51.174.138) port 50000
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

When i try to ssh chris@pi.azthir-terra.com -p 1387 i get the following:

ssh: connect to host pi.azthir-terra.com port 1387: Operation timed out

attempting to connect via web browser to either pi.azthir-terra.com:50000 or 108.51.174.138:50000 gives me an error that a secure connection can’t be made.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

via homerew. No modifications

a. System environment:

MacOS 14.5,

b. Command:

caddy is running as a service. I use the folllowing commands

brew services restart caddy
sudo nano /opt/homebrew/etc/caddyfile

d. My complete Caddy config:

# This replaces the existing content in /etc/caddy/Caddyfile

# Global options block
{
        debug
}

192.168.5.86 {
        # PROXY ALL REQUEST TO PORT 30000
        tls internal
        reverse_proxy localhost:30000
        encode zstd gzip
}

foundry.azthir-terra.com {
        # PROXY ALL REQUEST TO PORT 30000
        reverse_proxy localhost:30000
        encode zstd gzip
}

5e.azthir-terra.com {
        # PROXY ALL REQUEST TO PORT 5000
        root * /Volumes/foundrydisk/5etools-mirror-2.github.io/
        file_server browse
        encode zstd gzip
}

pi.azthir-terra.com {
        reverse_proxy 192.168.7.121:1387
        encode zstd gzip
}

pi.azthir-terra.com:50000 {
        reverse_proxy 192.168.7.121:50000
        encode zstd gzip
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

Any suggestions on what the problem might be would be greatly appreciated

Thanks!
C

2

Is that your public IP address? Are you port forwarding port 50000 to your Pi’s LAN IP address in your router?

This doesn’t sound like a problem with Caddy at all, it sounds like a networking issue.

I’m not sure what relevance SSH has here; in case you’re not aware, Caddy is an HTTP server, it can’t proxy SSH traffic. For that, you’d need a TCP proxy like GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy but keep in mind that SSH does not have the concept of “hostnames” or TLS-SNI, so SSH traffic cannot be routed by domain name.

1 Like
3

Yes, the 108. number is my public address.

The issue is that something — I suspect caddy’s certificates? — is preventing a secure connection to the pi. Like I said, http to the domain works. Https doesn’t. And ssh doesn’t work. And since caddy is providing the cert and handling the reverse proxy, I think there is an issue with the caddy setup. But I can’t seem to suss out what’s blocking it.

4

Show your Caddy logs. You have the debug global option already which should show it there’s any failed TLS handshakes. But to me it sounds like networking issues.

1 Like
5

how do I show my caddy logs?

6

Caddy emits its logs to stdout. I don’t use a Mac so I don’t know how that’s set up there.

1 Like
7

i can’t figure out how to access stdout, and logging just locks up caddy. I am giving up.

8

this literally used to work perfectly with no issues on my previous Pi. and I diidn’t change anything in my caddy file. So guiess I just won’t access my pi from outside my local network.

9

Add this to your Caddyfile’s global options block, it should write the logs to a file next to Caddy’s present working directory (I don’t know what that ends up being on Mac, I don’t know how you’re actually running it):

{
	log {
		output file caddy.log
	}
}
10

Here’s the log output:

{"level":"warn","ts":1716402647.1361039,"msg":"unable to determine directory for user configuration; falling back to current directory","error":"$HOME is not defined"}
{"level":"info","ts":1716402647.137442,"msg":"using provided configuration","config_file":"/opt/homebrew/etc/Caddyfile","config_adapter":""}
{"level":"info","ts":1716402647.1393962,"msg":"redirected default logger","from":"stderr","to":"/caddy.log"}
2024-05-22 14:30:47.141531 -0400 EDT m=+0.034150335 write error: can't open new logfile: open caddy.log: read-only file system
2024-05-22 14:30:47.146531 -0400 EDT m=+0.039149960 write error: can't open new logfile: open caddy.log: read-only file system
2024-05-22 14:30:47.150263 -0400 EDT m=+0.042882293 write error: can't open new logfile: open caddy.log: read-only file system
Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 1: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system
{"level":"warn","ts":1716402657.257578,"msg":"unable to determine directory for user configuration; falling back to current directory","error":"$HOME is not defined"}
{"level":"info","ts":1716402657.2596521,"msg":"using provided configuration","config_file":"/opt/homebrew/etc/Caddyfile","config_adapter":""}
{"level":"info","ts":1716402657.26206,"msg":"redirected default logger","from":"stderr","to":"/caddy.log"}
2024-05-22 14:30:57.264687 -0400 EDT m=+0.084201126 write error: can't open new logfile: open caddy.log: read-only file system
2024-05-22 14:30:57.269022 -0400 EDT m=+0.088536876 write error: can't open new logfile: open caddy.log: read-only file system
2024-05-22 14:30:57.272914 -0400 EDT m=+0.092428709 write error: can't open new logfile: open caddy.log: read-only file system
Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 1: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system
11

Try changing the log file path to somewhere else you know is writable. I don’t use a Mac so I don’t know what path is best.

But clearly Caddy’s storage directory is marked as read-only for some reason. That’s a problem with your system setup, you’ll need to figure out a solution for that.

12

how do I find out what the storage directory is?

13

Well it’s meant to be in your $HOME but clearly $HOME isn’t set with the way you’re running Caddy. See Conventions — Caddy Documentation

14

ok. how on earth i see what $HOME is for caddy? Like, where is config_file stored?

15

Just seeing this over and over in my caddy.log in /opt/homebrew/var/log

{"level":"warn","ts":1716409658.563532,"msg":"unable to determine directory for user configuration; falling back to current directory","error":"$HOME is not defined"}
{"level":"info","ts":1716409658.565645,"msg":"using provided configuration","config_file":"/opt/homebrew/etc/Caddyfile","config_adapter":""}
{"level":"info","ts":1716409658.570483,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
{"level":"info","ts":1716409658.570661,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x14000097d00"}
{"level":"info","ts":1716409658.571702,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x14000097d00"}
Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 1: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisioning CA 'local': generating root: saving root certificate: mkdir caddy: read-only file system
16

That’s a system env var, it’s not something defined in a config file. What is considered “home” depends on how the program is run. I don’t know why it’s not set in your case, I don’t know how Homebrew sets up the program to run etc.

17

Ok. Here’s my latest caddy file

# This replaces the existing content in /etc/caddy/Caddyfile

# Global options block

192.168.5.86 {
        # PROXY ALL REQUEST TO PORT 30000
        tls internal
        reverse_proxy localhost:30000
        encode zstd gzip
}

foundry.azthir-terra.com {
        # PROXY ALL REQUEST TO PORT 30000
        reverse_proxy localhost:30000
        encode zstd gzip
}

5e.azthir-terra.com {
        # PROXY ALL REQUEST TO PORT 5000
        root * /Volumes/foundrydisk/5etools-mirror-2.github.io/
        file_server browse
        encode zstd gzip
}

192.168.7.121 {
        tls internal
        reverse_proxy raspberrypi.local
        encode zstd gzip
}

pi.azthir-terra.com {
        tls internal
        reverse_proxy raspberrypi.local:1387
        encode zstd gzip
}

pi.azthir-terra.com:50000 {
        tls internal
        reverse_proxy raspberrypi.local:50000
        encode zstd gzip
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

and here’s the message I get when I curl -v https://pi.azthir-terra.com:50000

* Host pi.azthir-terra.com:50000 was resolved.
* IPv6: (none)
* IPv4: 108.51.174.138
*   Trying 108.51.174.138:50000...
* Connected to pi.azthir-terra.com (108.51.174.138) port 50000
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
18

and here’s the output from curl -v http://pi.azthir-terra.com:50000

* Host pi.azthir-terra.com:50000 was resolved.
* IPv6: (none)
* IPv4: 108.51.174.138
*   Trying 108.51.174.138:50000...
* Connected to pi.azthir-terra.com (108.51.174.138) port 50000
> GET / HTTP/1.1
> Host: pi.azthir-terra.com:50000
> User-Agent: curl/8.6.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< X-Powered-By: Express
< Location: /join
< Vary: Accept
< Content-Type: text/plain; charset=utf-8
< Content-Length: 27
< Date: Wed, 22 May 2024 20:58:26 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host pi.azthir-terra.com left intact
Found. Redirecting to /join%

As you can see, http works. https doesn’t. i’m convinced its a certificate problem somehow. Or I don’t have my caddyfile configured properly.

19

Homebrew sets the relevant environment variable, i.e. XDG_DATA_HOME, to #{HOMEBREW_PREFIX}/var/lib, per:

You can check the value of HOMEBREW_PREFIX by running brew config. Once you identify the directory, ensure that directory permissions are correct and allow the user running the service to write. I don’t know which user would that be, but I assume it’s the current user, unless you’re using sudo to run the service. You can ask the Homebrew team that on GitHub to know for sure. Their website doesn’t say anything, and I’m not familiar with their codebase.

Something similar was reported to Homebrew before, but the ever helpful stalebot closed it :man_shrugging:

Ah, lastly, the reason Caddy cannot find $HOME is because Homebrew filters the environment variable list before running the service :man_shrugging: To help troubleshoot this, they need to modify the service running command to include the flag --environ, which causes Caddy to print the environment variables as it sees them in the system and helps knowing how it perceives the world.

2 Likes
20

@ctbritt what changed, beside going from Pi 4 to Pi 5?
Can diffing the Pi 4 configs with the Pi 5 be done?

1 Like