Error getting cert validation

1. The problem I’m having:

I’m trying to enable HTTPS on my server. It seems that Let’s Encrypt cannot reach caddy to get the challenges. My router forwards all traffic to my server (DMZ is on). The Jellyfin server (on the same machine as caddy) is reachable from the internet when accessed through the domain name and the server’s 8096 port (caddy doesn’t handle that port). But it seems that caddy is not reachable on ports 80 / 443.

Ports 80 et 443 are open but on tcp6. Apparently tcp6 includes IP V4 addresses.

My config file worked on Windows.

Netstat output:

sudo netstat -nlp
Connexions Internet actives (seulement serveurs)
Proto Recv-Q Send-Q Adresse locale          Adresse distante        Etat        PID/Program name    
tcp        0      0 127.0.0.1:2019          0.0.0.0:*               LISTEN      10623/caddy         
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      2499/jellyfin       
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1336/cupsd          
tcp        0      0 127.0.0.1:8384          0.0.0.0:*               LISTEN      2097/syncthing      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      869/systemd-resolve 
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      869/systemd-resolve 
tcp        0      0 0.0.0.0:27500           0.0.0.0:*               LISTEN      2587/passimd        
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN      869/systemd-resolve 
tcp6       0      0 :::22000                :::*                    LISTEN      2097/syncthing      
tcp6       0      0 :::5355                 :::*                    LISTEN      869/systemd-resolve 
tcp6       0      0 ::1:631                 :::*                    LISTEN      1336/cupsd          
tcp6       0      0 :::443                  :::*                    LISTEN      10623/caddy         
tcp6       0      0 :::80                   :::*                    LISTEN      10623/caddy         
udp        0      0 0.0.0.0:7359            0.0.0.0:*                           2499/jellyfin       
udp        0      0 127.0.0.54:53           0.0.0.0:*                           869/systemd-resolve 
udp        0      0 127.0.0.53:53           0.0.0.0:*                           869/systemd-resolve 
udp        0      0 127.0.0.1:323           0.0.0.0:*                           979/chronyd         
udp        0      0 0.0.0.0:58574           0.0.0.0:*                           7053/python3        
udp        0      0 0.0.0.0:59890           0.0.0.0:*                           2097/syncthing      
udp        0      0 192.168.1.20:3702       0.0.0.0:*                           7053/python3        
udp        0      0 239.255.255.250:3702    0.0.0.0:*                           7053/python3        
udp        0      0 0.0.0.0:21027           0.0.0.0:*                           2097/syncthing      
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           902/avahi-daemon: r 
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           869/systemd-resolve 
udp        0      0 0.0.0.0:54984           0.0.0.0:*                           902/avahi-daemon: r 
udp6       0      0 :::39313                :::*                                2097/syncthing      
udp6       0      0 ::1:323                 :::*                                979/chronyd         
udp6       0      0 :::57711                :::*                                7053/python3        
udp6       0      0 :::443                  :::*                                10623/caddy         
udp6       0      0 :::42020                :::*                                902/avahi-daemon: r 
udp6       0      0 fe80::aee2:b837:75:3702 :::*                                7053/python3        
udp6       0      0 ff02::c:3702            :::*                                7053/python3        
udp6       0      0 :::21027                :::*                                2097/syncthing      
udp6       0      0 :::5353                 :::*                                902/avahi-daemon: r 
udp6       0      0 :::5355                 :::*                                869/systemd-resolve 
udp6       0      0 :::22000                :::*                                2097/syncthing      
raw6       0      0 :::58                   :::*                    7           1307/NetworkManager 

2. Error messages and/or full log output:

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Caddy is installed from the package manager. It runs as a systemd service.

a. System environment:

Fedora 41

b. Command:

caddy run --environ --config /etc/caddy/caddy.json

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/caddy.json
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla1.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla1",
					"http.log.error.blabla1"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla2.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla2",
					"http.log.error.blabla2"
				]
			},
			"blabla": {
				"writer": {
					"filename": "/var/log/caddy/blabla3.log",
					"output": "file",
					"roll_local_time": true
				},
				"include": [
					"http.log.access.blabla3",
					"http.log.error.blabla3"
				]
			}
		}
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8096"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"upstreams": [
														{
															"dial": "localhost:8080"
														}
													]
												}
											]
										}
									]
								}
							],
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"blabla.domain1.duckdns.org"
									]
								},
								{
									"host": [
										"blabla.domain1.net"
									]
								},
								{
									"host": [
										"fichiers.domain2.fr"
									]
								}
							],
							"handle": [
								{
									"handler": "file_server",
									"browse": {},
									"root": "/home/user/Public/caddy"
								}
							],
							"terminal": true
						}
					],
					"logs": {
						"logger_names": {
							"blabla.domain1.net": "blabla1",
							"blabla.domain1.duckdns.org": "blabla1",
							"blabla.domain1.net": "blabla2",
							"blabla.domain1.duckdns.org": "blabla2",
							"blabla.domain1.net": "blabla3",
							"blabla.domain1.duckdns.org": "blabla3",
							"fichiers.domain2.fr": "blabla3"
						},
						"should_log_credentials": true
					}
				}
			}
		},
		"tls": {
			"certificates": {
				"automate": [
					"blabla1.domain1.duckdns.org",
					"blabla2.domain1.duckdns.org",
					"blabla3.domain1.duckdns.org",
					"blabla1.domain1.net",
					"blabla2.domain1.net",
					"blabla3.domain1.net",
					"fichiers.domain2.fr"
				]
			}
		}
	}
}

5. Links to relevant resources:

I forgot to add DNS entries for my subdomains, this is probably the cause. I will report back after doing so.

I’ve updated the DNS records. I’ve run sudo caddy trust. It doesn’t seem to change anything, I’m still getting the error about cert validation.

New log: user@hostname:~$ journalctl --follow -u caddy.serviceavril 20 22:54:18 hostnam - Pastebin.com

After Googling the error "http: TLS handshake error from 127.0.0.1:43062: EOF" one suggestion was to run the command openssl s_client -msg; here is its output:

80E242A0127F0000:error:8000006F:system library:BIO_connect:Connection refused:crypto/bio/bio_sock2.c:178:calling connect()
80E242A0127F0000:error:10000067:BIO routines:BIO_connect:connect error:crypto/bio/bio_sock2.c:180:
connect:errno=111

The command tcpdump port 443 and '(tcp-syn|tcp-ack)!=0 shows incoming traffic when port 443 is accessed from the outside, so my router seems to forward traffic correctly.

12:43:25.603132 IP ip16.ip-137-74-154.eu.39266 > hostname.https: Flags [S], seq 714490359, win 65535, options [mss 1358,nop,wscale 8,nop,nop,sackOK], length 0
12:43:25.843034 IP ip16.ip-137-74-154.eu.16676 > hostname.https: Flags [S], seq 3568474721, win 65535, options [mss 1358,nop,wscale 8,nop,nop,sackOK], length 0
12:43:29.561264 IP ip16.ip-137-74-154.eu.39266 > hostname.https: Flags [S], seq 714490359, win 65535, options [mss 1358,nop,wscale 8,nop,nop,sackOK], length 0
12:43:29.842301 IP ip16.ip-137-74-154.eu.16676 > hostname.https: Flags [S], seq 3568474721, win 65535, options [mss 1358,nop,wscale 8,nop,nop,sackOK], length 0

I would appreciate any help.

Ports 80 and 443 were not open despite netstat’s output.

I opened them in firewalld by adding the http and https services :

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

Caddy complained about lack of authorizations on the root dir of the file server, despite being granted to the caddy user. I followed this answer to configure SELinux contexts on the root dir:

sudo chcon -t httpd_sys_content_t /var/www/example.com -R
sudo chcon -t httpd_sys_rw_content_t /var/www/example.com -R

Everything works now.

2 Likes