Caddy not working through SystemCTL

Help
1

1. The problem I’m having:

Running caddy with sudo systemctl start caddy, appears to start caddy but my website is not working when I try to access it. It works when I run manually:

/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

In this thread the issue was the lack of permissions to the caddy user. I checked as well and it seems good:
ls -la /etc/caddy/Caddyfile \\ -rw-r--r-- 1 caddy caddy 742 Dec 2 19:19 /etc/caddy/Caddyfile
ls -la /var/www/html/ \\ -rwxr-xr-x 1 caddy caddy 1497 Oct 1 13:37 index.html

2. Error messages and/or full log output:

I have not seen any error messages, it loooks to be running fine.

ubuntu@ip-172-31-37-189:~$ sudo systemctl status caddy
● caddy.service - Caddy
     Loaded: loaded (/usr/lib/systemd/system/caddy.service; disabled; preset: enabled)
     Active: active (running) since Tue 2024-12-03 18:24:40 UTC; 4min 14s ago
       Docs: https://caddyserver.com/docs/
   Main PID: 12662 (caddy)
      Tasks: 6 (limit: 1130)
     Memory: 9.9M (peak: 11.8M)
        CPU: 71ms
     CGroup: /system.slice/caddy.service
             └─12662 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: HOME=/var/lib/caddy
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: INVOCATION_ID=85aaef45b6434db68198f5155dc8d6df
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: JOURNAL_STREAM=8:77128
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: SYSTEMD_EXEC_PID=12662
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: {"level":"info","ts":1733250280.0189543,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: {"level":"info","ts":1733250280.021471,"msg":"adapted config to JSON","adapter":"caddyfile"}
Dec 03 18:24:40 ip-172-31-37-189 caddy[12662]: {"level":"info","ts":1733250280.0219426,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/website-access-log.log"}
Dec 03 18:24:40 ip-172-31-37-189 systemd[1]: Started caddy.service - Caddy.

3. Caddy version: v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy: installed with linux package and have been running so far with:

sudo caddy run --config /etc/caddy/Caddyfile

But this occupies the terminal so I wanted to use systemctl

a. System environment:

Ubuntu, ec2 machine

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

        debug
        log {
                output file /var/log/caddy/website-access-log.log
        }
}

# Redirect all www traffic to non-www
www.website.com {
        redir https://website.com{uri} # Ensure the `www` subdomain redirects to the root domain
}


website.com {
        root * /var/www/html # Path to your React app's build folder


        #Route is used to ensure that first reverse proxy is executed and only resort to try files if its not an api call
        route {
                reverse_proxy /api/* localhost:8080

                # Redirect all routes not matching files to index.html for client-side routing
                try_files {path} /index.html # This allows React Router to take over for non-file URLs

                # Serve static files (React app)
                file_server
        }
}

5. Links to relevant resources:

type or paste code here
2

What does that mean? In what way is it not working? Show evidence of that. Show an example request with curl -v. Show debug logs. See Keep Caddy Running — Caddy Documentation

3

By not working I mean I get the error:

# This site can’t provide a secure connection

**website.com** sent an invalid response.

With curl


curl -v website.com
* Host website.com:80 was resolved.
* IPv6: (none)
* IPv4: 3.66.134.133
*   Trying 3.66.134.133:80...
* Connected to website.com (3.66.134.133) port 80
> GET / HTTP/1.1
> Host: website.com
> User-Agent: curl/8.9.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://website.com/
< Server: Caddy
< Date: Thu, 05 Dec 2024 19:18:50 GMT
< Content-Length: 0
<

The logs in the log file show this:

{"level":"debug","ts":1733426243.48332,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733426243.483323,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733426243.48333,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"51013","server_name":"website.com","remote":"130.226.165.154:51013","identifier":"website.com","cipher_suites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733426243.483383,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:51013: no certificate available for 'website.com'"}
{"level":"debug","ts":1733426243.5265095,"logger":"events","msg":"event","name":"tls_get_certificate","id":"89a16f86-7ea3-4ea4-997f-8a709bba3d11","origin":"tls","data":{"client_hello":{"CipherSuites":[64250,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"website.com","SupportedCurves":[51914,25497,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[6682,772,771],"RemoteAddr":{"IP":"130.226.165.154","Port":51014,"Zone":""},"LocalAddr":{"IP":"172.31.37.189","Port":443,"Zone":""}}}}
{"level":"debug","ts":1733426243.5265455,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"website.com"}
{"level":"debug","ts":1733426243.5265503,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733426243.5265534,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733426243.5265641,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"51014","server_name":"website.com","remote":"130.226.165.154:51014","identifier":"website.com","cipher_suites":[64250,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733426243.5266166,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:51014: no certificate available for 'website.com'"}
4

You only made an HTTP request to Caddy, and it responded as expected with an HTTP->HTTPS redirect. Use curl -v https://website.com instead.

Are sure sure that’s all your logs? Please post everything, there must be errors earlier relating to cert issuance.

5 
* Host website.com:443 was resolved.
* IPv6: (none)
* IPv4: 3.66.134.133
*   Trying 3.66.134.133:443...
* Connected to website.com (3.66.134.133) port 443
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.
* closing connection #0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

Here is the entire log.

{"level":"info","ts":1733432880.1155927,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1733432880.115596,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.website.com","website.com"]}
{"level":"error","ts":1733432880.1157107,"msg":"unable to create folder for config autosave","dir":"/var/lib/caddy/.config/caddy","error":"mkdir /var/lib/caddy: permission denied"}
{"level":"info","ts":1733432880.1162271,"msg":"serving initial configuration"}
{"level":"warn","ts":1733432880.116334,"logger":"tls","msg":"unable to get instance ID; storage clean stamps will be incomplete","error":"mkdir /var/lib/caddy: permission denied"}
{"level":"error","ts":1733432880.1163802,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: creating lock file: open /var/lib/caddy/.local/share/caddy/locks/storage_clean.lock: no such file or directory"}
{"level":"info","ts":1733432880.1163857,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1733432880.117486,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0008f0380"}
{"level":"error","ts":1733432880.1176958,"logger":"tls","msg":"job failed","error":"www.website.com: obtaining certificate: failed storage check: mkdir /var/lib/caddy: permission denied - storage is probably misconfigured"}
{"level":"error","ts":1733432880.117776,"logger":"tls","msg":"job failed","error":"website.com: obtaining certificate: failed storage check: mkdir /var/lib/caddy: permission denied - storage is probably misconfigured"}
{"level":"debug","ts":1733432897.852266,"logger":"events","msg":"event","name":"tls_get_certificate","id":"fa619a89-cb1a-49de-ba8b-913a64304930","origin":"tls","data":{"client_hello":{"CipherSuites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"website.com","SupportedCurves":[14906,25497,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[56026,772,771],"RemoteAddr":{"IP":"130.226.165.154","Port":55590,"Zone":""},"LocalAddr":{"IP":"172.31.37.189","Port":443,"Zone":""}}}}
{"level":"debug","ts":1733432897.852313,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"website.com"}
{"level":"debug","ts":1733432897.8523188,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733432897.852322,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733432897.8523285,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"55590","server_name":"website.com","remote":"130.226.165.154:55590","identifier":"website.com","cipher_suites":[51914,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733432897.8523846,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:55590: no certificate available for 'website.com'"}
{"level":"debug","ts":1733432897.8526075,"logger":"events","msg":"event","name":"tls_get_certificate","id":"4505e6c0-deba-4d93-ad43-b9053473ed60","origin":"tls","data":{"client_hello":{"CipherSuites":[56026,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"website.com","SupportedCurves":[60138,25497,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[6682,772,771],"RemoteAddr":{"IP":"130.226.165.154","Port":55591,"Zone":""},"LocalAddr":{"IP":"172.31.37.189","Port":443,"Zone":""}}}}
{"level":"debug","ts":1733432897.8526366,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"website.com"}
{"level":"debug","ts":1733432897.8526423,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733432897.8526454,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733432897.8526497,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"55591","server_name":"website.com","remote":"130.226.165.154:55591","identifier":"website.com","cipher_suites":[56026,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733432897.852673,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:55591: no certificate available for 'website.com'"}
{"level":"debug","ts":1733432897.8953831,"logger":"events","msg":"event","name":"tls_get_certificate","id":"332753f7-cb60-40c9-a00d-0689dee3cf91","origin":"tls","data":{"client_hello":{"CipherSuites":[64250,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"website.com","SupportedCurves":[43690,25497,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[10794,772,771],"RemoteAddr":{"IP":"130.226.165.154","Port":55593,"Zone":""},"LocalAddr":{"IP":"172.31.37.189","Port":443,"Zone":""}}}}
{"level":"debug","ts":1733432897.8954122,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"website.com"}
{"level":"debug","ts":1733432897.8954175,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733432897.895421,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733432897.895434,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"55593","server_name":"website.com","remote":"130.226.165.154:55593","identifier":"website.com","cipher_suites":[64250,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733432897.8954718,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:55593: no certificate available for 'website.com'"}
{"level":"debug","ts":1733432897.8959491,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ebc421aa-0fdc-4ef1-ab5c-664656407f70","origin":"tls","data":{"client_hello":{"CipherSuites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"website.com","SupportedCurves":[51914,25497,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[27242,772,771],"RemoteAddr":{"IP":"130.226.165.154","Port":55592,"Zone":""},"LocalAddr":{"IP":"172.31.37.189","Port":443,"Zone":""}}}}
{"level":"debug","ts":1733432897.8960395,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"website.com"}
{"level":"debug","ts":1733432897.8960888,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1733432897.896112,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1733432897.8961537,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"130.226.165.154","remote_port":"55592","server_name":"website.com","remote":"130.226.165.154:55592","identifier":"website.com","cipher_suites":[19018,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1733432897.8963072,"logger":"http.stdlib","msg":"http: TLS handshake error from 130.226.165.154:55592: no certificate available for 'website.com'"}

This is what I get once I initialize and make one attempt to connect to the website

6

Check the permissions for /var/lib/caddy. It should have user:group caddy:caddy

7

It seems that the directory /var/lib/caddy does not even exist. How should I proceed?

8

Sorry for the delay - you can just create it, make sure it’s owned by caddy:caddy as I said (use chown to change its permissions) then restart Caddy.

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.