Firefox SSL_ERROR_INTERNAL_ERROR_ALERT on cloud.hlarc.me, but cURL Works

lord · March 31, 2025, 2:04am

1. The problem I’m having:

I’m running into an issue where Firefox throws SSL_ERROR_INTERNAL_ERROR_ALERT when accessing cloud.hlarc.me, but everything works fine with cloud[.]home[.]hlarc[.]me. Strangely, cURL always works, and sometimes a different browser does too.

2. Error messages and/or full log output:

Mar 31 01:17:43 nas caddy[422255]: {"level":"debug","ts":1743383863.4040074,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:8081","duration":1.069213687,"request":{"remote_ip":"192.168.1.26","remote_port":"55104","proto":"HTTP/1.1","method":"PROPFIND","host":"cloud.hlarc.me","uri":"/remote.php/dav/files/lord/","headers":{"X-Forwarded-Host":["cloud.hlarc.me"],"Content-Type":["text/xml; charset=utf-8"],"X-Forwarded-Proto":["https"],"X-Request-Id":["a8734191-760c-4937-b34e-f753a94b6c5a"],"X-Forwarded-For":["192.168.1.26"],"Accept-Language":["en-US,*"],"User-Agent":["Mozilla/5.0 (Linux) mirall/3.15.3daily (Nextcloud, endeavouros-6.13.8-arch1-1 ClientArchitecture: x86_64 OsArchitecture: x86_64)"],"Content-Length":["105"],"Cookie":[],"Authorization":[],"Accept":["*/*"],"Accept-Encoding":["zstd, br, gzip, deflate"],"Depth":["0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"cloud.hlarc.me"}},"headers":{"Referrer-Policy":["no-referrer"],"Vary":["Brief,Prefer"],"Content-Type":["application/xml; charset=utf-8"],"Server":["Apache/2.4.62 (Debian)"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Powered-By":["PHP/8.3.19"],"Content-Security-Policy":["default-src 'none';"],"Dav":["1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-paginate, nextcloud-checksum-update, nc-calendar-search, nc-enable-birthday-calendar"],"Date":["Mon, 31 Mar 2025 01:17:42 GMT"],"X-Robots-Tag":["noindex, nofollow"],"X-Debug-Token":["1fCgUcnIfSFBzaNs5wjO"],"X-Xss-Protection":["1; mode=block"],"X-Request-Id":["1fCgUcnIfSFBzaNs5wjO"],"Content-Encoding":["gzip"],"Content-Length":["234"]},"status":207}
Mar 31 01:17:43 nas caddy[422255]: {"level":"debug","ts":1743383863.437905,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:8081","duration":1.103621083,"request":{"remote_ip":"192.168.1.26","remote_port":"55092","proto":"HTTP/1.1","method":"GET","host":"cloud.hlarc.me","uri":"/ocs/v2.php/apps/notifications/api/v2/notifications?format=json","headers":{"Accept-Encoding":["zstd, br, gzip, deflate"],"User-Agent":["Mozilla/5.0 (Linux) mirall/3.15.3daily (Nextcloud, endeavouros-6.13.8-arch1-1 ClientArchitecture: x86_64 OsArchitecture: x86_64)"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["cloud.hlarc.me"],"Cookie":[],"X-Forwarded-For":["192.168.1.26"],"Accept":["*/*"],"Authorization":[],"X-Request-Id":["d206161f-c449-4597-9d09-c240d317cbf4"],"Accept-Language":["en-US,*"],"Ocs-Apirequest":["true"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"cloud.hlarc.me"}},"headers":{"X-Powered-By":["PHP/8.3.19"],"Content-Security-Policy":["default-src 'none';base-uri 'none';manifest-src 'self';frame-ancestors 'none'"],"X-Request-Id":["4IyFNgqe9u6rLbwoy0FA"],"Feature-Policy":["autoplay 'none';camera 'none';fullscreen 'none';geolocation 'none';microphone 'none';payment 'none'"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Xss-Protection":["1; mode=block"],"Content-Type":["application/json; charset=utf-8"],"Server":["Apache/2.4.62 (Debian)"],"Content-Length":["895"],"Referrer-Policy":["no-referrer"],"X-Nextcloud-User-Status":["offline"],"Date":["Mon, 31 Mar 2025 01:17:42 GMT"],"X-Content-Type-Options":["nosniff"],"Cache-Control":["no-cache, no-store, must-revalidate"],"Etag":["\"e6b024e8c6a26856095b0fcc8cf2b1b0\""],"Content-Encoding":["gzip"],"X-Frame-Options":["SAMEORIGIN"],"X-Robots-Tag":["noindex, nofollow"]},"status":200}

3. Caddy version:

Caddy Version 2.6.2 (The Most recent Version on the ubuntu Repos)

4. How I installed and ran Caddy:

Installed on Ubuntu (apt install caddy) 24.04.2 LTS enabled on Systemd and started

a. System environment:

Ubuntu 24.04.2 LTS x86_64 Kernel: 6.8.0-56-generic
I have a local DNS server that authoritatively resolves home[.]hlarc[.]me. and forwards all requests besides cloud[.]hlarc[.]me
Caddy serves cloud[.]hlarc[.]me and home[.]hlarc[.]me, with A & AAAA records pointing to the correct server.

A script copies SSL certificates to a directory accessible by Caddy and reloads Caddy.

Certbot manages +the certificates for home[.]hlarc[.]me, using a deploy hook to copy and set permissions for Caddy.

All other domains work fine—only cloud.hlarc.me is affected.

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

# /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

#{
#       tls /etc/letsencrypt/live/hlarc.me/fullchain.pem /etc/letsencrypt/live/hlarc.me/privkey.pem
#}
{
        debug
}
cloud.hlarc.me {
        tls /etc/caddy/certs/hlarc.me/fullchain.pem /etc/caddy/certs/hlarc.me/privkey.pem
        reverse_proxy localhost:8081
}

home.hlarc.me {
        tls /etc/caddy/certs/home.hlarc.me/fullchain.pem /etc/caddy/certs/home.hlarc.me/privkey.pem
        # Set this path to your site's directory.
        root * /var/www/html/

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

dns.home.hlarc.me {
        reverse_proxy localhost:5380
}
pihole.home.hlarc.me {
        reverse_proxy localhost:1234
}
fritz.home.hlarc.me {
        reverse_proxy 192.168.1.1:80
}
collabora.home.hlarc.me {
        reverse_proxy https://localhost:9980
}
cloud.home.hlarc.me {
        reverse_proxy localhost:8081
}
admin.home.hlarc.me {
        reverse_proxy https://localhost:9090
}
jf.home.hlarc.me {
        reverse_proxy http://localhost:8096
}

Mohammed90 · March 31, 2025, 7:39am

Except for cloud and home, the certs of all other sites are managed by Caddy.

Are you sure this is the right certification? The path says hlarc.me only. Is the path accessible by Caddy?

Don’t use their repo. They’re too many versions behind. Use ours that’s listed on the docs.

lord · April 1, 2025, 10:55pm

Thanks for the Tip with the Repo i switched over to the caddy Repo and updated to the 2.9.1.

Caddy does not manage any Certificates for me at the moment. That is the Case because i also need the certs for another service. Thats why I use Certbot. But caddy has access to the certificates because i chown them with the caddy user+group. And if i remember correctly it would throw an error while validating the config.

Weirdly the Problem went away completely by it self (before updating). But it will probably come back.

Mohammed90 · April 2, 2025, 3:28pm

You can do it the other way around You can specify the issuer to be Let’s Encrypt only, if you’d like, though not recommended. @timelordx has script to find Caddy’s certs to be used elsewhere, here:

github.com/timelordx/synology-cert-deploy

synology-cert-deploy.sh

main


      
          ## Extract privkey, cert, chain and fullchain from new_key and new_fullchain
          privkey_pem="$( openssl pkey -in "$new_key" -out - 2>/dev/null )"
          cert_pem="$( openssl x509 -in "$new_fullchain" -outform PEM -out - 2>/dev/null  )"
          fullchain_pem="$( grep -v -E '^\s*$' "$new_fullchain" 2>/dev/null )"
          chain_pem="$( diff <( echo "$cert_pem" ) <( echo "$fullchain_pem" ) 2>/dev/null | grep -E '^> ' | sed 's/^> //g' )"

lord · April 4, 2025, 1:43am

I think I worded it a little poorly. What I meant is that another server deploys the certificate to the Caddy server and reloads Caddy. But that probably won’t be the issue, as the renewal hook hasn’t been triggered yet. And if there is an issue with the deployment of the certificates, then why does it sometimes work without me doing anything?

lord · April 7, 2025, 4:36pm

I found the reason. My Deployment script used just caddy reload as the command but it did not know the correct Caddyfile to load so it just changed the files caddy was referencing what threw the error.