1. The problem I’m having:
We’re having users and internal team members report random occurrences of a SSL_ERROR_INTERNAL_ERROR_ALERT when trying to connect to our Caddy fleet. Having the user refresh the page seems to fix the issue for them and the page loads correctly.
Example URLs:
2. Error messages and/or full log output:
I’m not sure which Caddy logs would be helpful. We have 42,000+ items in our Dynamo certificate storage table. I also haven’t been able to replicate the issue myself which makes it even more difficult to track down.
Other helpful info is that app.aryeo.com is proxied through Cloudflare, whereas our wildcard subdomains go through Caddy first and then are reverse proxied to app.aryeo.com (so they get the benefit of Cloudflare, but they hit Caddy first).
3. Caddy version:
2.5.2
4. How I installed and ran Caddy:
a. System environment:
Ubuntu 20.04 on EC2, not using Docker
b. Command:
sudo systemctl daemon-reload
sudo systemctl enable caddy
sudo service caddy reload
d. My complete Caddy config:
{
servers {
timeouts {
read_body 25s
read_header 5s
write 30s
idle 60s
}
}
# Ensure we validate that the custom domain exists in Aryeo before
# trying to obtain a certificate for the domain
on_demand_tls {
ask https://app.aryeo.com/ask
}
# Store certificates in DynamoDB to share amongst nodes in the cluster
storage dynamodb caddy_ssl_certificates {
aws_region us-east-1
}
storage_clean_interval 32d
}
:8001 {
respond /health "I'm healthy!" 200
}
:8002 {
metrics
}
(modify-headers) {
# Drop the Caddy identifier header
header -Server
# Add a header to identify the region that served the request
header AryeoRegion "us-east-1"
header AryeoNode ""
}
(reverse-proxy) {
import modify-headers
reverse_proxy https://app.aryeo.com {
header_up Host app.aryeo.com
header_up User-Custom-Domain {host}
header_up X-Forwarded-Host {host}
header_up X-Forwarded-Port 443
health_timeout 5s
lb_try_duration 5s
lb_try_interval 250ms
transport http {
dial_timeout 5s
}
}
}
(cloudflare-tls) {
# When obtaining certificates for any *.aryeo.com domain
# use the installed Cloudflare module to allow Caddy to
# create any necessary TXT records for domain validation
tls {
dns cloudflare REDACTED
resolvers 1.1.1.1
}
}
(access-logs) {
log {
output net udp/localhost:10519
format filter {
wrap json
fields {
request>headers>Accept delete
request>headers>Accept-Encoding delete
request>headers>Accept-Language delete
request>headers>Sec-Fetch-Dest delete
request>headers>Sec-Fetch-Mode delete
}
}
}
}
(php-app) {
import access-logs
import modify-headers
root * /home/forge/aryeo.com/current/public
file_server
php_fastcgi unix//run/php/php8.0-fpm.sock {
root /home/forge/aryeo.com/current/public
header_down AryeoStatic false
# how long to try selecting available backends for each request
lb_try_duration 10s
lb_try_interval 500ms
# how long to wait when connecting to the upstream socket
dial_timeout 3s
# how long to wait when reading from the FastCGI server
read_timeout 30s
# how long to wait when sending to the FastCGI server
write_timeout 30s
# Expose these env vars to PHP for the Datadog trace extension to use
env DD_SERVICE laravel
env DD_ENV production
env DD_TRACE_LARAVEL_ENABLED true
# Allow the following IPs to proxy to us
# TODO: Update to Cloudflare's IP range in the future
trusted_proxies 0.0.0.0/0
}
@blocked {
path */wp-* *wlwmanifest.xml *xmlrpc.php *.php* *.ini* *.html* *.jsp* *.srf* */etc/passwd* */administrator/* *.pem* *.crt* *.key* *.p12* *.csr*
}
respond @blocked 403
}
(aryeo-app) {
import php-app
import cloudflare-tls
}
www.aryeo.com {
import aryeo-app
}
app.aryeo.com {
import aryeo-app
}
api.aryeo.com {
import aryeo-app
}
webhook.aryeo.com {
import aryeo-app
}
*.aryeo.com {
import cloudflare-tls
import reverse-proxy
import access-logs
}
https:// {
tls support@aryeo.com {
on_demand
}
import reverse-proxy
import access-logs
}
5. Links to relevant resources:
N/A