1. Caddy version (caddy version):
2.3.0. It was also happening with 2.2.1.
2. How I run Caddy:
One caddy server behind a AWS network load balancer with two zones and cross region enabled.
a. System environment:
Amazon linux 2
b. Command:
caddy run --config /home/ec2-user/caddy.json
c. Service/unit/compose file:
n/a
d. My complete Caddyfile or JSON config:
{
"admin": {
"disabled": true
},
"logging": {
"sink": {
"writer": {
"output": "stdout"
}
},
"logs": {
"log2stdout": {
"writer": {
"output": "stdout"
},
"level": "DEBUG"
}
}
},
"apps": {
"http": {
"servers": {
"websites": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"*.stampr.io"
]
}
],
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:8080"
}
],
"flush_interval": -1,
"buffer_requests": false
}
],
"terminal": true
},
{
"match": [
{
"not": [
{
"header_regexp": {
"host": {
"name": "disallow_branded_host",
"pattern": ".+\\.stampr\\.io$"
}
}
}
]
}
],
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "localhost:8080"
}
],
"flush_interval": -1,
"buffer_requests": false
}
],
"terminal": true
}
],
"strict_sni_host": true
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": [
"*.stampr.io"
],
"issuer": {
"module": "acme",
"ca": "https://acme-v02.api.letsencrypt.org/directory",
"email": "cory@stam.pr",
"challenges": {
"dns": {
"provider": {
"name": "route53"
}
}
},
"trusted_roots_pem_files": []
},
"storage": {
"module": "dynamodb",
"table": "websites-edge-certificate-storage",
"aws_region": "us-east-1"
},
"on_demand": false
},
{
"issuer": {
"module": "acme",
"ca": "https://acme-v02.api.letsencrypt.org/directory",
"email": "cory@stam.pr",
"challenges": {
"http": {
"disabled": false
},
"tls-alpn": {
"disabled": false
}
},
"trusted_roots_pem_files": []
},
"storage": {
"module": "dynamodb",
"table": "websites-edge-certificate-storage",
"aws_region": "us-east-1"
},
"on_demand": true
}
],
"on_demand": {
"ask": "http://localhost:8080/.stampr/.system/verify-tenant-ssl-provisioning-status"
}
}
}
}
}
3. The problem I’m having:
Caddy is responding with 403 when requesting a site and logs nothing with debug enabled. It seems somehow related to the load balancer IP I hit but both IPs hit the same server and caddy is responding to both requests. Just differently. (See videos at end for details)
4. Error messages and/or full log output:
Nothing is logged.
5. What I already tried:
I tried adding another server to avoid cross region routing from the load balancer but it didn’t work.
6. Links to relevant resources:
Here’s a couple videos of the issue:
- Shows the problem. 403s in one window/IP, go to another and the page is displayed when it gets the other IP from the load balancer. Caddy is servicing both requests though as evidenced by the headers returned along with updated date. https://drive.google.com/file/d/1Eo_rZ5i6umQf8NYBKcyFY9ZEOkwU3kT2/view?usp=sharing
- Shows the problem again this time includes the log. 403s lead to no log entries. Visiting a page and they show up. https://drive.google.com/file/d/1K99i-jsiwRRCQv0Aauv7TxfaKugztwsX/view?usp=sharing
I’m not sure why caddy would be treating the requests differently. They’re both hitting the same server.
Whenever I say this during troubleshooting I end up being wrong: either the statement itself is wrong, or I’m looking for the difference/bug in the wrong place and I need to step back and reassess.
– but I would start by using new incognito sessions at every trial, and inspect every request carefully. I wish Chrome showed more info about making the TLS handshake. Fortunately, you can get Caddy to emit almost everything the client sends in the TLS handshake though using placeholders.
Would be happy to have your sponsorship, but yeah… giving out a free service makes that difficult!