1. Caddy version (caddy version):
v2.5.1
2. How I run Caddy:
on local, I use docker. (it works)
my issue is when using AWS ECS (it forces https and throws an ERR_SSL_PROTOCOL_ERROR).
a. System environment:
Docker with linux alpine and my machine is Windows 11 on local
linux containers on AWS ECS
b. Command:
Caddy runs automatically, this is because I am using a boilerplate from github. I trust that this is not my issue.
c. Service/unit/compose file:
This is my docker-compose file
version: "3.4"
services:
php:
build:
context: .
target: symfony_php
args:
SYMFONY_VERSION: ${SYMFONY_VERSION:-}
SKELETON: ${SKELETON:-symfony/skeleton}
STABILITY: ${STABILITY:-stable}
restart: unless-stopped
volumes:
- php_socket:/var/run/php
healthcheck:
interval: 10s
timeout: 3s
retries: 3
start_period: 30s
environment:
# Run "composer require symfony/orm-pack" to install and configure Doctrine ORM
DATABASE_URL: postgresql://${POSTGRES_USER:-symfony}:${POSTGRES_PASSWORD:-ChangeMe}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-13}
# Run "composer require symfony/mercure-bundle" to install and configure the Mercure integration
MERCURE_URL: ${CADDY_MERCURE_URL:-http://caddy/.well-known/mercure}
MERCURE_PUBLIC_URL: https://${SERVER_NAME:-localhost}/.well-known/mercure
MERCURE_JWT_SECRET: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
caddy:
build:
context: .
target: symfony_caddy
depends_on:
- php
environment:
SERVER_NAME: ${SERVER_NAME:-localhost, caddy:80}
MERCURE_PUBLISHER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
MERCURE_SUBSCRIBER_JWT_KEY: ${CADDY_MERCURE_JWT_SECRET:-!ChangeMe!}
restart: unless-stopped
volumes:
- php_socket:/var/run/php
- caddy_data:/data
- caddy_config:/config
ports:
# HTTP
- target: 80
published: ${HTTP_PORT:-80}
protocol: tcp
# HTTPS
- target: 443
published: ${HTTPS_PORT:-443}
protocol: tcp
# HTTP/3
- target: 443
published: ${HTTP3_PORT:-443}
protocol: udp
# Mercure is installed as a Caddy module, prevent the Flex recipe from installing another service
###> symfony/mercure-bundle ###
###< symfony/mercure-bundle ###
###> doctrine/doctrine-bundle ###
database:
image: postgres:${POSTGRES_VERSION:-13}-alpine
ports:
- "5432:5432"
environment:
POSTGRES_DB: ${POSTGRES_DB:-app}
# You should definitely change the password in production
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ChangeMe}
POSTGRES_USER: ${POSTGRES_USER:-symfony}
volumes:
- db-data:/var/lib/postgresql/data:rw
# You may use a bind-mounted host directory instead, so that it is harder to accidentally remove the volume and lose all your data!
# - ./docker/db/data:/var/lib/postgresql/data:rw
###< doctrine/doctrine-bundle ###
volumes:
php_socket:
caddy_data:
caddy_config:
###> symfony/mercure-bundle ###
###< symfony/mercure-bundle ###
###> doctrine/doctrine-bundle ###
db-data:
###< doctrine/doctrine-bundle ###
d. My complete Caddyfile or JSON config:
{
# Debug
{$DEBUG}
# HTTP/3 support
servers {
protocol {
experimental_http3
}
}
}
{$SERVER_NAME}
log
route {
root * /srv/app/public
mercure {
# Transport to use (default to Bolt)
transport_url {$MERCURE_TRANSPORT_URL:bolt:///data/mercure.db}
# Publisher JWT key
publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
# Subscriber JWT key
subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
# Allow anonymous subscribers (double-check that it's what you want)
anonymous
# Enable the subscription API (double-check that it's what you want)
subscriptions
# Extra directives
{$MERCURE_EXTRA_DIRECTIVES}
}
vulcain
push
php_fastcgi unix//var/run/php/php-fpm.sock
encode zstd gzip
file_server
}
3. The problem I’m having:
My setup works on my local environment, But when I get the containers up on ECS, and try to access the site from the public IP, I get a
ERR_SSL_PROTOCOL_ERROR
and it is automatically converting http to https (even though on local it is not doing so).
the logs of the caddy container are showing that my request is reaching, and getting hit with a 308 permanently moved. I suspect that my problem is that I cant keep “localhost” as my server name in the CADDYFILE, since im trying to reach the server using an IP and not the localhost. Is this the issue? and if so, is there no other way than to create a domain name?
Also, if that is the case, how should I modify my CADDYFILE, so that I can reach the server without https.
4. Error messages and/or full log output:
Below are the logs for the caddy container. What I found weird is that at startup it states that http->https forcing is enabled, and then right atter it says that it is disabled (even though it still forces it when I try to reach the server via its IP).
here is the log at startup:
{
"level": "info",
"ts": 1653033952.7034664,
"msg": "using provided configuration",
"config_file": "/etc/caddy/Caddyfile",
"config_adapter": "caddyfile"
}
{
"level": "warn",
"ts": 1653033952.7076328,
"msg": "Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies",
"adapter": "caddyfile",
"file": "/etc/caddy/Caddyfile",
"line": 2
}
{
"level": "info",
"ts": 1653033952.7090356,
"logger": "admin",
"msg": "admin endpoint started",
"address": "tcp/localhost:2019",
"enforce_origin": false,
"origins": [
"//localhost:2019",
"//[::1]:2019",
"//127.0.0.1:2019"
]
}
{
"level": "info",
"ts": 1653033952.7092228,
"logger": "http",
"msg": "server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS",
"server_name": "srv0",
"https_port": 443
}
{
"level": "info",
"ts": 1653033952.7092412,
"logger": "http",
"msg": "enabling automatic HTTP->HTTPS redirects",
"server_name": "srv0"
}
{
"level": "warn",
"ts": 1653033952.7092586,
"logger": "http",
"msg": "server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server",
"server_name": "srv1",
"http_port": 80
}
{
"level": "info",
"ts": 1653033952.7097282,
"logger": "tls.cache.maintenance",
"msg": "started background certificate maintenance",
"cache": "0xc0004f4b60"
}
{
"level": "info",
"ts": 1653033952.7847414,
"logger": "tls",
"msg": "cleaning storage unit",
"description": "FileStorage:/data/caddy"
}
{
"level": "info",
"ts": 1653033952.7848887,
"logger": "tls",
"msg": "finished cleaning storage units"
}
{
"level": "warn",
"ts": 1653033952.8502672,
"logger": "pki.ca.local",
"msg": "installing root certificate (you might be prompted for password)",
"path": "storage:pki/authorities/local/root.crt"
}
{
"level": "info",
"ts": 1653033952.8508463,
"msg": "define JAVA_HOME environment variable to use the Java trust"
}
{
"level": "info",
"ts": 1653033952.8509572,
"msg": "Warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"
}
{
"level": "info",
"ts": 1653033952.9620202,
"msg": "certificate installed properly in linux trusts"
}
{
"level": "info",
"ts": 1653033952.9622743,
"logger": "http",
"msg": "enabling experimental HTTP/3 listener",
"addr": ":443"
}
{
"level": "info",
"ts": 1653033952.9623816,
"msg": "failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."
}
{
"level": "info",
"ts": 1653033952.9625163,
"logger": "http",
"msg": "enabling automatic TLS certificate management",
"domains": [
"localhost"
]
}
{
"level": "info",
"ts": 1653033952.9627492,
"msg": "autosaved config (load with --resume flag)",
"file": "/config/caddy/autosave.json"
}
{
"level": "info",
"ts": 1653033952.9627666,
"msg": "serving initial configuration"
}
{
"level": "info",
"ts": 1653033952.9631817,
"logger": "tls.obtain",
"msg": "acquiring lock",
"identifier": "localhost"
}
{
"level": "info",
"ts": 1653033953.0010908,
"logger": "tls.obtain",
"msg": "lock acquired",
"identifier": "localhost"
}
{
"level": "info",
"ts": 1653033953.0030353,
"logger": "tls.obtain",
"msg": "certificate obtained successfully",
"identifier": "localhost"
}
{
"level": "info",
"ts": 1653033953.0030713,
"logger": "tls.obtain",
"msg": "releasing lock",
"identifier": "localhost"
}
{
"level": "warn",
"ts": 1653033953.0035007,
"logger": "tls",
"msg": "stapling OCSP",
"error": "no OCSP stapling for [localhost]: no OCSP server specified in certificate",
"identifiers": [
"localhost"
]
}
And here is the log of me trying to access the server
{
"level": "info",
"ts": 1653034005.6129005,
"logger": "http.log.access",
"msg": "handled request",
"request": {
"remote_ip": "178.135.2.191",
"remote_port": "4573",
"proto": "HTTP/1.1",
"method": "GET",
"host": "34.249.213.26",
"uri": "/",
"headers": {
"Connection": [
"keep-alive"
],
"Upgrade-Insecure-Requests": [
"1"
],
"User-Agent": [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36"
],
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"Accept-Encoding": [
"gzip, deflate"
],
"Accept-Language": [
"en-US,en;q=0.9"
]
}
},
"user_id": "",
"duration": 0.000068327,
"size": 0,
"status": 308,
"resp_headers": {
"Server": [
"Caddy"
],
"Connection": [
"close"
],
"Location": [
"https://34.249.213.26/"
],
"Content-Type": []
}
}
5. What I already tried:
I tried modifying the servername variable in the docker-compose file to disable http->https
tried
localhost:80
:80
but it is not working on ECS.
I still have not tried creating a domain and changing the servername to http://domainname.com, as I do not have immediate access to my company’s DNS, and did not want to request access before being sure that this is the issue
6. Links to relevant resources:
In case it is relevant, This also is the Task definition on ECS:
"taskDefinitionArn": "arn:aws:ecs:eu-west-1:MYACCOUNTID:task-definition/prototype-TF:4",
"containerDefinitions": [
{
"name": "database",
"image": "MYACCOUNTID.dkr.ecr.eu-west-1.amazonaws.com/prototype/postgres",
"cpu": 0,
"links": [],
"portMappings": [
{
"containerPort": 5432,
"hostPort": 5432,
"protocol": "tcp"
}
],
"essential": true,
"entryPoint": [],
"command": [],
"environment": [
{
"name": "POSTGRES_USER",
"value": "symfony"
},
{
"name": "POSTGRES_PASSWORD",
"value": "ChangeMe"
},
{
"name": "POSTGRES_DB",
"value": "app"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"secrets": [],
"dnsServers": [],
"dnsSearchDomains": [],
"extraHosts": [],
"dockerSecurityOptions": [],
"dockerLabels": {},
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/prototype-TF",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
},
"systemControls": []
},
{
"name": "php",
"image": "MYACCOUNTID.dkr.ecr.eu-west-1.amazonaws.com/prototype/perlo_php",
"cpu": 0,
"links": [],
"portMappings": [],
"essential": true,
"entryPoint": [],
"command": [],
"environment": [
{
"name": "MERCURE_URL",
"value": "http://caddy/.well-known/mercure"
},
{
"name": "MERCURE_JWT_SECRET",
"value": "!ChangeMe!"
},
{
"name": "DATABASE_URL",
"value": "postgresql://symfony:ChangeMe@localhost:5432/app?serverVersion=13"
},
{
"name": "MERCURE_PUBLIC_URL",
"value": "http://localhost/.well-known/mercure"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"secrets": [],
"dnsServers": [],
"dnsSearchDomains": [],
"extraHosts": [],
"dockerSecurityOptions": [],
"dockerLabels": {},
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/prototype-TF",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
},
"systemControls": []
},
{
"name": "caddy",
"image": "MYACCOUNTID.dkr.ecr.eu-west-1.amazonaws.com/prototype/perlo_caddy",
"cpu": 0,
"links": [],
"portMappings": [
{
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp"
},
{
"containerPort": 443,
"hostPort": 443,
"protocol": "tcp"
},
{
"containerPort": 443,
"hostPort": 443,
"protocol": "udp"
}
],
"essential": true,
"entryPoint": [],
"command": [],
"environment": [
{
"name": "MERCURE_SUBSCRIBER_JWT_KEY",
"value": "!ChangeMe!"
},
{
"name": "SERVER_NAME",
"value": "localhost, localhost:80"
},
{
"name": "MERCURE_PUBLISHER_JWT_KEY",
"value": "!ChangeMe!"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"secrets": [],
"dnsServers": [],
"dnsSearchDomains": [],
"extraHosts": [],
"dockerSecurityOptions": [],
"dockerLabels": {},
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/prototype-TF",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "ecs"
},
"secretOptions": []
},
"systemControls": []
}
],
"family": "prototype-TF",
"executionRoleArn": "arn:aws:iam::MYACCOUNTID:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"revision": 4,
"volumes": [],
"status": "ACTIVE",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
},
{
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "ecs.capability.task-eni"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
}
],
"placementConstraints": [],
"compatibilities": [
"EC2",
"FARGATE"
],
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
},
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "3072",
"registeredAt": "2022-05-20T08:02:10.133Z",
"registeredBy": "arn:aws:iam::MYACCOUNTID:user/MYAWSUSERNAME",
"tags": [
{
"key": "ecs:taskDefinition:createdFrom",
"value": "ecs-console-v2"
},
{
"key": "ecs:taskDefinition:stackId",
"value": "arn:aws:cloudformation:eu-west-1:MYACCOUNTID:stack/ECS-Console-V2-TaskDefinition-135fd1f2-e700-4d73-99ee-6aea8b4b3d6c/227194d0-d813-11ec-9f8f-029688cd0bc9"
}
]
}