Unsure on Caddyfile in Portainer

1. Output of caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

Ran though Portainer with command

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

a. System environment:

Debian Bullseye host
Docker version 20.10.5
Portainer version Community Edition 2.16.2

b. Command:

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

c. Service/unit/compose file:

Do not know how to retrieve from Portainer/Docker

d. My complete Caddy config:

172.16.20.4:443 {
	tls internal

	encode gzip

	reverse_proxy /notifications/hub 172.16.20.4:3012

	reverse_proxy 172.16.20.4:80 {
		header_up X-Real-IP {remote_host}
	}
}

port.garth.one {
	reverse_proxy 172.16.9.1:8006
}

3. The problem I’m having:

When Caddy and the two services (Portainer and Vaultwarden) are all up I am unable to reach the HTTPS version of Vaultwarden (HTTP still works fine) and the port.garth.one does not send me to portainer

I get this from https://172.16.20.4

This site can’t be reached 172.16.9.1 refused to connect.
Try:

Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED

and this from https://port.garth.one

Connection timed out Error code 522

4. Error messages and/or full log output:

INF ts=1671160716.3389893 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile
WRN ts=1671160716.3417804 msg=Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies adapter=caddyfile file=/etc/caddy/Caddyfile line=6
INF ts=1671160716.3439555 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]
INF ts=1671160716.3443856 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443
INF ts=1671160716.3444228 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0
INF ts=1671160716.3443775 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc00069ca80
INF ts=1671160716.3450954 logger=tls msg=cleaning storage unit description=FileStorage:/data/caddy
INF ts=1671160716.3451052 logger=http msg=enabling HTTP/3 listener addr=:443
INF ts=1671160716.3452008 msg=failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
DBG ts=1671160716.3453057 logger=http msg=starting server loop address=[::]:443 tls=true http3=true
INF ts=1671160716.3453798 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]
DBG ts=1671160716.3454835 logger=http msg=starting server loop address=[::]:80 tls=false http3=false
INF ts=1671160716.345529 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]
INF ts=1671160716.3455563 logger=http msg=enabling automatic TLS certificate management domains=["port.garth.one","172.16.20.4"]
INF ts=1671160716.3461924 logger=tls msg=finished cleaning storage units
INF ts=1671160716.3464015 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json
INF ts=1671160716.3464792 msg=serving initial configuration
INF ts=1671160716.3465142 logger=tls.obtain msg=acquiring lock identifier=172.16.20.4
INF ts=1671160716.3467722 logger=tls.obtain msg=acquiring lock identifier=port.garth.one
INF ts=1671160716.497243 logger=tls.obtain msg=lock acquired identifier=172.16.20.4
INF ts=1671160716.4972227 logger=tls.obtain msg=lock acquired identifier=port.garth.one
INF ts=1671160716.497526 logger=tls.obtain msg=obtaining certificate identifier=172.16.20.4
INF ts=1671160716.4975977 logger=tls.obtain msg=obtaining certificate identifier=port.garth.one
DBG ts=1671160716.4976492 logger=events msg=event name=cert_obtaining id=abf9f728-db4b-47d7-88a9-fb043ea6eb4b origin=tls data={"identifier":"172.16.20.4"}
DBG ts=1671160716.4977076 logger=events msg=event name=cert_obtaining id=a339bc84-b240-4375-bd20-f85fd6ecc2ea origin=tls data={"identifier":"port.garth.one"}
DBG ts=1671160716.498341 logger=tls.obtain msg=trying issuer 1/2 issuer=acme-v02.api.letsencrypt.org-directory
DBG ts=1671160716.4983613 logger=tls.obtain msg=trying issuer 1/2 issuer=acme-v02.api.letsencrypt.org-directory
DBG ts=1671160716.4983876 logger=tls.obtain msg=trying issuer 2/2 issuer=acme.zerossl.com-v2-DV90
DBG ts=1671160716.4984128 logger=events msg=event name=cert_failed id=81f9dac1-a93e-49b5-92ba-2da64d909e29 origin=tls data={"error":{},"identifier":"172.16.20.4","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}
ERR ts=1671160716.4985318 logger=tls.obtain msg=will retry error=[172.16.20.4] Obtain: subject does not qualify for a public certificate: 172.16.20.4 attempt=1 retrying_in=60 elapsed=0.00123513 max_duration=2592000
WRN ts=1671160716.689143 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority
WRN ts=1671160717.0543773 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority
WRN ts=1671160717.4184477 logger=http.acme_client msg=HTTP request failed; retrying url=https://acme-v02.api.letsencrypt.org/directory error=performing request: Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority
ERR ts=1671160717.418548 logger=tls.obtain msg=could not get certificate from issuer identifier=port.garth.one issuer=acme-v02.api.letsencrypt.org-directory error=registering account [] with server: provisioning client: performing request: Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority
DBG ts=1671160717.418572 logger=tls.obtain msg=trying issuer 2/2 issuer=acme.zerossl.com-v2-DV90
WRN ts=1671160717.4188077 logger=http msg=missing email address for ZeroSSL; it is strongly recommended to set one for next time
ERR ts=1671160717.622286 logger=tls.obtain msg=could not get certificate from issuer identifier=port.garth.one issuer=acme.zerossl.com-v2-DV90 error=account pre-registration callback: performing EAB credentials request: Post "https://api.zerossl.com/acme/eab-credentials-email": x509: certificate signed by unknown authority
DBG ts=1671160717.6223865 logger=events msg=event name=cert_failed id=f7914102-8aae-4841-b53f-157eb2172308 origin=tls data={"error":{},"identifier":"port.garth.one","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}
ERR ts=1671160717.6224327 logger=tls.obtain msg=will retry error=[port.garth.one] Obtain: account pre-registration callback: performing EAB credentials request: Post "https://api.zerossl.com/acme/eab-credentials-email": x509: certificate signed by unknown authority attempt=1 retrying_in=60 elapsed=1.125058499 max_duration=2592000

5. What I already tried:

I have tried changing the Caddyfile to be both more and less verbose with no changes, I have recently wiped the whole system and reinstalled it with the same outcome, I’ve followed multiple articles with similar issues, but either had issues understanding the solution or after trying the solution provided no extra luck.

Any help would be much appreciated, I do not quite understand the docker networking section other than being able to successfully setup a macvlan to give each container their own IP

6. Links to relevant resources:

Past 4 link limit

Forgot to post this, but this is the curl -v

curl -v 172.16.20.4:443
*   Trying 172.16.20.4:443...
* connect to 172.16.20.4 port 443 failed: Connection refused
* Failed to connect to 172.16.20.4 port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.16.20.4 port 443: Connection refused

Took a suuuuuper quick look at this and thought to myself: “They’ve probably bind mounted /etc or something”. I see that particular mistake relatively often. It breaks outgoing HTTPS because you lose your trusted root certificate store and therefore can’t trust any certificate on the internet. You need outgoing HTTPS to place an ACME order. No trust → no HTTPS → no certificate order → no certificate → no HTTPS for your Caddy server.

Mmm, still fitting the profile.

Then, I see:

So, I can’t verify whether my hunch is or isn’t the case. We’d really need to see exactly how you’ve configured your container to verify specifically how your ca-certificates are broken, and if you don’t know how you’re running it, we’ve got no chance.

2 Likes

Thanks for giving it even a quick look, i do have /etc/ssl bound if that’s what you mean, i can give it a try later today.
As for running it, what i meant is i couldn’t find a way to export it to a run cmd or compose file. I was able to get this from the inspection, which seems to have everything, just not in an ideal format

{
    "AppArmorProfile": "docker-default",
    "Args": [
        "run",
        "--config",
        "/etc/caddy/Caddyfile",
        "--adapter",
        "caddyfile"
    ],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "caddy",
            "run",
            "--config",
            "/etc/caddy/Caddyfile",
            "--adapter",
            "caddyfile"
        ],
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "CADDY_VERSION=v2.6.2",
            "XDG_CONFIG_HOME=/config",
            "XDG_DATA_HOME=/data"
        ],
        "ExposedPorts": {
            "2019/tcp": {},
            "443/tcp": {},
            "443/udp": {},
            "80/tcp": {}
        },
        "Hostname": "Caddy",
        "Image": "caddy:latest",
        "Labels": {
            "org.opencontainers.image.description": "a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go",
            "org.opencontainers.image.documentation": "https://caddyserver.com/docs",
            "org.opencontainers.image.licenses": "Apache-2.0",
            "org.opencontainers.image.source": "https://github.com/caddyserver/caddy-docker",
            "org.opencontainers.image.title": "Caddy",
            "org.opencontainers.image.url": "https://caddyserver.com",
            "org.opencontainers.image.vendor": "Light Code Labs",
            "org.opencontainers.image.version": "v2.6.2"
        },
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {
            "/config": {},
            "/data": {},
            "/etc/caddy/Caddyfile": {},
            "/etc/ssl": {},
            "/logs": {}
        },
        "WorkingDir": "/srv"
    },
    "Created": "2022-12-15T21:25:48.841921028Z",
    "Driver": "overlay2",
    "ExecIDs": null,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/965992d1754b8df9ee78efd6094ab635f7dcf36945a9392ac466f7f1ea83a9aa-init/diff:/var/lib/docker/overlay2/bde5d2eee141525608ee004ec5a45b67a96fbcef567a198e9abd1995409e1c8b/diff:/var/lib/docker/overlay2/9ef5175e49944b82452d02b750f3cf799a52407142fd60b6b3423474daa490f3/diff:/var/lib/docker/overlay2/42c55fbf394f6e1e9d3526273612057d719abe86b207a515d243ad0668d23383/diff:/var/lib/docker/overlay2/e9bd2c0fe7f08eb15c4b2af62042287d22195f2c3a08d81105d6d4338e1d0867/diff:/var/lib/docker/overlay2/1e7efc472519b9fe9b0bac5f7c0f2d26a38a2e8c209014ec0dc73dd41b9fb1cc/diff",
            "MergedDir": "/var/lib/docker/overlay2/965992d1754b8df9ee78efd6094ab635f7dcf36945a9392ac466f7f1ea83a9aa/merged",
            "UpperDir": "/var/lib/docker/overlay2/965992d1754b8df9ee78efd6094ab635f7dcf36945a9392ac466f7f1ea83a9aa/diff",
            "WorkDir": "/var/lib/docker/overlay2/965992d1754b8df9ee78efd6094ab635f7dcf36945a9392ac466f7f1ea83a9aa/work"
        },
        "Name": "overlay2"
    },
    "HostConfig": {
        "AutoRemove": false,
        "Binds": [
            "/home/nortrix0/Caddyfile:/etc/caddy/Caddyfile",
            "/home/nortrix0/caddy-ssl:/etc/ssl",
            "/home/nortrix0/caddy-config:/config",
            "/home/nortrix0/caddy-data:/data",
            "/home/nortrix0/caddy-logs:/logs"
        ],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "CapAdd": [
            "AUDIT_WRITE",
            "CHOWN",
            "DAC_OVERRIDE",
            "FOWNER",
            "FSETID",
            "KILL",
            "MKNOD",
            "NET_BIND_SERVICE",
            "NET_RAW",
            "SETFCAP",
            "SETGID",
            "SETPCAP",
            "SETUID",
            "SYS_CHROOT"
        ],
        "CapDrop": [
            "AUDIT_CONTROL",
            "BLOCK_SUSPEND",
            "DAC_READ_SEARCH",
            "IPC_LOCK",
            "IPC_OWNER",
            "LEASE",
            "LINUX_IMMUTABLE",
            "MAC_ADMIN",
            "MAC_OVERRIDE",
            "NET_ADMIN",
            "NET_BROADCAST",
            "SYSLOG",
            "SYS_ADMIN",
            "SYS_BOOT",
            "SYS_MODULE",
            "SYS_NICE",
            "SYS_PACCT",
            "SYS_PTRACE",
            "SYS_RAWIO",
            "SYS_RESOURCE",
            "SYS_TIME",
            "SYS_TTY_CONFIG",
            "WAKE_ALARM"
        ],
        "Cgroup": "",
        "CgroupParent": "",
        "CgroupnsMode": "private",
        "ConsoleSize": [
            0,
            0
        ],
        "ContainerIDFile": "",
        "CpuCount": 0,
        "CpuPercent": 0,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpuShares": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "DeviceCgroupRules": null,
        "DeviceRequests": [],
        "Devices": [],
        "Dns": [],
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": [],
        "GroupAdd": null,
        "IOMaximumBandwidth": 0,
        "IOMaximumIOps": 0,
        "Init": false,
        "IpcMode": "private",
        "Isolation": "",
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "Links": null,
        "LogConfig": {
            "Config": {},
            "Type": "json-file"
        },
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "NanoCpus": 0,
        "NetworkMode": "172.16.20",
        "OomKillDisable": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "PidsLimit": null,
        "PortBindings": {
            "443/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "443"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "80"
                }
            ]
        },
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ],
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "always"
        },
        "Runtime": "runc",
        "SecurityOpt": null,
        "ShmSize": 67108864,
        "UTSMode": "",
        "Ulimits": null,
        "UsernsMode": "",
        "VolumeDriver": "",
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771/hostname",
    "HostsPath": "/var/lib/docker/containers/c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771/hosts",
    "Id": "c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771",
    "Image": "sha256:006d393a4e6a01f82413e41b3e0f06dfb1872d5ca6a37aba315e4ec9e2cc6c4c",
    "LogPath": "/var/lib/docker/containers/c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771/c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771-json.log",
    "MountLabel": "",
    "Mounts": [
        {
            "Destination": "/data",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/home/nortrix0/caddy-data",
            "Type": "bind"
        },
        {
            "Destination": "/logs",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/home/nortrix0/caddy-logs",
            "Type": "bind"
        },
        {
            "Destination": "/etc/caddy/Caddyfile",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/home/nortrix0/Caddyfile",
            "Type": "bind"
        },
        {
            "Destination": "/etc/ssl",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/home/nortrix0/caddy-ssl",
            "Type": "bind"
        },
        {
            "Destination": "/config",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/home/nortrix0/caddy-config",
            "Type": "bind"
        }
    ],
    "Name": "/Caddy",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "172.16.20": {
                "Aliases": [
                    "Caddy",
                    "c2e7f279792d"
                ],
                "DriverOpts": null,
                "EndpointID": "2c905ce62e0043065f99fe7844da7b62fed4c79148b1f95c21d5f4d1a9f21148",
                "Gateway": "172.16.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": {
                    "IPv4Address": "172.16.20.3"
                },
                "IPAddress": "172.16.20.3",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:ac:10:14:03",
                "NetworkID": "b94cb7e7344cf90a2085a919070df9047e9265da60800c89201eb79f136689dc"
            }
        },
        "Ports": {},
        "SandboxID": "58c5cf7cfe4d603ec01049e5330c6ec5a8bc18553405faf24bc22b0e9112ea67",
        "SandboxKey": "/var/run/docker/netns/58c5cf7cfe4d",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "Path": "caddy",
    "Platform": "linux",
    "Portainer": {
        "ResourceControl": {
            "Id": 26,
            "ResourceId": "c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771",
            "SubResourceIds": [],
            "Type": 1,
            "UserAccesses": [],
            "TeamAccesses": [],
            "Public": false,
            "AdministratorsOnly": true,
            "System": false
        }
    },
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/c2e7f279792dde0c9666e42fa0f69e00bbeac91ab9baa107b5f889e3d9597771/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Dead": false,
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "2022-12-16T03:18:22.469710712Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 49609,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2022-12-16T03:18:36.313485577Z",
        "Status": "running"
    }
}

If that doesn’t help i can explain how i set it up, but other than posting multiple screenshots of my portainer web interface this is best i can find to get the information readily available for you.

This topic was automatically closed after 30 days. New replies are no longer allowed.