On_demand fails suddenly in 2.7.4

1. The problem I’m having:

CADDY FAILING TO START
gives error

2. Error messages and/or full log output:

ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.0882225,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.090713,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.091754,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00046bf00"}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.7430162,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.091754,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00046bf00"}
ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.0917861,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00046bf00"}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.7461374,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641637.9972196,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.747761,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004f6080"}
ca.1.gk88kh9t90p1@vvstream1    | {"level":"info","ts":1696641631.0917861,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00046bf00"}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.747761,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004f6080"}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.1398861,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
ca.1.gk88kh9t90p1@vvstream1    | Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641637.999855,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.142838,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.7480164,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0004f6080"}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.1433713,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ebf00"}
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641638.0011666,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00052d400"}
ca.1.x76nemof1jbr@vvstream1    | {"level":"info","ts":1696641623.7480164,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0004f6080"}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.1433713,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ebf00"}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.1436057,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003ebf00"}
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641638.0011666,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00052d400"}
ca.1.9mowf66wd8oj@vvstream1    | {"level":"info","ts":1696641645.1436057,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003ebf00"}
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641638.00124,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00052d400"}
ca.1.kf3nehao6ocw@vvstream1    | {"level":"info","ts":1696641638.00124,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00052d400"}
ca.1.x76nemof1jbr@vvstream1    | Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
ca.1.9mowf66wd8oj@vvstream1    | Error: loading initial config: loading new config: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
ca.1.kf3nehao6ocw@vvstream1    | Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details

3. Caddy version:

docker run caddy caddy version
v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

a. System environment:

docker --version
Docker version 20.10.17, build 100c701
uname -a
Linux vvstream1 5.4.0-135-generic #152~18.04.2-Ubuntu SMP Tue Nov 29 08:23:49 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

docker service was created via an app, not via command line. Added service inspect above.

[
    {
        "Id": "ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf",
        "Created": "2023-03-11T11:15:40.52758391Z",
        "Path": "node",
        "Args": [
            "src/mon.js",
            "{\"mode\":\"docker\",\"machineIp\":\"192.168.1.2\",\"coreSources\":[{\"t\":\"mon\"}],\"sources\":[{\"t\":\"mon\"}]}"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5136,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-03-11T11:15:42.393229935Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:dc47f2b90ecd263f3232d3270f3448188f5290ec395f239e8041f3ba81ff3104",
        "ResolvConfPath": "/var/lib/docker/containers/ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf/hostname",
        "HostsPath": "/var/lib/docker/containers/ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf/hosts",
        "LogPath": "/var/lib/docker/containers/ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf/ca71bf48d10705f4681e105fe597b682b87c45117718014c3df59dcf46c978bf-json.log",
        "Name": "/aman-tracker-4-s.1.up9xmi937w1shs7f4ogcjauom",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "3",
                    "max-size": "10M"
                }
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "default",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": [],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/mo/upload",
                    "Target": "/mo/upload"
                },
                {
                    "Type": "bind",
                    "Source": "/var/run",
                    "Target": "/var/run"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/ad283923f03a2c57e93e1326a72f7d473e693c9381b8b5488fe3918f357a9db4-init/diff:/var/lib/docker/overlay2/5293b6dc691471cf55301e3a261c4133bfae8ae553e2f2ca5d32344950342f88/diff:/var/lib/docker/overlay2/02fe869774a9be8032e596cd8f0f0bc20cf4e88b77217e96f02513709cfaec11/diff:/var/lib/docker/overlay2/0ba4466859ade4576d04935c3d323d0888f19012536f0eb9669f372fc12b89fe/diff:/var/lib/docker/overlay2/a3da6938ba2a0284958994860f25c5b5eb6fca118669fa101e82e74bb90c0095/diff:/var/lib/docker/overlay2/a217b4fe0b07e5266ff0c4de28bd078a6d1ce200db1595da5b584fde13cce935/diff:/var/lib/docker/overlay2/efb4f92f56acd003a419e0a20ef7f0f0608cebf6207c44df8d6daa289ffe9234/diff:/var/lib/docker/overlay2/5e95aa315b491e351c0cfb749e5b86d41e609b3f5e1f3220576b3cd5446da30a/diff:/var/lib/docker/overlay2/8b7d014a210ac6b3f3a92b1175ecf87d79b7c2afcb7ddcf483b73a591ca68b5b/diff:/var/lib/docker/overlay2/8beedcb22d18e6ebb63a5b8d10c25e7c762cb53503d3c240c906c2f5072eb7b4/diff:/var/lib/docker/overlay2/1b55e2c8cd1fde69e32666b0cc4c478e6e9638516a39c25537d69111c6518492/diff:/var/lib/docker/overlay2/60afa877892fa67b398807c15a5a752a6696dfbc6ffd9e3f9901b5d2b3515488/diff",
                "MergedDir": "/var/lib/docker/overlay2/ad283923f03a2c57e93e1326a72f7d473e693c9381b8b5488fe3918f357a9db4/merged",
                "UpperDir": "/var/lib/docker/overlay2/ad283923f03a2c57e93e1326a72f7d473e693c9381b8b5488fe3918f357a9db4/diff",
                "WorkDir": "/var/lib/docker/overlay2/ad283923f03a2c57e93e1326a72f7d473e693c9381b8b5488fe3918f357a9db4/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mo/upload",
                "Destination": "/mo/upload",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run",
                "Destination": "/var/run",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ca71bf48d107",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NODE_TLS_REJECT_UNAUTHORIZED=0",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NODE_VERSION=14.15.1",
                "YARN_VERSION=1.22.5"
            ],
            "Cmd": null,
            "Image": "amantracker:222",
            "Volumes": null,
            "WorkingDir": "/mo/codeAll",
            "Entrypoint": [
                "node",
                "src/mon.js",
                "{\"mode\":\"docker\",\"machineIp\":\"192.168.1.2\",\"coreSources\":[{\"t\":\"mon\"}],\"sources\":[{\"t\":\"mon\"}]}"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.swarm.node.id": "qkjd0z13cvtz8pt77eos0w48p",
                "com.docker.swarm.service.id": "109pn7uhqib410ly76n9cu5na",
                "com.docker.swarm.service.name": "aman-tracker-4-s",
                "com.docker.swarm.task": "",
                "com.docker.swarm.task.id": "up9xmi937w1shs7f4ogcjauom",
                "com.docker.swarm.task.name": "aman-tracker-4-s.1.up9xmi937w1shs7f4ogcjauom"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "1da7ed2f3a9346a4546e2334d27b310434ec082d27c6a87a5acfd32b6c4a48e9",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/1da7ed2f3a93",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ingress": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.0.10"
                    },
                    "Links": null,
                    "Aliases": [
                        "ca71bf48d107"
                    ],
                    "NetworkID": "wp4ed2difmhhmqcv8ewyio3lk",
                    "EndpointID": "2cd48f48158923093c0ccf1e87e777328165bc46a14cf41b7361fbbf0c793bd0",
                    "Gateway": "",
                    "IPAddress": "10.0.0.10",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:00:0a",
                    "DriverOpts": null
                },
                "nma": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.1.10"
                    },
                    "Links": null,
                    "Aliases": [
                        "ca71bf48d107"
                    ],
                    "NetworkID": "bqd6i4ilq8yqpzqtmhpjcl2lq",
                    "EndpointID": "bd9c4f6a3cd1b6695d4d6455c490f24b5febdf946a9b5934c7b90e5e5985de90",
                    "Gateway": "",
                    "IPAddress": "10.0.1.10",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:01:0a",
                    "DriverOpts": null
                }
            }
        }
    }
]

c. Service/unit/compose file:

docker service created via app, see inspect command above for details

d. My complete Caddy config:


 "tls": {
            "automation": {
                "policies": [
                    {
                        "on_demand": true
                    }
                ],
                "on_demand": {
                    "rate_limit": {
                        "interval": "10m",
                        "burst": 2
                    }
                }
            }
        } 

5. Links to relevant resources:

You must set an ask endpoint, as the error says, otherwise you’re vulnerable to DDoS. It was made a hard requirement to ensure users don’t have their servers attacked.

It was a warning in prior versions, which you must have ignored.

1 Like

pls suggest the changes in json config.

An attacker can point a wildcard domain like *.evil.com to your server, then start making requests to a.evil.com then b.evil.com to infinity, until your server runs out of storage space or hits rate limits.

So you must have validation in place to prevent domains you don’t know (that users haven’t registered with your server) from getting a certificate.

See https://caddyserver.com/docs/json/apps/tls/automation/on_demand/

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.