Shared storage not working between two instances of Caddy

1. Caddy version (caddy version):

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

To avoid the “context deadline exceeded” issue (Context Deadline Exceeded on Admin API Call - #2 by matt) I have to run two instances of Caddy: one to serve the dashboard website (the public-facing instance) and another to configure how to serve users’ websites.

a. System environment:

rod@staging:~$ uname -a
Linux staging 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

b. Command:

# Public-facing
sudo systemctl start caddy

# Internal
sudo systemctl start caddy-api

c. Service/unit/compose file:

I had to remove the units’ properties because there is a 32000 charater limit so I’ll post them as a separate comment.

d. My complete Caddyfile or JSON config:

Public instance’s Caddyfile:

rod@staging:~$ caddy fmt /etc/caddy/Caddyfile
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	debug
	admin localhost:2018
}

staticful.net {
	handle /api {
		reverse_proxy localhost:4000
	}

	# Taken from https://socket.io/docs/v3/reverse-proxy/#caddy-2
	rewrite /sio /sio/
	handle /sio/* {
		uri strip_prefix /sio
		rewrite * /socket.io{path}
		reverse_proxy localhost:5000 {
			header_up Host {host}
			header_up X-Real-IP {remote}
		}
	}

	handle {
		# Set this path to your site's directory.
		# root * /usr/share/caddy
		root * /var/www/html

		try_files {path} /index.html
		# Enable the static file server.
		file_server
	}
}

http:// {
	reverse_proxy http://localhost:9010
}

https:// {
	reverse_proxy 127.0.0.1:9001
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

I added sink logs to the public instance via the admin API, so here is the public instance’s config via curl http://localhost:2018/config/ | jq:

{
  "admin": {
    "listen": "localhost:2018"
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "group": "group4",
                      "handle": [
                        {
                          "handler": "rewrite",
                          "uri": "/sio/"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/sio"
                          ]
                        }
                      ]
                    },
                    {
                      "group": "group3",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "group": "group1",
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "uri": "/socket.io{http.request.uri.path}"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "strip_path_prefix": "/sio"
                                },
                                {
                                  "handler": "reverse_proxy",
                                  "headers": {
                                    "request": {
                                      "set": {
                                        "Host": [
                                          "{http.request.host}"
                                        ],
                                        "X-Real-Ip": [
                                          "{http.request.remote}"
                                        ]
                                      }
                                    }
                                  },
                                  "upstreams": [
                                    {
                                      "dial": "localhost:5000"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/sio/*"
                          ]
                        }
                      ]
                    },
                    {
                      "group": "group3",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "localhost:4000"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/api"
                          ]
                        }
                      ]
                    },
                    {
                      "group": "group3",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "vars",
                                  "root": "/var/www/html"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "uri": "{http.matchers.file.relative}"
                                }
                              ],
                              "match": [
                                {
                                  "file": {
                                    "try_files": [
                                      "{http.request.uri.path}",
                                      "/index.html"
                                    ]
                                  }
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "file_server",
                                  "hide": [
                                    "/etc/caddy/Caddyfile"
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "staticful.net"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "127.0.0.1:9001"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {}
          ]
        },
        "srv1": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:9010"
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "level": "DEBUG"
      }
    },
    "sink": {
      "writer": {
        "filename": "/var/log/caddy/sink-pub.log",
        "output": "file"
      }
    }
  }
}

Internal instance’s config via curl http://localhost:2019/config/ | jq:

{
  "apps": {
    "http": {
      "http_port": 9010,
      "https_port": 9001,
      "servers": {
        "srv0": {
          "listen": [
            ":9001"
          ],
          "routes": [
            {
              "@id": "randomer-html5-boilerplate",
              "group": "user-site",
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "headers": {
                    "request": {
                      "set": {
                        "Host": [
                          "{upstream_hostport}"
                        ]
                      }
                    }
                  },
                  "rewrite": {
                    "uri": "/randomer-html5-boilerplate{http.request.orig_uri}"
                  },
                  "upstreams": [
                    {
                      "dial": "s3-website.localhost:8000"
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "prrtz.com"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  },
  "logging": {
    "logs": {
      "default": {
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        },
        "level": "DEBUG",
        "writer": {
          "filename": "/var/log/caddy/default.log",
          "output": "file"
        }
      }
    },
    "sink": {
      "writer": {
        "filename": "/var/log/caddy/sink-api.log",
        "output": "file"
      }
    }
  }
}

3. The problem I’m having:

I’m expecting both Caddy instances to share the certificate storage as it says here: Automatic HTTPS — Caddy Documentation but looking at the logs it seems like the public instance isn’t finding the prrtz.com certificate issued earlier by the internal instance at timestamp 2022-07-22T21:22:22.663+0300 (logs from the previous run via sudo cat /var/log/caddy/default.log):

(I had to remove full logs because there is a limit of 32000 characters per post)

{"level":"info","ts":"2022-07-22T21:22:22.663+0300","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"prrtz.com"}
{"level":"info","ts":"2022-07-22T21:22:22.663+0300","logger":"tls.obtain","msg":"releasing lock","identifier":"prrtz.com"}
{"level":"debug","ts":"2022-07-22T21:22:22.663+0300","logger":"tls","msg":"loading managed certificate","domain":"prrtz.com","expiration":"2022-10-20T17:22:20.000Z","issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
{"level":"debug","ts":"2022-07-22T21:22:22.862+0300","logger":"tls.cache","msg":"added certificate to cache","subjects":["prrtz.com"],"expiration":"2022-10-20T17:22:20.000Z","managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753","cache_size":1,"cache_capacity":10000}

When I curl a user’s site (prrtz.com is pointed at this machine’s IP address) I expect the public Caddy instance to find the certificate in the shared storage and serve a successful response, but looks like it doesn’t (error no certificate matching TLS ClientHello at the bottom of public instance’s logs).

rod@staging:~$ curl -v https://prrtz.com
*   Trying 89.223.64.2:443...
* TCP_NODELAY set
* Connected to prrtz.com (89.223.64.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

4. Error messages and/or full log output:

Public instance

rod@staging:~$ sudo systemctl status caddy | cat
● caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-07-23 18:52:37 MSK; 25min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 45053 (caddy)
      Tasks: 7 (limit: 2274)
     Memory: 8.1M
     CGroup: /system.slice/caddy.service
             └─45053 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Jul 23 18:52:41 staging caddy[45053]: {"level":"info","ts":1658591561.5238678,"logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2018","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"60876","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}
Jul 23 18:52:46 staging caddy[45053]: {"level":"info","ts":1658591566.154822,"logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2018","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"60880","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}
Jul 23 18:55:28 staging caddy[45053]: {"level":"debug","ts":1658591728.0174782,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9010","total_upstreams":1}
Jul 23 18:55:28 staging caddy[45053]: {"level":"debug","ts":1658591728.023302,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9010","duration":0.003604083,"request":{"remote_ip":"185.7.214.104","remote_port":"57202","proto":"HTTP/1.1","method":"POST","host":"89.223.64.2:80","uri":"/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php","headers":{"Content-Type":["application/x-www-form-urlencoded"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["185.7.214.104"],"X-Forwarded-Proto":["http"],"X-Forwarded-Host":["89.223.64.2:80"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"],"Content-Length":["19"]}},"headers":{"Date":["Sat, 23 Jul 2022 15:55:28 GMT"],"Content-Length":["0"],"Location":["https://89.223.64.2/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php"],"Server":["Caddy"]},"status":308}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9190598,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"prrtz.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204192,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204297,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.920449,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":"prrtz.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204834,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"prrtz.com","remote":"89.223.64.2:49004","identifier":"prrtz.com","cipher_suites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"cert_cache_fill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9213223,"logger":"http.stdlib","msg":"http: TLS handshake error from 89.223.64.2:49004: no certificate available for 'prrtz.com'"}


Public instance logs (sudo journalctl -u caddy --no-pager):

Jul 23 18:52:37 staging systemd[1]: Starting Caddy...
Jul 23 18:52:37 staging caddy[45053]: caddy.HomeDir=/var/lib/caddy
Jul 23 18:52:37 staging caddy[45053]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Jul 23 18:52:37 staging caddy[45053]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Jul 23 18:52:37 staging caddy[45053]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Jul 23 18:52:37 staging caddy[45053]: caddy.Version=v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=
Jul 23 18:52:37 staging caddy[45053]: runtime.GOOS=linux
Jul 23 18:52:37 staging caddy[45053]: runtime.GOARCH=amd64
Jul 23 18:52:37 staging caddy[45053]: runtime.Compiler=gc
Jul 23 18:52:37 staging caddy[45053]: runtime.NumCPU=1
Jul 23 18:52:37 staging caddy[45053]: runtime.GOMAXPROCS=1
Jul 23 18:52:37 staging caddy[45053]: runtime.Version=go1.18.3
Jul 23 18:52:37 staging caddy[45053]: os.Getwd=/
Jul 23 18:52:37 staging caddy[45053]: LANG=en_US.UTF-8
Jul 23 18:52:37 staging caddy[45053]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Jul 23 18:52:37 staging caddy[45053]: NOTIFY_SOCKET=/run/systemd/notify
Jul 23 18:52:37 staging caddy[45053]: HOME=/var/lib/caddy
Jul 23 18:52:37 staging caddy[45053]: LOGNAME=caddy
Jul 23 18:52:37 staging caddy[45053]: USER=caddy
Jul 23 18:52:37 staging caddy[45053]: INVOCATION_ID=a5972514769b4a7a9547c70985fd5e4d
Jul 23 18:52:37 staging caddy[45053]: JOURNAL_STREAM=9:1510010
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4328818,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 23 18:52:37 staging caddy[45053]: {"level":"warn","ts":1658591557.440443,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4494333,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2018","enforce_origin":false,"origins":["//localhost:2018","//[::1]:2018","//127.0.0.1:2018"]}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4501128,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jul 23 18:52:37 staging caddy[45053]: {"level":"warn","ts":1658591557.4502773,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Jul 23 18:52:37 staging caddy[45053]: {"level":"debug","ts":1658591557.4540946,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Jul 23 18:52:37 staging caddy[45053]: {"level":"debug","ts":1658591557.4544847,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4546406,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["staticful.net"]}
Jul 23 18:52:37 staging caddy[45053]: {"level":"debug","ts":1658591557.4555428,"logger":"tls","msg":"loading managed certificate","domain":"staticful.net","expiration":1666276983,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 23 18:52:37 staging caddy[45053]: {"level":"debug","ts":1658591557.4574065,"logger":"tls.cache","msg":"added certificate to cache","subjects":["staticful.net"],"expiration":1666276983,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"45df2719cbbd113833576c3e1c68d6b9e39e66ac139112fdb89f2d9e0845e2b7","cache_size":1,"cache_capacity":10000}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4584486,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jul 23 18:52:37 staging systemd[1]: Started Caddy.
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4661975,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003eaf50"}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4665065,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4681902,"logger":"tls","msg":"finished cleaning storage units"}
Jul 23 18:52:37 staging caddy[45053]: {"level":"info","ts":1658591557.4705985,"msg":"serving initial configuration"}
Jul 23 18:52:41 staging caddy[45053]: {"level":"info","ts":1658591561.5238678,"logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2018","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"60876","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}
Jul 23 18:52:46 staging caddy[45053]: {"level":"info","ts":1658591566.154822,"logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2018","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"60880","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}
Jul 23 18:55:28 staging caddy[45053]: {"level":"debug","ts":1658591728.0174782,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9010","total_upstreams":1}
Jul 23 18:55:28 staging caddy[45053]: {"level":"debug","ts":1658591728.023302,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9010","duration":0.003604083,"request":{"remote_ip":"185.7.214.104","remote_port":"57202","proto":"HTTP/1.1","method":"POST","host":"89.223.64.2:80","uri":"/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php","headers":{"Content-Type":["application/x-www-form-urlencoded"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["185.7.214.104"],"X-Forwarded-Proto":["http"],"X-Forwarded-Host":["89.223.64.2:80"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"],"Content-Length":["19"]}},"headers":{"Date":["Sat, 23 Jul 2022 15:55:28 GMT"],"Content-Length":["0"],"Location":["https://89.223.64.2/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php"],"Server":["Caddy"]},"status":308}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9190598,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"prrtz.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204192,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204297,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.920449,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":"prrtz.com"}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9204834,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"prrtz.com","remote":"89.223.64.2:49004","identifier":"prrtz.com","cipher_suites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"cert_cache_fill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
Jul 23 19:01:11 staging caddy[45053]: {"level":"debug","ts":1658592071.9213223,"logger":"http.stdlib","msg":"http: TLS handshake error from 89.223.64.2:49004: no certificate available for 'prrtz.com'"}

Public instance sink log (sudo cat /var/log/caddy/sink-pub.log):

[INFO] Redirected sink to here (/var/log/caddy/sink-pub.log)

Internal instance

rod@staging:~$ sudo systemctl status caddy-api | cat
● caddy-api.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy-api.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/caddy-api.service.d
             └─override.conf
     Active: active (running) since Sat 2022-07-23 18:53:30 MSK; 25min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 45081 (caddy)
      Tasks: 6 (limit: 2274)
     Memory: 7.4M
     CGroup: /system.slice/caddy-api.service
             └─45081 /usr/bin/caddy run --environ --resume

Jul 23 18:53:30 staging caddy[45081]: HOME=/var/lib/caddy
Jul 23 18:53:30 staging caddy[45081]: LOGNAME=caddy
Jul 23 18:53:30 staging caddy[45081]: USER=caddy
Jul 23 18:53:30 staging caddy[45081]: INVOCATION_ID=6aa7f710450b4cd2982d2dfdd25f99f1
Jul 23 18:53:30 staging caddy[45081]: JOURNAL_STREAM=9:1510286
Jul 23 18:53:30 staging caddy[45081]: XDG_CONFIG_HOME=/var/lib/caddy-api
Jul 23 18:53:30 staging caddy[45081]: {"level":"info","ts":1658591610.716091,"msg":"resuming from last configuration","autosave_file":"/var/lib/caddy-api/caddy/autosave.json"}
Jul 23 18:53:30 staging caddy[45081]: {"level":"info","ts":1658591610.717915,"msg":"[INFO] Redirecting sink to: /var/log/caddy/sink-api.log"}
Jul 23 18:53:30 staging caddy[45081]: {"level":"info","ts":1658591610.7183805,"msg":"redirected default logger","from":"stderr","to":"/var/log/caddy/default.log"}
Jul 23 18:53:30 staging systemd[1]: Started Caddy.

Internal instance default log (sudo cat /var/log/caddy/default.log):

{"level":"info","ts":"2022-07-23T18:53:30.725+0300","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
{"level":"info","ts":"2022-07-23T18:53:30.726+0300","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":9001}
{"level":"info","ts":"2022-07-23T18:53:30.726+0300","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"debug","ts":"2022-07-23T18:53:30.726+0300","logger":"http","msg":"starting server loop","address":"[::]:9001","http3":false,"tls":true}
{"level":"debug","ts":"2022-07-23T18:53:30.726+0300","logger":"http","msg":"starting server loop","address":"[::]:9010","http3":false,"tls":false}
{"level":"info","ts":"2022-07-23T18:53:30.726+0300","logger":"http","msg":"enabling automatic TLS certificate management","domains":["prrtz.com"]}
{"level":"debug","ts":"2022-07-23T18:53:30.727+0300","logger":"tls","msg":"loading managed certificate","domain":"prrtz.com","expiration":"2022-10-20T17:22:20.000Z","issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
{"level":"debug","ts":"2022-07-23T18:53:30.727+0300","logger":"tls.cache","msg":"added certificate to cache","subjects":["prrtz.com"],"expiration":"2022-10-20T17:22:20.000Z","managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753","cache_size":1,"cache_capacity":10000}
{"level":"info","ts":"2022-07-23T18:53:30.727+0300","msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy-api/caddy/autosave.json"}
{"level":"info","ts":"2022-07-23T18:53:30.734+0300","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000285490"}
{"level":"info","ts":"2022-07-23T18:53:30.734+0300","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
{"level":"info","ts":"2022-07-23T18:53:30.736+0300","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2022-07-23T18:53:30.738+0300","msg":"serving initial configuration"}
{"level":"info","ts":"2022-07-23T18:53:37.118+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42028","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}

Internal instance sink log (sudo cat /var/log/caddy/sink-api.log):

[INFO] Redirected sink to here (/var/log/caddy/sink-api.log)

5. What I already tried:

At first I ran the internal Caddy instance via the sudo -H -u caddy bash -c 'caddy run' command, but that didn’t work either and gave me the same error. Unfortunately I’m stuck now as I’m not sure what else I should try.

6. Links to relevant resources:

1 Like

Public-facing instance’s unit properties (nothing is overridden here):

rod@staging:~$ systemctl show caddy.service | cat
Type=notify
Restart=no
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=5s
TimeoutAbortUSec=5s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=[not set]
GID=[not set]
NRestarts=0
OOMPolicy=stop
ExecMainStartTimestamp=Fri 2022-07-22 18:43:02 MSK
ExecMainStartTimestampMonotonic=601990042403
ExecMainExitTimestamp=Sat 2022-07-23 17:43:42 MSK
ExecMainExitTimestampMonotonic=684829767073
ExecMainPID=40330
ExecMainCode=1
ExecMainStatus=0
ExecStart={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile ; ignore_errors=no ; start_time=[Fri 2022-07-22 18:43:02 MSK] ; stop_time=[Sat 2022-07-23 17:43:42 MSK] ; pid=40330 ; code=exited ; status=0 }
ExecStartEx={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile ; flags= ; start_time=[Fri 2022-07-22 18:43:02 MSK] ; stop_time=[Sat 2022-07-23 17:43:42 MSK] ; pid=40330 ; code=exited ; status=0 }
ExecReload={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReloadEx={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
MemoryCurrent=[not set]
CPUUsageNSec=[not set]
EffectiveCPUs=
EffectiveMemoryNodes=
TasksCurrent=[not set]
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=no
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
AllowedCPUs=
AllowedMemoryNodes=
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=2274
IPAccounting=no
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=512
LimitNPROCSoft=512
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=7580
LimitSIGPENDINGSoft=7580
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUAffinity=
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
NUMAMask=
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read
AmbientCapabilities=cap_net_bind_service
User=caddy
Group=caddy
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=yes
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectKernelLogs=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
PrivateMounts=no
ProtectHome=no
ProtectSystem=full
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
TimeoutCleanUSec=infinity
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictSUIDSGID=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
ProtectHostname=no
KillMode=control-group
KillSignal=15
RestartKillSignal=15
FinalKillSignal=9
SendSIGKILL=yes
SendSIGHUP=no
WatchdogSignal=6
Id=caddy.service
Names=caddy.service
Requires=system.slice network-online.target -.mount sysinit.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=-.mount systemd-tmpfiles-setup.service systemd-journald.socket network.target network-online.target basic.target sysinit.target system.slice
RequiresMountsFor=/tmp /var/tmp
Documentation=https://caddyserver.com/docs/
Description=Caddy
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/etc/systemd/system/caddy.service
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Sat 2022-07-23 17:43:42 MSK
StateChangeTimestampMonotonic=684829767160
InactiveExitTimestamp=Fri 2022-07-22 18:43:02 MSK
InactiveExitTimestampMonotonic=601990043207
ActiveEnterTimestamp=Fri 2022-07-22 18:43:02 MSK
ActiveEnterTimestampMonotonic=601990228415
ActiveExitTimestamp=Sat 2022-07-23 17:43:42 MSK
ActiveExitTimestampMonotonic=684829750320
InactiveEnterTimestamp=Sat 2022-07-23 17:43:42 MSK
InactiveEnterTimestampMonotonic=684829767160
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2022-07-22 18:43:02 MSK
ConditionTimestampMonotonic=601990035496
AssertTimestamp=Fri 2022-07-22 18:43:02 MSK
AssertTimestampMonotonic=601990035497
Transient=no
Perpetual=no
StartLimitIntervalUSec=10s
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=53ad3b48851a4c6ebf4aa4ac18c60607
CollectMode=inactive

Internal instance’s unit properties (only the Environment property is overridden to avoid clashing with the public instance’s config):

rod@staging:~$ systemctl show caddy-api.service | cat
Type=notify
Restart=no
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=1min 30s
TimeoutStopUSec=5s
TimeoutAbortUSec=5s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestampMonotonic=0
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=0
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
ReloadResult=success
CleanResult=success
UID=[not set]
GID=[not set]
NRestarts=0
OOMPolicy=stop
ExecMainStartTimestamp=Fri 2022-07-22 19:54:15 MSK
ExecMainStartTimestampMonotonic=606262753187
ExecMainExitTimestamp=Sat 2022-07-23 17:43:43 MSK
ExecMainExitTimestampMonotonic=684831468233
ExecMainPID=40617
ExecMainCode=1
ExecMainStatus=0
ExecStart={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy run --environ --resume ; ignore_errors=no ; start_time=[Fri 2022-07-22 19:54:15 MSK] ; stop_time=[Sat 2022-07-23 17:43:43 MSK] ; pid=40617 ; code=exited ; status=0 }
ExecStartEx={ path=/usr/bin/caddy ; argv[]=/usr/bin/caddy run --environ --resume ; flags= ; start_time=[Fri 2022-07-22 19:54:15 MSK] ; stop_time=[Sat 2022-07-23 17:43:43 MSK] ; pid=40617 ; code=exited ; status=0 }
Slice=system.slice
MemoryCurrent=[not set]
CPUUsageNSec=[not set]
EffectiveCPUs=
EffectiveMemoryNodes=
TasksCurrent=[not set]
IPIngressBytes=[no data]
IPIngressPackets=[no data]
IPEgressBytes=[no data]
IPEgressPackets=[no data]
IOReadBytes=18446744073709551615
IOReadOperations=18446744073709551615
IOWriteBytes=18446744073709551615
IOWriteOperations=18446744073709551615
Delegate=no
CPUAccounting=no
CPUWeight=[not set]
StartupCPUWeight=[not set]
CPUShares=[not set]
StartupCPUShares=[not set]
CPUQuotaPerSecUSec=infinity
CPUQuotaPeriodUSec=infinity
AllowedCPUs=
AllowedMemoryNodes=
IOAccounting=no
IOWeight=[not set]
StartupIOWeight=[not set]
BlockIOAccounting=no
BlockIOWeight=[not set]
StartupBlockIOWeight=[not set]
MemoryAccounting=yes
DefaultMemoryLow=0
DefaultMemoryMin=0
MemoryMin=0
MemoryLow=0
MemoryHigh=infinity
MemoryMax=infinity
MemorySwapMax=infinity
MemoryLimit=infinity
DevicePolicy=auto
TasksAccounting=yes
TasksMax=2274
IPAccounting=no
Environment=XDG_CONFIG_HOME=/var/lib/caddy-api
UMask=0022
LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=1048576
LimitNOFILESoft=1048576
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=512
LimitNPROCSoft=512
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=7580
LimitSIGPENDINGSoft=7580
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
CPUAffinity=
CPUAffinityFromNUMA=no
NUMAPolicy=n/a
NUMAMask=
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardInputData=
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
LogLevelMax=-1
LogRateLimitIntervalUSec=0
LogRateLimitBurst=0
SecureBits=0
CapabilityBoundingSet=cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read
AmbientCapabilities=cap_net_bind_service
User=caddy
Group=caddy
DynamicUser=no
RemoveIPC=no
MountFlags=
PrivateTmp=yes
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectKernelLogs=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
PrivateMounts=no
ProtectHome=no
ProtectSystem=full
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
LockPersonality=no
RuntimeDirectoryPreserve=no
RuntimeDirectoryMode=0755
StateDirectoryMode=0755
CacheDirectoryMode=0755
LogsDirectoryMode=0755
ConfigurationDirectoryMode=0755
TimeoutCleanUSec=infinity
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictSUIDSGID=no
RestrictNamespaces=no
MountAPIVFS=no
KeyringMode=private
ProtectHostname=no
KillMode=control-group
KillSignal=15
RestartKillSignal=15
FinalKillSignal=9
SendSIGKILL=yes
SendSIGHUP=no
WatchdogSignal=6
Id=caddy-api.service
Names=caddy-api.service
Requires=sysinit.target -.mount network-online.target system.slice
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=multi-user.target shutdown.target
After=systemd-journald.socket basic.target -.mount network.target systemd-tmpfiles-setup.service sysinit.target network-online.target system.slice
RequiresMountsFor=/tmp /var/tmp
Documentation=https://caddyserver.com/docs/
Description=Caddy
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/etc/systemd/system/caddy-api.service
DropInPaths=/etc/systemd/system/caddy-api.service.d/override.conf
UnitFileState=enabled
UnitFilePreset=enabled
StateChangeTimestamp=Sat 2022-07-23 17:43:43 MSK
StateChangeTimestampMonotonic=684831468327
InactiveExitTimestamp=Fri 2022-07-22 19:54:15 MSK
InactiveExitTimestampMonotonic=606262754069
ActiveEnterTimestamp=Fri 2022-07-22 19:54:15 MSK
ActiveEnterTimestampMonotonic=606262867518
ActiveExitTimestamp=Sat 2022-07-23 17:43:43 MSK
ActiveExitTimestampMonotonic=684831454083
InactiveEnterTimestamp=Sat 2022-07-23 17:43:43 MSK
InactiveEnterTimestampMonotonic=684831468327
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2022-07-22 19:54:15 MSK
ConditionTimestampMonotonic=606262743081
AssertTimestamp=Fri 2022-07-22 19:54:15 MSK
AssertTimestampMonotonic=606262743082
Transient=no
Perpetual=no
StartLimitIntervalUSec=10s
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
SuccessAction=none
InvocationID=7e8bdaed3c054679a23e9f2b9fcf2d59
CollectMode=inactive
1 Like

Thanks for the detailed help topic, we really appreciate it when people fill out the template!

So, I noticed from a quick Ctrl+F that your config doesn’t specify storage. Meaning Caddy just uses its default storage, which according to your logs, is:

FileStorage:/var/lib/caddy/.local/share/caddy

(The “FileStorage” is just the name of the Storage type implementation.)

So, you’d have to be sharing the /var/lib/caddy/.local/share/caddy folder with other instances in order for Caddy to share the resources within it. (That’s something external to Caddy.) There are ways of sharing a file system folder (mounting a network folder, for example – though maybe not ideal for very-high-volume sites), or you can use another storage module like a database (redis, postgres, etc).

But yeah, as long as Caddy shares its storage, it will coordinate cert management.

1 Like

Thanks for the quick reply!

That’s correct! So my logic was because both Caddy instances are running on the same virtual machine (which I wasn’t clear about, sorry) to which I connect via ssh and both Caddys point to the same storage folder according to both of the Caddys’ logs, doesn’t that mean that the /var/lib/caddy/.local/share/caddy folder is already being shared between them? Or should I do any additional setup to make it “shared” Caddy-wise?

I apologize in advance because I’m not super proficient in Linux and I’ve only been using Caddy for about two weeks so I might be missing something obvious here :-/

1 Like

Oh, I gotchu. Thanks for clarifying. (That’s a bit unusual to run 2 different Caddy instances on the same machine! May I ask why you’re doing that?)

I think the issue is that the public instance wasn’t configured with the hostname prrtz.com.

  • It’s not in the site name (only https://)
  • On-demand TLS isn’t enabled
  • Certificates aren’t loaded manually (which is good, since we automate things these days)

Basically, Caddy has no idea what hostnames to serve, nor does it have any configuration for when/how to get certificates, so it gives “no certificate available”.

I’d recommend just configuring your public instance with the site name prrtz.com, i.e. replace https:// with prrtz.com.

Although, this does make me wonder, should we look in storage for a cert anyway?

2 Likes

Ok, so these are the scenarios that I’m talking about:

  1. Serve dashboard assets: https://staticful.net (user dashboard) -> [public Caddy] static assets (html, css, js, etc.)
  2. API request (e.g. by making an input and pressing a button in the dashboard): "link domain (e.g. prrtz.com) to a user's website" API request -> [public Caddy] reverse proxy to the node.js server (port 4000) -> [update internal Caddy's config via API] insert 'prrtz.com' into 'host' of the user's website config
  3. Serve user’s site assets: https://prrtz.com (user's website) -> [public Caddy] reverse proxy to internal Caddy -> [internal Caddy] match by domain and reverse proxy to the user-specific S3 bucket

At first I had a single Caddy server, but the any API requests that modified its config (2nd scenario) resulted in the context deadline exceeded error with the admin API shutting down. So I followed the advice in the Context Deadline Exceeded on Admin API Call post where you explain the problem and later say that another server is needed (although you also say that it’s possible to “bind the admin API to a non-localhost interface”, but I’m not sure if it applies in my case because I’m running everything on a single machine).

I know about the on-demand TLS feature, but I don’t know if it’s going to be helpful in my case because:

  1. In the second scenario where a user links a domain to their website I want to generate a free SSL certificate right away (that’s where Caddy helps a lot :pray:) and notify the user if there are any errors in case the Caddy API doesn’t return 200 or when issuing a certificate.
  2. I need to match multiple domains to a single S3 bucket (now that I think of it maybe it’s not a problem?)
  3. Soon I’m going to implement a feature where users will be able to add their own custom certificates instead of the default Let’s Encrypt or ZeroSSL ones (didn’t look into this yet, but I’m guessing I should be looking at JSON Config Structure - Caddy Documentation)

In my case that would be super useful because everything would just work :slightly_smiling_face: But I guess the question is how frequently are people hitting this problem? :man_shrugging:

Hopefully this clarifies my setup further (and if not, feel free to ask if anything is unclear). Also, I wanted to say thanks for making Caddy (especially making it configurable via an API and automatic HTTPS — those are life savers!) and providing great support for it!

1 Like

Ok, so I turned on on-demand TLS on the public Caddy server, but curl responds with Client sent an HTTP request to an HTTPS server.

Here is the updated public Caddy’s config:

rod@staging:~/repos$ caddy fmt /etc/caddy/Caddyfile
# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	debug
	admin localhost:2018
	on_demand_tls {
		ask http://127.0.0.1:6000
	}
}

staticful.net {
	handle /api {
		reverse_proxy localhost:4000
	}

	# Taken from https://socket.io/docs/v3/reverse-proxy/#caddy-2
	rewrite /sio /sio/
	handle /sio/* {
		uri strip_prefix /sio
		rewrite * /socket.io{path}
		reverse_proxy localhost:5000 {
			header_up Host {host}
			header_up X-Real-IP {remote}
		}
	}

	handle {
		# Set this path to your site's directory.
		# root * /usr/share/caddy
		root * /var/www/html

		try_files {path} /index.html
		# Enable the static file server.
		file_server
	}
}

http:// {
	reverse_proxy http://localhost:9010
}

https:// {
	tls {
		on_demand
	}
	reverse_proxy 127.0.0.1:9001
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

Curl result:

rod@staging:~/repos$ curl -v https://prrtz.com
*   Trying 89.223.64.2:443...
* TCP_NODELAY set
* Connected to prrtz.com (89.223.64.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=prrtz.com
*  start date: Jul 22 17:22:21 2022 GMT
*  expire date: Oct 20 17:22:20 2022 GMT
*  subjectAltName: host "prrtz.com" matched cert's "prrtz.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f4872bd8c0)
> GET / HTTP/2
> Host: prrtz.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 400 
< server: Caddy
< content-type: text/plain; charset=utf-8
< date: Sun, 24 Jul 2022 16:17:41 GMT
< 
Client sent an HTTP request to an HTTPS server.
* Connection #0 to host prrtz.com left intact

Public Caddy’s logs:

Jul 24 19:17:41 staging caddy[45053]: {"level":"debug","ts":1658679461.947629,"logger":"tls.handshake","msg":"choosing certificate","identifier":"prrtz.com","num_choices":1}
Jul 24 19:17:41 staging caddy[45053]: {"level":"debug","ts":1658679461.950081,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"prrtz.com","subjects":["prrtz.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753"}
Jul 24 19:17:41 staging caddy[45053]: {"level":"debug","ts":1658679461.9501576,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["prrtz.com"],"managed":true,"expiration":1666286540,"hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753"}
Jul 24 19:17:41 staging caddy[45053]: {"level":"debug","ts":1658679461.9562702,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"127.0.0.1:9001","total_upstreams":1}
Jul 24 19:17:41 staging caddy[45053]: {"level":"debug","ts":1658679461.9578557,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9001","duration":0.001251376,"request":{"remote_ip":"89.223.64.2","remote_port":"49860","proto":"HTTP/2.0","method":"GET","host":"prrtz.com","uri":"/","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"],"X-Forwarded-For":["89.223.64.2"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["prrtz.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"prrtz.com"}},"headers":{},"status":400}

Internal Caddy’s logs:

{"level":"info","ts":"2022-07-24T18:53:30.819+0300","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
{"level":"info","ts":"2022-07-24T18:53:30.843+0300","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2022-07-24T19:03:58.420+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42722","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:03:58.593+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42724","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:06:48.008+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42746","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:06:48.086+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42748","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:06:49.718+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42864","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:13:03.189+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42886","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}

Edit: Looks like my internal node API ask-domain endpoint isn’t responding, looking into that now…

Ok, I fixed the on-demand ask url and curl’d it to make sure it works. However, the problem is still there: it still responds with Client sent an HTTP request to an HTTPS server. Also, it looks like the internal instance isn’t getting the request.

Public config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.
{
	debug
	admin localhost:2018
	on_demand_tls {
		ask http://127.0.0.1:6000/is-own-domain
	}
}

staticful.net {
	handle /api {
		reverse_proxy localhost:4000
	}

	# Taken from https://socket.io/docs/v3/reverse-proxy/#caddy-2
	rewrite /sio /sio/
	handle /sio/* {
		uri strip_prefix /sio
		rewrite * /socket.io{path}
		reverse_proxy localhost:5000 {
			header_up Host {host}
			header_up X-Real-IP {remote}
		}
	}

	handle {
		# Set this path to your site's directory.
		# root * /usr/share/caddy
		root * /var/www/html

		try_files {path} /index.html
		# Enable the static file server.
		file_server
	}
}

http:// {
	reverse_proxy http://localhost:9010
}

https:// {
	tls {
		on_demand
	}
	reverse_proxy 127.0.0.1:9001
}

# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

Curl result:

rod@staging:~/repos$ curl -v https://prrtz.com
*   Trying 89.223.64.2:443...
* TCP_NODELAY set
* Connected to prrtz.com (89.223.64.2) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=prrtz.com
*  start date: Jul 22 17:22:21 2022 GMT
*  expire date: Oct 20 17:22:20 2022 GMT
*  subjectAltName: host "prrtz.com" matched cert's "prrtz.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5612f5f478c0)
> GET / HTTP/2
> Host: prrtz.com
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 400 
< server: Caddy
< content-type: text/plain; charset=utf-8
< date: Sun, 24 Jul 2022 16:35:25 GMT
< 
Client sent an HTTP request to an HTTPS server.
* Connection #0 to host prrtz.com left intact

Public logs:

Jul 24 19:35:03 staging systemd[1]: Reloading Caddy.
Jul 24 19:35:03 staging caddy[54249]: {"level":"info","ts":1658680503.8533006,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 24 19:35:03 staging caddy[54249]: {"level":"warn","ts":1658680503.859238,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
Jul 24 19:35:03 staging caddy[54249]: {"level":"info","ts":1658680503.859731,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Jul 24 19:35:03 staging caddy[54249]: {"level":"warn","ts":1658680503.8620868,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":10}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8664362,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2018","uri":"/load","remote_ip":"127.0.0.1","remote_port":"33578","headers":{"Accept-Encoding":["gzip"],"Cache-Control":["must-revalidate"],"Content-Length":["1715"],"Content-Type":["application/json"],"Origin":["http://localhost:2018"],"User-Agent":["Go-http-client/1.1"]}}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8698092,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2018","enforce_origin":false,"origins":["//localhost:2018","//[::1]:2018","//127.0.0.1:2018"]}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8705332,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Jul 24 19:35:03 staging caddy[45053]: {"level":"warn","ts":1658680503.8706985,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Jul 24 19:35:03 staging caddy[45053]: {"level":"debug","ts":1658680503.8726292,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Jul 24 19:35:03 staging caddy[45053]: {"level":"debug","ts":1658680503.872922,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8729327,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["staticful.net"]}
Jul 24 19:35:03 staging caddy[45053]: {"level":"debug","ts":1658680503.874047,"logger":"tls","msg":"loading managed certificate","domain":"staticful.net","expiration":1666276983,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 24 19:35:03 staging caddy[45053]: {"level":"debug","ts":1658680503.8747725,"logger":"tls.cache","msg":"added certificate to cache","subjects":["staticful.net"],"expiration":1666276983,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"45df2719cbbd113833576c3e1c68d6b9e39e66ac139112fdb89f2d9e0845e2b7","cache_size":1,"cache_capacity":10000}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.875346,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003eae00"}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8788621,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003ea7e0"}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8792162,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.8802536,"logger":"admin.api","msg":"load complete"}
Jul 24 19:35:03 staging systemd[1]: Reloaded Caddy.
Jul 24 19:35:03 staging caddy[45053]: {"level":"info","ts":1658680503.897645,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2018"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7831936,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"prrtz.com"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7852805,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7852993,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7853265,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":"prrtz.com"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7875595,"logger":"tls","msg":"loading managed certificate","domain":"prrtz.com","expiration":1666286540,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7887642,"logger":"tls.cache","msg":"added certificate to cache","subjects":["prrtz.com"],"expiration":1666286540,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753","cache_size":2,"cache_capacity":10000}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7890036,"logger":"tls.handshake","msg":"loaded certificate from storage","subjects":["prrtz.com"],"managed":true,"expiration":1666286540,"hash":"33887b87cafc18802ce19b2b12bc6da820a407669a15da27c3debc956fd43753"}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7937932,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"127.0.0.1:9001","total_upstreams":1}
Jul 24 19:35:25 staging caddy[45053]: {"level":"debug","ts":1658680525.7997308,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9001","duration":0.00535812,"request":{"remote_ip":"89.223.64.2","remote_port":"49926","proto":"HTTP/2.0","method":"GET","host":"prrtz.com","uri":"/","headers":{"X-Forwarded-Host":["prrtz.com"],"User-Agent":["curl/7.68.0"],"Accept":["*/*"],"X-Forwarded-For":["89.223.64.2"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"prrtz.com"}},"headers":{},"status":400}
Jul 24 19:36:26 staging caddy[45053]: {"level":"debug","ts":1658680586.3823166,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9010","total_upstreams":1}
Jul 24 19:36:26 staging caddy[45053]: {"level":"debug","ts":1658680586.3871658,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9010","duration":0.003374855,"request":{"remote_ip":"2.57.122.98","remote_port":"33900","proto":"HTTP/1.1","method":"GET","host":"89.223.64.2","uri":"/","headers":{"Accept-Language":["en US,en;q=0.9,sv;q=0.8"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"],"X-Forwarded-Host":["89.223.64.2"],"X-Forwarded-Proto":["http"],"Accept-Encoding":["gzip, deflate","gzip, deflate"],"X-Requested-With":["XMLHttpRequest"],"Accept":["*/*"],"X-Forwarded-For":["2.57.122.98"],"Content-Type":["application/x-www-form-urlencoded; charset=UTF-8"]}},"headers":{"Location":["https://89.223.64.2/"],"Server":["Caddy"],"Date":["Sun, 24 Jul 2022 16:36:26 GMT"],"Content-Length":["0"]},"status":308}

Internal logs (looks like the request didn’t get through the public Caddy server):

rod@staging:~/repos$ sudo tail -n 10 /var/log/caddy/default.log
{"level":"info","ts":"2022-07-24T19:06:48.086+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42748","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:06:49.718+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42864","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:13:03.189+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42886","headers":{"Accept":["*/*"],"User-Agent":["curl/7.68.0"]}}
{"level":"debug","ts":"2022-07-24T19:21:59.542+0300","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"127.0.0.1"}
{"level":"debug","ts":"2022-07-24T19:21:59.543+0300","logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":""}
{"level":"debug","ts":"2022-07-24T19:21:59.543+0300","logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"127.0.0.1:40054","identifier":"127.0.0.1","cipher_suites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"cert_cache_fill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":"2022-07-24T19:21:59.543+0300","logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:40054: no certificate available for '127.0.0.1'"}
{"level":"info","ts":"2022-07-24T19:32:18.915+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42932","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:32:19.032+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42934","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}
{"level":"info","ts":"2022-07-24T19:32:19.159+0300","logger":"admin.api","msg":"received request","method":"GET","host":"localhost:2019","uri":"/config/","remote_ip":"127.0.0.1","remote_port":"42942","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip,deflate"],"Connection":["close"],"User-Agent":["node-fetch/1.0 (+https://github.com/bitinn/node-fetch)"]}}

Ok, solved it! The problem was that the public Caddy was reverse proxying to the HTTPS port of the internal Caddy instead of its HTTP port and the internal Caddy was listening on the HTTPS port. I’ll still need to figure out how to add users’ custom TLS certificates, but that will come later :slight_smile:

Thanks again for all the help! Onwards!

Glad you figured it out @rod! Yep, looks like that message was coming from your proxied backend rather than the proxy itself.

Thanks for documenting your process, I’m sure it will be helpful to future searchers.

Let us know when you have other questions!

1 Like