Caddy V2 tls internal only work when port 80/443 are free?

1. Caddy version (caddy version):

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

Installed Caddy via copr YUM repo

yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/g/caddy/caddy/repo/epel-7/group_caddy-caddy-epel-7.repo
yum -y install caddy --disablerepo=epel
sed -i 's|:80|http://caddy.domain.com:81\n\nheader x-powered-by "caddy centminmod"\nheader vary Accept-Encoding\nencode gzip\n|' /etc/caddy/Caddyfile
caddy trust
cp -a /usr/share/caddy/index.html /usr/share/caddy/caddy-index.html
service caddy start
service caddy status
chkconfig caddy on

a. System environment:

  • CentOS 7.8
  • Nginx running on port 80 and 443
  • VirtualBox on Windows 10 Pro

b. Command:

service caddy start

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

http://caddy.domain.com:81 {

header x-powered-by "caddy centminmod"
header vary Accept-Encoding
encode gzip

# Set this path to your site's directory.
root * /usr/share/caddy

# Enable the static file server.
file_server

# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
}

https://caddy.domain.com:4444 {

tls internal
header x-powered-by "caddy centminmod"
header vary Accept-Encoding
encode gzip

# Set this path to your site's directory.
root * /usr/share/caddy

# Enable the static file server.
file_server

# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
}

3. The problem I’m having:

I am installing and testing Caddy v2 besides my own Nginx installation so Nginx runs on port 80/443 and intend for Caddy v2 to run on port 81 for non-HTTPS and port 4444 for HTTPS.

The problem Caddy is refusing to start up with tls internal for local dev SSL certificates on HTTPS port 4444. It seems it’s hard coded for port 80 and 443 as when I stop Nginx on port 80 and 443, then Caddy does start up and the Caddyfile defined port 81 and 4444 does work.

4. Error messages and/or full log output:

when Nginx still running

service caddy status
Redirecting to /bin/systemctl status caddy.service
● caddy.service - Caddy
   Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2020-05-08 14:13:42 UTC; 24s ago
     Docs: https://caddyserver.com/docs/
  Process: 29154 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
 Main PID: 29154 (code=exited, status=1/FAILURE)

May 08 14:13:42 centos7.localdomain caddy[29154]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
May 08 14:13:42 centos7.localdomain caddy[29154]: HOME=/var/lib/caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: LOGNAME=caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: USER=caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: SHELL=/sbin/nologin
May 08 14:13:42 centos7.localdomain caddy[29154]: {"level":"info","ts":1588947222.328688,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 08 14:13:42 centos7.localdomain caddy[29154]: {"level":"info","ts":1588947222.3311186,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:201...127.0.0.1:2019"]}
May 08 14:13:42 centos7.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
May 08 14:13:42 centos7.localdomain systemd[1]: Unit caddy.service entered failed state.
May 08 14:13:42 centos7.localdomain systemd[1]: caddy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
journalctl -u caddy --no-pager

May 08 14:13:36 centos7.localdomain systemd[1]: Stopping Caddy...
May 08 14:13:36 centos7.localdomain systemd[1]: Stopped Caddy.
May 08 14:13:42 centos7.localdomain systemd[1]: Started Caddy.
May 08 14:13:42 centos7.localdomain caddy[29154]: caddy.HomeDir=/var/lib/caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.GOOS=linux
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.GOARCH=amd64
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.Compiler=gc
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.NumCPU=2
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.GOMAXPROCS=2
May 08 14:13:42 centos7.localdomain caddy[29154]: runtime.Version=go1.14.2
May 08 14:13:42 centos7.localdomain caddy[29154]: os.Getwd=/
May 08 14:13:42 centos7.localdomain caddy[29154]: LANG=en_US.UTF-8
May 08 14:13:42 centos7.localdomain caddy[29154]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
May 08 14:13:42 centos7.localdomain caddy[29154]: HOME=/var/lib/caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: LOGNAME=caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: USER=caddy
May 08 14:13:42 centos7.localdomain caddy[29154]: SHELL=/sbin/nologin
May 08 14:13:42 centos7.localdomain caddy[29154]: {"level":"info","ts":1588947222.328688,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 08 14:13:42 centos7.localdomain caddy[29154]: {"level":"info","ts":1588947222.3311186,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
May 08 14:13:42 centos7.localdomain systemd[1]: caddy.service: main process exited, code=exited, status=1/FAILURE
May 08 14:13:42 centos7.localdomain systemd[1]: Unit caddy.service entered failed state.
May 08 14:13:42 centos7.localdomain systemd[1]: caddy.service failed.
caddy validate
validate: decoding config: unexpected end of JSON input

versus when nginx stopped

ngxstop
service caddy restart
service caddy status 
Redirecting to /bin/systemctl status caddy.service
● caddy.service - Caddy
   Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-05-08 14:06:34 UTC; 1s ago
     Docs: https://caddyserver.com/docs/
 Main PID: 28968 (caddy)
   CGroup: /system.slice/caddy.service
           ├─28968 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
           └─28975 sudo -- tee /etc/pki/ca-trust/source/anchors/Caddy_Local_Authority_-_2020_ECC_Root_152059614568658987024318939517890672913.pem

May 08 14:06:34 centos7.localdomain caddy[28968]: {"level":"info","ts":1588946794.1281571,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 08 14:06:34 centos7.localdomain caddy[28968]: {"level":"info","ts":1588946794.1307392,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["[::1]:2019","127.0.0.1:201...localhost:2019"]}
May 08 14:06:34 centos7.localdomain caddy[28968]: 2020/05/08 14:06:34 [INFO][cache:0xc0005e3770] Started certificate maintenance routine
May 08 14:06:34 centos7.localdomain caddy[28968]: {"level":"info","ts":1588946794.1444044,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
May 08 14:06:34 centos7.localdomain caddy[28968]: {"level":"warn","ts":1588946794.1809726,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
May 08 14:06:34 centos7.localdomain caddy[28968]: 2020/05/08 14:06:34 not NSS security databases found
May 08 14:06:34 centos7.localdomain caddy[28968]: 2020/05/08 14:06:34 define JAVA_HOME environment variable to use the Java trust
May 08 14:06:34 centos7.localdomain sudo[28975]: pam_unix(sudo:auth): conversation failed
May 08 14:06:34 centos7.localdomain sudo[28975]: pam_unix(sudo:auth): auth could not identify password for [caddy]
May 08 14:06:34 centos7.localdomain sudo[28975]: pam_succeed_if(sudo:auth): requirement "uid >= 1000" not met by user "caddy"
Hint: Some lines were ellipsized, use -l to show in full.
caddy validate       
validate: decoding config: unexpected end of JSON input

however, at least the domains in Caddyfile resolve

curl -Ik http://caddy.domain.com:81/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Powered-By: caddy centminmod
Date: Fri, 08 May 2020 14:08:22 GMT
curl -Ik https://caddy.domain.com:4444/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Powered-By: caddy centminmod
Date: Fri, 08 May 2020 14:08:09 GMT
curl -Ikv https://caddy.domain.com:4444/caddy-index.html
* About to connect() to caddy.domain.com port 4444 (#0)
*   Trying 192.168.0.18...
* Connected to caddy.domain.com (192.168.0.18) port 4444 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: (nil)
*       start date: May 08 13:52:25 2020 GMT
*       expire date: May 09 01:53:25 2020 GMT
*       common name: (nil)
*       issuer: CN=Caddy Local Authority - ECC Intermediate
> HEAD /caddy-index.html HTTP/1.1
> User-Agent: curl/7.29.0
> Host: caddy.domain.com:4444
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 12226
Content-Length: 12226
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Etag: "q9xapl9fm"
Etag: "q9xapl9fm"
< Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
< Server: Caddy
Server: Caddy
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Powered-By: caddy centminmod
X-Powered-By: caddy centminmod
< Date: Fri, 08 May 2020 14:18:44 GMT
Date: Fri, 08 May 2020 14:18:44 GMT

< 
* Connection #0 to host caddy.domain.com left intact

So question is what is correct way to setup local dev HTTPS SSL certificates for HTTPS site on a port other than 443 i.e. 4444 while still allowing my Nginx server to run on port 80 and 443 ? As it seems only way to even test Caddy on custom port 81/4444 is to stop Nginx.

Example h2load test

test Caddy on port 4444 need to stop Nginx

sleep 120; ngxstop; kill -9 $(pidof caddy); service caddy start; sleep 10
curl -Ik https://caddy.domain.com:4444/caddy-index.html
h2load -t1 -c500 -n2000 -m100 -H "Accept-Encoding:gzip" https://caddy.domain.com:4444/caddy-index.html

curl -Ik https://caddy.domain.com:4444/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 15:10:15 GMT

h2load -t1 -c500 -n2000 -m100 -H "Accept-Encoding:gzip" https://caddy.domain.com:4444/caddy-index.html
starting benchmark...
spawning thread #0: 500 total client(s). 2000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.08s, 649.68 req/s, 3.31MB/s
requests: 2000 total, 2000 started, 2000 done, 2000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10.18MB (10669759) total, 105.72KB (108259) headers (space savings 81.40%), 9.99MB (10480000) data
                     min         max         mean         sd        +/- sd
time for request:    49.99ms       1.99s       1.08s    501.20ms    55.00%
time for connect:   668.11ms       1.16s       1.04s    116.78ms    90.80%
time to 1st byte:      1.09s       3.01s       2.10s    565.89ms    59.40%
req/s           :       1.31        3.67        2.05        0.66    72.60%

test Nginx on port 443 stop Caddy

sleep 120; echo; service caddy stop; kill -9 $(pidof caddy); ngxstart; sleep 10
curl -Ik https://ngx.domain.com/caddy-index.html
h2load -t1 -c500 -n2000 -m100 -H "Accept-Encoding:gzip" https://ngx.domain.com/caddy-index.html

curl -Ik https://ngx.domain.com/caddy-index.html
HTTP/1.1 200 OK
Date: Fri, 08 May 2020 15:12:28 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12226
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5eb30579-2fc2"
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Accept-Ranges: bytes

h2load -t1 -c500 -n2000 -m100 -H "Accept-Encoding:gzip" https://ngx.domain.com/caddy-index.html
starting benchmark...
spawning thread #0: 500 total client(s). 2000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.30s, 1535.31 req/s, 7.82MB/s
requests: 2000 total, 2000 started, 2000 done, 2000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10.19MB (10682500) total, 404.30KB (414000) headers (space savings 26.86%), 9.74MB (10208000) data
                     min         max         mean         sd        +/- sd
time for request:   159.91ms    617.56ms    295.12ms    100.07ms    63.60%
time for connect:   177.77ms       1.10s    856.74ms    167.31ms    64.20%
time to 1st byte:   795.31ms       1.26s       1.15s     71.78ms    68.20%
req/s           :       3.17        5.03        3.48        0.23    78.20%

Thanks

The HTTP and HTTPS ports are 80 and 443, respectively. Caddy has to use those to solve the ACME challenges and set up redirects.

But you can change which ports it uses internally as the HTTP and HTTPS ports (obviously, this won’t change which ports clients use for them, since you can’t change that): Global options (Caddyfile) — Caddy Documentation - look for the http_port and https_port options.

1 Like

Sweet thanks @matt that fixed it

{
    http_port   81
    https_port  4444
}
curl -Ik https://caddy.domain.com:4444/caddy-index.html                                               
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 16:37:13 GMT

curl -Ik https://ngx.domain.com/caddy-index.html                                               
HTTP/1.1 200 OK
Date: Fri, 08 May 2020 16:37:25 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 12226
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5eb30579-2fc2"
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Accept-Ranges: bytes
1 Like

Awesome! Thank you for taking the initiative to figure it out from the docs :slight_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.