V2: How to test Caddyfile etc. before deploying to production?

skyfaller · April 9, 2020, 3:39pm

1. My Caddy version (`caddy version`):

v2.0.0-rc.2 h1:7NOaxYrsnQ5D3rcLGn2UnLZHLfBpgrfM/JNTLhjCJ1c=

2. How I run Caddy:

a. System environment:

Arch Linux and/or Ubuntu 19.10 eoan on Linode “nanode” virtual machines. Possibly macOS in the future if I have to test locally? Systemd, no containers yet.

b. Command:

/usr/local/bin/caddy run --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

[Unit]
Description=Caddy v2 web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service

[Service]
#Restart=on-abnormal

; Do not allow the process to be restarted in a tight loop. If the
; process fails to start, something critical needs to be fixed.
;StartLimitIntervalSec=14400
;StartLimitBurst=10

; User and group the process will run as.
User={{ web_server_user }}
Group={{ web_server_user }}

; caddy command assumes the caddyfile adapter if filename starts with Caddyfile
ExecStart=/usr/local/bin/caddy run --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile

; Use graceful shutdown with a reasonable timeout
ExecStop=/usr/local/bin/caddy stop
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=4096

; The following additional security directives only work with systemd v229 or later.
; They further restrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

# global options block
{
	storage file_system {
		root	/etc/caddy/storage
	}
	experimental_http3
	email 	{{ admin_email }}
}

# reusable snippets
(boilerplate) {
	encode gzip zstd
	file_server
}

# start site blocks

# test page
www.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/www/public/
#	try_files {path}.html {path}
	import boilerplate
}

# redirect no-www to www
sunrisemovement.dev {
	redir https://www.sunrisemovement.dev
}

# handcoded Rhode Island site
ri.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/ri/public
	import boilerplate
	php_fastcgi unix//var/run/php/php7.3-fpm.sock
}

# handcoded SF Bay website (originally Github Pages)
sfbay.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/sfbay/public
	import boilerplate
}
sfbay.sunrisemovement.org {
	root * /srv/sunrisemovement.org/sfbay/public
	import boilerplate
}

philadelphia.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/philadelphia/public
	import boilerplate
}

3. The problem I’m having:

I need to test my Caddyfile and other aspects of my environment before deploying changes to production.

I understand that if I changed my Caddyfile to use IP addresses instead of domain names, Caddy would generate a local CA and provide certificates on the local machine. I also understand that I could disable HTTPS and serve HTTP.

Unfortunately, neither option is very attractive. Some web apps such as Wordpress flip out if you try to serve them from a different URL than they expect. Even if my websites weren’t such precious fragile children, I would still want to test my production Caddyfile with as few modifications as possible (ideally zero changes), to reduce the odds that I’ll discover new problems when I deploy to production. I do not want to use naked IP addresses or unencrypted HTTP.

Are there any other (better) options for testing my configuration before deploying? Before HTTPS, I would have tested by setting my /etc/hosts file to fool my computer into thinking my test server owned the relevant domains so it would serve up my site as normal, but clearly that won’t work with Let’s Encrypt.

skyfaller · April 9, 2020, 11:41pm

OK, my apologies, it appears I didn’t read the documentation closely enough. It looks like I can set individual sites to use the local ca with the “tls internal” Caddyfile directive, or set the whole Caddyfile to use the local ca with the global option “local_certs”.

If I want to have every site except one use the local ca, what’s the best way to do that? Set the local_certs global option, and then for the exception… ? Would "tls <email>" switch back on Let’s Encrypt for the one functional site?

And then if I am testing “locally” on a remote server… how do I view the website? I can’t think of anything except installing a browser on the server and X forwarding.

matt · April 10, 2020, 12:33am

Great question, Nelson!

You can serve them at the same URL, but you’ll have to trick/convince your system to resolve a public/production address to your local machine. (This is usually just a matter of editing the hosts file.)

Of course, then you have to tell Caddy to use its own internal CA (as it looks like you discovered), because the actual production domain name won’t point to your dev machine. You can do that, as you found, with the local_certs global option.

Normally, this hassle is not necessary; should only be needed if services will not allow the URL to be anything else. For example, in the basic reverse proxying scenario you could just replace the domain name in your Caddyfile with localhost and bam, it’ll work over HTTPS in dev. But if you can’t do that, you’ll have to:

Update your hosts file to resolve the production domain to your loopback
Tell Caddy to use the local issuer (either through local_certs global option, easier; or tls internal per-site)

Yes, it does. Try it with caddy adapt | jq and you’ll see the automation policies:

Caddyfile:

{
	local_certs
}

1.example.com {
}

2.example.com {
	tls foo@bar.com
}

Adapted JSON:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "1.example.com"
                  ]
                }
              ],
              "terminal": true
            },
            {
              "match": [
                {
                  "host": [
                    "2.example.com"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [
              "2.example.com"
            ],
            "issuer": {
              "email": "foo@bar.com",
              "module": "acme"
            }
          },
          {
            "issuer": {
              "module": "internal"
            }
          }
        ]
      }
    }
  }
}

See how that works? It should do what you expect.

You don’t have to serve the site just locally, you can open it up to any network interface you want.

skyfaller · April 10, 2020, 11:01pm

Maybe this should be a new thread or a bug report, but I’m having trouble getting the local CA working. Maybe this just needs more documentation so I know what dependencies the local CA has?

Here’s my current Caddyfile for the test server running Arch Linux:

# global options block
{
	storage file_system {
		root	/etc/caddy/storage
	}
	experimental_http3
	local_certs
}

# reusable snippets
(boilerplate) {
	encode gzip zstd
	file_server
}

# start site blocks

# public test page
brockovich.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/www/public/
	import boilerplate
	tls	{{ admin_email }}
}

# local test page
www.sunrisemovement.dev {
	root * /srv/sunrisemovement.dev/www/public/
	import boilerplate
}
# redirect no-www to www
sunrisemovement.dev {
	redir https://www.sunrisemovement.dev
}

The public site using Let’s Encrypt is still functional. However, the local CA is not doing well. First, it complained I didn’t have certutil / nss installed:

Apr 10 16:40:45 brockovich systemd[1]: Reloading Caddy v2 web server.
Apr 10 16:40:45 brockovich caddy[142957]: {"level":"info","ts":1586551245.1005006,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.1050384,"logger":"admin.api","msg":"received request","method":"POST","uri":"/load","remote_addr":"127.0.0.1:44560","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1167"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.1059146,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["localhost:2019"]}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.1086466,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.1089435,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 [INFO][cache:0xc000788a50] Started certificate maintenance routine
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"warn","ts":1586551245.3589947,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 define JAVA_HOME environment variable to use the Java trust
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"error","ts":1586551245.3625145,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.3633766,"logger":"tls","msg":"cleaned up storage units"}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.3635137,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 [DEBUG] udp/:443: Usage counter should not go above 2 or maybe 3, is now: 2
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.363721,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["brockovich.sunrisemovement.dev","www.sunrisemovement.dev","sunrisemovement.dev"]}
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 [DEBUG] Fake-closing underlying packet conn
Apr 10 16:40:45 brockovich caddy[139462]: 2020/04/10 16:40:45 [INFO][cache:0xc000788410] Stopped certificate maintenance routine
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.3648434,"msg":"autosaved config","file":"/srv/http/.config/caddy/autosave.json"}
Apr 10 16:40:45 brockovich caddy[139462]: {"level":"info","ts":1586551245.3649354,"logger":"admin.api","msg":"load complete"}
Apr 10 16:40:45 brockovich systemd[1]: Reloaded Caddy v2 web server.

So I “fixed” that by installing the nss Arch package. Unfortunately, Caddy is still unhappy, and I don’t fully understand the error messages:

Apr 10 18:42:55 brockovich systemd[1]: Stopped Caddy v2 web server.
Apr 10 18:42:55 brockovich systemd[1]: Started Caddy v2 web server.
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.306561,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.3166182,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["localhost:2019"]}
Apr 10 18:42:55 brockovich caddy[144238]: 2020/04/10 18:42:55 [INFO][cache:0xc00069e9b0] Started certificate maintenance routine
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.3389246,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.3391316,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"warn","ts":1586558575.5540116,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 10 18:42:55 brockovich caddy[144238]: 2020/04/10 18:42:55 define JAVA_HOME environment variable to use the Java trust
Apr 10 18:42:55 brockovich caddy[144238]: 2020/04/10 18:42:55 not NSS security databases found
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"error","ts":1586558575.561442,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.5627358,"logger":"tls","msg":"cleaned up storage units"}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.5630877,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.5633273,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["brockovich.sunrisemovement.dev","www.sunrisemovement.dev","sunrisemovement.dev"]}
Apr 10 18:42:55 brockovich caddy[144238]: 2020/04/10 18:42:55 [WARNING] Stapling OCSP: no OCSP stapling for [www.sunrisemovement.dev]: no OCSP server specified in certificate
Apr 10 18:42:55 brockovich caddy[144238]: 2020/04/10 18:42:55 [WARNING] Stapling OCSP: no OCSP stapling for [sunrisemovement.dev]: no OCSP server specified in certificate
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.5698445,"msg":"autosaved config","file":"/srv/http/.config/caddy/autosave.json"}
Apr 10 18:42:55 brockovich caddy[144238]: {"level":"info","ts":1586558575.5700452,"msg":"serving initial configuration"}
Apr 10 18:43:36 brockovich caddy[144238]: 2020/04/10 18:43:36 http2: panic serving [2601:42:0:6200:d111:db58:1bc9:319]:58335: runtime error: invalid memory address or nil pointer dereference
Apr 10 18:43:36 brockovich caddy[144238]: goroutine 35 [running]:
Apr 10 18:43:36 brockovich caddy[144238]: net/http.(*http2serverConn).runHandler.func1(0xc00027a3c0, 0xc000395f8e, 0xc0002b1380)
Apr 10 18:43:36 brockovich caddy[144238]:         net/http/h2_bundle.go:5713 +0x16b
Apr 10 18:43:36 brockovich caddy[144238]: panic(0x144d380, 0x2470800)
Apr 10 18:43:36 brockovich caddy[144238]:         runtime/panic.go:969 +0x166
Apr 10 18:43:36 brockovich caddy[144238]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc00069aea0, 0x192e1e0, 0xc00027a3c0, 0xc0004d3c00)
Apr 10 18:43:36 brockovich caddy[144238]:         github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
Apr 10 18:43:36 brockovich caddy[144238]: net/http.serverHandler.ServeHTTP(0xc0009deb60, 0x192e1e0, 0xc00027a3c0, 0xc0004d2d00)
Apr 10 18:43:36 brockovich caddy[144238]:         net/http/server.go:2807 +0xa3
Apr 10 18:43:36 brockovich caddy[144238]: net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc000903b30, 0xc000674e00, 0xc0009deb60, 0x192e1e0, 0xc00027a3c0, 0xc0004d2d00)
Apr 10 18:43:36 brockovich caddy[144238]:         net/http/server.go:3381 +0x8d
Apr 10 18:43:36 brockovich caddy[144238]: net/http.(*http2serverConn).runHandler(0xc0002b1380, 0xc00027a3c0, 0xc0004d2d00, 0xc0007e27e0)
Apr 10 18:43:36 brockovich caddy[144238]:         net/http/h2_bundle.go:5720 +0x8b
Apr 10 18:43:36 brockovich caddy[144238]: created by net/http.(*http2serverConn).processHeaders
Apr 10 18:43:36 brockovich caddy[144238]:         net/http/h2_bundle.go:5454 +0x4e1
Apr 10 18:43:37 brockovich caddy[144238]: 2020/04/10 18:43:37 http2: panic serving [2601:42:0:6200:d111:db58:1bc9:319]:58336: runtime error: invalid memory address or nil pointer dereference
Apr 10 18:43:37 brockovich caddy[144238]: goroutine 49 [running]:
Apr 10 18:43:37 brockovich caddy[144238]: net/http.(*http2serverConn).runHandler.func1(0xc00027a570, 0xc000395f8e, 0xc000001800)
Apr 10 18:43:37 brockovich caddy[144238]:         net/http/h2_bundle.go:5713 +0x16b
Apr 10 18:43:37 brockovich caddy[144238]: panic(0x144d380, 0x2470800)
Apr 10 18:43:37 brockovich caddy[144238]:         runtime/panic.go:969 +0x166
Apr 10 18:43:37 brockovich caddy[144238]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc00069aea0, 0x192e1e0, 0xc00027a570, 0xc0000cd700)
Apr 10 18:43:37 brockovich caddy[144238]:         github.com/caddyserver/caddy/v2@v2.0.0-rc.2/modules/caddyhttp/server.go:203 +0x932
Apr 10 18:43:37 brockovich caddy[144238]: net/http.serverHandler.ServeHTTP(0xc0009deb60, 0x192e1e0, 0xc00027a570, 0xc0000ccd00)
Apr 10 18:43:37 brockovich caddy[144238]:         net/http/server.go:2807 +0xa3
Apr 10 18:43:37 brockovich caddy[144238]: net/http.initALPNRequest.ServeHTTP(0x19335a0, 0xc0002c1740, 0xc000675180, 0xc0009deb60, 0x192e1e0, 0xc00027a570, 0xc0000ccd00)
Apr 10 18:43:37 brockovich caddy[144238]:         net/http/server.go:3381 +0x8d
Apr 10 18:43:37 brockovich caddy[144238]: net/http.(*http2serverConn).runHandler(0xc000001800, 0xc00027a570, 0xc0000ccd00, 0xc0007e3480)
Apr 10 18:43:37 brockovich caddy[144238]:         net/http/h2_bundle.go:5720 +0x8b
Apr 10 18:43:37 brockovich caddy[144238]: created by net/http.(*http2serverConn).processHeaders
Apr 10 18:43:37 brockovich caddy[144238]:         net/http/h2_bundle.go:5454 +0x4e1

The money quotes appear to be “define JAVA_HOME environment variable to use the Java trust” and “not NSS security databases found”, but I don’t know where to start with those. Any ideas?

matt · April 10, 2020, 11:21pm

The errors about JAVA_HOME and certutil just means that since you don’t have Java or Firefox installed it won’t install trust into those stores. It should have still installed into your system trust store. You don’t need to install Java, Firefox, or nss libraries for system-level trust.

As for the panic, this has been fixed already; try building the latest from master or download CI artifact

skyfaller · April 11, 2020, 3:35am

Awesome, I’ve got it working! I installed the latest from master and the panic is gone. I pointed the domain to my test server using /etc/hosts on my laptop, and then followed the directions at CA/Changing Trust Settings - MozillaWiki to install the Caddy local CA cert in my Firefox dev browser. I was then able to load the website as normal. That was just one test website, but I feel confident that I can figure out the others now.

matt · April 11, 2020, 4:05am

Awesome! That’s weird that it didn’t install into the Firefox trust store automatically. I wonder why…

skyfaller · April 11, 2020, 4:07am

Oh, it’s because Caddy is running on the remote Arch Linux server, and Firefox is running on my truly local Macbook Pro. So Caddy had no way to do its automatic install thing on a different computer If there is any way to automate such a setup, I would be very interested, though!

matt · April 11, 2020, 4:17am

Oh, that’s right. I forgot about that. hmm yeah maybe someday. but for now, that will have to do!

system · May 11, 2020, 4:30am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.