Invalid Character Error When Trying To Setup Reverse Proxy

Help
1

1. Caddy version (caddy version):

v.2.3.0

2. How I run Caddy:

Running Caddy as a systemd service

a. System environment:

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

b. Command:

Managing Caddy using standard systemctl commands

sudo systemctl {start|stop} caddy

c. Service/unit/compose file:

None (at least not that I’m aware of)

d. My complete Caddyfile or JSON config:

# This replaces the existing content in /etc/caddy/Caddyfile
# A CONFIG SECTION FOR YOUR HOSTNAME
lordstrahdsfoundry.ddns.net {
        # PROXY ALL REQUEST TO PORT 30000
        reverse_proxy localhost:30000
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

3. The problem I’m having:

I am running an application server (specifically Foundry VTT) on a Rasberry Pi. External users to the application currently connect via a browser pointed at http://lordstrahdsfoundry(dot)ddns(dot)net:30000. However, the Foundry community suggests configuring a reverse proxy to handle signed SSL certificates and to obfuscate port numbers.
I’ve attempted to get this working with Caddy, but the logs appear to show that Caddy is having some problems with an invalid character (at least, I believe this to be the root problem) and as such is not issuing SSL certificates.

4. Error messages and/or full log output:

Taken from syslog on start of Caddy service

Apr 27 16:54:42 rpi-foundryserver systemd[1]: Started Caddy.
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: caddy.HomeDir=/var/lib/caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: caddy.Version=v2.3.0
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.GOOS=linux
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.GOARCH=arm
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.Compiler=gc
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.NumCPU=4
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.GOMAXPROCS=4
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: runtime.Version=go1.15.6
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: os.Getwd=/
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: LANG=en_GB.UTF-8
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: HOME=/var/lib/caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: LOGNAME=caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: USER=caddy
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: INVOCATION_ID=7e90d9b3d802434f8576dac64ec29964
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: JOURNAL_STREAM=8:45138
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8338163,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8406475,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8412714,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x414b680"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8416348,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8416839,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.842737,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["lordstrahdsfoundry.ddns.net"]}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8434865,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8435326,"msg":"serving initial configuration"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8439078,"logger":"tls","msg":"cleaned up storage units"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"info","ts":1619538882.8443065,"logger":"tls.obtain","msg":"acquiring lock","identifier":"lordstrahdsfoundry.ddns.net"}
Apr 27 16:54:42 rpi-foundryserver caddy[11830]: {"level":"error","ts":1619538882.8445034,"logger":"tls","msg":"job failed","error":"lordstrahdsfoundry.ddns.net: obtaining certificate: decoding lockfile contents: invalid character '\\x00' looking for beginning of value"}

5. What I already tried:

I followed recommended installation instructions for Caddy here.

I followed Caddyfile creation instructions here.

I’ve deleted and recreated the caddyfile, both copy/paste and manually.

I’ve tried grepping for the specific character \x00 using

grep --color='auto' -P -n "[\x00]" /etc/caddy/Caddyfile

but that didn’t return any results.

I’ve gone through the Caddyfile, deleted extra whitespace, and recreated all new lines to no effect.

And as a last attempt to see if I could find any problems, I did hexdump -C on the Caddyfile but that showed no \x00 characters.

I’m now at a loss and any help will be greatly appreciated!

6. Links to relevant resources:

Foundry VTT Knowledge Base incase it’s needed.

2

That’s the first time I’ve seen that. It’s not complaining about your Caddyfile. What are the contents of your Caddy data directory, speficially in the locks folder (and the contents of the files)?

3

Here’s the file tree for the data directory:

/var/lib/caddy/.local/share/caddy/
├── acme
│   ├── acme-staging-v02.api.letsencrypt.org-directory
│   │   ├── challenge_tokens
│   │   └── users
│   │       └── default
│   │           ├── default.json
│   │           └── default.key
│   ├── acme-v02.api.letsencrypt.org-directory
│   │   ├── challenge_tokens
│   │   └── users
│   │       └── default
│   │           ├── default.json
│   │           └── default.key
│   └── acme.zerossl.com-v2-dv90
│       ├── challenge_tokens
│       └── users
│           └── default
│               ├── default.json
│               └── default.key
├── certificates
│   ├── acme-staging-v02.api.letsencrypt.org-directory
│   ├── acme-v02.api.letsencrypt.org-directory
│   └── acme.zerossl.com-v2-dv90
└── locks
    └── issue_cert_lordstrahdsfoundry.ddns.net.lock

The data directory is located in /var/lib/caddy/.local instead of $HOME. Assuming that isn’t an issue, then the contents of the single file in the locks appears, to my eye, to contain garbage unless this is encrypted:

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
4

That’s because you’re running Caddy as a service, which runs Caddy as the caddy user, whose home is /var/lib/caddy.

5

That’s kind of what I figured to be honest, but just felt it was worth mentioning as it deviates from the list of data directory locations on the Conventions page.

6

Technically it doesn’t, because $HOME when running as a service is /var/lib/caddy, because $HOME means “the home of the current user”.

7

@matt
Hi Matt, I was just wondering if you’d got any thoughts about what was going on here? I put the directory and file contents above as you asked for.

8

I don’t know… ^@ reminds me of the terminal’s rendering of a Ctrl+ key combination:

Was there only one time you ran Caddy for that domain, i.e. was this the first run? Does the error reliably happen repeatedly?

If there’s any sort of virtualization or containerization going on, try cutting that out, also any weird file systems or abstractions. Looks like the disk has all 0’s instead of actual data for some reason, but Go didn’t report any errors.

9

Well, I’ve stopped and started the service a few times to see if anything changed, but its always the same error, so it seems pretty repeatable.

About the only thing I haven’t done yet in uninstall Caddy and start clean to see if that fixes it.

If I delete the lock file will Caddy auto-generate a new one?

10

Yeah you definitely should delete it. Clearly Caddy won’t run with it there. But it would be good to figure out how that ever happened. But I’ll accept that it might remain a mystery.

11

OK, well that’s some progress made.

For what it’s worth, I uninstalled Caddy, deleted /var/lib/caddy and re-installed. For some reason, on re-installation, /var/lib/caddy wasn’t created, but a quick mkdir and chown put that back in place.

The contents of the lock file now appears to be normal so that’s resolved the unknown character error, but I’ve now got a host of HTTP 400 connection errors instead. As they are a new problem though, I’ll make a new post for them.

Thanks @matt and @francislavoie for your help.

1 Like
12

This topic was automatically closed after 30 days. New replies are no longer allowed.