Caddy time (zone)

[Rob789 (Robbert)]

1. Caddy version (caddy version):

v2.4.0-beta.1

2. How I run Caddy:

a. System environment:

Debian 10

b. Command:

caddy run

3. The problem I’m having:

When looking at the logs, I notice that the time in Caddy is 1 hour different than the syetem time. I think this is causing the local certificate to be invalid.

I did not have this error a few days ago and I can’t get my head around on what I could have changed to have this mismatch. The system time(zone) is correct.

4. Error messages and/or full log output:

2021/03/07 16:28:59.735 ERROR http.log.error x509: certificate has expired or is not yet valid: current time 2021-03-07T17:28:59+01:00 is after 2021-03-05T03:52:06Z {“request”: {“remote_addr”: “192.168.5.1:4954”, “proto”: “HTTP/2.0”, “method”: “POST”, “host”: “bpass.intrafit.nl”, “uri”: “/identity/connect/token”, “headers”: {“Accept-Language”: [“en-GB,en;q=0.5”], “Accept-Encoding”: [“gzip, deflate, br”], “Device-Type”: [“3”], “Origin”: [“moz-extension://fa1d4d8a-3175-4671-9c06-8150bf64cb83”], “Content-Length”: [“151”], “Pragma”: [“no-cache”], “Te”: [“trailers”], “User-Agent”: [“Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0”], “Content-Type”: [“application/x-www-form-urlencoded; charset=utf-8”], “Cache-Control”: [“no-cache”], “Accept”: [“application/json”]}, “tls”: {“resumed”: false, “version”: 772, “cipher_suite”: 4865, “proto”: “h2”, “proto_mutual”: true, “server_name”: “bpass.intrafit.nl”}}, “duration”: 0.018063684, “status”: 502, “err_id”: “70h04ts6z”, “err_trace”: “reverseproxy.statusError (reverseproxy.go:783)”}

5. What I already tried:

Athough I’m pretty sure I had this working OK with the latest Caddy, I went back to Caddy 2.3.0 without any noticable change.

[francislavoie]

What user are you using to run Caddy? You can check if there’s a TZ environment variable set for that user. Running with the --environ flag will have Caddy print its environment at startup.

[Rob789 (Robbert)]

I’m running Caddy directly as root.

I checked the environment and I don’t see any time settings.

root@RJ-CaddyTK ~# caddy run --environ
caddy.HomeDir=/root
caddy.AppDataDir=/root/.local/share/caddy
caddy.AppConfigDir=/root/.config/caddy
caddy.ConfigAutosavePath=/root/.config/caddy/autosave.json
caddy.Version=v2.3.0
runtime.GOOS=linux
runtime.GOARCH=amd64
runtime.Compiler=gc
runtime.NumCPU=1
runtime.GOMAXPROCS=1
runtime.Version=go1.16
os.Getwd=/root

SHELL=/bin/bash
LANGUAGE=en_US.UTF-8
EDITOR=vim
PWD=/root
LOGNAME=root
HOME=/root
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SSH_CONNECTION=192.168.2.200 49384 192.168.2.2 8022
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm
LESSOPEN=| /usr/bin/lesspipe %s
USER=root
SHLVL=1
PAGER=less -X -R -F
LC_CTYPE=C
SSH_CLIENT=192.168.2.200 49384 8022
LC_ALL=C
PATH=/usr/lib/git-core:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAIL=/var/mail/root
SSH_TTY=/dev/pts/0
_=/usr/bin/caddy

[francislavoie]

Maybe this might reveal what’s going on?

How to Set or Change the Time Zone in Linux

[Rob789 (Robbert)]

Well, I changed the timezone (back) to UTC which is the time Caddy uses in the log. So system time date is the same as the timestamps in the Caddy log.

I can’t recal to have changed it but I’ve been tweaking so many systems over the last couple of days that I may have… :-

I still have the same error message about the certificate. Could this be because it has not been regenerated yet? Is there a way to enforce this?

[francislavoie]

You can delete the certificate from Caddy’s storage and it’ll force a re-issue. But that error looks to be that Caddy can’t trust the certificate of your upstream app, not the certificate it’s managing itself on that server. So check your upstream server’s logs, something must’ve happened for it to not renew and end up with an expired cert.

[Rob789 (Robbert)]

I did not see any error message in the upstream Caddy that could give away the issue.

Didn’t think about just deleting the cert

I deleted the certificates for (all) the local hostsnames and restarted Caddy up- and downstream. New certificates were generated and all is working as expected.

No clue what could have caused this issue but I’ll monitor it for a few days before making changes again.