Caddy behind Caddy reverse_proxy?

Help
1

1. Caddy version (caddy version):

v2.3.0

2. How I run Caddy:

As a systemd service on the main server, and as a FreeBSD service on my computer

a. System environment:

Main server: Archlinux systemd service, my computer: FreeBSD 13.0 service

b. Command:

none

c. Service/unit/compose file:

caddy.service:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s

LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

caddy FreeBSD service file:

#!/bin/sh

# PROVIDE: caddy
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

# To enable caddy, add 'caddy_enable="YES"' to /etc/rc.conf or
# /etc/rc.conf.local

# Optional settings:
# caddy_config (string):      Full path to caddy config file
#                             (/usr/local/etc/caddy/Caddyfile)
# caddy_adapter (string):     Config adapter type (caddyfile)
# caddy_directory (string):   Root for caddy storage (ACME certs, etc.)
#                             (/var/db/caddy)
# caddy_extra_flags (string): Extra flags passed to caddy start
# caddy_logdir (string):      Where caddy logs are stored
#                             (/var/log/caddy)
# caddy_logfile (string):     Location of process log (${caddy_logdir}/caddy.log)
#                             This is for startup/shutdown/error messages.
#                             To create an access log, see:
#                             https://caddyserver.com/docs/caddyfile/directives/log
# caddy_user (user):          User to run caddy (root)
# caddy_group (group):        Group to run caddy (wheel)
#
# This script will honor XDG_CONFIG_HOME/XDG_DATA_HOME. Caddy will create a
# .../caddy subdir in each of those. By default, they are subdirs of /var/db/caddy.
# See https://caddyserver.com/docs/conventions#data-directory

. /etc/rc.subr

name=caddy
rcvar=caddy_enable
desc="Powerful, enterprise-ready, open source web server with automatic HTTPS written in Go"

load_rc_config $name

# Defaults
: ${caddy_enable:=NO}
: ${caddy_adapter:=caddyfile}
: ${caddy_config:=/usr/local/etc/caddy/Caddyfile}
: ${caddy_directory:=/var/db/caddy}
: ${caddy_extra_flags:=""}
: ${caddy_logdir:="/var/log/${name}"}
: ${caddy_logfile:="${caddy_logdir}/${name}.log"}
: ${caddy_user:="root"}
: ${caddy_group:="wheel"}

# Config and base directories
: ${XDG_CONFIG_HOME:="${caddy_directory}/config"}
: ${XDG_DATA_HOME:="${caddy_directory}/data"}
export XDG_CONFIG_HOME XDG_DATA_HOME

command="/usr/local/bin/${name}"
caddy_flags="--config ${caddy_config} --adapter ${caddy_adapter}"
pidfile="/var/run/${name}/${name}.pid"

required_files="${caddy_config} ${command}"

start_precmd="caddy_precmd"
start_cmd="caddy_start"
stop_cmd="caddy_stop"

# Extra Commands
extra_commands="configtest reload"
configtest_cmd="caddy_command validate ${caddy_flags}"
reload_cmd="caddy_command reload ${caddy_flags}"

caddy_command()
{
        /usr/bin/su -m "${caddy_user}" -c "${command} $*"
}

caddy_precmd()
{
        # Create required directories and set permissions
        /usr/bin/install -d -m 755 -o "${caddy_user}" -g "${caddy_group}" ${caddy_directory}
        /usr/bin/install -d -m 700 -o "${caddy_user}" -g "${caddy_group}" ${caddy_directory}/config
        /usr/bin/install -d -m 700 -o "${caddy_user}" -g "${caddy_group}" ${caddy_directory}/data
        /usr/bin/install -d -m 755 -o "${caddy_user}" -g "${caddy_group}" ${caddy_logdir}
        /usr/bin/install -d -m 700 -o "${caddy_user}" -g "${caddy_group}" /var/run/caddy
}

caddy_start()
{
        echo -n "Starting caddy... "
        /usr/bin/su -m ${caddy_user} -c "${command} start ${caddy_flags} \
                ${caddy_extra_flags} --pidfile ${pidfile}" >> ${caddy_logfile} 2>&1
        if [ $? -eq 0 ] && ps -ax -o pid | grep -q "$(cat ${pidfile})"; then
                echo "done"
                echo "Log: ${caddy_logfile}"
        else
                echo "Error: Caddy failed to start"
                echo "Check the caddy log: ${caddy_logfile}"
        fi
}

caddy_stop()
{
        echo -n "Stopping caddy... "
        if caddy_command stop; then
                echo "done"
        else
                echo "Error: Unable to stop caddy"
                echo "Check the caddy log: ${caddy_logfile}"
        fi
}

run_rc_command "$1"

d. My complete Caddyfile or JSON config:

Server:

daniel.dev.talent.careers {
        reverse_proxy http://localhost:11005 {
        }
}

Behind the proxy:

{
        auto_https off
}

localhost, lvh.me, daniel.dev.talent.careers {
        tls /etc/ssl/localcerts/bundle.crt /etc/ssl/localcerts/server.key
        root * /opt/sompani/web
        file_server
        php_fastcgi * unix//var/run/php-fpm.sock {
                index "app_dev.php"
        }
}

3. The problem I’m having:

I am trying to use Caddy behind a reverse_proxy from caddy. I do a reverse port forward using ssh -N -R 11005:localhost:80 zilti@sompani-live and want to use that to proxy the traffic to my local machine.

But that doesn’t work. When I just forward it to the local machine, caddy ends up in a redirect loop because the caddy behind the proxy tries to forward to https. When I disable https redirects on the local machine, caddy will simply reply with a 502 and claims it couldn’t connect to the reverse proxy.

4. Error messages and/or full log output:

Apr 20 12:41:25 sompani.com caddy[16801]: {"level":"error","ts":1618915285.6971805,"logger":"http.log.error","msg":"read tcp [::1]:50442->[::1]:11005: read: connection reset by peer","request":{"remote_addr":"[2a02:8109:86c0:1ed8::aebf]:36750","proto":"HTTP/2.0","method":"GET","host":"daniel.dev.talent.careers","uri":"/favicon.ico","headers":{"Cache-Control":["no-cache"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Referer":["https://daniel.dev.talent.careers/"],"Sec-Gpc":["1"],"Pragma":["no-cache"],"Accept":["image/avif,image/webp,*/*"],"Accept-Language":["de-CH"],"Authorization":["Basic c29tcGFuaTppc3JlYWR5"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"daniel.dev.talent.careers"}},"duration":0.024448411,"status":502,"err_id":"4qtrs1zcn","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
Apr 20 12:42:54 sompani.com caddy[16801]: {"level":"error","ts":1618915374.5730531,"logger":"http.log.error","msg":"read tcp [::1]:50668->[::1]:11005: read: connection reset by peer","request":{"remote_addr":"[2a02:8109:86c0:1ed8::aebf]:36750","proto":"HTTP/2.0","method":"GET","host":"daniel.dev.talent.careers","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["de-CH"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Authorization":["Basic c29tcGFuaTppc3JlYWR5"],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"],"Te":["trailers"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"daniel.dev.talent.careers"}},"duration":0.03086192,"status":502,"err_id":"5ymf54frk","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
Apr 20 12:42:55 sompani.com caddy[16801]: {"level":"error","ts":1618915375.096938,"logger":"http.log.error","msg":"read tcp [::1]:50670->[::1]:11005: read: connection reset by peer","request":{"remote_addr":"[2a02:8109:86c0:1ed8::aebf]:36750","proto":"HTTP/2.0","method":"GET","host":"daniel.dev.talent.careers","uri":"/favicon.ico","headers":{"Sec-Gpc":["1"],"Cache-Control":["no-cache"],"Te":["trailers"],"Accept":["image/avif,image/webp,*/*"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://daniel.dev.talent.careers/"],"Authorization":["Basic c29tcGFuaTppc3JlYWR5"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"],"Accept-Language":["de-CH"],"Dnt":["1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"daniel.dev.talent.careers"}},"duration":0.026124128,"status":502,"err_id":"pd77cc4qr","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
Apr 20 12:43:30 sompani.com caddy[16801]: {"level":"error","ts":1618915410.0436592,"logger":"http.log.error","msg":"read tcp [::1]:50764->[::1]:11005: read: connection reset by peer","request":{"remote_addr":"[2a02:8109:86c0:1ed8::aebf]:36750","proto":"HTTP/2.0","method":"GET","host":"daniel.dev.talent.careers","uri":"/api.html","headers":{"Accept-Language":["de-CH"],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"],"Pragma":["no-cache"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"],"Dnt":["1"],"Authorization":["Basic c29tcGFuaTppc3JlYWR5"],"Cache-Control":["no-cache"],"Te":["trailers"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"daniel.dev.talent.careers"}},"duration":0.030108041,"status":502,"err_id":"q8s2ccy1c","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}
Apr 20 12:43:30 sompani.com caddy[16801]: {"level":"error","ts":1618915410.478982,"logger":"http.log.error","msg":"read tcp [::1]:50766->[::1]:11005: read: connection reset by peer","request":{"remote_addr":"[2a02:8109:86c0:1ed8::aebf]:36750","proto":"HTTP/2.0","method":"GET","host":"daniel.dev.talent.careers","uri":"/favicon.ico","headers":{"Cache-Control":["no-cache"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"],"Dnt":["1"],"Authorization":["Basic c29tcGFuaTppc3JlYWR5"],"Referer":["https://daniel.dev.talent.careers/api.html"],"Pragma":["no-cache"],"Accept":["image/avif,image/webp,*/*"],"Accept-Language":["de-CH"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Gpc":["1"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"daniel.dev.talent.careers"}},"duration":0.026271992,"status":502,"err_id":"9d5ig4qgb","err_trace":"reverseproxy.statusError (reverseproxy.go:783)"}

5. What I already tried:

See above.

6. Links to relevant resources:

My last-ditch effort was to follow this: 🕳 Tunneling Local Applications with SSH and Caddy - Bruno Scheufler

2

Caddy will default to listening on port 443 even if you turn off auto_https, but you’re trying to proxy to it over HTTP (port 80).

Either proxy over HTTPS (port 443), (and ideally configure trust by using the tls_trusted_ca_certs option of reverse_proxy (Caddyfile directive) — Caddy Documentation), or explicitly tell your upstream Caddy to listen for HTTP by prefixing the site address with http://.

1 Like
3

For those getting here via online searches, yes, proxying via HTTPS was the solution for me. Here is my proxy config:

daniel.dev.talent.careers {
        reverse_proxy https://localhost:11005 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

Of course, be careful with the tls_insecure_skip_verify, don’t use in production environments, yaddayadda, you know the drill :slight_smile:

4

This topic was automatically closed after 30 days. New replies are no longer allowed.