1. The problem I’m having:
I have enabled routes to the caddyserver via hyper-v wsl network
and after i enabled those special routes, the irc server will work by vlan ip address and nat works to show that it connects from the windows hyper-v wsl interface, but the gitea instance is 500, or caddy is 500.
let me look at the logs and share what is relevant.
something with these rules broke the webserver, i will debug the process i went through tomorrow and most likely fix it also.
caddy server ip 10.10.0.2
WSL ip 172.18.230.3
what i did on windows
route add 10.10.0.0 mask 255.255.255.0 172.18.230.3
what i did on linux
sudo iptables -A FORWARD -i eth0.141 -o eth0 -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o eth0.141 -j ACCEPT
there are other iptables rules too, but they are just for the
vlan<->bridge<->other-vlan communication
i construct my docker network specifically to stop it from doing all of the stuff docker does in order to forward through the WSL ip. it is below in the B. comand section
cgi.gate.internal is the caddyserver (10.10.0.2) and it’s using the caddy-cgi plugin.
CURL from my windows desktop, that hosts the WSL instance that runs the VLAN and docker networks, seems to work just fine.
Even a CURL on code.gate.internal (the one returning the error below) returns the html code in the terminal, however upon accessing from google chrome, it just spits out the simple error in the error section below.
C:\Users\lucyfurnice>curl --insecure -v https://cgi.gate.internal/info.cgi
* Host cgi.gate.internal:443 was resolved.
* IPv6: (none)
* IPv4: 10.10.0.2
* Trying 10.10.0.2:443...
* schannel: disabled automatic use of client certificate
* ALPN: curl offers http/1.1
* ALPN: server accepted http/1.1
* Connected to cgi.gate.internal (10.10.0.2) port 443
* using HTTP/1.x
> GET /info.cgi HTTP/1.1
> Host: cgi.gate.internal
> User-Agent: curl/8.10.1
> Accept: */*
>
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* Request completely sent off
< HTTP/1.1 200 OK
< Alt-Svc: h3=":443"; ma=2592000
< Content-Type: text/plain
< Server: Caddy
< Date: Wed, 08 Jan 2025 21:10:44 GMT
< Content-Length: 529
<
AUTH_TYPE []
CONTENT_LENGTH []
CONTENT_TYPE []
GATEWAY_INTERFACE [CGI/1.1]
PATH_INFO []
PATH_TRANSLATED []
POST_DATA []
QUERY_STRING []
REMOTE_ADDR [172.18.224.1]
REMOTE_HOST [172.18.224.1]
REMOTE_IDENT []
REMOTE_USER []
REQUEST_METHOD [GET]
SCRIPT_EXEC [/var/www/cgi-bin/info.sh]
SCRIPT_EXEC [someargument]
SCRIPT_NAME [/info.cgi]
SERVER_NAME [cgi.gate.internal]
SERVER_PORT [443]
SERVER_PROTOCOL [HTTP/1.1]
SERVER_SOFTWARE [go]
* Connection #0 to host cgi.gate.internal left intact
caddy logs show that within the wsl network(s) the gitea actions runner can reach the gitea server @ 10.10.0.3
{"level":"debug","ts":1736371737.859761,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.0.3:3000","total_upstreams":1}
{"level":"debug","ts":1736371737.8702698,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.0.3:3000","duration":0.010419485,"request":{"remote_ip":"10.10.0.4","remote_port":"49836","client_ip":"10.10.0.4","proto":"HTTP/1.1","method":"POST","host":"code.gate.internal","uri":"/api/actions/runner.v1.RunnerService/FetchTask","headers":{"Connect-Protocol-Version":["1"],"X-Forwarded-For":["10.10.0.4"],"X-Forwarded-Host":["code.gate.internal"],"User-Agent":["connect-go/1.16.2 (go1.23.1)"],"Connect-Timeout-Ms":["4999"],"X-Runner-Token":["98607dce5bce1ff32f2df806965f1f6076e93ea5"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"X-Runner-Uuid":["b88d3e97-395a-403a-b4f1-e428ab19b711"],"X-Runner-Version":["v0.2.11"],"Content-Length":["2"],"Content-Type":["application/proto"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"code.gate.internal"}},"headers":{"Accept-Encoding":["gzip"],"Content-Type":["application/proto"],"Date":["Wed, 08 Jan 2025 21:28:57 GMT"],"Content-Length":["2"]},"status":200}
{"level":"debug","ts":1736371739.858961,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.0.3:3000","total_upstreams":1}
{"level":"debug","ts":1736371739.8680327,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.0.3:3000","duration":0.008979797,"request":{"remote_ip":"10.10.0.4","remote_port":"49836","client_ip":"10.10.0.4","proto":"HTTP/1.1","method":"POST","host":"code.gate.internal","uri":"/api/actions/runner.v1.RunnerService/FetchTask","headers":{"User-Agent":["connect-go/1.16.2 (go1.23.1)"],"X-Forwarded-For":["10.10.0.4"],"Accept-Encoding":["gzip"],"Connect-Protocol-Version":["1"],"Connect-Timeout-Ms":["4999"],"Content-Type":["application/proto"],"X-Runner-Version":["v0.2.11"],"Content-Length":["2"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["code.gate.internal"],"X-Runner-Token":["98607dce5bce1ff32f2df806965f1f6076e93ea5"],"X-Runner-Uuid":["b88d3e97-395a-403a-b4f1-e428ab19b711"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"code.gate.internal"}},"headers":{"Content-Type":["application/proto"],"Date":["Wed, 08 Jan 2025 21:28:59 GMT"],"Content-Length":["2"],"Accept-Encoding":["gzip"]},"status":200}
{"level":"debug","ts":1736371741.8586657,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.0.3:3000","total_upstreams":1}
{"level":"debug","ts":1736371741.8674862,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.0.3:3000","duration":0.008710989,"request":{"remote_ip":"10.10.0.4","remote_port":"49836","client_ip":"10.10.0.4","proto":"HTTP/1.1","method":"POST","host":"code.gate.internal","uri":"/api/actions/runner.v1.RunnerService/FetchTask","headers":{"X-Runner-Token":["98607dce5bce1ff32f2df806965f1f6076e93ea5"],"Content-Length":["2"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["10.10.0.4"],"User-Agent":["connect-go/1.16.2 (go1.23.1)"],"Connect-Timeout-Ms":["4999"],"X-Runner-Version":["v0.2.11"],"Connect-Protocol-Version":["1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["code.gate.internal"],"Content-Type":["application/proto"],"X-Runner-Uuid":["b88d3e97-395a-403a-b4f1-e428ab19b711"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"code.gate.internal"}},"headers":{"Accept-Encoding":["gzip"],"Content-Type":["application/proto"],"Date":["Wed, 08 Jan 2025 21:29:01 GMT"],"Content-Length":["2"]},"status":200}
{"level":"debug","ts":1736371743.858141,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.10.0.3:3000","total_upstreams":1}
{"level":"debug","ts":1736371743.8672585,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.0.3:3000","duration":0.009025597,"request":{"remote_ip":"10.10.0.4","remote_port":"49836","client_ip":"10.10.0.4","proto":"HTTP/1.1","method":"POST","host":"code.gate.internal","uri":"/api/actions/runner.v1.RunnerService/FetchTask","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip"],"User-Agent":["connect-go/1.16.2 (go1.23.1)"],"Content-Type":["application/proto"],"X-Runner-Uuid":["b88d3e97-395a-403a-b4f1-e428ab19b711"],"X-Runner-Version":["v0.2.11"],"Content-Length":["2"],"Connect-Protocol-Version":["1"],"Connect-Timeout-Ms":["4999"],"X-Runner-Token":["98607dce5bce1ff32f2df806965f1f6076e93ea5"],"X-Forwarded-For":["10.10.0.4"],"X-Forwarded-Host":["code.gate.internal"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"code.gate.internal"}},"headers":{"Content-Length":["2"],"Accept-Encoding":["gzip"],"Content-Type":["application/proto"],"Date":["Wed, 08 Jan 2025 21:29:03 GMT"]},"status":200}
2. Error messages and/or full log output:
there is no sink.log
This page could not be displayed. An internal error has occurred.
3. Caddy version:
caddy:2.8.4
4. How I installed and ran Caddy:
#syntax=docker/dockerfile:1
FROM caddy:2.8.4-builder-alpine AS builder
RUN apk update
RUN apk --no-cache add make git gcc libtool musl-dev ca-certificates dumb-init
RUN echo "http://dl-cdn.alpinelinux.org/alpine/v3.19/community" | tee -a /etc/apk/repositories
RUN apk update
RUN apk --no-cache add php82 php82-dev
WORKDIR /src
ENV CGO_ENABLED=1
ENV XCADDY_GO_BUILD_FLAGS="-ldflags '-w -s'"
ENV XCADDY_SETCAP=1
RUN xcaddy build v2.8.4 --with github.com/abiosoft/caddy-exec \
--with github.com/baldinof/caddy-supervisor \
--with github.com/caddyserver/cache-handler \
--with github.com/aksdb/caddy-cgi/v2@v2.2.0 \
--with github.com/greenpau/caddy-git \
--with github.com/Elegant996/scgi-transport \
--with github.com/mholt/caddy-l4 \
--with github.com/caddy-dns/cloudflare \
--with github.com/greenpau/caddy-security \
--with github.com/greenpau/caddy-trace \
--with github.com/caddyserver/jsonc-adapter \
--with github.com/hslatman/caddy-crowdsec-bouncer \
--with github.com/caddyserver/transform-encoder \
--with github.com/dunglas/mercure/caddy \
--with github.com/dunglas/vulcain/caddy \
--with github.com/zhangjiayin/caddy-mysql-storage
FROM caddy:2.8.4-alpine
COPY --from=builder /src/caddy /usr/bin/caddy
RUN apk update \
&& apk add moonscript tcl tk python3 perl curl wget git \
gcc musl-dev openssl-dev make tcl-dev bash
WORKDIR /srv
RUN cd /srv \
&& wget "https://core.tcl-lang.org/tcltls/uv/tcltls-1.7.22.tar.gz" \
&& tar xzvf ./tcltls-1.7.22.tar.gz \
&& cd tcltls-1.7.22 \
&& ./configure --prefix=/usr/ \
&& make \
&& make install
RUN cd /srv && git clone https://github.com/jorge-leon/ton.git ton \
&& mv /srv/ton /usr/lib/tcl8.6
CMD ["caddy", "run", "--config", "/etc/caddy/config.json", "--adapter", ""]
a. System environment:
Linux bluebloom 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Microsoft Windows [Version 10.0.26100.2605]
b. Command:
./start.sh start_www
start_www() {
sudo bash -c 'source ./scripts/999-utils.sh; call_create_network "www_up"'
screen -dmS my_gateway bash -c "./500-www/500-gateway/run.sh; exec bash"
screen -dmS my_gitea bash -c "./500-www/550-gitea/run.sh; exec bash"
# actions runner boots faster than gitea so sleep for a few.
sleep 5s
screen -dmS my_gitea_actionsrunner bash -c "./500-www/551-gitea_actions/run.sh; exec bash"
}
stop_www() {
docker container stop {my_gateway,my_gitea}
screen -S my_gateway -X quit
screen -S my_gitea -X quit
screen -S my_gitea_actionsrunner -X quit
sudo bash -c 'source ./scripts/999-utils.sh; call_create_network "www_down"'
}
fname=$1
shift
eval $fname $@
999-utils.sh:
www_up() {
ip link add link eth0 name eth0.14 type vlan id 14
ip link set eth0.14 up
ip addr add 10.9.0.0/24 dev eth0.14
ip addr add fe81:CCCC::21 dev eth0.14
docker network create -d bridge \
--subnet=10.10.0.0/24 \
--subnet=fe81:DDDD::/64 \
--gateway=10.10.0.1 \
--gateway=fe81:DDDD::22 \
--ipv6 \
-o parent=eth0.14 \
-o com.docker.network.bridge.name=eth0.141 \
-o name=net_www \
$nn_www
};
www_down() {
ip link del eth0.14
docker network rm $nn_www
};
fname=$1
shift
eval $fname $@
c. Service/unit/compose file:
d. My complete Caddy config:
{
"admin": {
"disabled": false,
"listen": "0.0.0.0:2019",
"enforce_origin": false,
"origins": [
"127.0.0.1",
"10.10.0.2",
"10.10.0.3"
],
"identity": {
"identifiers": [
"10.10.0.2",
"gate.internal",
"code.gate.internal",
"www.gate.internal"
],
"issuers": [
{
"module": "internal",
"ca": "",
"lifetime": 0,
"sign_with_root": false
}
]
},
"remote": {
"listen": "0.0.0.0:2021"
}
},
"logging": {
"sink": {
"writer": {
"output": "file",
"filename": "/var/log/caddy/sink.log",
"roll": true,
"roll_size_mb": 4,
"roll_gzip": true,
"roll_local_time": true,
"roll_keep": 32,
"roll_keep_days": 90
}
},
"logs": {
"access": {
"writer": {
"output": "file",
"filename": "/var/log/caddy/access.json",
"roll": true,
"roll_size_mb": 4,
"roll_gzip": true,
"roll_local_time": true,
"roll_keep": 32,
"roll_keep_days": 90
},
"encoder": {
"format": "json"
},
"level": "INFO",
"sampling": {
"interval": 0,
"first": 0,
"thereafter": 0
},
"include": [
"http.*",
"https.*"
]
},
"info": {
"writer": {
"output": "file",
"filename": "/var/log/caddy/info.json",
"roll": true,
"roll_size_mb": 4,
"roll_gzip": true,
"roll_local_time": true,
"roll_keep": 32,
"roll_keep_days": 90
},
"encoder": {
"format": "json"
},
"level": "INFO",
"sampling": {
"interval": 0,
"first": 0,
"thereafter": 0
},
"exclude": [
"admin.*"
]
},
"debug": {
"writer": {
"output": "file",
"filename": "/var/log/caddy/debug.json",
"roll": true,
"roll_size_mb": 4,
"roll_gzip": true,
"roll_local_time": true,
"roll_keep": 32,
"roll_keep_days": 90
},
"encoder": {
"format": "json"
},
"level": "DEBUG",
"sampling": {
"interval": 0,
"first": 0,
"thereafter": 0
},
"exclude": [
"admin.*",
"http.*",
"https.*"
]
},
"admin_api": {
"writer": {
"output": "file",
"filename": "",
"roll": true,
"roll_size_mb": 8,
"roll_gzip": true,
"roll_local_time": true,
"roll_keep": 32,
"roll_keep_days": 90
},
"encoder": {
"format": "console"
},
"level": "INFO",
"sampling": {
"interval": 0,
"first": 0,
"thereafter": 0
},
"exclude": [
"http.log.*"
]
}
}
},
"storage": {
"module": "file_system",
"root": "/data"
},
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [
":443",
":80"
],
"routes": [
{
"match": [
{
"host": [
"www.gate.internal"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"encodings": {
"gzip": {},
"zstd": {}
},
"handler": "encode",
"prefer": [
"zstd",
"gzip"
]
}
]
},
{
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "10.10.0.4:3000"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"code.gate.internal"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"encodings": {
"gzip": {},
"zstd": {}
},
"handler": "encode",
"prefer": [
"zstd",
"gzip"
]
},
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "10.10.0.3:3000"
}
]
}
]
}
]
}
],
"terminal": true
},
{
"match": [
{
"host": [
"cgi.gate.internal"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"args": [
"someargument"
],
"executable": "/var/www/cgi-bin/info.sh",
"handler": "cgi",
"scriptName": "/info.cgi"
}
],
"match": [
{
"path": [
"/info.cgi"
]
}
]
},
{
"handle": [
{
"args": [
"someargument"
],
"executable": "/var/www/cgi-bin/irc.tcl",
"handler": "cgi",
"scriptName": "/irc.cgi"
}
],
"match": [
{
"path": [
"/irc.cgi"
]
}
]
},
{
"handle": [
{
"args": [
"someargument"
],
"executable": "/var/www/cgi-bin/ldap.py",
"handler": "cgi",
"scriptName": "/irc2.cgi"
}
],
"match": [
{
"path": [
"/irc2.cgi"
]
}
]
},
{
"handle": [
{
"args": [
"someargument"
],
"executable": "/var/www/cgi-bin/irc.lua",
"handler": "cgi",
"scriptName": "/irc3.cgi"
}
],
"match": [
{
"path": [
"/irc3.cgi"
]
}
]
}
]
}
]
}
],
"logs": {
"logger_names": {
"www.gate.internal": "log0"
},
"skip_hosts": [
"code.gate.internal"
]
}
}
}
},
"tls": {
"certificates": {"automate": ["gate.internal","www.gate.internal","code.gate.internal","cgi.gate.internal"] },
"automation": {
"policies": [{
"subjects": ["code.gate.internal"],
"issuers":[{
"module": "internal",
"lifetime": 0,
"sign_with_root": false
}],
"key_type":"rsa4096",
"storage": {
"module": "file_system",
"root": "/data/_tls/gate.internal"
}
},
{
"subjects": ["www.gate.internal"],
"issuers":[{
"module": "internal",
"lifetime": 0,
"sign_with_root": false
}],
"key_type":"rsa4096",
"storage": {
"module": "file_system",
"root": "/data/_tls/gate.internal"
}
},
{
"subjects": ["gate.internal"],
"issuers":[{
"module": "internal",
"lifetime": 0,
"sign_with_root": false
}],
"key_type":"rsa4096",
"storage": {
"module": "file_system",
"root": "/data/_tls/gate.internal"
}
},
{
"subjects": ["cgi.gate.internal"],
"issuers":[{
"module": "internal",
"lifetime": 0,
"sign_with_root": false
}],
"key_type":"rsa4096",
"storage": {
"module": "file_system",
"root": "/data/_tls/gate.internal"
}
}]
},
"session_tickets": {
"key_source": {
"provider": "distributed",
"storage": {
"module": "file_system",
"root": "/data/_session_tickets/"
}
}
}
},
"pki": {
},
"layer4": {
"servers": {
"code.gate.internal.ssh": {
"listen": [":22"],
"routes": [
{
"match":[{
"ssh": {
}
}],
"handle":[{
"handler":"proxy",
"upstreams": [{
"dial": ["10.10.0.3:22"]
}]
}]
}
]
}
}
}
}
}
5. Links to relevant resources:
6. My thoughts
i am spinning up a fresh wsl because it seems that my tls for caddy just dropped around the same time that i decided to add a network route
The error seems to persist through the wsl instance, after refreshing all the images i use.
I verified my config with caddy validate
Full documentation is available at:
https://caddyserver.com/docs/command-line
27f8b95f0726:/etc/caddy# caddy validate --config ./config.json
2025/01/09 19:39:55.724 INFO using config from file {"file": "./config.json"}
2025/01/09 19:39:55.726 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2025/01/09 19:39:55.726 INFO tls.cache.maintenance started background certificate maintenance{"cache": "0xc001728c00"}
2025/01/09 19:39:55.726 INFO tls.cache.maintenance stopped background certificate maintenance{"cache": "0xc001728c00"}
Valid configuration
This page could not be displayed. An internal error has occurred.
why such a simple error message?
why does curl work but google chrome and MS Edge get the fake error message?
i do not see that error message in the golang source for net/http
and i dont think its in the caddy source code either.
i have Norton as a virus scanner, and i checked to make sure it wasnt a norton browser extension.
im going to try uninstalling and reinstalling hyper-v and wsl.
I also want to express my frustration here,
I wasted all day yesterday, and now all day today trying to fix this.