1. Output of caddy version
:
sudo docker exec caddy caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=
2. How I run Caddy:
a. System environment:
Docker
uname -a
Linux zenon-info 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
b. Command:
sudo docker-compose up -d
c. Service/unit/compose file:
version: '3'
services:
homer:
image: b4bz/homer
container_name: homer
volumes:
- ./config/homer/assets/:/www/assets
# ports:
# - 8080:8080
user: 1000:1000 # default
environment:
- INIT_ASSETS=1 # default
networks:
- znn-network
znnd:
build: .
container_name: znnd
ports:
- 35995:35995
- 35995:35995/udp
# - 35997:35997
# - 35998:35998
restart: always
volumes:
- data:/root/.znn
networks:
- znn-network
caddy:
image: caddy:2
container_name: caddy
restart: always
ports:
- 80:80 # Needed for the ACME HTTP-01 challenge.
- 443:443
- 35997:35997
- 35998:35998
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- ./caddy-config:/config
- ./caddy-data:/data
# - ./test_code:/wdata
environment:
DOMAIN: "https://00.deeznnutz.com"
EMAIL: "REMOVE"
LOG_FILE: "/data/access.log"
networks:
- znn-network
volumes:
data:
networks:
znn-network:
driver: bridge
d. My complete Caddy config:
This is my actual Caddyfile with nothing changed
00.deeznnutz.com {
reverse_proxy homer:8080
}
00.deeznnutz.com:35997 {
reverse_proxy znnd:35997
}
3. The problem I’m having:
When I run the following command on the endpoint I get a 403
error
curl -X GET https://00.deeznnutz.com:35997 -H "content-type: application/json" -d '{"jsonrpc": "2.0", "id": 40, "method": "stats.networkInfo", "params": []}'
4. Error messages and/or full log output:
curl https://00.deeznnutz.com:35997 -v
* Trying 70.34.152.117:35997...
* TCP_NODELAY set
* Connected to 00.deeznnutz.com (70.34.152.117) port 35997 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=00.deeznnutz.com
* start date: Dec 25 13:43:31 2022 GMT
* expire date: Mar 25 13:43:30 2023 GMT
* subjectAltName: host "00.deeznnutz.com" matched cert's "00.deeznnutz.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffeac6b7c0)
> GET / HTTP/2
> Host: 00.deeznnutz.com:35997
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403
< alt-svc: h3=":35997"; ma=2592000
< content-type: text/plain; charset=utf-8
< date: Sun, 25 Dec 2022 17:08:02 GMT
< server: Caddy
< x-content-type-options: nosniff
<
invalid host specified
* Connection #0 to host 00.deeznnutz.com left intact
{"level":"debug","ts":1671992417.902228,"logger":"events","msg":"event","name":"tls_get_certificate","id":"4c3beee6-e52e-4488-8570-3b214edb0ce9","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"00.deeznnutz.com","SupportedCurves":[29,23,30,25,24],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
{"level":"debug","ts":1671992417.902292,"logger":"tls.handshake","msg":"choosing certificate","identifier":"00.deeznnutz.com","num_choices":1}
{"level":"debug","ts":1671992417.902355,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"00.deeznnutz.com","subjects":["00.deeznnutz.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"e878ff5b5d4f0969b1a0105b8dc6c12d76d0b2026fee24b39f2dde1230b18d67"}
{"level":"debug","ts":1671992417.9023762,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"50.247.129.189","remote_port":"32636","subjects":["00.deeznnutz.com"],"managed":true,"expiration":1679751811,"hash":"e878ff5b5d4f0969b1a0105b8dc6c12d76d0b2026fee24b39f2dde1230b18d67"}
{"level":"debug","ts":1671992417.9269614,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"znnd:35997","total_upstreams":1}
{"level":"debug","ts":1671992417.927543,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"znnd:35997","duration":0.000525095,"request":{"remote_ip":"50.247.129.189","remote_port":"32636","proto":"HTTP/2.0","method":"GET","host":"00.deeznnutz.com:35997","uri":"/","headers":{"Accept":["*/*"],"Content-Type":["application/json"],"Content-Length":["73"],"User-Agent":["curl/7.68.0"],"X-Forwarded-For":["50.247.129.189"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["00.deeznnutz.com:35997"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"00.deeznnutz.com"}},"headers":{"Content-Type":["text/plain; charset=utf-8"],"X-Content-Type-Options":["nosniff"],"Date":["Sun, 25 Dec 2022 18:20:17 GMT"]},"status":403}
5. What I already tried:
- I can
telnet 00.deeznnutz.com 35997
- I tried to change this:
00.deeznnutz.com:35997 {
reverse_proxy znnd:35997
}
to
00.deeznnutz.com:35997 {
reverse_proxy homer:8080 #different container on same network
}
And I was able to access the endpoint with SSL. I believe the endpoint at znnd:35997 needs different headers or settings (maybe) but I’m not sure.
-
I tried the same general setup with nginx on docker and got the same 403 error
-
I am able to setup nginx native with the following
.config
file on a different domain and can run the command (on a different domain) as expected. That command for that domain and setup is:curl -X GET https://secure.deeznnodez.com:35997 -H "content-type: application/json" -d '{"jsonrpc": "2.0", "id": 40, "method": "stats.networkInfo", "params": []}'
server {
listen 35997 ssl;
server_name secure.deeznnodez.com;
ssl_certificate /etc/letsencrypt/live/secure.deeznnodez.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/secure.deeznnodez.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location / {
proxy_pass http://[LOCAL IP]:35997; #using one local node rather than load balancer. See note above
}
}
So I can run this command successfully with a native (not in docker) setup with nginx as the proxy, but not in docker with either caddy or nginx.
-
I’m able to ping
znnd
fromcaddy
-
I can expose 35997:35997 directly from
znnd
and query the server directly on 35997. So the issue is how I’ve configured caddy -
I’m able to connect to wss://00.deeznnutz.com:35998 when I proxy through caddy with these Caddyfile settings:
00.deeznnutz.com:35998 {
reverse_proxy znnd:35998
}
6. Links to relevant resources:
NA. Thank you for taking a look.