Reverse Proxy 502

Help
1

1. The problem I’m having:

I receive a 502 error when connecting to one of my reverse proxy endpoints via caddy. I two configured, one works, one does not. I am attempting to connect to a Splunk instance, and Splunk is configured for http only. Referring to my Caddyfile, the photos reverse proxy works, but splunk does not. I am using macvlan networking for Splunk but the bridge networking for my photos endpoint. Everything is running in Docker on the same physical host.

curl -vL splunk.reimerfamily.net

* Trying 70.169.10.9:80...

* Connected to splunk.reimerfamily.net (70.169.10.9) port 80 (#0)

> GET / HTTP/1.1

> Host: splunk.reimerfamily.net

> User-Agent: curl/7.88.1

> Accept: */*

>

< HTTP/1.1 308 Permanent Redirect

< Connection: close

< Location: https://splunk.reimerfamily.net/

< Server: Caddy

< Date: Tue, 17 Sep 2024 11:22:50 GMT

< Content-Length: 0

<

* Closing connection 0

* Clear auth, redirects to port from 80 to 443

* Issue another request to this URL: 'https://splunk.reimerfamily.net/'

* Trying 70.169.10.9:443...

* Connected to splunk.reimerfamily.net (70.169.10.9) port 443 (#1)

* ALPN: offers h2,http/1.1

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* CAfile: /etc/ssl/certs/ca-certificates.crt

* CApath: /etc/ssl/certs

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

* TLSv1.3 (IN), TLS handshake, Certificate (11):

* TLSv1.3 (IN), TLS handshake, CERT verify (15):

* TLSv1.3 (IN), TLS handshake, Finished (20):

* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.3 (OUT), TLS handshake, Finished (20):

* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256

* ALPN: server accepted h2

* Server certificate:

* subject: CN=splunk.reimerfamily.net

* start date: Sep 13 09:13:29 2024 GMT

* expire date: Dec 12 09:13:28 2024 GMT

* subjectAltName: host "splunk.reimerfamily.net" matched cert's "splunk.reimerfamily.net"

* issuer: C=US; O=Let's Encrypt; CN=E5

* SSL certificate verify ok.

* using HTTP/2

* h2h3 [:method: GET]

* h2h3 [:path: /]

* h2h3 [:scheme: https]

* h2h3 [:authority: splunk.reimerfamily.net]

* h2h3 [user-agent: curl/7.88.1]

* h2h3 [accept: */*]

* Using Stream ID: 1 (easy handle 0x55a6e7cedce0)

> GET / HTTP/2

> Host: splunk.reimerfamily.net

> user-agent: curl/7.88.1

> accept: */*

>

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

< HTTP/2 502

< alt-svc: h3=":443"; ma=2592000

< server: Caddy

< content-length: 0

< date: Tue, 17 Sep 2024 11:22:53 GMT

<

* Connection #1 to host splunk.reimerfamily.net left intact

2. Error messages and/or full log output:

{"level":"debug","ts":1726572542.0957484,"logger":"events","msg":"event","name":"tls_get_certificate","id":"38beca5d-50b9-4198-824d-32003480ea40","origin":"tls","data":{"client_hello":{"CipherSuites":[23130,4865,4866,4867,49196,49195,52393,49200,49199,52392,49162,49161,49172,49171,157,156,53,47,49160,49170,10],"ServerName":"splunk.reimerfamily.net","SupportedCurves":[60138,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[23130,772,771,770,769],"RemoteAddr":{"IP":"192.168.6.252","Port":63747,"Zone":""},"LocalAddr":{"IP":"172.18.0.2","Port":443,"Zone":""}}}}
{"level":"debug","ts":1726572542.0957925,"logger":"tls.handshake","msg":"choosing certificate","identifier":"splunk.reimerfamily.net","num_choices":1}
{"level":"debug","ts":1726572542.0958216,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"splunk.reimerfamily.net","subjects":["splunk.reimerfamily.net"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"efd0dfe09d0d851f869d35ef6d70edcbb43ccc9474eb1cc3954e27df0f077fa5"}
{"level":"debug","ts":1726572542.0958354,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"192.168.6.252","remote_port":"63747","subjects":["splunk.reimerfamily.net"],"managed":true,"expiration":1733994809,"hash":"efd0dfe09d0d851f869d35ef6d70edcbb43ccc9474eb1cc3954e27df0f077fa5"}
{"level":"debug","ts":1726572542.103056,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"192.168.7.70:8000","total_upstreams":1}
{"level":"debug","ts":1726572545.1046853,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"192.168.7.70:8000","duration":3.001523407,"request":{"remote_ip":"192.168.6.252","remote_port":"63747","client_ip":"192.168.6.252","proto":"HTTP/2.0","method":"GET","host":"splunk.reimerfamily.net","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Safari/605.1.15"],"Priority":["u=0, i"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["192.168.6.252"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["splunk.reimerfamily.net"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"splunk.reimerfamily.net"}},"error":"dial tcp 192.168.7.70:8000: i/o timeout"}
{"level":"error","ts":1726572545.1048007,"logger":"http.log.error","msg":"dial tcp 192.168.7.70:8000: i/o timeout","request":{"remote_ip":"192.168.6.252","remote_port":"63747","client_ip":"192.168.6.252","proto":"HTTP/2.0","method":"GET","host":"splunk.reimerfamily.net","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Priority":["u=0, i"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Safari/605.1.15"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"splunk.reimerfamily.net"}},"duration":3.001773372,"status":502,"err_id":"ecud2gef3","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Docker compose

a. System environment:

Debian bookworm running docker engine version 27.2.1 and docker compose version 2.29.2.

b. Command:

docker compose up -d

c. Service/unit/compose file:

name: caddy

services:
  caddy:
    image: caddy:latest
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /reimerfamdata/docks/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /reimerfamdata/docks/caddy/data:/data
      - /reimerfamdata/docks/caddy/config:/config

d. My complete Caddy config:

{
	debug
}
photos.reimerfamily.net {
    reverse_proxy 192.168.7.5:2342
}

splunk.reimerfamily.net {
    reverse_proxy http://192.168.7.70:8000
}

5. Links to relevant resources:

2

Howdy @Hack3rDan, welcome to the Caddy community.

"error":"dial tcp 192.168.7.70:8000: i/o timeout"

This looks like a pretty bog standard instance of an unreachable upstream.

Then you have a networking problem.

Linux macvlan networking does not allow for guests to communicate with hosts.

However, when a guest virtual machine is configured to use a type='direct' network interface such as macvtap, despite having the ability to communicate with other guests and other external hosts on the network, the guest cannot communicate with its own host.

This situation is actually not an error — it is the defined behavior of macvtap. Due to the way in which the host’s physical Ethernet is attached to the macvtap bridge, traffic into that bridge from the guests that is forwarded to the physical interface cannot be bounced back up to the host’s IP stack. Additionally, traffic from the host’s IP stack that is sent to the physical interface cannot be bounced back up to the macvtap bridge for forwarding to the guests.

—B.9. Guest Can Reach Outside Network, but Cannot Reach Host when Using macvtap Interface | Red Hat Product Documentation

You’ll see this issue all over the place if you google Docker macvlan container can’t reach host or similar terms.

To fix this you’d need to pick a different network driver (like a default bridge) or connect the physical host to a dedicated external networking switch that can hairpin packets back out on the same port to handle this specific use case.

1 Like
3

@Whitestrake, thank you for that info. Changed Splunk to used a bridge network and now everything is working as expected.

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.