How do I configure caddy proxy to work - 502 bad gateway plague

Help
1

I am at a loss. I have pasted curl output below from the caddy computer console, another machine attempting to access content via caddy, and caddy logs.

I can use curl via the IPs to get the content successfully. However, once caddy is introduced into the chain, I get a 502 bad gateway. I have attempted to configure caddy to match the curl command.

What am I doing wrong?

FROM CADDY CONSOLE

$ curl --insecure https://app.internal.foo.com/webclient/          
<!DOCTYPE html>
<html>
  <head>
    <title>Web Client</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap-theme.min.css">
  </head>
  <body>
    <div id="app">Loading...</div>
    <script src="/static/webclient/bundle.e0d8ab921f98.js"></script>
  </body>
</html>




$ curl --insecure --resolve 'app.internal.foo.com:443:172.28.14.51' https://app.internal.foo.com/webclient/

<!DOCTYPE html>
<html>
  <head>
    <title>Web Client</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap-theme.min.css">
  </head>
  <body>
    <div id="app">Loading...</div>
    <script src="/static/webclient/bundle.e0d8ab921f98.js"></script>
  </body>
</html>

Caddyfile:

$ cat /etc/caddy/Caddyfile
http://*.internal.foo.com {
  proxy / http://172.28.14.51:80 {
    transparent
  }
  log stdout
  errors stderr
}

https://*.internal.foo.com {
  proxy / https://172.28.14.51:443 {
    transparent
    insecure_skip_verify
  }
  tls {
    max_certs 100
    alpn http/1.1
    protocols tls1.0 tls1.2
  }
  log stdout
  errors stderr
}

FROM ANOTHER HOST:

$ curl --insecure https://app.internal.foo.com/webclient/       
<!DOCTYPE html>
<html>
  <head>
    <title>Web Client</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap-theme.min.css">
  </head>
  <body>
    <div id="app">Loading...</div>
    <script src="/static/webclient/bundle.e0d8ab921f98.js"></script>
  </body>
</html>


$ curl --insecure --resolve 'app.internal.foo.com:443:172.28.14.51' https://app.internal.foo.com/webclient/

<!DOCTYPE html>
<html>
  <head>
    <title>Web Client</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap-theme.min.css">
  </head>
  <body>
    <div id="app">Loading...</div>
    <script src="/static/webclient/bundle.e0d8ab921f98.js"></script>
  </body>
</html>


$ curl --insecure --resolve 'app.internal.foo.com:443:172.28.14.50' https://app.internal.foo.com/webclient/
502 Bad Gateway

CADDY LOGS FROM A FEW ATTEMPTS:

Oct 08 17:49:06 caddy caddy[4157]: 08/Oct/2019:17:49:06 +0000 [ERROR 502 /webclient/] read tcp 172.28.14.50:34490->172.28.14.51:443: read: connection reset by peer
Oct 08 17:49:06 caddy caddy[4157]: 172.28.14.107 - - [08/Oct/2019:17:49:06 +0000] "GET /webclient/ HTTP/1.1" 502 16
Oct 08 17:49:14 caddy caddy[4157]: 08/Oct/2019:17:49:14 +0000 [ERROR 502 /webclient/] read tcp 172.28.14.50:34492->172.28.14.51:443: read: connection reset by peer
Oct 08 17:49:14 caddy caddy[4157]: 172.28.14.107 - - [08/Oct/2019:17:49:14 +0000] "GET /webclient/ HTTP/1.1" 502 16
Oct 08 17:49:18 caddy caddy[4157]: 08/Oct/2019:17:49:18 +0000 [ERROR 502 /webclient/] read tcp 172.28.14.50:34494->172.28.14.51:443: read: connection reset by peer
Oct 08 17:49:18 caddy caddy[4157]: 172.28.14.107 - - [08/Oct/2019:17:49:18 +0000] "GET /webclient/ HTTP/1.1" 502 16
2

Compare your curl versions between hosts.

It’s possible you are using an outdated curl - where either one of two things is not supported:

  • --resolve parameter not supported by the curl version
  • The version of OpenSSL compiled into / used by curl is less than version 1.0.0 (I think) - causing problems with SNI.
3

So I here are some more verbose logs. I’ve demonstrated that the caddy reverse proxy works for http but not for https. So that should mean that network connectivity is not the issue:

Curl version from client I am using:

$ curl --version
curl 7.54.0 (x86_64-redhat-linux-gnu) libcurl/7.54.0 OpenSSL/1.0.1e zlib/1.2.3 c-ares/1.12.0 libssh2/1.8.0 nghttp2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy Metalink

Curl output for https request:

$ curl -v --include --resolve 'app.internal.foo.com:443:172.28.14.50' https://app.internal.foo.com/webclient/
* Added app.internal.foo.com:443:172.28.14.50 to DNS cache
* Hostname app.internal.foo.com was found in DNS cache
*   Trying 172.28.14.50...
* TCP_NODELAY set
* Connected to app.internal.foo.com (172.28.14.50) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP2 (h2)
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* Server certificate:
*  subject: CN=app.internal.foo.com
*  start date: Oct  8 14:09:01 2019 GMT
*  expire date: Jan  6 14:09:01 2020 GMT
*  subjectAltName: host "app.internal.foo.com" matched cert's "app.internal.foo.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xa43310)
> GET /webclient/ HTTP/2
> Host: app.internal.foo.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 502
HTTP/2 502
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< server: Caddy
server: Caddy
< x-content-type-options: nosniff
x-content-type-options: nosniff
< content-length: 16
content-length: 16
< date: Wed, 09 Oct 2019 14:38:46 GMT
date: Wed, 09 Oct 2019 14:38:46 GMT

<
502 Bad Gateway
* Connection #0 to host app.internal.foo.com left intact

Curl output for http request:

$ curl -v --include --resolve 'app.internal.foo.com:80:172.28.14.50' http://app.internal.foo.com/webclient/
* Added app.internal.foo.com:80:172.28.14.50 to DNS cache
* Hostname app.internal.foo.com was found in DNS cache
*   Trying 172.28.14.50...
* TCP_NODELAY set
* Connected to app.internal.foo.com (172.28.14.50) port 80 (#0)
> GET /webclient/ HTTP/1.1
> Host: app.internal.foo.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Content-Length: 170
Content-Length: 170
< Content-Type: text/html
Content-Type: text/html
< Date: Wed, 09 Oct 2019 14:40:05 GMT
Date: Wed, 09 Oct 2019 14:40:05 GMT
< Location: https://app.internal.foo.com/webclient/
Location: https://app.internal.foo.com/webclient/
< Server: Caddy
Server: Caddy
< Server: nginx/1.15.10 Microsoft-HTTPAPI/2.0
Server: nginx/1.15.10 Microsoft-HTTPAPI/2.0

<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.15.10</center>
</body>
</html>
* Connection #0 to host app.internal.foo.com left intact

Here is a copy of the caddy file exported from the caddy server console:

$ sudo cat /etc/caddy/Caddyfile
http://*.internal.foo.com {
  proxy / http://wap.internal.foo.com {
    transparent
  }

  log stdout
  errors stderr
}

https://*.internal.foo.com {
  proxy / https://wap.internal.foo.com {
    transparent
    insecure_skip_verify
  }

  tls {
    max_certs 100
  }

  log stdout
  errors stderr
}
4

It may be your OpenSSL version, your on “OpenSSL/1.0.1e” that is from 2014/06/4…
… pretty sure running a 5+ year old version of OpenSSL isn’t a good thing.

I could be completely wrong here…
… I just remember experiencing something similar previously and it was related to the OpenSSL version, it was post HeartBleed so perhaps you need at least OpenSSL 1.0.2 or later.

As an aside TLS 1.3 was added in OpenSSL 1.1.1 which is the latest version:

5

That is just a random computer I am running curl commands from. Caddy is freshly installed on Ubuntu 18.04.

6

This error, emitted by Caddy’s proxying:

Oct 08 17:49:06 caddy caddy[4157]: 08/Oct/2019:17:49:06 +0000 [ERROR 502 /webclient/] read tcp 172.28.14.50:34490->172.28.14.51:443: read: connection reset by peer

means that while reading from the remote’s socket, the connection was reset by the remote.

Is Caddy running on the machine with IP 172.28.14.50? Apparently, that machine is having trouble communicating cleanly with 172.28.14.51:443 – either the backend app is closing/resetting the connection, or some network piece in between is doing it.

7

Caddy is 172.28.14.50. WAP is 172.28.14.51

@caddy:~$ ifconfig | grep -i inet
        inet 172.28.14.50  netmask 255.255.255.0  broadcast 172.28.14.255
        inet6 fe80::215:5dff:fe0e:880f  prefixlen 64  scopeid 0x20<link>
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>

Here is console output from the caddy machine showing curl can communicate with APP through WAP successfully

@caddy:~$ curl -v --include --resolve 'app.internal.foo.com:443:172.28.14.51' https://app.internal.foo.com/webclient/
* Added app.internal.foo.com:443:172.28.14.51 to DNS cache
* Hostname app.internal.foo.com was found in DNS cache
*   Trying 172.28.14.51...
* TCP_NODELAY set
* Connected to app.internal.foo.com (172.28.14.51) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=app.internal.foo.com
*  start date: Oct  8 22:29:00 2019 GMT
*  expire date: Oct  7 22:29:00 2020 GMT
*  subjectAltName: host "app.internal.foo.com" matched cert's "app.internal.foo.com"
*  issuer: DC=com; DC=foo; DC=internal; CN=Bar
*  SSL certificate verify ok.
> GET /webclient/ HTTP/1.1
> Host: app.internal.foo.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 422
Content-Length: 422
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< ETag: "2fd1291e170048245ad8a8a8c157ee6e"
ETag: "2fd1291e170048245ad8a8a8c157ee6e"
< Server: nginx/1.15.10 Microsoft-HTTPAPI/2.0
Server: nginx/1.15.10 Microsoft-HTTPAPI/2.0
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Date: Wed, 09 Oct 2019 17:08:33 GMT
Date: Wed, 09 Oct 2019 17:08:33 GMT

<

<!DOCTYPE html>
<html>
  <head>
    <title>Web Client</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/latest/css/bootstrap-theme.min.css">
  </head>
  <body>
    <div id="app">Loading...</div>
    <script src="/static/webclient/bundle.e0d8ab921f98.js"></script>
  </body>
</html>
* Connection #0 to host app.internal.foo.com left intact

And here is ping from caddy:

@caddy:~$ ping wap.internal.foo.com -c 3
PING wap21.internal.foo.com (172.28.14.51) 56(84) bytes of data.
64 bytes from wap01.internal.foo.com (172.28.14.51): icmp_seq=1 ttl=128 time=0.768 ms
64 bytes from wap01.internal.foo.com (172.28.14.51): icmp_seq=2 ttl=128 time=0.662 ms
64 bytes from wap01.internal.foo.com (172.28.14.51): icmp_seq=3 ttl=128 time=0.703 ms

--- wap21.internal.foo.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.662/0.711/0.768/0.043 ms
8

Why do you have to tell curl to resolve that hostname to that IP address? Does your system resolver not return the right IP?

9

Because the local network flow doesn’t go through the proxies so I have to force it.

caddy → microsoft web application proxy → app

When I do app.internal.foo.com locally I get that endpoint directly.

Internet traffic goes through the proxy chain. In order to mimic that flow I have to force curl to resolve app.internal.foo.com to wap.internal.foo.com on the caddy host machine.

I was demonstrating that the machine caddy.internal.foo.com is able to curl https through microsoft web application proxy → app successfully but the proxy configuration is failing.

10

I still don’t fully understand your network setup, but it seems that the Microsoft proxy is resetting the connection. It’s not clear why, you’d probably have to check the Microsoft proxy’s logs.

11

Can I configure Caddy to provide verbose logging for the proxy connection? Like how curl will show all of the TLS and http negotiation?

12

WAP is a reverse proxy that gives access from the internet to local services. I am trying to place Caddy infront of it for the easy Let’s Encrypt certificate generation.

My goal is to let Caddy serve up the public facing ssl cert and everything else continue as is.

13

Hmm, not without recompiling it.

But what is in the WAP’s logs regarding the connection(s)? That has yet to be posted here, and would be very helpful I think. Right now just looking at it from one end all we can do is guess: we don’t know why the remote is resetting the connection without a log from it.

14

I understand – was hoping to be able to skip the problem from this direction

That is where I started but I am still working on getting them. It doesn’t seem to be as trivial as opening the log file. WAP doesn’t seem to offer those logs and I am not having much luck so far.

I had hoped it would be trivial to make the proxy request look like the curl request that was succeeding. I don’t really understand why it doesn’t. The configuration reads to me such that it should be similar.

15

We can definitely implement more fine-grained/verbose logging into Caddy 2. Chat with Miguel (from our email conversation) and he can get it prioritized for you.

Getting logs from WAP would probably be the most helpful. Even Caddy’s logs, when it rejects or closes a connection, will often tell you why. So I’m assuming the Microsoft product does too.

16

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.