Caddy in docker container not serving https

Johnathan_Harding · September 9, 2019, 5:22pm

Hello. I’m new to Caddy. Thanks in advance for the help.

What Am I Trying to Do
Serve my server page over https. Caddy logs says its serving my website over https, but the page doesn’t load.

What Have I tried
Environment - my website and caddy (abiosoft/caddy) are in docker containers. They are on the same docker network, can ping each other. My host server is Ubuntu. Host server is on cloud, using vendor’s firewall with ports 443, 2015, 3000 open. My server alias is “talk” receiving traffic on port 3000. The url for my server should be https://devtalk.mm4a.io:3000

Inside the Caddy container is my Caddyfile, not in a host directory. Looks like this:

~/caddy2$ docker exec -it caddy3 sh
/srv # cat /etc/Caddyfile
devtalk.mm4a.io 
                                   
proxy   /  http://talk:3000 { 
        websocket   
        transparent 
}                                                   
                                                   
proxy   /api/v1/graph/ql  http://talk:3000 { 
        websocket   
        transparent 
}

Error msgs, logs

2019/09/09 16:08:57 [INFO][cache:0xc00008c780] Scanning for stale OCSP staples
2019/09/09 16:08:57 [INFO][cache:0xc00008c780] Done checking OCSP staples
2019/09/09 16:17:17 http: TLS handshake error from 45.33.80.76:37138: no certificate available for ''
2019/09/09 16:33:02 [INFO] SIGTERM: Shutting down servers then terminating
2019/09/09 16:33:02 [INFO][cache:0xc00008c780] Stopped certificate maintenance routine
Activating privacy features... 2019/09/09 16:33:04 [INFO][cache:0xc0000327d0] Started certificate maintenance routine
done.

Serving HTTPS on port 3000 
https://devtalk.mm4a.io:3000

2019/09/09 16:33:04 [INFO] Serving https://devtalk.mm4a.io:3000 

Serving HTTP on port 80 
http://devtalk.mm4a.io

2019/09/09 16:33:04 [INFO] Serving http://devtalk.mm4a.io 
2019/09/09 16:33:04 [NOTICE] Sending telemetry: we were too early; waiting 9h3m33.182301869s before trying again
2019/09/09 16:43:23 [INFO] SIGTERM: Shutting down servers then terminating
2019/09/09 16:43:23 [INFO][cache:0xc0000327d0] Stopped certificate maintenance routine
Activating privacy features... done.

Serving HTTPS on port 443 
https://devtalk.mm4a.io

2019/09/09 16:43:24 [INFO] Serving https://devtalk.mm4a.io 

Serving HTTP on port 80 
http://devtalk.mm4a.io

2019/09/09 16:43:24 [INFO] Serving http://devtalk.mm4a.io 
2019/09/09 16:43:24 [INFO][cache:0xc00008c780] Started certificate maintenance routine
2019/09/09 16:43:24 [NOTICE] Sending telemetry: we were too early; waiting 8h53m13.090148534s before trying again

So what could be doing wrong? Last week I got a limit reached message, figured I’d try again today. When I inspect the page on the browser, I seen the server is trying to server content, but errors out.

Whitestrake · September 10, 2019, 6:48am

Hi @Johnathan_Harding, welcome to the Caddy community!

This one appears to be a bit of a mess of different factors, and misconceptions.

First up:

No. Whatever’s listening on port 3000, it ain’t Caddy. It’s probably Talk itself. And it’s not HTTPS, it’s serving regular plain old HTTP over port 3000.

➜ curl -I http://devtalk.mm4a.io:3000/
HTTP/1.1 302 Found
X-Talk-Trace-ID: 45ecd400-d394-11e9-9950-cfe6931f68f3
X-DNS-Prefetch-Control: off
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Location: /admin
Vary: Accept, Accept-Encoding
Content-Type: text/plain; charset=utf-8
Content-Length: 28
Date: Tue, 10 Sep 2019 06:28:59 GMT
Connection: keep-alive

I can see two problems here.

Caddy isn’t serving it (no Server: Caddy header).
It’s serving a Strict-Transport-Security header despite the fact it’s serving HTTP on this non-standard port.

You probably just straight up wanna close port 3000 on your host. It doesn’t need to be open if you want Caddy to be proxying it for you. You probably also want to remove the HSTS header. Reimplement it in Caddy if you’re sure you want it.

This is weird. Nothing in your Caddyfile is telling Caddy to serve a HTTPS site on port 3000. This shouldn’t be here - it does not conform to the Caddyfile you posted.

And we know that external requests aren’t reaching it anyway, because:

no Server: Caddy header, and
we’re getting plain HTTP on port 3000, not HTTPS (can’t have both).

Therefore, there must be another HTTP listener on port 3000. I’d wager that this is talk:3000, routed to port 3000 on the host, as I mentioned a bit earlier in this post.

This is where your Caddy site is available.

➜ curl -I https://devtalk.mm4a.io
HTTP/2 302
content-type: text/plain; charset=utf-8
date: Tue, 10 Sep 2019 06:34:29 GMT
location: /admin
server: Caddy
strict-transport-security: max-age=15552000; includeSubDomains
vary: Accept, Accept-Encoding
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-talk-trace-id: 0a9697a0-d395-11e9-9950-cfe6931f68f3
x-xss-protection: 1; mode=block
content-length: 28

Note the server: Caddy in the response headers. HSTS makes a comeback here, but not because it’s implemented in Caddy, but because it’s proxying to talk:3000.

Inaccessible, but harmlessly so. As you noted, the vendor firewall ports open are 443, 2015, and 3000. 80 isn’t listed here and appears to be timing out (consistent with firewalled port). Not a problem.

This image is useful, but it unfortunately doesn’t show the whole story. Bringing it up in my own web inspector:

The elephant in the room here is: Request URL: https://devtalk.mm4a.io:3000/static/coral-admin/bundle.1e8a5f7f108e77759ed6.js

This URL has some inherent issues. The big one is that we’re requesting HTTPS (that we know Caddy is serving on port 443) from port 3000 (which we have demonstrated is in fact a HTTP listener and not served by Caddy itself).

Lets try to make this request manually, EITHER:

With the correct scheme (HTTP) for the port (3000):

➜ curl -I http://devtalk.mm4a.io:3000/static/coral-admin/bundle.1e8a5f7f108e77759ed6.js
HTTP/1.1 200 OK
X-Talk-Trace-ID: 98981690-d396-11e9-9950-cfe6931f68f3
X-DNS-Prefetch-Control: off
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Vary: Accept-Encoding
Accept-Ranges: bytes
Cache-Control: public, max-age=604800, immutable
Last-Modified: Mon, 12 Aug 2019 14:54:02 GMT
ETag: W/"311a40-16c86526310"
Content-Type: application/javascript; charset=UTF-8
Content-Length: 3217984
Date: Tue, 10 Sep 2019 06:45:37 GMT
Connection: keep-alive

Looks good! OR:

With the correct port (443) for the scheme (HTTPS):

➜ curl -I https://devtalk.mm4a.io/static/coral-admin/bundle.1e8a5f7f108e77759ed6.js
HTTP/2 200
accept-ranges: bytes
cache-control: public, max-age=604800, immutable
content-type: application/javascript; charset=UTF-8
date: Tue, 10 Sep 2019 06:45:45 GMT
etag: W/"311a40-16c86526310"
last-modified: Mon, 12 Aug 2019 14:54:02 GMT
server: Caddy
strict-transport-security: max-age=15552000; includeSubDomains
vary: Accept-Encoding
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-talk-trace-id: 9d6ad450-d396-11e9-9950-cfe6931f68f3
x-xss-protection: 1; mode=block
content-length: 3217984

Well, looks like it’s working, both through Talk directly, and through Caddy via the proxy. So what’s going wrong?

Probably, there is a setting somewhere in Talk where you configure what your site name is. It is probably using this to generate its links for your browser to fetch these assets. It is almost certainly currently set to https://devtalk.mm4a.io:3000, but there is no circumstance where this URL makes sense - not for Caddy proxying upstream, not for the client talking to Caddy.

It should either be https://devtalk.mm4a.io (standard HTTPS port) or http://devtalk.mm4a.io:3000 (HTTP on non-standard port).

Johnathan_Harding · September 10, 2019, 3:46pm

Matt,

Thanks a mill for the thorough breakdown. You what the fix was? I changed the Talk server url (container) from devtalk.mm4a.io:3000 to devtalk.mm4a.io. Then everything worked!

   $ curl -I https://devtalk.mm4a.io
HTTP/2 302 
content-type: text/plain; charset=utf-8
date: Tue, 10 Sep 2019 15:44:09 GMT
location: /admin
server: Caddy
strict-transport-security: max-age=15552000; includeSubDomains
vary: Accept, Accept-Encoding
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-talk-trace-id: d430a190-d3e1-11e9-a9ba-9d228ef0f8cd
x-xss-protection: 1; mode=block
content-length: 28

When you said I could close port 3000 on firewall, that got me thinking like yeah. Caddy is proxying from 443, so why does 3000 need to be open? That’s its an internal routing thing. I should have stopped and imagined how the packets flow and it would have clicked.

Thanks again.

Whitestrake · September 10, 2019, 11:17pm

Heh. You could say I had a hunch this would be the case

system · December 9, 2019, 11:17pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.