Switch service from HTTP to HTTPS


(giopas) #1

Hi folks,

I would kindly ask your help to move one of my proxy services from HTTP to HTTPS.

Caddy runs with the following command:

caddy -agree -port 20015 -root /home/Qhttpd -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid -conf caddy.conf

and in the caddy.conf file I have the following block:

http://service.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://192.168.2.87:8989 {
		keepalive 32
		transparent
		websocket
	}
}

Indeed on my router I have forwarded the port 80 to port 8989 of the IP where Caddy is running.

So far so good.

What I would like is to change “service” in HTTPS (in this example, using port 9898 in the service config).

I have therefore tried to change the config file as follows:

https://service.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / https://192.168.2.87:9898 {
		keepalive 32
		transparent
		websocket
	}
}

and I forwarded port 443 of the router to the same port 8989 if the IP where Caddy is running.

But this does not work. What am I missing here?

Thanks in advance for your help!


(Matthew Fay) #2

Without more information on exactly what part is going wrong, it’s difficult to speculate.

  1. What did you do? (e.g. “browsed to http://example.com in Chrome”, or “ran caddy from CLI”)
  2. What did you expect to happen? (e.g. “my website appears”, or “Caddy starts serving sites”)
  3. What happened instead? (e.g. “Chrome gave a certificate error”, or “Caddy couldn’t bind a port”)

(giopas) #3

Thank you Whitestrake, I think I simplified a part of the configuration indeed.

In my case I already have port 444 used by another service (QTS interface of my QNAP). I want therefore still use SSL on another port, let’s say 8443 this time dedicated to Caddy.

I therefore tried to portforward 8443 to the Caddy internal port 8963, hoping that it would then reforward to my https service 9898.

However if I try to open the browser and go to
https://service.mydomain.com:8443 I am forwarded to QTS.

I just need to understand the logic of port forwarding to Caddy, then I could step by step fix certificate issues and alike.

Thanks!


(Matthew Fay) #4

I think I understand what you’re trying to achieve, by accessing https://service.mydomain.com:8443:

Router (HTTPS, port 8443) -> Caddy (HTTPS, port 8963) -> Internal service (HTTPS, port 9898)

So you browse to https://service.mydomain.com:8443, but instead of your internal service, it loads the QNAP web interface.

Can you tell me what website appears when you browse directly to https://192.168.2.87:9898?


(giopas) #5

Exactly!

What I get is to access to my service in HTTPS (after receiving a SSL certificate error of course).

So, either the problem seems the port forwarding or my Caddy configuration, right?


(Matthew Fay) #6

The Caddy configuration looks fine to me.

Can you tell me what output you get from Caddy when you start it with this configuration?

Also worth noting that if you’d be getting a certificate error in a browser, Caddy should be returning 502s when trying to proxy to https://192.168.2.87:9898 unless you add insecure_skip_verify, because it also attempts to verify the upstream certificate.

You can also try to verify that Caddy is working correctly by taking the router port forwarding out of the question and cURLing the Caddy host directly:

curl -iL https://[Caddyhost]:8963 -H "Host:service.mydomain.com"


(giopas) #7

Hi,

Thanks for the guidance!

If I run curl, I get:

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol


(Matthew Fay) #8

Kinda looks like the error you get when you make a HTTPS request to a HTTP endpoint… Try curl -iL http://[Caddyhost]:8963 -H "Host:service.mydomain.com" (replaced https:// with http://).


(giopas) #9

Thank you again Whitestrake.

I summarize:

If I have on my caddy.conf file the following block:

http://service.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://192.168.2.87:8989 {
		keepalive 32
		transparent
		websocket
	}
}

I can access the service via browser (both from inside and outside my LAN, in the last case having portforwarded port 80 to 8963) and on curl I get the following:

# curl -iL http://192.168.1.3:8963 -H "Host:service.mydomain.com"                                                                                                                                                                                                        
HTTP/1.1 401 Unauthorized                                                                                                                                                                                                                                                           
Content-Length: 149                                                                                                                                                                                                                                                                 
Content-Type: text/html; charset=utf-8                                                                                                                                                                                                                                              
Date: Mon, 09 Jul 2018 08:06:15 GMT                                                                                                                                                                                                                                                 
Referrer-Policy: strict-origin-when-cross-origin                                                                                                                                                                                                                                    
Set-Cookie: TWISTED_SESSION=e7fa4a49137b7647168e0a2c0541f0cc; Path=/                                                                                                                                                                                                                
Strict-Transport-Security: max-age=31536000;                                                                                                                                                                                                                                        
Www-Authenticate: Basic realm="Service"                                                                                                                                                                                                                                           
X-Content-Type-Options: nosniff                                                                                                                                                                                                                                                     
X-Frame-Options: DENY                                                                                                                                                                                                                                                               
X-Xss-Protection: 1; mode=block                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                    
<html>                                                                                                                                                                                                                                                                              
  <head><title>401 - Unauthorized</title></head>                                                                                                                                                                                                                                    
  <body>                                                                                                                                                                                                                                                                            
    <h1>Unauthorized</h1>                                                                                                                                                                                                                                                           
    <p>401 Authentication required</p>                                                                                                                                                                                                                                              
  </body>                                                                                                                                                                                                                                                                           
</html>

However if I change the caddy.conf file with the following (and I portforward port 8443 to 8963), I get:

https://service.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / https://192.168.2.87:9898 {
		keepalive 32
		transparent
		websocket
	}
}

in this case, with the browser I can access the service from within the LAN, but I can’t from outside the LAN (i.e. using https://service.mydomain.com:8443).

However I just noticed that as soon as I make the above change in the config, all other reverse proxies which are HTTP (like transmission, etc…) fails to work.

curl: (7) Failed to connect to 192.168.1.3 port 8963: Connection refused

Indeed caddy does not start!

ps aux |grep caddy
 7267 admin       956 S   grep caddy 

It seems that I miss a config in caddy somewhere.

Here is the beginning of the caddy.conf file, in case it is useful:

(gzipconf) {
	gzip {
		ext *
		level 7
		min_length 1
	}
}

(addheader) {
	header / {
		Strict-Transport-Security "max-age=31536000;"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
	}
}

Probably I am missing something really stupid here!

Thanks!


(Matthew Fay) #10

So you’re actually making two sets changes between those two Caddyfiles:

  1. http://service.mydomain.com:8963
    Changing scheme to HTTPS
  2. http://192.168.2.87:8989
    Changing scheme to HTTPS and port to 98898

The multiple sets of changes are confusing the issue. Lets throw everything out for now - move your Caddyfile to Caddyfile.bak or similar - and start fresh with this Caddyfile:

https://service.mydomain.com:8963 {
  tls self_signed
  status 200 /
}

Once Caddy is started, run curl -kIL https://service.mydomain.com:8443 from outside your network and lets see what you get. We want to see HTTP/1.1 200 OK and Server: Caddy along with the current date. Once we get that working, we can look into adding the reverse proxy back in.


(giopas) #11

Thank you Matthew,

I did what you asked and lunched caddy as follows:

~# ./caddy -agree -port 20015 -root /home/Qhttpd -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid -conf test.conf          
Activating privacy features... done.                              
https://service.mydomain.com:8963                               
WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192".

Then I run curl from outside my network (installing termux on my smartphone) and I got:

~# curl -kIL https://service.mydomain.com:8443
HTTP/2 200
server: Caddy
date: Tue, 10 Jul 2018 08:06:11 GMT

So far, so good :slight_smile:

What’s then now? :blush:


(Matthew Fay) #12

Alright, so we know that part works.

Lets add:

proxy / https://192.168.2.87:9898 {
  transparent
  websocket
}

And remove status 200 /. Then run the same cURL command again.

We want it to have a very similar response to the result of curl -IL https://192.168.2.87:9898.


(giopas) #13

Thank you again Matthew,

I confirm that I receive a very similar status:

~# curl -kIL https://service.mydomain.xyz:8443
HTTP/2 502
content-type-options: nosniff
content-lenght: 16
date: Wed, 11 Jul 2018 07:53:56 GMT

If I open via browser (https://service.mydomain.com:8443), I receive the usual SSL certificate error and then:

502 Bad Gateway

However if I open the browser and use the internal IP (https://192.168.2.87:9898) I can access the resource without any problem. Of course the router forwards port 8443 to the caddy server port 8963. :thinking:


(Matthew Fay) #14

Alright, one more thing to add then. Change the proxy block to this:

proxy / https://192.168.2.87:9898 {
  transparent
  websocket
  insecure_skip_verify
}

Let us know if that makes a difference.


(giopas) #15

Ok, this time it seems working, even if I still need to accept invalid certificate on the browser.

For anyone’s reference, here is the working block:

https://service.mydomain.com:8963 {
  tls self_signed
  proxy / https://192.168.2.87:9898 {
  transparent
  websocket
  insecure_skip_verify
  }
}

In this case curl gives me the following:

~# curl -kIL https://service.mydomain.com:8443
HTTP/2 401
content-type: text/html; charset=utf-8
date: Wed, 11 Jul 2018 12:22:43 GMT
server: Caddy
server"TwistedWeb/12.0.0
set-cookie: TWISTED_SESION=[.......]; Path=/
www-authenticate: Basic realm="[Service]"
content-lenght: 149

Perfect! Now:

  1. how can I get rid of the ssl certificate error (i.e. how to enable LetsEncrypt certificate)?

  2. as I cannot start SSL connection on port 443, how can I automatically redirect queries from http://service.mydomain.com to https://service.mydomain.com:8443?

  3. Why if I mix the above https block we made together with other http blocks in my caddy.conf, caddy does not start anymore?

(gzipconf) {
	gzip {
		ext *
		level 7
		min_length 1
	}
}

(addheader) {
	header / {
		Strict-Transport-Security "max-age=31536000;"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
	}
}

http://repository.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://192.168.2.87:9745 {
		keepalive 32
		transparent
		websocket
	}
}

https://service.mydomain.com:8963 {
  tls self_signed
  proxy / https://192.168.2.87:9898 {
  transparent
  websocket
  insecure_skip_verify
  }
}
  1. [shall I open a different topic here?] if there is another service which does not work with SSL certificates, can I use Caddy reverse proxy to make it SSL encrypted? For this other service I have the caddy.conf block already in place like the first one in the first post.

Thanks a million for your excellent help!


(Matthew Fay) #16

Remove tls self_signed from the Caddyfile and make sure your domain name is publicly resolvable and Caddy is reachable on port 80. The reason I suggested this is it removes one level of complexity while troubleshooting Caddy (the certificate requisition/Automatic HTTPS stage), removing it returns the site back to validated certificates.

There’s an implicit HTTP version of your site that Caddy deploys with an automatic HTTP->S redirect. You just need to override its behaviour by explicitly defining your own. For example:

http://example.com {
  redir https://example.com:8443
}
https://example.com:8443 {
  ...
}

In your case, naturally the second block will have a different port number as you are translating the port when you forward it from your NAT device.

Not sure. What does Caddy output when you try to start it and it fails?

Generally, yes. You set Caddy to serve an arbitrary hostname via HTTPS, and in the site definition, you proxy internally to a HTTP-only service, as we’ve done here.


(giopas) #17

Thank you again Alex, I feel I’m getting close thanks to your invaluable help and patience!

In relation to tls self_signed, if I take it out, I get the following error, which makes caddy fail to start:

2018/07/11 17:41:31 [service.mydomain.com] failed to get certificate: [service.mydomain.com] error presenting token: presenting with standard HTTP provider server: Could not start HTTP server for challenge -> listen tcp :80: listen: address already in use

What is wrong here?

In relation to having both the HTTP and HTTPS reverse proxy working, I changed test.conf file as follows:

http://service.mydomain.com:8963 {
  redir https://service.mydomain.com:8963
}

https://service.mydomain.com:8963 {
  proxy / https://192.168.2.87:9898 {
  transparent
  websocket
  insecure_skip_verify
  }
}

But when running on the terminal I got:

2018/07/11 17:56:27 cannot multiplex service.mydomain.com (not TLS) and service.mydomain.com (TLS) on same listener

I think this is related to when you say:

In your case, naturally the second block will have a different port number as you are translating the port when you forward it from your NAT device.

What should I do in practice to make it work? Shall I provide additional options to the launch command (see below)?

./caddy -agree -port 20015 -root /home/Qhttpd -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid -conf caddy.conf

Thanks again!


(Matthew Fay) #18

Something on your server is already listening on port 80, which blocks Caddy from doing the same. Since it can’t do that, it can’t get a certificate for your site, so it aborts the startup and lets you know.

You could use sudo netstat -tulpn | grep :80 to try determine which process has port 80.

It is not possible for a HTTP listener and a HTTPS listener to share the same port. They must be separate. The default certificate validation that Caddy will use for Automatic HTTPS requires your HTTP listener be accessible on port 80, so it’s a good idea to leave your HTTP site there. You can move your HTTPS site to the higher port without issue.

This isn’t related to the HTTP(S) listener multiplexing, but it WILL cause you a problem.

Specifically, with the directive redir https://service.mydomain.com:8963, you’ve configured Caddy to tell clients to make a new request to that address. But clients on the public internet don’t see port 8963, because they don’t see your Caddy host; they see your router, and your router has port 8443.

So, change the port in the redir directive to the port you want clients on the public internet to access your HTTPS site. Caddy has to tell connecting clients to reconnect on port 8443 instead, but must listen on port 8963, because your router is in the middle and that’s how you’ve connected the traffic.


(giopas) #19

Ok, I modified the file as follows:

http://service.mydomain.com:8963 {
  redir https://service.mydomain.com:8443
}

https://service.mydomain.com:8964 {
  proxy / https://192.168.2.87:9898 {
  transparent
  websocket
  insecure_skip_verify
  }
}

In addition, on the router I have done following port-forwards:

Incoming HTTP connection port 80 -> [Caddyhost]:8963
Incoming HTTPS connection port 8443 -> [Caddyhost]:8964

You are right, as I am running a QNAP, even if I disable webserver on it, it still uses port 80 [1]. I should try to play with service binding, using the two ethernet ports I have on the NAS to try to free port 80 at least on one interface.

[1] https://forum.qnap.com/viewtopic.php?f=32&t=131636&start=15#p664856


(Matthew Fay) #20

Nice, it seems like all those ports should line up. Does your site work as expected when you browse to https://service.mydomain.com:8443?