On Demand SSL on ports 80,443 with health checks?

Moe · May 22, 2020, 3:20am

1. Caddy version (`caddy version`):

Version 2

2. How I run Caddy:

On EC2 servers on AWS behind a Load Balancer.

a. System environment:

Ubuntu 18.04

b. My complete Caddyfile or JSON config:

{
  storage file_system {
    root /mnt/efs/fs1
  }
}


:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


*.amazonaws.com:80 {

respond "Working!"
}

3. The problem I’m having:

My main problem is I want on_demand SSL while also needing health checks from the load balancer. (I have several Caddy servers behind a load balancer all sharing a filesystem).

Everything works fine if I use the healthcheck from the load balancer on port 80. However, the rest of the traffic on port 80 is not redirected to 443 and in turn not generating certificates

5. What I already tried:

I tried changing the healthchecks to use Port 81. However, that does’t work and the healthchecks from Amazon always fail.

Here’s the updated Caddyfile I tried with no luck. I tried many variations of this and am hoping someone can point out a proper approach or what I’m doing wrong.

{
  storage file_system {
    root /mnt/efs/fs1
  }
}


:80,:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


*.amazonaws.com:81 {

respond "Working!"
}

I’m also trying to avoid using port 443 for healthchecks since when I did that it was trying to generate certificates for internal load balancer.

Last note, I don’t have an option of choosing the hostname the load balancer uses. This is a restriction with AWS.

Any pointers would be greatly appreciated.

Whitestrake · May 22, 2020, 4:22am

In what manner, specifically, do they always fail? What information does Amazon provide on its health check? It’s really important to get the exact error. Did their check timeout? Was there a protocol error? Is Amazon seeing non-200 responses for the endpoint? The nature of Amazon’s error will tell us what we need to troubleshoot.

Moe · May 22, 2020, 6:24am

Unfortunately, Amazon does not supply any sort of information on the type of load balancer I’m using. Simply says “unhealthy”

From my understanding Amazon load balancer is simply asking looking for a 200 success code message. And this does work perfectly on port 80. Just when I try this on port 81 (or any other port) it doesn’t work.

I guess what my bottom line is:
Assuming the issue is let’s say on Amazon, does CaddyServer support having ports 80 and 443 do reverse proxy, while a different port just reponse to the health check.

From my Caddyfile above, am I missing something?

I’m testing for hours different things and it seems very inconsistent (to me at least).

For example, this would work:

    *.amazonaws.com:80 {

    respond "Working!"
    }

But this won't: 

    :80 {

    respond "Working!"
    }

I'm hoping you can shed some more light on this, I'd really appreciate even random ideas of what I can attempt and test.

Whitestrake · May 22, 2020, 6:27am

I’m going to guess that one possibility is that Amazon requires a HTTP endpoint, not a HTTPS endpoint.

*.amazonaws.com:80 will produce a HTTP listener, *.amazonaws.com:81 will produce a self-signed/locally-trusted HTTPS listener with a redirect set up on port 80 (Caddy is HTTPS-first, HTTP only on explicit scheme or default HTTP port).

Try http://*.amazonaws.com:81 and see if that makes a difference.

Moe · May 22, 2020, 7:01am

Thanks @Whitestrake
I just tried that but unfortunately it didn’t work. However it gave me several ideas of things I can try along these lines. I’ll try different ports, protocols and combinations of those. Hopefully if I shoot enough I’ll hit something.

Moe · May 22, 2020, 3:51pm

Still no luck for me.

I have another way of asking this. Is it possible to do this:

All port 80 traffic gets routed to HTTPS
All HTTPS traffic get on_demand certificate
Traffic from a specific source, or traffic to a specific endpoint skips the on_demand certificate.

The bottom line for me is that everything works if I set it up on port 80. But I’m trying to avoid certificates being generated for my load-balancer.

francislavoie · May 22, 2020, 4:25pm

If by “routed”, you mean “redirected”, then yes, Caddy automatically sets up HTTP->HTTPS redirects for you via the Automatic HTTPS feature:

I think you’re looking for something like this:

{
	on_demand_tls {
		ask http://example.com/ask
	}
}

*.example.com {
	# your site's directives

	tls {
		on_demand
	}
}

http://for-aws.example.com {
	respond "OK!"
}

Moe · May 22, 2020, 5:54pm

Yayyy!! This finally worked. I appreciate it.

I modified the code just a bit as follows:

{
        storage file_system {
         root /mnt/efs/fs1
        }
        on_demand_tls {
                ask https://api.nvssolutions.com/api/domain-checker-base
        }
}

:443 {
    reverse_proxy sites.nvssolutions.io:443

        tls moe@nvssolutions.com {
          on_demand
        }
}


http://*.amazonaws.com {
        respond "OK!"
}

Can I ask one more? Now when I go to a site it doesn’t seem to auto redirect to HTTPS.

If I go to https:// it works fine, but http:// it says: Client sent an HTTP request to an HTTPS server.

This shows error => http://acl1.tagahash.com
This works correctly => https://acl1.tagahash.com

Can you guide me how I can finish this and get it redirected to https?

francislavoie · May 22, 2020, 6:34pm

I’m pretty sure that the redirects are only enabled if you use an actual domain as the site label, but I’m not certain.

I think you can add your own redirects like this though:

http:// {
	redir https://{host}{uri}
}

Also just as an additional tip, you can use the caddy fmt command to make your Caddyfile prettier.

matt · May 22, 2020, 6:48pm

It should enable redirects regardless.

Moe · May 22, 2020, 7:06pm

Thanks. It’s giving me an error now.

Job for caddy.service failed because the control process exited with error code. See "systemctl status caddy.service" and "journalctl -xe" for details.

Error I"m seeing:

cannot make a TLS automation policy from a server block that has a host-less address when there are other server block addresses lacking a host

This is really where I keep getting stuck. The ondemand SSL is for domains that are not predefined so I must just keep that as host-less.

matt · May 22, 2020, 11:40pm

What’s your entire and unaltered Caddyfile at this point?

Moe · May 23, 2020, 2:04am

@matt thanks for looking at this for me. Great product btw.

#server1

{
        storage file_system {
         root /mnt/efs/fs1
        }
        on_demand_tls {
                ask https://api.nvssolutions.com/api/domain-checker-base
        }
}


:443 {
    reverse_proxy sites.nvssolutions.com:443

        tls moe@nvssolutions.com {
          on_demand
        }
}

http:// {
        redir https://{host}{uri}
}

https://*.amazonaws.com {
        respond "OK!"
}

matt · May 23, 2020, 8:04pm

You can get rid of the http:// server block, that redirect happens automatically.

Moe · May 24, 2020, 4:34am

Thanks. But for some reason it’s still not forwarding my traffic to HTTPS.

When i got to an http site I get this message:

“Client sent an HTTP request to an HTTPS server.”

matt · May 24, 2020, 4:55am

Your reverse proxy isn’t configured to use TLS. Replace :443 with https:// at the beginning, see how that goes.

Moe · May 24, 2020, 5:21am

Thanks again Matt.

I tried that and still no luck, then it occurred to me that this might be an issue with my AWS Global Accelerator. https://aws.amazon.com/global-accelerator/

Here’s my findings.
If I forward my domain directly to the Caddy server it does forward correctly to HTTPS. However if I forward my domain to the static IP address on my load balancer, that doesn’t seem to forward to https. Clearly not an issue with Caddy.

Here’s a screenshot of the network tab for these requests. I don’t see the “server : Caddy” in the header, I guess that means it’s not even hitting the Caddy server.

Moe · May 25, 2020, 1:27am

Thanks everyone for your help. This ended up mostly being an issue with my configuration on AWS.

All good now.

In case anyone else is building something similar using AWS. Keep in mind you must use a TCP load balancer but since each target group only accepts one type of traffic (http or https) and you can’t redirect on network load balancer, you must create 2 different target groups. One of http and one for https.

system · June 21, 2020, 3:30am

This topic was automatically closed after 30 days. New replies are no longer allowed.