HTTP Issues with Reverse Proxy

Help
1

1. Caddy version (caddy version):

2.0.0

2. How I run Caddy:

a. System environment:

Ubuntu 20.04 LTS.
Installed natively with:

> wget https://github.com/caddyserver/caddy/releases/download/v2.0.0/caddy_2.0.0_linux_amd64.deb
> sudo dpkg -i caddy_2.0.0_linux_amd64.deb

I have a properly-registered domain (for the sake of argument, let’s say my.example.com) pointing to an Internal IP on my network, so it’s not resolvable by public, but it is resolvable to all my computers (using Tailscale).

b. Command:

I restart my Caddy server using the following command, so it picks up my config:

> caddy reload --config /etc/caddy/Caddyfile
2020/05/25 17:00:58.290	INFO	using provided configuration	{"config_file": "/etc/caddy/Caddyfile", "config_adapter": ""}

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

Caddyfile

http://myone.example.com {
    reverse_proxy 0.0.0.0:8080
}

mytwo.example.com:80 {
    reverse_proxy 0.0.0.0:8181
}

Note: The real domain in my /etc/caddy/Caddifile is here changed to “my.example.com” for security.

3. The problem I’m having:

The first syntax (http://) does not work, I get 404. Documentation implies that the first syntax should work, so that seems like a bug?

The second syntax (:80 in the end) does work if you access the URL with curl from command line, but it fails in any major GUI browser. I suspect due to dangling HSTS somewhere that curl ignores but real browsers don’t?

4. Error messages and/or full log output:

n/a

5. What I already tried:

n/a

6. Links to relevant resources:

2

0.0.0.0 isn’t a real IP address. It just means “any IP”. I think curl accepts it as a loopback though.

Is this a service running on the same machine? If so, use localhost or 127.0.0.1 for the reverse_proxy address.

Also you might have issues with hairpinning if you use the domain on the same machine, unless you have domain in your hosts file mapped to 127.0.0.1 or something like that. Depends on the router.

Edit: Scratch that part, I missed your note regarding the domain in 2a of your post.

3

They just go to stderr (i.e. your terminal, or whatever parent process is capturing standard error).

Replace your reverse_proxy with respond Hello or something just as a sanity check that you’re even hitting the server you think are you, first.

4

You can run journalctl -u caddy to see the logs when running with a systemd service

2 Likes
5

Thanks for the responses.

I updated the report. Something extra bad was happening with Caddy (I think the process was in some kind of corrupt state), that became less bad after a server reboot.

Bottomline:
“domain.com:80” syntax “works” (but not in any major browser, just in curl), http://domain.com syntax doesn’t work at all.

6

Thank you, Matt. I was able to find an operator error with the “respond” trick.

I believe both “http://domain.com” and “domain.com:80” syntax work. But as I mentioned - only in curl, Postman or such client, not in any GUI Browser. I even tried to delete HSTS records manually and install fresh browsers, but none of that seems to help.

Any suggestions?

Many thanks.

P.S. @francislavoie “journalctl” command was great to see startup messages, I still cannot see access log messages anywhere, though? Thank you!

7

domain.com:80 and http://domain.com are the same thing from Caddy’s perspective. If you’re getting a 404, that’s coming from your upstream service, not Caddy! If Caddy failed to match a request, you’d see an empty 200 response instead.

I’m not certain why your access logs wouldn’t be writing to disk, but wherever it is should be writable by the caddy user.

For the time being, you can just set log on each of your sites (no options block) which will make access logs output to stdout, which you’ll find via journalctl.

HSTS should have no effect here, all that would do is force the browser to load the site via HTTPS.

1 Like
8

@francislavoie correct, the difference between http:// and :80 was my fault. :sorry:

The log trick worked!

I now have:

http://one.example.com {
    reverse_proxy 127.0.0.1:8080
    log 
}

http://two.example.com {
    reverse_proxy 127.0.0.1:8081
    log 
}

and I can see the access logs from both when I run journalctl -u caddy -f. Thank you!

As for HSTS - I think that’s what is going on, indeed. It is forcing browser to load HTTPS, but there is no HTTPS router, so the requests fail. Makes sense?

1 Like
9

Did you put your site on the HSTS preload? Based on clearing browser caches and using fresh installs and still getting this issue, I’d assume this is at least a possibility? (You’d know if you did; you have to set a specific header and then go to a website to request being preloaded.)

If you ARE on the list, you’re a bit out of luck unless you can get public facing, validated HTTPS up and running again and start sending HSTS headers without preload: HSTS Preload List Removal

10

Thank you, @Whitestrake. No, I haven’t done anything conscious with HSTS and these hosts. The only thing - I was using them over HTTPS with Caddy, and I am trying to to switch them to HTTP now.

11

In that case, fully clearing caches - or using a freshly installed (i.e. completely wiped or never previously installed) browser - should resolve this issue with the browsers inappropriately assuming HTTPS.

1 Like
12

Tried both of those things but still getting the same issue. Weird :frowning:

13

I don’t think regular cache clears or reinstalls do the trick. You have to tell the browser specifically to forget HSTS. How to clear HSTS settings in Chrome and Firefox

14

This topic was automatically closed after 30 days. New replies are no longer allowed.