Unexpected "bind: address already in use" for unrelated port

Alex_C · December 24, 2022, 5:03pm

1. Output of `caddy version`:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

caddy reverse-proxy --debug --from 10.128.0.48:9002 --to 127.0.0.1:9001
initially tried to proxy from the same port:
caddy reverse-proxy --debug --from 10.128.0.48:9001 --to 127.0.0.1:9001

a. System environment:

Ubuntu 20.04.5 LTS

b. Command:

# ip -c -br -4 a l dev eth0
eth0             UP             10.128.0.48/24
# lsof -Pni :9001,9002
COMMAND       PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
docker-pr 1614549 root    4u  IPv4 19798428      0t0  TCP 127.0.0.1:9001 (LISTEN)

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane. -->

d. My complete Caddy config:

None

3. The problem I’m having:

In case of proxying 10.128.0.48:9002 to 127.0.0.1:9001 Caddy tries to bind to 80 by some reason what is confusing behavior from the user’s point of view
In case of proxying 10.128.0.48:9001 to 127.0.0.1:9001 “bind: address already in use” is even more confusing, seems Caddy is trying to bind to 0.0.0.0 instead

4. Error messages and/or full log output:

# caddy reverse-proxy --debug --from 10.128.0.48:9002 --to 127.0.0.1:9001
2022/12/24 16:40:56.752 WARN    admin   admin endpoint disabled
2022/12/24 16:40:56.752 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "proxy"}
2022/12/24 16:40:56.752 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00048d260"}
2022/12/24 16:40:56.753 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/12/24 16:40:56.753 INFO    tls     finished cleaning storage units
2022/12/24 16:40:56.767 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2022/12/24 16:40:56.767 INFO    http    enabling HTTP/3 listener        {"addr": ":9002"}
2022/12/24 16:40:56.767 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022/12/24 16:40:56.767 DEBUG   http    starting server loop    {"address": "[::]:9002", "tls": true, "http3": true}
2022/12/24 16:40:56.767 INFO    http.log        server running  {"name": "proxy", "protocols": ["h1", "h2", "h3"]}
2022/12/24 16:40:56.767 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0xc00048d260"}
Error: loading new config: http app module: start: listening on :80: listen tcp :80: bind: address already in use

And for the same port:

# caddy reverse-proxy --debug --from 10.128.0.48:9001 --to 127.0.0.1:9001
2022/12/24 16:44:39.546 WARN    admin   admin endpoint disabled
2022/12/24 16:44:39.546 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "proxy"}
2022/12/24 16:44:39.548 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0005f63f0"}
2022/12/24 16:44:39.548 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/12/24 16:44:39.548 INFO    tls     finished cleaning storage units
2022/12/24 16:44:39.562 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2022/12/24 16:44:39.562 INFO    tls.cache.maintenance   stopped background certificate maintenance      {"cache": "0xc0005f63f0"}
Error: loading new config: http app module: start: listening on :9001: listen tcp :9001: bind: address already in use

5. What I already tried:

Google, community forum

6. Links to relevant resources:

francislavoie · December 26, 2022, 11:24pm

This is because Caddy is HTTPS by default, and will try to set up HTTP->HTTPS redirects which involves port 80.

You can explicitly tell Caddy that you want HTTP by prefixing your address with http://

The “from” address is the site address, which is used for hostname matching and certificate automation. It doesn’t control the bind address.

If you want to bind to a particular interface, you should use a Caddyfile and use the bind directive to do so.

Alex_C · December 27, 2022, 7:13am

Hello @francislavoie
Thanks a lot for the explanation!
This statement seems a bit confusing:

Perhaps you mean it’s not about what I want, but about what I have to do (so this is the only option)?
Because in this scenario I don’t care about bind, I just want to make it work out-of-the-box ¯_(ツ)_/¯

Let me try to summarize: if you proxy to the same port on the localhost - you MUST use bind to specific interface and you MUST use Caddyfile
Correct?
If yes - I believe that should be added in documentation, coz current behaviour is really confusing

emilylange · December 27, 2022, 11:42am

You cannot use the caddy reverse-proxy cli feature.
You can use a Caddyfile with the bind directive.

Technically speaking, you could also use any of the other config adapters or just plain json, but that’s not something I would recommend.

So mostly yes - yes.

Alex_C · December 27, 2022, 12:01pm

Thanks for the clarification!
Is there any chance that adding --bind to reverse-proxy cli feature can be considered as a future improvement (someday somehow)?
@emilylange @francislavoie

francislavoie · December 27, 2022, 9:07pm

Possibly, you could open a feature request.

But I strongly recommend using a Caddyfile if you plan to leave the proxy running long-term, it’s a much better way to keep it running because you have a file that keeps your config so you don’t need to remember the exact arguments to use for the command to run it, and you can use systemd to keep Caddy running if you reboot your server.

The CLI commands are really just meant for quick-and-dirty use when testing stuff or to proxy a development server. If you’re doing anything more complex than that, a config file is the way to go.

Alex_C · December 28, 2022, 7:56am

Yes @francislavoie that was exactly for testing.
I was thinking “why spend time for writing 20 lines of nginx.conf if I can do it as usual just in 1 command with Caddy”.
And got stuck.
And had to get back to nginx.conf

So I will submit feature request if you don’t mind.
Frankly speaking all 3 parts could be improved:

documentation
diagnostic message (I believe Caddy could try to give us a hint for such scenario, not just “address already in use”)
additional option for CLI reverse-proxy

francislavoie · December 28, 2022, 8:14am

Fair enough.

Well, that message is accurate. You can’t listen to port 9001 when you already have a service using that port. That message is coming from the kernel (or if not, the Go stdlib, anyway).

Using bind would allow you to specify the interface (i.e. the IP address) to bind to, and that might work if your other service is also binding to a specific interface (i.e. not 0.0.0.0, which means “all IPv4 interfaces”).

Otherwise, you should just use a different port number.

Alex_C · December 28, 2022, 11:26am

I got your point
But the fact that some message comes from kernel doesn’t make user interaction perfect by default and it doesn’t mean things can’t be done better.

Recently I was debugging Wireguard and also got that “address already in use”.
I re-checked my setup dozen times (I have a lot of network interfaces and a lot of suntetworks in use and some IP conflict was somehow possible) so I really was confused because found nothing.

And only after some time I realized that it was not about (IP) address conflict but about port conflict.

So this kernel message is far from perfect, believe me

francislavoie · December 28, 2022, 3:22pm

Well a source address includes the port. The message is accurate. You made the assumption that the address does not include the port, which is incorrect.

Alex_C · January 19, 2023, 8:08pm

Sorry for bothering you again @francislavoie , I met almost the same issue, and don’t want to duplicate request.
So here is the case - just for test I need simplest reverse proxy, plain HTTP.
Based on your previous advice I used HTTP prefixes, but got the same unexpected result as in my 1st scenario :

$ caddy reverse-proxy --from http://10.128.0.48:12080 --to http://127.0.0.1:42249
2023/01/19 19:58:59.063 WARN admin admin endpoint disabled
2023/01/19 19:58:59.063 INFO http enabling automatic HTTP->HTTPS redirects {“server_name”: “proxy”}
2023/01/19 19:58:59.064 INFO tls.cache.maintenance started background certificate maintenance {“cache”: “0xc0003fd030”}
2023/01/19 19:58:59.064 INFO tls cleaning storage unit {“description”: “FileStorage:/root/.local/share/caddy”}
2023/01/19 19:58:59.065 INFO tls finished cleaning storage units
2023/01/19 19:58:59.087 INFO pki.ca.local root certificate is already trusted by system {“path”: “storage:pki/authorities/local/root.crt”}
2023/01/19 19:58:59.087 INFO http enabling HTTP/3 listener {“addr”: “:12080”}
2023/01/19 19:58:59.087 INFO failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See UDP Receive Buffer Size · quic-go/quic-go Wiki · GitHub for details.
2023/01/19 19:58:59.087 INFO http.log server running {“name”: “proxy”, “protocols”: [“h1”, “h2”, “h3”]}
2023/01/19 19:58:59.087 INFO tls.cache.maintenance stopped background certificate maintenance {“cache”: “0xc0003fd030”}
Error: loading new config: http app module: start: listening on :80: listen tcp :80: bind: address already in use

Is there some simple way(one-liner in CLI would be perfect) to disable all those great automatic features and just get the simplest reverse proxy?
Thanks in advance!

emilylange · January 19, 2023, 8:23pm

That’s a bug that got fixed in reverseproxy: Don't enable auto-https when `--from` flag is http by IndeedNotJames · Pull Request #5269 · caddyserver/caddy · GitHub, which will be included in the next release.

You could either wait for an upcoming release, build that unreleased version from source using xcaddy, or use a Caddyfile

system · January 23, 2023, 5:04pm

This topic was automatically closed after 30 days. New replies are no longer allowed.