Caddy behind nginx reverse proxy result in infinite 301 redirections

Help
1

1. The problem I’m having:

My infrastructure has a public server running nginx as a reverse proxy.
Then, I have a server with caddy running that redirect the requests to different Docker containers. Caddy itself is running under Docker.

The request is redirected by Caddy itself, not the service. To ensure this, I stopped the service (frontend), and the redirection still persist.

I have no clue why this happen.

The https if managed by NGINX (reverse proxy), and caddy is serving as http in a private network.

2. Error messages and/or full log output:

caddy-1     | {"level":"debug","ts":1748963671.2795904,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"cacafrontend:80","total_upstreams":1}
caddy-1     | {"level":"debug","ts":1748963671.2806203,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"cacafrontend:80","duration":0.000991199,"request":{"remote_ip":"192.168.102.103","remote_port":"50824","client_ip":"redacted_ipv6:9374","proto":"HTTP/1.1","method":"GET","host":"redacted.example.net","uri":"/","headers":{"X-Forwarded-For":["redacted_ipv6:9374, 192.168.102.103"],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["cross-site"],"X-Forwarded-Server":["redacted.example.net"],"X-Real-Ip":["redacted_ipv6:9374"],"X-Forwarded-Proto":["https"],"Sec-Gpc":["1"],"X-Forwarded-Host":["redacted.example.net"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Priority":["u=0, i"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Sec-Fetch-Mode":["navigate"]}},"headers":{"X-Cache-Lookup":["HIT from proxy1:82"],"Connection":["keep-alive"],"Server":["nginx"],"Date":["Tue, 03 Jun 2025 15:14:32 GMT"],"Content-Type":["text/html"],"Content-Length":["162"],"Location":["https://redacted.example.net/"],"X-Cache":["MISS from proxy1"],"Via":["1.1 proxy1 (squid/5.7)"]},"status":301}
caddy-1     | {"level":"info","ts":1748963671.2810364,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.168.102.103","remote_port":"50824","client_ip":"redacted_ipv6:9374","proto":"HTTP/1.1","method":"GET","host":"redacted.example.net","uri":"/","headers":{"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.5"],"Priority":["u=0, i"],"X-Forwarded-For":["redacted_ipv6:9374"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["cross-site"],"X-Forwarded-Host":["redacted.example.net"],"X-Forwarded-Server":["redacted.example.net"],"Connection":["close"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"X-Real-Ip":["redacted_ipv6:9374"]}},"bytes_read":0,"user_id":"","duration":0.001466639,"size":162,"status":301,"resp_headers":{"Content-Type":["text/html"],"Content-Length":["162"],"Location":["https://redacted.example.net/"],"X-Cache":["MISS from proxy1"],"Server":["Caddy","nginx"],"X-Cache-Lookup":["HIT from proxy1:82"],"Via":["1.1 proxy1 (squid/5.7)"],"Date":["Tue, 03 Jun 2025 15:14:32 GMT"]}}
caddy-1     | {"level":"debug","ts":1748963671.3094547,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"cacafrontend:80","total_upstreams":1}
caddy-1     | {"level":"debug","ts":1748963671.3142984,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"cacafrontend:80","duration":0.004796872,"request":{"remote_ip":"192.168.102.103","remote_port":"50826","client_ip":"redacted_ipv6:9374","proto":"HTTP/1.1","method":"GET","host":"redacted.example.net","uri":"/","headers":{"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Gpc":["1"],"X-Real-Ip":["redacted_ipv6:9374"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"],"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Server":["redacted.example.net"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Priority":["u=0, i"],"X-Forwarded-Host":["redacted.example.net"],"X-Forwarded-For":["redacted_ipv6:9374, 192.168.102.103"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["cross-site"]}},"headers":{"Content-Length":["162"],"Location":["https://redacted.example.net/"],"X-Cache-Lookup":["HIT from proxy1:82"],"Via":["1.1 proxy1 (squid/5.7)"],"Connection":["keep-alive"],"Content-Type":["text/html"],"Date":["Tue, 03 Jun 2025 15:14:32 GMT"],"X-Cache":["MISS from proxy1"],"Server":["nginx"]},"status":301}
caddy-1     | {"level":"info","ts":1748963671.3143983,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.168.102.103","remote_port":"50826","client_ip":"redacted_ipv6:9374","proto":"HTTP/1.1","method":"GET","host":"redacted.example.net","uri":"/","headers":{"Connection":["close"],"Sec-Fetch-Dest":["document"],"Priority":["u=0, i"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"X-Forwarded-Host":["redacted.example.net"],"X-Forwarded-Server":["redacted.example.net"],"X-Forwarded-For":["redacted_ipv6:9374"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"X-Real-Ip":["redacted_ipv6:9374"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["cross-site"]}},"bytes_read":0,"user_id":"","duration":0.004950677,"size":162,"status":301,"resp_headers":{"Via":["1.1 proxy1 (squid/5.7)"],"Date":["Tue, 03 Jun 2025 15:14:32 GMT"],"X-Cache":["MISS from proxy1"],"Server":["Caddy","nginx"],"Content-Type":["text/html"],"Content-Length":["162"],"Location":["https://redacted.example.net/"],"X-Cache-Lookup":["HIT from proxy1:82"]}}

3. Caddy version:

caddy 2.10.0 (docker)

4. How I installed and ran Caddy:

Caddy is running under Docker:

  caddy:
    image: caddy
    ports:
      - 8080:8080
    volumes:
      - ./caddy:/etc/caddy

a. System environment:

Debian 12
Docker 28.2.2

b. Command:

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
services:
  frontend:
    init: true
    build:
      context: ./frontend
      target: dev
  backend:
    init: true
    tty: true
    build:
      context: ./backend
      target: dev
    environment:
      - {redacted}
  caddy:
    image: caddy:2.7.2
    ports:
      - 8080:8080
    volumes:
      - ./caddy:/etc/caddy

d. My complete Caddy config:

{
  debug
  servers {
    trusted_proxies static 192.168.102.103/24 192.168.102.88/24
  }
}

http://redacted.example.net:8080 {
  log {
      output stdout
      format console
  }
  reverse_proxy frontend:80 {
    trusted_proxies 192.168.102.103/24 192.168.102.88/24
  }
}

http://redactedback.example.net:8080 {
log {
    output stdout
    format console
}
  reverse_proxy backend:5000
}

e. My complete nginx config:

server {
        listen          80;
        listen          [::]:80;
        server_name     redacted.example.net;

        rewrite         ^(.*)$ https://$host$1 permanent;
}

server {
        listen           443 ssl http2;
        listen          [::]:443 ssl http2;
        server_name     redacted.example.net;

        access_log      /var/log/nginx/redacted_front.access.log;
        error_log       /var/log/nginx/redacted_front.error.log;

        location / {
                proxy_pass      http://192.168.102.39:8080;
                proxy_http_version 1.1;

                allow           all;
        }
}

proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-Host $host;
proxy_set_header        X-Forwarded-Server $host;
proxy_set_header        X-Forwarded-Proto https;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_hide_header       X-Powered-By;
proxy_intercept_errors on;
proxy_buffering on;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

other headers:

add_header Strict-Transport-Security $hsts_header always;

If you need any other information I can share other configs.

I tested a lot, searched a lot. I know caddy could be used as standalone without nginx and it would be much easier but I don’t want to expose the server publicly.

Thanks

2

You seem to have a Squid server in the picture as well.

1 Like
3

In fact, we use squid as a http proxy (for web requests).
It is configured as http_proxy env variable.
It is not used as reverse proxy.

How can the http proxy be a problem here ?
I didn’t think about it as a source of the issue, I will investigate on this side too.

4

There’s too many REDACTED things in your post that it’s hard to tell.

Can you please bypass Nginx and test Caddy directly by running this?

curl -vs http://192.168.102.39:8080 -H 'Host: redacted.example.net'

and since you’re running a forward proxy in your network, can you also try this to bypass Squid as well

curl -vs http://192.168.102.39:8080 -H 'Host: redacted.example.net' -x ""

and post the result?

2 Likes
5

From the reverse proxy running nginx

# curl -vs http://192.168.102.39:8080 -H 'Host: redacted.example.net'
*   Trying 192.168.102.39:8080...
* Connected to 192.168.102.39 (192.168.102.39) port 8080 (#0)
> GET / HTTP/1.1
> Host: redacted.example.net
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 162
< Content-Type: text/html
< Date: Tue, 03 Jun 2025 16:21:18 GMT
< Location: https://redacted.example.net/
< Server: Caddy
< Server: nginx
< Via: 1.1 proxy1 (squid/5.7)
< X-Cache: MISS from proxy1
< X-Cache-Lookup: HIT from proxy1:82
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host 192.168.102.39 left intact

I get exactly the same result when using -x “”
I see the “Via” using proxy1 (squid), but http_proxy variables are not defined on the reverse_proxy server. I am not exactly sure from where the “Via” comes from.

I also did this from my own PC, connected through VPN (I don’t have proxy), and got exactly the same response.

But from the server running caddy:

# curl -vs http://192.168.102.39:8080 -H 'Host: redacted.example.net'
* Uses proxy env variable http_proxy == 'http://192.168.102.61:82'
*   Trying 192.168.102.61:82...
* Connected to 192.168.102.61 (192.168.102.61) port 82 (#0)
> GET http://192.168.102.39:8080/ HTTP/1.1
> Host: redacted.example.net
> User-Agent: curl/7.88.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Server: Caddy
< Date: Tue, 03 Jun 2025 16:23:20 GMT
< Content-Length: 0
< X-Cache: MISS from proxy1
< X-Cache-Lookup: MISS from proxy1:82
< Via: 1.1 proxy1 (squid/5.7)
< Connection: keep-alive
< 
* Connection #0 to host 192.168.102.61 left intact

This server use the proxy. It is configured in the Docker config for containers and env variable for host (via http_proxy variables).
However, if I don’t use the proxy (with -x “”), I get 301 redirected.
So, I can conclude that by not using the proxy, I get 301 redirected, but when using the proxy, I get 200 response.
I don’t know what can cause this.

Sorry for all the redacted things. “redacted” always refer to the same string.
(sorry for the delayed response, caddy forum needed me to wait 22minutes)

6

OK, the squid proxy was in fact a problem.
In fact, the Caddy container had the http_proxy and https_proxy variables configured.

I didn’t know Caddy was relying on these variables for anything.
I don’t know Caddy enough to know why this change something.

Do Caddy try to use the http proxy when doing reverse proxy to local containers like frontend:80 ?
And why caddy act differently when the request comes from the proxy ?

Thanks for your observation and your help

1 Like
7

Yes, Caddy will automatically pick them up from the environment if set. You can disable this behavior for revere_proxy by adding this into the reverse_proxy block:

transport http {
	forward_proxy_url none
}

This part is, unfortunately, not documented yet. We’re behind on the documentation update.

I don’t think it’s Caddy behaving differently as much as it’s the proxy having side-effect, unless I missed parts of the conversation here.

1 Like
8

I’m wondering if the 301 is originating from Caddy or Nginx, or if this message is just something Nginx adds to the redirect coming from Caddy. I don’t think I’ve ever seen Nginx inject its own message page into proxied content, so seeing “nginx” in the message makes me think the 301 is actually coming from Nginx rather than Caddy.

9

You are right, it seems that nginx is making the redirection (I didn’t even see this “nginx” content), but this is maybe due to Caddy using the proxy.

On this server (192.168.102.39), no nginx is running, especially not on port 8080.
So Caddy should only proxy to the container “frontend:80”.

But, because of the proxy (http_proxy env var), I guess the request is captured somewhere by a Nginx and 301 redirect is made.

This could also explain the “< Server: nginx” next to the “< Server: Caddy” on the request.

I will investigate more on the “why”, but the solution was in fact to disable the proxy for the Caddy container, using the undocumented yet “forward_proxy_url” directive.

1 Like