Caddy is not working well with NTLM. Continuous 401 Responses

1. The problem I’m having:

NTLM based application repeatedly keeps asking for authentication (401).

2. Error messages and/or full log output:

As you can see, after I enter the credentials couple of requests go through but then 401 response along with Www-Authenticate: Negotiate is back. As far as the client is concerned login doesn’t work

3. Caddy version:

v2.7.6

4. How I installed and ran Caddy:

a. System environment:

Ubuntu 22

b. Command:

  1. Start Caddy
  2. Make API calls to the endpoint

c. Service/unit/compose file:

I am using it directly for now;

d. My complete Caddy config:


{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "@id": "6617dfefa69a11bcada86df9",
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "encodings": {
                                                        "gzip": {},
                                                        "zstd": {}
                                                    },
                                                    "handler": "encode"
                                                },
                                                {
                                                    "handler": "reverse_proxy",
                                                    "headers": {
                                                        "request": {
                                                            "set": {
                                                                "Host": [
                                                                    "{http.reverse_proxy.upstream.hostport}"
                                                                ],
                                                                "X-Forwarded-For": [
                                                                    "{http.request.remote.host}"
                                                                ],
                                                                "X-Real-IP": [
                                                                    "{http.request.remote.host}"
                                                                ],
                                                                "X-Forwarded-Port": [
                                                                    "{http.request.remote.port}"
                                                                ]
                                                            },
                                                            "add": {
                                                                "X-Forwarded-Proto": [
                                                                    "{http.request.scheme}"
                                                                ]
                                                            }
                                                        },
                                                        "response": {
                                                            "replace": {
                                                                "Location": [
                                                                    {
                                                                        "replace": "6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io",
                                                                        "search": "{http.reverse_proxy.upstream.host}"
                                                                    }
                                                                ],
                                                                "Set-Cookie": [
                                                                    {
                                                                        "search": "secure;",
                                                                        "replace": ""
                                                                    },
                                                                    {
                                                                        "search": "SameSite=None",
                                                                        "replace": "SameSite=Lax"
                                                                    }
                                                                ]
                                                            },
                                                            "set": {
                                                                "Content-Security-Policy": [
                                                                    "frame-ancestors *"
                                                                ]
                                                            }
                                                        }
                                                    },
                                                    "transport": {
                                                        "protocol": "http_ntlm",
                                                        "tls": {
                                                            "insecure_skip_verify": true
                                                        }
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": "etest.DOMAIN.com:443"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ],
                    "tls_connection_policies": [
                        {
                            "match": {
                                "sni": [
                                    "*.SLUG-38.gw.DOMAIN.io"
                                ]
                            }
                        },
                        {}
                    ]
                }
            }
        },
        "tls": {
            "certificates": {
                "load_files": [
                    {
                        "certificate": "/opt/DOMAIN/sdpgw/letsencrypt/fullchain.pem",
                        "key": "/opt/DOMAIN/sdpgw/letsencrypt/privatekey.pem"
                    }
                ]
            }
        }
    },
    "logging": {
        "logs": {
            "Empty": {
                "writer": {
                    "filename": "/opt/myapp/caddy.logs",
                    "output": "file"
                },
                "level": "DEBUG"
            }
        },
        "sink": {
            "writer": {
                "filename": "/opt/myapp/caddy.logs",
                "output": "file"
            },
            "level": "DEBUG"
        }
    }
}

This is a combination of translating the haproxy config (given below) and surfing through the Caddy Community for any NTLM related article. I am not sure what else to change

5. Links to relevant resources:

But the same NTLM app works perfectly in haproxy. But when I moved to Caddy it doesn’t work anymore.

This is the haproxy config that was being used


defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
  default-server init-addr last,libc,none no-tls-tickets


frontend ourwebsitefrontend
    mode http
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/
    #6617dfefa69a11bcada86df9_START_FRONTEND
    acl host6617dfefa69a11bcada86df9 hdr_dom(host) -i 6617dfefa69a11bcada86df9.SLUG-54.gw.app.DOMAIN.io
    use_backend 6617dfefa69a11bcada86df9_backend if xtoken_present jwtverify host6617dfefa69a11bcada86df9
    #6617dfefa69a11bcada86df9_END_FRONTEND


#6617dfefa69a11bcada86df9_START_BACKEND
backend 6617dfefa69a11bcada86df9_backend
        mode http
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    #option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    http-send-name-header Host
    server etest.wns.com etest.wns.com:443 ssl verify none
    http-response replace-header Location (http|https)://(.+?(?=/))/(.*) \1://%[lua.get_redirection_location()]/\3
#6617dfefa69a11bcada86df9_END_BACKEND

What am I missing here?

Addition Logs

Here are two entries (from caddy debug logs) where the first log is after authentication is successful but immediately follows with 401. Its like as if the backend server suddenly decides this is not the same user as before

{"level":"debug","ts":1712981254.1765645,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"etest.wns.com:443","duration":0.020057676,"request":{"remote_ip":"X.X.X.X","remote_port":"54236","client_ip":"X.X.X.X","proto":"HTTP/3.0","method":"GET","host":"etest.wns.com:443","uri":"/Resources/bootstrap/css/bootstrap.min.css","headers":{"Accept":["text/css,*/*;q=0.1"],"Sec-Fetch-Mode":["no-cors"],"X-Real-Ip":["X.X.X.X"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-For":["X.X.X.X"],"Referer":["https://6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io/"],"Sec-Ch-Ua-Mobile":["?0"],"Cookie":[],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""],"Sec-Fetch-Dest":["style"],"Dnt":["1"],"Pragma":["no-cache"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io"],"Accept-Language":["en-US,en;q=0.9"],"Cache-Control":["no-cache"],"Sec-Ch-Ua-Platform":["\"macOS\""]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io"}},"headers":{"Cache-Control":["no-cache, no-store"],"Persistent-Auth":["true"],"Date":["Sat, 13 Apr 2024 04:07:33 GMT"],"Content-Length":["32182"],"Content-Encoding":["gzip"],"Expires":["0"],"Vary":["Accept-Encoding"],"X-Frame-Options":["SAMEORIGIN"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"Pragma":["no-cache"],"Content-Type":["text/css"],"Last-Modified":["Fri, 22 Dec 2023 15:29:02 GMT"],"Accept-Ranges":["bytes"],"Etag":["\"c93c8c97eb34da1:0\""],"X-Xss-Protection":["1; mode=block"],"X-Content-Type-Options":["nosniff"]},"status":200}



{"level":"debug","ts":1712981254.1861625,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"etest.wns.com:443","duration":0.029091655,"request":{"remote_ip":"X.X.X.X","remote_port":"54236","client_ip":"X.X.X.X","proto":"HTTP/3.0","method":"GET","host":"etest.wns.com:443","uri":"/Resources/StyleSheet/style.css","headers":{"X-Forwarded-Host":["6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["same-origin"],"Sec-Ch-Ua-Platform":["\"macOS\""],"X-Real-Ip":["X.X.X.X"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"],"Sec-Fetch-Dest":["style"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-For":["X.X.X.X"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/css,*/*;q=0.1"],"Referer":["https://6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io/"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Cookie":[],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h3","server_name":"6617dfefa69a11bcada86df9.SLUG-38.gw.DOMAIN.io"}},"headers":{"Pragma":["no-cache"],"Content-Type":["text/html"],"Www-Authenticate":["Negotiate","NTLM"],"X-Xss-Protection":["1; mode=block"],"X-Frame-Options":["SAMEORIGIN"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"Date":["Sat, 13 Apr 2024 04:07:33 GMT"],"Content-Length":["1293"],"Cache-Control":["no-cache, no-store"],"Expires":["0"],"X-Content-Type-Options":["nosniff"]},"status":401}

So when I enabled HTTP/1.1 it got resolved. But for whatever reason setting versions at http_ntlm level it does not work

Ah, I totally forgot that yeah, NTLM only works with HTTP/1.1 servers.

Thanks for following up with the answer! (I only ever worked on that module once, just for fun.)

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.