Proxy header_upstream randomly not sent to upstream

[jimjimovich (Jim Mendenhall)]

I have a problem where one of my custom headers is randomly not sent to my upstream server. The code looks something like this:

proxy /reports/ https://upstream.com/ {
   header_upstream X-Something {>Token-Claim-HTTPS://WWW.SITE.COM/SOMETHING}
   header_upstream -Token-Claim-HTTPS://WWW.SITE.COM/SOMETHING
}

The headers come from an Auth0 token and the jwt plugin (that’s why they’re oddly formatted) and cause a problem for my upstream server. I try set a new header to the value of the problematic header and then delete the problematic header. This works 90% of the time, but fails quite often to forward the custom header.

I was wondering if these rules execute in order or if they happen concurrently and sometimes the JTW header is getting deleted before it is set to my custom header.

Thanks in advance for your help!

[matt (Matt Holt)]

Hmm, they are processed in order and not concurrently. If you can somehow narrow this down so it is reproducible, please open an issue so we can take a look at it!

[jimjimovich (Jim Mendenhall)]

I did some more testing and it seems to be completely random as to when the headers are not passed. I checked my logs and all other headers are passed along (the others from the JWT plugin for example) and also custom headers that are passed with header_upstream like this header_upstream X-FOO BAR.

The only headers that are not consistently passed are those which should contain the data found in a header that is later deleted. (Like above). Even then, it may fail for one but not another.

I also tried passing the content of an existing header to a new header and NOT deleting the original. Those seem to always be passed as well.

It really has the feel of a race condition to me.

[matt (Matt Holt)]

Huh. Strange. I wonder if the map underlying the headers is being reused? I think @tw4452852 looked into something like this once, but I’m only speculating that that’s what is happening.

Can you compile with -race? Then report if something bad happens? I’m in the middle of getting a release ready so I don’t have time to try to reproduce it right now.