Attach CORS header to the response from external server

Help
1

So I’m trying to host a SPA website, and this website sends at one point a request to a different server using external library. But it fails - there is no CORS header.
(the external server is currently hosted on my internal network just for testing purposes)

I added cors to my config file but it didn’t help.
This is my config file:

    :2019
    root ./dist
    rewrite {
        regexp .*
        to {path} /
    }
    cors

Error from console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://10.55.50.111:8069/xmlrpc/2/common. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

Request sent:

    Host: 10.55.50.111:8069
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
    Accept: */*
    Accept-Language: en-US,en;q=0.7,pl;q=0.3
    Accept-Encoding: gzip, deflate
    Access-Control-Request-Method: POST
    Access-Control-Request-Headers: content-type,user-agent
    Referer: http://localhost:2019/
    Origin: http://localhost:2019
    Connection: keep-alive

Response:
    HTTP/1.0 200 OK
    Content-Type: text/xml; charset=utf-8
    Content-Length: 807
    Server: Werkzeug/0.14.1 Python/3.6.8
    Date: Mon, 09 Sep 2019 10:47:34 GMT

I thought I need a proxy of some kind. Unfortunately I’m a newbie and I don’t really understand how proxies work. But still I tried to make one.
I added this my config:

    proxy /xmlrpc/2/common http:10.55.50.111:8069 {
        transparent
        header_downstream Access-Control-Allow-Origin "*"
    }

And changed request IP inside my website’s javascript from 10.55.50.111:8069 to localhost.
But it broke the request:

Sent request:

    Host: localhost:2019
    User-Agent: NodeJS XML-RPC Client
    Accept: text/xml
    Accept-Language: en-US,en;q=0.7,pl;q=0.3
    Accept-Encoding: gzip, deflate
    Referer: http://localhost:2019/
    Content-Type: text/xml
    Origin: http://localhost:2019
    Content-Length: 317
    Connection: keep-alive

Response:

    HTTP/1.1 405 Method Not Allowed
    Access-Control-Allow-Origin: *
    Content-Type: text/plain; charset=utf-8
    Server: Caddy
    Vary: Origin
    X-Content-Type-Options: nosniff
    Date: Mon, 09 Sep 2019 10:37:00 GMT
    Content-Length: 23

As you can see there is Method not allowed in the response and I don’t know why.

How do I do this?

2

Hi @Gumaa, welcome to the Caddy community, and thanks for putting so much information in your post!

Man, CORS… Great fun.

I suppose that “just add CORS headers to your external server” won’t suffice? :smiley:

Did you add it to the config of the “external server [that] is currently hosted on my internal network just for testing purposes”, or did you add it to your main site?

CORS headers need to be received from the external origin. Adding it to your site is like saying all other sites can now come and make CORS requests for your site’s resources.

This looks good, actually. Whenever people run into CORS issues, the options are basically:

  1. Disable the CORS requirement locally
  2. Ask the external server operator to add CORS headers
  3. Spoof the upstream by proxying to them and supplying your own CORS headers

You’re at #3. It might be less helpful if the upstream server tries to enforce its CORS policy (such by denying requests from the incorrect origin, or whatever), but it’ll work to fool your browser into thinking it’s OK to make that request.

It looks like what you’ve got there is basically what I’d have done. The weirdness here is in the method. It looks like Caddy is actually rejecting the request from the SPA client.

    HTTP/1.1 405 Method Not Allowed
    Server: Caddy

Contrast with a working response…

    HTTP/1.0 200 OK
    Server: Werkzeug/0.14.1 Python/3.6.8

So this request isn’t getting upstream before it gets a 405 response. Caddy v1 core only issues a 405 status in two locations (see: Repository search results · GitHub): the markdown directive and the staticfiles handler.

This is a bit of a clue, and since I see you don’t appear to have markdown in your Caddy config, we can assume that request “fell through” to the Caddy static file handler. That’s the last middleware in the chain, essentially, the “final destination” for a request - if no other directive has yet written a response, the static file handler looks for a file that matches the request and tries to serve it.
If the request had been marked out for the proxy, it could never have reached the static file handler.

So we can infer that it skipped the proxy. Most likely reason is because the SPA client didn’t actually target the /xmlrpc/2/common path as you were expecting.

3

Thank you for the response @Whitestrake ! And sure, such a great fun…

I suppose that “just add CORS headers to your external server” won’t suffice?

Exactly

I made some progress since yesterday - I got it working on my testing server. But then I hit another error on a production server caused maybe by a bug inside an external library that I’m using, just some browser thing or I need to make my proxy HTTPS.

So what I did:

  • Split the Caddy server to two separate ports - one for serving the website and other just for proxy (I’m not really sure why but it does not work if the proxy is on the same port and I get method not allowed error on the post request, maybe some config colision with the part for serving the website)
  • Removed the header_downstream and replace it entirely with cors
  • Add some additional allowed headers to a default cors parameters (with technique known as trial and error: send request → check error for missing header → add one)

And this is my config now:

:2019 {
    root ./dist

    rewrite {
        regexp .*
        to {path} /
    }
}


:2020 {
    cors / {
        allowed_headers Content-Type,User-Agent
    }
    proxy /xmlrpc http://10.55.50.111:8069 {
        transparent
    }
}

<Javascript related> (you can skip this part because it has nothing to do with Caddy)
In the meantime I encounter small quirk on the javascript side . I had to provide URL and a port to the external library and I did this just like this:

url: 'localhost:2020',
port: '',

As you can see I left port empty and put it inside URL. This is because the library would actually send a request to something like this: http://localhost:2019/:2020/xmlrpc… if I moved port to its place. So the solution was to add http:// before localhost to the URL and then move port to a proper place.

URL: 'http://localhost'
port: '2020'

And also if URL was just strait localhost it wasn’t preflighting the OPTION request (just POST straight away and it confused me for a moment) and if it was http://localhost it was sending OPTION first.
</Javascript related>

And now it is working fine but only with my testing server and not on the production. The main difference and probably cause of the error is that testing server is running standard HTTP and production server is on HTTPS.

When inside the config I replace http://10.55.50.111:8069 with https://<production public url> and again send the request I get the response 400: Bad request`

Console error (with stack trace):

Error: "Unknown XML-RPC tag 'TITLE'"
    onError deserializer.js:109
    onClosetag deserializer.js:202
    emit events.js:146
    <anonymous> sax.js:258
    emit sax.js:624
    emitNode sax.js:629
    closeTag sax.js:889
    write sax.js:1436
    write sax.js:239
    ondata _stream_readable.js:619
    emit events.js:151
    addChunk _stream_readable.js:291
    readableAddChunk _stream_readable.js:276
    push _stream_readable.js:245
    read response.js:87

Request:

Host: localhost:2020
User-Agent: NodeJS XML-RPC Client
Accept: text/xml
Accept-Language: en-US,en;q=0.7,pl;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:2019/
Content-Type: text/xml
Origin: http://localhost:2019
Content-Length: 310
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

Response:

HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Content-Length: 226
Content-Type: text/html; charset=iso-8859-1
Date: Tue, 10 Sep 2019 08:19:20 GMT
Server: Caddy
Server: Apache
Vary: Origin

I tried couple things to fix this.

  • Production server is an older instance than my testing server but I checked the request with the exact same body in Postman and it is working fine.
  • I also tried inside Firefox dev tools to modify Content-Type of the request form text/xml to application/xml and manualy resend but it didn’t help as well.
  • I even tried to send this request with standard fetch but outcome is the same.
  • The most recent thing I tried is to send requests not to http://localhost:2020 but https://localhost:2020 but when I do this there is no response comming back. So maybe I need to upgrade my Caddy server to HTTPS? And maybe HTTPS response is not allowed to go to HTTP?

I saw similar error 2 year ago on the library github but I already have patched version so… I guess this one day of work never happened :slight_smile: Just wonder how to send this goddamn requests…

EDIT
I tried to run Caddy with self-signed certificate like this:

localhost:2019 {
    root ./dist

    rewrite {
        regexp .*
        to {path} /
    }

    tls self_signed
}


localhost:2020 {
    cors / {
        allowed_headers Content-Type,User-Agent
    }
    proxy /xmlrpc https://<production public url> {
        transparent
    }
    
    tls self_signed
}

And after whitelisting both of this untrusted certs inside Firefox I still get 400: Bad request :c

4

If you got a 400 Bad Request, that’s a valid response and it means that whichever scheme you were using, the client successfully talked with the server. By the time you get any HTTP status response, you’re already past the connection (and TLS negotiation if applicable) stage, so we’re good there.

The issue is that the server had an issue with the actual request itself; some kind of corruption or malformed request on the client’s part is usually the cause of this one.

5

But could it be that the server sees that there is a problem with request comming from HTTP rather than HTTPS and deny it?

6

No, this is impossible.

If you make a HTTPS request to a HTTP listener, the TLS negotiation will fail and you will not be able to communicate. If you make a HTTP request to a HTTPS listener, you will get a gibberish response. Neither scenario allows for actual communication to take place.

Getting a HTTP status code means you connected successfully to the server, sent a request, and it returned one. Status 400 then means that we have already passed the connection stage.

1 Like
7

I can send the request with Postman, Python and even vanilla Go but I’m not able to send it with the browser (either running script or manually inside dev tools). Even if the body is copy-pasted.
And the only common factor is that when I’m using browser I’m sending the request through Caddy (as a proxy to attach CORS headers to the response) and with every other tool I’m not using it at all.

Everything is pointing to a mistake with Caddy configuration.

I tried again making sure everything is on HTTPS and this is what I get:

Console log:
20190911_121314

Request:

Host: localhost:2020
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
Accept: */*
Accept-Language: en-US,en;q=0.7,pl;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://localhost:2019/
Content-Type: application/xml
Origin: https://localhost:2019
Content-Length: 264
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
TE: Trailers

Response:

HTTP/2.0 400 Bad Request
access-control-allow-origin: *
content-type: text/html; charset=iso-8859-1
date: Wed, 11 Sep 2019 10:08:47 GMT
server: Caddy
server: Apache
vary: Origin
content-length: 226
X-Firefox-Spdy: h2

So I will probably write my own server, send the request from inside the Go and then put data to a website from there…

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.