V2 reverse_proxy: please add CORS example to the docs

I had a working version which has stopped working after a few weeks now. Probably due to changes in the beta versions.

Trying to solve this with all the necessary data is a LOT of work.

So to make this simpler for me and others: Can you please add a working example of how to add CORS to a subdomain serving the api, for instance api.example.com?

Maybe here: Reverse proxy quick-start — Caddy Documentation

What kind of examples do you want to see? What is the configuration that is working for you? What about it was a lot of work?

1. My Caddy version (caddy version):

v2.0.0-beta.13

2. How I run Caddy:

a. System environment:

Ubuntu Docker 5:19.03.1~3 on 18.04 on digital ocean

b. Command:

No idea what is meant.

c. Service/unit/compose file:

version: '3.7'
services:
  db:
    # specify container name to make it easier to run commands.
    # for example, you could run docker exec -i postgres psql -U postgres postgres < schema.sql to run an SQL file against the Postgres database
    container_name: bb_db
    restart: always
    #image: couchdb:2.3.1
    image: db
    build:
      context: ./db
    env_file:
      - ./.env
    networks:
      - network
    expose:
      - '5984'
      - '4369'
      - '9100'
    ports:
      - '5984:5984'
    volumes:
      - db_data:/opt/couchdb/data
      - db_config_data:/opt/couchdb/etc/local.d
  caddy:
    build:
      context: ./caddy
    container_name: bb_caddy
    networks:
      - network
    depends_on:
      - db
    restart: always
    # original image downgrades user but that seems not to work
    # see: https://caddy.community/t/basic-docker-compose-setup-failing/6892/7?u=alexander_gabriel
    user: root
    ports:
      - '80:80'
      - '443:443'
    env_file:
      - ./.env
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
      - caddy_certs:/root/.local/share/caddy
      - caddy_config:/root/.config/caddy
volumes:
  db_data:
  db_config_data:
  caddy_certs:
  caddy_config:
networks:
  network:

d. My complete Caddyfile or JSON config:

{
  email alex.barbalex@gmail.com
  #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

blue-borders.ch, www.blue-borders.ch {
  root * /html
  encode zstd gzip
  file_server
}

api.blue-borders.ch {
  reverse_proxy bb_db:5984 {
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_down Access-Control-Allow-Origin  https://blue-borders.ch
    header_down Access-Control-Allow-Credentials=true
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
}

mediterranean-migration.com, www.mediterranean-migration.com {
  root * /html
  encode zstd gzip
  file_server
}

api.mediterranean-migration.com {
  reverse_proxy bb_db:5984 {
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_down Access-Control-Allow-Origin https://mediterranean-migration.com
    header_down Access-Control-Allow-Credentials=true
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
}

3. The problem I’m having:

With this config the app now errors:

Access to fetch at 'https://api.blue-borders.ch/bb/' from origin 'https://blue-borders.ch' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Credentials' header in the response is '' which must be 'true' when the request's credentials mode is 'include'.

But that worked until today when using an earlier version of caddy.

What I changed today was:

  • using the production domains instead of the one to test on
  • using two domains instead of one
  • rebuilding, so using the newest version of caddy

A very similar app with the almost exact same Caddyfile is still running here: https://ortinfo.ch. And cross-domain works.

5. What I already tried:

I tried serving couchdb not from a subdomain and ended up with this:

{
  email alex.barbalex@gmail.com
  #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

blue-borders.ch, www.blue-borders.ch {
  root * /html
  encode zstd gzip
  file_server
  
  reverse_proxy /bb/* {
    to bb_db:5984
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
  
  # need to add this because it us used by couchdb when logging in
  reverse_proxy /_session {
    to bb_db:5984
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
}

mediterranean-migration.com, www.mediterranean-migration.com {
  root * /html
  encode zstd gzip
  file_server
  
  reverse_proxy /bb/* {
    to bb_db:5984
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
  
  # need to add this because it us used by couchdb when logging in
  reverse_proxy /_session {
    to bb_db:5984
    # unfortunately the transparent shorthand does not seem to exist (yet)
    # https://caddy.community/t/v2-reverse-proxy-transparent/6480/8
    #transparent
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote}
    header_up X-Forwarded-For {http.request.remote}
    header_up X-Forwarded-Port {http.request.port}
    header_up X-Forwarded-Proto {http.request.scheme}
  }
}

which I guess technically works but is not what I want.

6. Answers to your questions:

What kind of examples do you want to see?

The Caddyfile for a simple app with an api in a subdomain and the app in the parent domain. Serving the api with cors enabled. Similar to my case but only serving on a single domain.

It is simply about seeing how CORS can be enabled for a subdomain.

Maybe that would fit here: reverse_proxy (Caddyfile directive) — Caddy Documentation

What is the configuration that is working for you?

See above.

What about it was a lot of work?

Sorry for being so general and not more helpful.

What I meant: It took me several days of digging through the docs and issues to get my site to work a few weeks ago on caddy v2. And now it is not working any more so here I am, beginning at zero, digging through docs and searching the issues again.

Why would I do that using a beta version? Simply because I did all the same for version 1 and got stuck with issues that had (according to comments in issues) been solved in v2 but would not be solved in v1 any more.

If I remember right it had to do with hitting the 50 times limit for letsencrypt. Which happened to me so it was impossible for me to try anything using v1.

So I have spent a lot of time on this generally.

Which only distracts from the great job you guys are doing for us all. And from the fact that it is egoistic to complain that your great work caused me to loose some time.

Sorry for that!

Thanks for the detailed follow-up!

I only have a quick moment right now but I wanted to mention that this looks wrong, I think it should be:

header_down Access-Control-Allow-Credentials true

Anyway I gotta bounce but try that and see what happens. I’ll respond in more depth later.

Quick answer: header_down Access-Control-Allow-Credentials true made no difference

Alright, I’ve had a little more time.

This is because Caddy 2 passes the Host header thru by default. It also adds X-Forwarded-For automatically. (I’ve updated the linked post to clarify that. It was quite old.) So you do not need a “transparent” keyword; but if you really need the other headers set like X-Real-IP (is set to same thing as X-Forwarded-For, but whatever) you will need to add that yourself.

When figuring out proxy configs, I recommend setting debug in your global options to enable debug-level logging.

Very odd. :thinking:

A few things are different: the reverse_proxy uses a matcher /bb/* which the earlier one does not, and it doesn’t do any header_down with CORS headers. Why are CORS headers gone, how does it work without them?

This is the command you are running. :slight_smile: Like, caddy.

Are you by chance stopping and restarting Caddy instead of using caddy reload? How are you running Caddy? (Pinging @francislavoie since it seems like Docker is being used here and I’m in over my head trying to figure that out.)

I know it works because I just tried it.

The old way:

$ curl -v "http://localhost:1234/"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 1234 (#0)
> GET / HTTP/1.1
> Host: localhost:1234
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials=true: 
< Content-Length: 13
< Date: Mon, 30 Mar 2020 01:40:10 GMT
< Server: Caddy
< Server: Caddy
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
Hello, there!

(Notice the empty header which has =true in the field name instead of the value)

With the fixed line:

$ curl -v "http://localhost:1234/"
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 1234 (#0)
> GET / HTTP/1.1
> Host: localhost:1234
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Content-Length: 13
< Date: Mon, 30 Mar 2020 01:41:18 GMT
< Server: Caddy
< Server: Caddy
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host localhost left intact
Hello, there!

(See how the space is necessary?)

Did you use their staging endpoint for testing? https://caddyserver.com/v1/docs/automatic-https#testing

Sorry to hear you’ve spent a lot of time on things. Have you checked out our upgrade guide? I spent quite some time last week preparing it. It has some good recommendations for how to approach the v2 upgrade: Upgrading to Caddy 2 — Caddy Documentation

Thanks a lot Matt, for your great help. Unfortunately I have not gotten it to work yet.

This is because Caddy 2 passes the Host header thru by default. It also adds X-Forwarded-For automatically. (I’ve updated the linked post to clarify that. It was quite old.) So you do not need a “transparent” keyword; but if you really need the other headers set like X-Real-IP (is set to same thing as X-Forwarded-For, but whatever) you will need to add that yourself.

I am definitely not knowledgable when it comes to cors. I simply copied these X-Forwarded- settings from someone else who posted them. From my current experiments it seems that you are right and they make no difference.

When figuring out proxy configs, I recommend setting debug in your global options to enable debug-level logging

Thanks, that is a great help.

This is the command you are running. Like, caddy

I think docker-compose.yml is doing that for me. Which would explain why I am not aware of it. Blissfull noobness :slightly_smiling_face:

the reverse_proxy uses a matcher /bb/* which the earlier one does not

That is because I am proxying to CouchDB. To a database called bb. CouchDB connects to /bb/* on the CouchDB server. And also while authorizing to /_sessions which explaines that proxy. Both proxies are covered by reverse_proxy bb_db:5984 when serving from the sub-domain.

These have to be explicit when on the main domain because all other requests should be passed to the file_server.

and it doesn’t do any header_down with CORS headers. Why are CORS headers gone, how does it work without them?

In my understanding this is not cross domain: blue-borders.ch is requesting a ressource from blue-borders.ch/bb. Not from api.blue-borders.ch/bb.

Are you by chance stopping and restarting Caddy instead of using caddy reload? How are you running Caddy?

As mentioned above, I am (was) not even aware of the caddy command. And I still don’t know how I would use it.

I am reloading using:

docker-compose up -d --no-deps --build caddy
docker-compose up -d --force-recreate

See how the space is necessary?

Yes, thank you!

Did you use their staging endpoint for testing? https://caddyserver.com/v1/docs/automatic-https#testing

Yes. But that was exactly what caused the issues: I did not work, using staged, so I was forced to use non-staged.

Unfortunately I could not find the issue where this was elaborated again. But that is what I remember. Beware of my horrible memory and general confusedness though…

Have you checked out our upgrade guide?

Yes, but not very thoroughly. As I have never actually used v1 beyond trying to get this here to work. So I don’t grasp v1 concepts as well as v2 (with the caveat of v2 still changing…).

What I realized just now: Isn’t the matcher missing when I am reverse proxying my subdomains?

I now changed:

api.blue-borders.ch {
  reverse_proxy bb_db:5984 {
    ...cors settings
  }
}

to:

api.blue-borders.ch {
  reverse_proxy * bb_db:5984 {
    ...cors settings
  }
}

which is giving me other error messages:

  1. With: header_down Access-Control-Allow-Origin *:
Access to fetch at 'https://api.blue-borders.ch/bb/' from origin 'https://blue-borders.ch' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
  1. With: header_down Access-Control-Allow-Origin https://blue-borders.ch:
Access to fetch at 'https://api.blue-borders.ch/bb/' from origin 'https://blue-borders.ch' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

What may be missing in the cors settings is Access-Control-Allow-Methods but I could not figure out how to set them:

header_down Access-Control-Allow-Methods: POST, GET, OPTIONS

errored out:

Error during parsing: Wrong argument count or unexpected line ending after 'OPTIONS'

I then changed it to:

header_down Access-Control-Allow-Methods: POST,GET,OPTIONS

Which did not error out. But still did not work.

I then set them to:

header_down Access-Control-Allow-Methods: OPTIONS,DELETE,GET,HEAD,POST

which gives this error:

Access to fetch at 'https://api.blue-borders.ch/bb/' from origin 'https://blue-borders.ch' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

I just realized that the setting will depend on what CouchDB expects. Being a bit of a noob myself, maybe this here will make you see, what I have to add?

There is a short passage in the CouchDB docs specifying how to configure caddy v1: (I initially tried to use v1 because of this but as mentioned above did not succeed in using staged letsencrypt) 4.6. Reverse Proxies — Apache CouchDB® 3.2 Documentation

If you invest time to help me get this working I will try to get the couchdb docs updated. That may save you (and oder CouchDB devs) some time in the future.

Go take a look at the readme on the caddy-docker repo, it should clarify some things for you when it comes to running Caddy.

* is implicit, as long as the first non-matcher argument does not look like a path (i.e. starts with /)

Also, don’t use a colon for the header syntax. You can use double quotes to wrap the header value if it contains spaces

Hm. Not sure I am up to this task as I can’t seem to find what you seem to mention here: Docker Hub.

Where would I find this readme?

1 Like

Thanks, that helps. But I could not find it in the docs.

The first example is pretty clear

Yeah, you are right.

Only thing is: As I was looking for proxying, I was not even aware of the page you just linked but rather learned from here: reverse_proxy (Caddyfile directive) — Caddy Documentation.

And being the noob I am, I would not even have known whether “regular” headers are treated same as when proxying.

Just trying to make you aware of how noobs see the docs.

NOT trying to blame anyone for anything!

Thanks for your help!

Actually I am still not sure how this should be done.

Here a bad noob’s variants to choose from:

  • header_down Access-Control-Allow-Methods: OPTIONS,DELETE,GET,HEAD,POST,PUT
  • header_down Access-Control-Allow-Methods: "OPTIONS, DELETE, GET, HEAD, POST, PUT"
  • header_down Access-Control-Allow-Methods: "OPTIONS DELETE GET HEAD POST PUT"

Which one(s) are correct?

Just to show you how noobs get distracted: I actually tried this one too, while trying to get things to work:

  • header_down “Access-Control-Allow-Methods: OPTIONS, DELETE, GET, HEAD, POST, PUT”

That is why I believe a simple example would help a lot.

The four examples above are only for one single line of the Caddyfile.

Imagine what I can do to the entire file… :roll_eyes:

None of them, but the second one is the closest. Drop the colon. You need the quotes so as to keep the whole value together since there are spaces in it.

lol

Thank you, Matt.

I have now settled to serving CouchDB from the same domain, so cors is not bothering me right now.

I would still try to get it working and to update the CouchDB docs for caddy v2 if someone helps to get it running.

Otherwise feel free to close the issue.

And thanks a lot for helping. And for this great tool.

1 Like