I’m not sure this error is related to Caddy or not. If it’s related to Caddy, I may open a discussion here with more detail.
Thank you
Next time, please fill out the help topic template, and don’t simply link to a question on another site.
Use expose: in your docker-compose.yml rather than ports:. Using ports: publishes the ports on the host machine, but that’s not useful when trying to have a proxy connect to it from another container. Using expose: allows incoming connections inside the docker network.
1 Like
Thanks for the tip about exposing to ports only to containers. So, if this problem is related to Caddy, I’ll fill the template. Just can’t be sure, where’s the problem.
No worries! If/when that happens, feel free to either continue the topic here with the template, or start a new one if this one’s been automatically closed.
1 Like
1. Caddy version (caddy version):
v2.4.3
2. How I run Caddy:
Inside a Docker Container along with other services
a. System environment:
Docker Desktop for Mac OS 10.15.7
Docker Engine: v20.10.7
Docker Compuse: 1.29.2
b. Command:
docker-compose up -d proxy
c. Service/unit/compose file:
services:
firebase:
container_name: myapp_backend
build:
context: .
volumes:
- ${PWD}:/usr/src/app
- ./emulators/:/root/.cache/firebase/emulators
expose:
- "4000" # Emulator Hub
- "4400" # Emulator Hub
- "9393" # Emulator UI
- "9099" # Firebase Auth
- "5001" # Firebase Functions
- "8080" # Firebase Firestore
- "8085" # Firebase Pub/Pub
- "9199" # Firebase Storage
networks:
- mynetwork
environment:
FIREBASE_TOKEN: ${FIREBASE_TOKEN}
command: ["./docker-entrypoint.sh"]
# Caddy Proxy
proxy:
image: caddy
container_name: myapp_proxy
volumes:
- ./caddy/Caddyfile:/etc/caddy/Caddyfile
- ./caddy/certs:/etc/caddy/certs
- ./caddy/data:/data
- ./caddy/config:/config
networks:
- mynetwork
ports:
- "7372:7372"
- "80:80"
- "443:443"
networks:
mynetwork:
external: true
d. My complete Caddyfile or JSON config:
elasticsearch.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy / myapp_elasticsearch:9200
}
logstash.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy / myapp_logstash:5000
}
kibana.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy / myapp_kibana:5601
}
firebase.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy / myapp_backend:9393
reverse_proxy /auth myapp_backend:9099
reverse_proxy /functions myapp_backend:5001
reverse_proxy /firestore myapp_backend:8080
reverse_proxy /pubsub myapp_backend:8085
reverse_proxy /storage myapp_backend:9199
}
3. The problem I’m having:
I use Caddy server for reverse proxy into a docker container over self signed SSL using mkcert
Certificates are trusted, valid. When I access the website (It’s Firebase emulator) I get SSL related error.
Here’s the error message from Caddy.
4. Error messages and/or full log output:
{
"level": "error",
"ts": 1626251095.9275308,
"logger": "http.log.error",
"msg": "dial tcp 172.27.0.6:9393: connect: connection refused",
"request": {
"remote_addr": "172.27.0.1:63608",
"proto": "HTTP/2.0",
"method": "GET",
"host": "firebase.myapp.localhost",
"uri": "/",
"headers": {
"Sec-Ch-Ua": [
"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\""
],
"Upgrade-Insecure-Requests": ["1"],
"Accept": [
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
],
"Sec-Fetch-Site": ["none"],
"Sec-Fetch-User": ["?1"],
"Sec-Fetch-Dest": ["document"],
"Accept-Encoding": ["gzip, deflate, br"],
"Cache-Control": ["no-cache"],
"Sec-Ch-Ua-Mobile": ["?0"],
"User-Agent": [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
],
"Sec-Fetch-Mode": ["navigate"],
"Accept-Language": ["en-US"],
"Pragma": ["no-cache"]
},
"tls": {
"resumed": true,
"version": 772,
"cipher_suite": 4865,
"proto": "h2",
"proto_mutual": true,
"server_name": "firebase.myapp.localhost"
}
},
"duration": 0.0013572,
"status": 502,
"err_id": "hufgrkw73",
"err_trace": "reverseproxy.statusError (reverseproxy.go:857)"
}
5. What I already tried:
N/A
6. Links to relevant resources:
Alright, so we received the request:
Which, based on your config:
Should be proxied to myapp_backend:9393.
But we issued a 502 response:
Because:
Which indicates that myapp_backend resolved to 172.27.0.6 (via Docker DNS) but the container on the other end rejected the connection attempt.
Most frequently we see outright refused connections when there is no firewall (we’d be much more likely to see a timeout instead) but also no program is listening on that port of the host in question.
I’m not sure about the SSL error; the logs and config you’ve posted here don’t show any TLS-related issues as far as I can see.
Cool, okay, I changed firebase host to 0.0.0.0 so there’s a listener now. So, this might be related to firebase config. Because if I remove host (or set to 127.0.0.1) from the firebase config, it become inaccessible from outside or even inside containers. The ports are exposed to other containers, in the same networks. However with 0.0.0.0 with no hostname is partially working. Please see the screenshot.
I think we need to set the HTTP request header when proxy it. But what (0.0.0.0 or the domain?) to set and how to set?
Which request header?
If you’re referring to Host, Caddy automatically sets this to whatever Host was requested by the originating client. So, if you typed firebase.myapp.localhost in your browser, Host would be firebase.myapp.localhost. This is the default behaviour of a “transparent” reverse proxy.
Sorry, my bad! I was confused with that remote address. It’s all good then, no problem with Caddy at all as I was guess like that, now back to beginning. I think this problem is related to Firebase. It seems Caddy delivery the request to Firebase container and gets a html response but it’s broken. But when trying to access via http://127.0.0.1:9393 or http://0.0.0.0:9393 when the ports were enabled, not exposed (earlier) - It works.
What could possibly break from http to https? 
I’ve seen some sites get a bit ornery when the client has HTTPS but the backend doesn’t. Usually CORS, insecure scripts being denied, or redirects to HTTP content on the same port (which naturally fail).
Loading the site with the Web Console open on the Network tab might give us a good idea.
Hmm… Definitely not a lot here to indicate what’s going wrong.
The error from earlier indicated the manifest.json was an issue.
Maybe take a peek at the Network view of a direct request (i.e. http://127.0.0.1:9393) to see what a good request looks like by comparison? Might also be worth inspecting manifest.json closely between the two requests.
Googling this is frustrating; for some reason it seems like there’s absolutely zero documentation whatsoever on running Firebase behind a reverse proxy.
Html responses are equal but manifest.json is empty over https
Think I figured it out.
/ here is literal - this means, a request for / specifically gets proxied. Anything that’s not exactly / won’t get proxied.
Since /manifest.json (and probably a whole slew of other assets!) don’t fall under any of the other matchers, they don’t get processed at all. Caddy serves up the default response, an empty 200 OK.
In fact, all of those reverse_proxy matchers should probably have wildcards.
Path matching is an exact match by default; you must append a
*for a fast prefix match.
—Request matchers (Caddyfile) — Caddy Documentation
Yay! It worked after changed to this!!! Thanks a lot @Whitestrake
reverse_proxy * myapp_backend:9393
1 Like
Okay @Whitestrake one more thing to manage this in a proper way:
Firebase has path to show different modules like firestore, storage etc. So they use /auth /firestore etc.
But they also manage these applications via different ports to be connected directly. So, what I changed the Caddyfile to this
firebase.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy * myapp_backend:9393
}
Only home page works now, when I go to subpages like /firestore it tries to fetch some configs from :9099 which is auth module.
I can create another subdomains for each module but without doing that, is it possible to handle port based request to proxy related container, like:
firebase.myapp.localhost {
tls /etc/caddy/certs/_wildcard.myapp.localhost+4.pem /etc/caddy/certs/_wildcard.myapp.localhost+4-key.pem
reverse_proxy * myapp_backend:9393
:9099 {
reverse_proxy * myapp_backend:9099
}
}
and proper way to manage these requests?
reverse_proxy functions* myapp_backend:5001
reverse_proxy firestore* myapp_backend:8080
reverse_proxy pubsub* myapp_backend:8085
reverse_proxy storage* myapp_backend:9199
I think this all ports need to address from root level Caddyfile, isn’t it? I notice while discuss in here 
:5001 {
reverse_proxy * myapp_backend:5001
}
Does the client make the requests to the alternate ports?
Or does the client make all requests via port 443 (HTTPS)?
Hmm… That’s inconvenient. Can you configure the clients to make those requests over HTTPS? (Or configure the server to issue HTTPS links?)
Either way, you’ll need additional site blocks in your Caddyfile. It’d probably be simpler to do it like so:
http://firebase.myapp.localhost:5001/functions* {
reverse_proxy myapp_backend:5001
}
http://firebase.myapp.localhost:8080/firestore* {
reverse_proxy myapp_backend:8080
}
# ...etc
If you can make the clients send these requests over HTTPS, strip the http:// part at the start of each site address.
Alternately if you can’t get clients on HTTPS, just expose these ports directly to Firebase. It’s not like Caddy is doing much here if it’s just receiving HTTP and proxying HTTP.