Reverse Proxy not serving the build files

randikaisuru · July 24, 2023, 5:41pm

1. The problem I’m having:

Basically I am trying to setup multiple caddy servers inside the same host. One is working as a main caddy service. And all other caddy servers working as serving frontend applications.

2. Error messages and/or full log output:

No error messages inside the caddy logs or frontend applications sides.
But when after build and run my docker containers getting blank page as response not hitting the frontend application.

I am running all this caddy server in same host with different port bindings

can I know is there is something wrong with my reverse proxy configuration?

because when curl localhost:8080 from host server I am getting response as index file
and also I can see the all build files inside /srv path inside the each frontend containers

3. Caddy version:

I am running caddy on docker compose
version is caddy:2.4.6

4. How I installed and ran Caddy:

I am using docker and docker-compose to install and run the program.

a. System environment:

OS: Linux server
Docker version: 3

b. Command:

docker-compose -f docker-compose.yml up

c. Service/unit/compose file:

Main Caddy docker-compose file

version: "3"
services:
  caddy:
    image: caddy:2.4.6
    restart: unless-stopped
    container_name: caddy
    networks:
      - caddy-network
    ports:
      - 80:80
      - 443:443
    volumes:
      - $PWD/Caddyfile.prod:/etc/caddy/Caddyfile
      - /var/caddy-data:/data
      - /var/caddy-config:/config
      - /var/caddy-logs:/var/log/caddy
    deploy:
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
networks:
  caddy-network:
    external: true
volumes:
  caddy-data:
    driver: local
  caddy-config:
    driver: local
  caddy-logs:
    driver: local

Frontend app docker file

### first Stage

FROM node:16 AS builder

WORKDIR /usr/src/app

COPY ./package.json ./
COPY ./yarn.lock ./

RUN npm install

COPY . .

ARG BASE_URL
ENV REACT_APP_BASE_URL=${BASE_URL} NODE_ENV=${NODE_ENV}

RUN yarn build

### second stage

FROM caddy:2.4.6

ARG CADDYFILE
COPY ${CADDYFILE} /etc/caddy/Caddyfile

COPY --from=builder /usr/src/app/build/ /srv

EXPOSE 80

EXPOSE 443

Frontend apps docker-compose file

version: "3"
services:
  treble-ecommerce-variant-products-frontend-admin:
    image: treble-ecommerce-variant-products-frontend-admin-prod:${ENV}
    restart: unless-stopped
    ports:
      - "8080:80"
      - "8443:443"
    networks:
      - caddy-network
    volumes:
      - caddy-data:/data
      - caddy-config:/config
      - /root/treble/uploads/trebleecommercevariantproductsadminuploads/:/srv/trebleecommercevariantproductsadminuploads
  treble-ecommerce-variant-products-frontend:
    image: treble-ecommerce-variant-products-frontend-prod:${ENV}
    restart: unless-stopped
    ports:
      - "9090:80"
      - "9443:443"
    networks:
      - caddy-network
    volumes:
      - caddy-data:/data
      - caddy-config:/config
      - /root/treble/uploads/trebleecommercevariantproductsuploads/:/srv/trebleecommercevariantproductsuploads
  treble-ecommerce-variant-products-backend:
    image: treble-ecommerce-variant-products-backend
    restart: unless-stopped
    env_file: ./treble-ecommerce-variant-products-backend/config/${ENV}.env
    ports:
      - "5000:5000"
    networks:
      - caddy-network
    volumes:
      - /var/log/services/treble-ecommerce-variant-products-backend/:/var/log/services/treble-ecommerce-variant-products-backend/
      - /root/treble/uploads/trebleecommercevariantproductsuploads/:/usr/src/app/trebleecommercevariantproductsuploads
networks:
  caddy-network:
    external: true
volumes:
  mongo-data:
    driver: local
  caddy-data:
    driver: local
  caddy-config:
    driver: local

d. My complete Caddy config:

1 Main Caddy Server
2 Frontend applications run inside the Caddy server

My main caddy server Caddyfile configure like this

admin.treblemart.com:443 {
    tls treblesolution@gmail.com
    route {
        reverse_proxy treble-ecommerce-variant-products-frontend-admin:80
        reverse_proxy /api* treble-ecommerce-variant-products-backend:5000
    }
}


www.treblemart.com:443 {
    tls treblesolution@gmail.com
    route {
        reverse_proxy treble-ecommerce-variant-products-frontend:80
        reverse_proxy /api* treble-ecommerce-variant-products-backend:5000
    }
}

And my one of frontend Caddyfile looks like this

http://localhost:80 {
    tls treblesolution@gmail.com
    root * /srv
    route {
        reverse_proxy /api* treble-ecommerce-variant-products-backend:5000
        try_files {path} {path}/ /index.html
        file_server
    }
}

other one is looks like this

http://localhost:80 {
    tls treblesolution@gmail.com
    root * /srv
    route {
        reverse_proxy /api* treble-ecommerce-variant-products-backend:5000
        try_files {path} {path}/ /index.html
        file_server
    }
}

5. Links to relevant resources:

francislavoie · July 24, 2023, 10:58pm

Please fill out the help topic template as per the forum rules.

randikaisuru · July 25, 2023, 1:58am

@francislavoie Done.

francislavoie · July 25, 2023, 5:26am

That’s quite an old version. Please upgrade to the latest, v2.6.4.

With this config, because of route, only the first proxy will ever get used. Omitting the matcher means “match all requests”, so your frontend will receive all requests, even the ones with /api*. If you inverse them then it would work fine. But in this case you don’t need route anyway, and removing the route would also fix it.

This looks fine though; the proxy is first, so it will let through any non-/api* URLs to the file server.

We need to see your debug logs from Caddy. The help topic template explains how to do this. Use the debug global option (in every relevant Caddy config), make a request with curl -v and show the output, and show your Caddy logs (of every relevant Caddy container).

randikaisuru · July 25, 2023, 1:13pm

@francislavoie

curl -vL https://www.treblemart.com/
*   Trying 172.67.135.183:443...
* Connected to www.treblemart.com (172.67.135.183) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=treblemart.com
*  start date: Jul 12 08:04:36 2023 GMT
*  expire date: Oct 10 08:04:35 2023 GMT
*  subjectAltName: host "www.treblemart.com" matched cert's "*.treblemart.com"
*  issuer: C=US; O=Let's Encrypt; CN=E1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x560d7f2fd560)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: www.treblemart.com
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< date: Tue, 25 Jul 2023 13:00:41 GMT
< content-length: 0
< alt-svc: h3=":443"; ma=86400
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cZY692ejZmVmcQRmD0PZaOrR44nd%2B5GIw5rcXwf%2BXtIRgdbv34L8ln%2Fw253xTPFXbnm3bD6L2uDwbnhoLY0dmk%2FBHlw9OaufhYdyLC1Dwnb9W65e9Sw1fdzGi2mGdpvi1YCs2rQ%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 7ec496551f283657-FRA
<
* Connection #0 to host www.treblemart.com left intact

Main Caddy logs

Starting caddy ... done
Attaching to caddy
caddy    | {"level":"info","ts":1690290423.1389744,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy    | {"level":"warn","ts":1690290423.1411724,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":6}
caddy    | {"level":"info","ts":1690290423.1432507,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy    | {"level":"info","ts":1690290423.1436274,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy    | {"level":"info","ts":1690290423.1436522,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy    | {"level":"info","ts":1690290423.1441505,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy    | {"level":"info","ts":1690290423.1442451,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
caddy    | {"level":"debug","ts":1690290423.1443558,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
caddy    | {"level":"info","ts":1690290423.144372,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy    | {"level":"debug","ts":1690290423.1444337,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy    | {"level":"info","ts":1690290423.1444488,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy    | {"level":"info","ts":1690290423.1444528,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["admin.treblemart.com","www.treblemart.com"]}
caddy    | {"level":"debug","ts":1690290423.1450722,"logger":"tls","msg":"loading managed certificate","domain":"admin.treblemart.com","expiration":1697359935,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddy    | {"level":"debug","ts":1690290423.1455386,"logger":"tls.cache","msg":"added certificate to cache","subjects":["admin.treblemart.com"],"expiration":1697359935,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"5ce14ca8c0a4ee74427a6d1de25982b718fab4e932beea5847560e8c92030cf8","cache_size":1,"cache_capacity":10000}
caddy    | {"level":"debug","ts":1690290423.1456046,"logger":"events","msg":"event","name":"cached_managed_cert","id":"4d197154-e664-4bb4-8ff6-1e01b8e47e3b","origin":"tls","data":{"sans":["admin.treblemart.com"]}}
caddy    | {"level":"debug","ts":1690290423.1461003,"logger":"tls","msg":"loading managed certificate","domain":"www.treblemart.com","expiration":1697359509,"issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/data/caddy"}
caddy    | {"level":"debug","ts":1690290423.146513,"logger":"tls.cache","msg":"added certificate to cache","subjects":["www.treblemart.com"],"expiration":1697359509,"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"b0b49f8883177eb86e845ce5353f77fd58962bb0faaefd693b76cf71e4785d2b","cache_size":2,"cache_capacity":10000}
caddy    | {"level":"debug","ts":1690290423.1465464,"logger":"events","msg":"event","name":"cached_managed_cert","id":"36241da3-bd1d-4cd9-90d4-7f035758408d","origin":"tls","data":{"sans":["www.treblemart.com"]}}
caddy    | {"level":"info","ts":1690290423.1470122,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0004e42a0"}
caddy    | {"level":"info","ts":1690290423.1471553,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy    | {"level":"info","ts":1690290423.1472306,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy    | {"level":"info","ts":1690290423.147546,"msg":"serving initial configuration"}
caddy    | {"level":"info","ts":1690290423.148306,"logger":"tls","msg":"finished cleaning storage units"}
caddy    | {"level":"debug","ts":1690290468.4882154,"logger":"events","msg":"event","name":"tls_get_certificate","id":"d2c4882f-f707-4fd2-ab26-06de4e7cf449","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49196,49199,49200,49171,49192,156,157,47,53,10],"ServerName":"www.treblemart.com","SupportedCurves":[29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"Conn":{}}}}
caddy    | {"level":"debug","ts":1690290468.4883835,"logger":"tls.handshake","msg":"choosing certificate","identifier":"www.treblemart.com","num_choices":1}
caddy    | {"level":"debug","ts":1690290468.4884007,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"www.treblemart.com","subjects":["www.treblemart.com"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"b0b49f8883177eb86e845ce5353f77fd58962bb0faaefd693b76cf71e4785d2b"}
caddy    | {"level":"debug","ts":1690290468.4884088,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"172.69.150.199","remote_port":"38246","subjects":["www.treblemart.com"],"managed":true,"expiration":1697359509,"hash":"b0b49f8883177eb86e845ce5353f77fd58962bb0faaefd693b76cf71e4785d2b"}
caddy    | {"level":"debug","ts":1690290468.6231728,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"treble-ecommerce-variant-products-frontend:80","total_upstreams":1}
caddy    | {"level":"debug","ts":1690290468.6249497,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"treble-ecommerce-variant-products-frontend:80","duration":0.001672164,"request":{"remote_ip":"172.69.150.199","remote_port":"38246","proto":"HTTP/2.0","method":"GET","host":"www.treblemart.com","uri":"/","headers":{"Cf-Ray":["7ec4a0c2bf774dc0-FRA"],"User-Agent":["curl/7.81.0"],"X-Forwarded-Proto":["https"],"Cf-Ipcountry":["IN"],"X-Forwarded-For":["172.69.150.199"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Connecting-Ip":["157.245.110.30"],"Accept-Encoding":["gzip"],"Accept":["*/*"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Host":["www.treblemart.com"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.treblemart.com"}},"headers":{"Server":["Caddy"],"Date":["Tue, 25 Jul 2023 13:07:48 GMT"],"Content-Length":["0"]},"status":200}

Frontend caddy log

treble-ecommerce-variant-products-frontend_1        | 


{
  "level": "info",
  "ts": 1690290468.62473,
  "logger": "http.log.access",
  "msg": "handled request",
  "request": {
    "remote_ip": "172.18.0.5",
    "remote_port": "46384",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "www.treblemart.com",
    "uri": "/",
    "headers": {
      "Accept": [
        "*/*"
      ],
      "Accept-Encoding": [
        "gzip"
      ],
      "X-Forwarded-For": [
        "172.69.150.199"
      ],
      "Cf-Ray": [
        "7ec4a0c2bf774dc0-FRA"
      ],
      "Cf-Visitor": [
        "{\"scheme\":\"https\"}"
      ],
      "X-Forwarded-Host": [
        "www.treblemart.com"
      ],
      "X-Forwarded-Proto": [
        "https"
      ],
      "User-Agent": [
        "curl/7.81.0"
      ],
      "Cdn-Loop": [
        "cloudflare"
      ],
      "Cf-Connecting-Ip": [
        "157.245.110.30"
      ],
      "Cf-Ipcountry": [
        "IN"
      ]
    }
  },
  "user_id": "",
  "duration": 0.00000336,
  "size": 0,
  "status": 0,
  "resp_headers": {
    "Server": [
      "Caddy"
    ]
  }
}

francislavoie · July 25, 2023, 1:57pm

Ah I see what’s going on now.

This sets up a Host matcher looking for localhost. When proxying, Caddy uses the original Host header for the upstream request. Your request is for www.treblemart.com so it doesn’t match.

The solution is to simply change that site block for your upstreams to :80 so it doesn’t use a Host matcher.

randikaisuru · July 25, 2023, 2:50pm

@francislavoie YES! YES! YES! it’s works… It’s working now.
Thank you so much!

system · August 24, 2023, 2:51pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.