SSL error when making fetch() request from Node to Caddy

I run php-based backend API and Remix.js frontend. Both behind Caddy.

Caddy routes are as follows:

example.com → goes to and gets handled by Remix.js
example.com/api/* → routed to the php backend

When Remix.js makes an api call to PHP, the following happens:

  • the call is made by node.js (part of Remix fetching data before server-side rendering the website)
  • the call is made via fetch() (node.js)
  • the call is reverse_proxied by Remix to php

It throws an error, and I’m trying to understand if it is related to Remix in any way. Please advise.

FetchError: request to https://caddy/api/auth/prepared-signup failed, reason: write EPROTO 10F068ADFFFF0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1584:SSL alert number 80

Thinking aloud, I remember seing that Caddy does not use openssl. Whereas the error message talks about openssl. This makes me think that it is when Node tries to make an HTTP request to Caddy, and at that point the communication breaks.

Note that the host is caddy – that’s because I am running both Caddy and Remix in their respective Docker containers, via Docker Compose. I don’t think it matters though.

What could be wrong, and how to investigate? Any ideas are appreciated. Thanks.

Node uses OpenSSL, so Node is reporting the error the OpenSSL libs gave to it.

Please fill out the help template, we can’t help you otherwise.

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

I run php-based backend API and Remix.js frontend. Both behind Caddy.

Caddy routes are as follows:

example.com → goes to and gets handled by Remix.js
example.com/api/* → routed to the php backend

When Remix.js makes an api call to PHP, the following happens:

  • the call is made by node.js (part of Remix fetching data before server-side rendering the website)
  • the call is made via fetch() (node.js)
  • the call is reverse_proxied by Remix to php

It throws an error, and I’m trying to understand if it is related to Remix in any way. Please advise.

FetchError: request to https://caddy/api/auth/prepared-signup failed, reason: write EPROTO 10F068ADFFFF0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1584:SSL alert number 80

Thinking aloud, I remember seing that Caddy does not use openssl. Whereas the error message talks about openssl. This makes me think that it is when Node tries to make an HTTP request to Caddy, and at that point the communication breaks.

Note that the host is caddy – that’s because I am running both Caddy and Remix in their respective Docker containers, via Docker Compose. I don’t think it matters though.

What could be wrong, and how to investigate? Any ideas are appreciated. Thanks.

a. System environment:

  • Docker Compose
  • Node 18.8.0 (runs Remix)

b. Command:

I run Caddy as a Docker container, via Docker Compose:

    caddy:
        profiles: [ "dev" ]
        image: caddy:2.5.2
        depends_on:
            - db
        links:
            - db
        restart: always
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - ./docker/caddy/config:/config
            - ./docker/caddy/data:/data
            - ./docker/caddy/dev.caddyfile:/etc/caddy/.caddyfile
        environment:
            WEBSITE_HOST: "bot.127.0.0.1.nip.io"
        command: [ "caddy", "run", "--config", "/etc/caddy/.caddyfile", "--watch", "--adapter", "caddyfile" ]
        logging:
            driver: local
            options:
                max-size: "2m"
                max-file: "5"
    php:
        profiles: [ "dev", "local-prod" ]
        build:
            context: ./docker/php
            dockerfile: php.Dockerfile
            labels:
                # This label allows to identify the container image in Makefile commands
                - "botstore.service.title=php"
        # If you specify image as well as build, then Compose names the built image
        # with the name specified and optional tag specified in image:
        image: botstore_php_service
        depends_on:
            - db
            - mailhog
        links:
            - db
            - mailhog
        environment:
            PHP_IDE_CONFIG: "serverName=BotInDocker"
            GNUPGHOME: /app/cache/gnupg
        volumes:
            # The Web Application
            - ./src:/app/src
            # public assets
            - ./public:/app/public
            # The main configuration file
            - ./config.ini:/app/config.ini:ro
            # Additional PHP configuration to be loaded besides the standard ones
            - ./docker/php/bot.php.ini:/usr/local/etc/php/conf.d/bot.php.ini:ro
            - ./docker/php/bot.php.ini:/usr/local/etc/php-fpm.d/zz-bot.php.ini:ro
            # Caches
            - ./cache:/app/cache
            # Tools
            - ./tools:/app/tools
            # Database files
            - ./db:/app/db
            # Composer
            - ./composer:/app/composer:ro
            # .dev
            - .dev:/app/.dev

    web:
        profiles: [ "dev" ]
        build:
            context: ./docker/web
            dockerfile: web.Dockerfile
            labels:
                # This label allows to identify the container image in Makefile commands
                - "botstore.service.title=web"
        # If you specify image as well as build, then Compose names the built image
        # with the name specified and optional tag specified in image:
        image: botstore_web_service
        volumes:
            - ./web:/app
            - ./config.ini:/config.ini
        depends_on:
            - db
            - caddy
        links:
            - db
            - caddy
        logging:
            driver: local
            options:
                max-size: "2m"
                max-file: "5"

c. Service/unit/compose file:

See above.

d. My complete Caddy config:

{
	# Hide My Email by Apple
	email hidden@icloud.com

	# Issue local SSL certificates in development mode.
	local_certs

	debug
}

# WEBSITE_HOST environment variable will be set by Docker Compose.
{$WEBSITE_HOST:bot.127.0.0.1.nip.io} {
	@api {
		path /api/*
	}

    reverse_proxy @api php:9000 {
        transport fastcgi {
            # An example of env values for FastCGI:
            # https://www.nginx.com/resources/wiki/start/topics/examples/phpfcgi/
            # Also see: https://github.com/nginx/nginx/blob/master/conf/fastcgi_params
            env DOCUMENT_ROOT /app/public/php
            env SCRIPT_NAME /index.php
            env SCRIPT_FILENAME /app/public/php/index.php
            env REQUEST_URI {http.request.uri}
        }
    }

	encode gzip

	# "web" container in Docker Compose.
	reverse_proxy web:3000
}

3. The problem I’m having:

The problem is described in a section above, as it goes well together with “how I run Caddy”.

4. Error messages and/or full log output:

{
  "level": "debug",
  "ts": 1663084968.7999194,
  "logger": "http.handlers.reverse_proxy",
  "msg": "upstream roundtrip",
  "upstream": "web:3000",
  "duration": 0.003874541,
  "request": {
    "remote_ip": "172.18.0.1",
    "remote_port": "62452",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "bot.127.0.0.1.nip.io",
    "uri": "/favicon.ico",
    "headers": {
      "Accept": [
        "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"
      ],
      "Accept-Language": [
        "en-AU,en;q=0.7"
      ],
      "Sec-Gpc": [
        "1"
      ],
      "Referer": [
        "https://bot.127.0.0.1.nip.io/signup"
      ],
      "X-Forwarded-Proto": [
        "https"
      ],
      "X-Forwarded-Host": [
        "bot.127.0.0.1.nip.io"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "Sec-Fetch-Mode": [
        "no-cors"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
      ],
      "Sec-Fetch-Dest": [
        "image"
      ],
      "X-Forwarded-For": [
        "172.18.0.1"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "bot.127.0.0.1.nip.io"
    }
  },
  "headers": {
    "Accept-Ranges": [
      "bytes"
    ],
    "Content-Encoding": [
      "gzip"
    ],
    "Cache-Control": [
      "public, max-age=3600"
    ],
    "Last-Modified": [
      "Tue, 06 Sep 2022 15:51:42 GMT"
    ],
    "Vary": [
      "Accept-Encoding"
    ],
    "Date": [
      "Tue, 13 Sep 2022 16:02:48 GMT"
    ],
    "Connection": [
      "keep-alive"
    ],
    "Etag": [
      "W/\"423e-18313800ab0\""
    ],
    "Content-Type": [
      "image/x-icon"
    ],
    "Keep-Alive": [
      "timeout=5"
    ]
  },
  "status": 200
}

5. What I already tried:

Searched this forum and found out that Caddy does not use OpenSSL, so concluded it’s when Node talks to Caddy, but not sure.

6. Links to relevant resources:

Remix framework I’m using: https://remix.run/

Docker file I use for Node:

FROM node:18.8.0
ENV NODE_ENV development

WORKDIR /app
RUN cd /app && corepack enable && echo 'YARN VERSION IN BUILDER: ' && yarn --version

EXPOSE 3000

# Staying alive. Learn more: https://serverfault.com/a/972760
CMD while :; do :; done & kill -STOP $! && wait $!
# .. and alternative simpler version. Note that tail will still causes some file operations from time to time.
# CMD tail -f /dev/null

Thanks; you’ve posted a single debug log for the favicon… but that request is irrelevant, since you’re asking about the /api/auth/prepared-signup endpoint, right? What are the full logs, as asked for by the help template? Those will have the error message in it.

Both speak TLS, so the whole point of a protocol is that the implementation doesn’t matter as long as they both speak the same protocol. (And aren’t super outdated.) OpenSSL can definitely speak to Go, and vice-versa.

When sshing into Caddy Docker container 2.5.2,

/srv # journalctl -u caddy --no-pager
sh: journalctl: not found

I don’t use Docker, so you’ll either need to research how to get the logs out of it or wait until someone who knows Docker comes along to answer your question. In the meantime I recommend troubleshooting without Docker unless you know how to use Docker, because Docker only adds complexity and problems.

Caddy docker images are Alpine based, which doesn’t use systemd, and it isn’t run as a service inside the container, so you won’t catch its logs through journalctl anyways. Try:

docker-compose logs caddy
2 Likes

I used debug mode in Node to see what it says when trying to fetch() api proxied by Caddy:

FetchError: request to https://caddy/api/auth/prepared-signup failed, reason: write EPROTO 104003B8FFFF0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1584:SSL alert number 80

    at ClientRequest.<anonymous> (/app/.yarn/cache/@remix-run-web-fetch-npm-4.2.0-c89374c10b-ecd91b6942.zip/node_modules/@remix-run/web-fetch/src/fetch.js:111:11)
    at ClientRequest.emit (node:events:513:28)
    at TLSSocket.socketErrorListener (node:_http_client:494:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)

In the same time, looking at the full Caddy log, I cannot see any requests to prepared-signup. Ideas?

Attaching to bot_caddy_1
caddy_1    | {"level":"info","ts":1663122125.7924685,"msg":"using provided configuration","config_file":"/etc/caddy/.caddyfile","config_adapter":"caddyfile"}
caddy_1    | {"level":"warn","ts":1663122125.7933064,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/.caddyfile","line":17}
caddy_1    | {"level":"info","ts":1663122125.7937667,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1    | {"level":"info","ts":1663122125.7938762,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40003d4310"}
caddy_1    | {"level":"info","ts":1663122125.7997603,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1    | {"level":"info","ts":1663122125.799777,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1    | {"level":"warn","ts":1663122125.8075402,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
caddy_1    | {"level":"info","ts":1663122125.8077123,"msg":"Warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
caddy_1    | {"level":"info","ts":1663122125.8077157,"msg":"define JAVA_HOME environment variable to use the Java trust"}
caddy_1    | {"level":"info","ts":1663122125.825468,"msg":"certificate installed properly in linux trusts"}
caddy_1    | {"level":"debug","ts":1663122125.8256543,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
caddy_1    | {"level":"debug","ts":1663122125.8256783,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
caddy_1    | {"level":"info","ts":1663122125.8256807,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["bot.127.0.0.1.nip.io"]}
caddy_1    | {"level":"info","ts":1663122125.8256876,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1    | {"level":"warn","ts":1663122125.8325648,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [bot.127.0.0.1.nip.io]: no OCSP server specified in certificate","identifiers":["bot.127.0.0.1.nip.io"]}
caddy_1    | {"level":"debug","ts":1663122125.8325808,"logger":"tls.cache","msg":"added certificate to cache","subjects":["bot.127.0.0.1.nip.io"],"expiration":1663155366,"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7","cache_size":1,"cache_capacity":10000}
caddy_1    | {"level":"info","ts":1663122125.8344803,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1    | {"level":"info","ts":1663122125.8365893,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1    | {"level":"info","ts":1663122125.8366196,"msg":"serving initial configuration"}
caddy_1    | {"level":"info","ts":1663122125.8371675,"logger":"watcher","msg":"watching config file for changes","config_file":"/etc/caddy/.caddyfile"}
caddy_1    | {"level":"debug","ts":1663122147.6842191,"logger":"tls.handshake","msg":"choosing certificate","identifier":"bot.127.0.0.1.nip.io","num_choices":1}
caddy_1    | {"level":"debug","ts":1663122147.6842763,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"bot.127.0.0.1.nip.io","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122147.6842895,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"expiration":1663155366,"hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122147.686909,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.21.0.1:62560: remote error: tls: unknown certificate"}
caddy_1    | {"level":"debug","ts":1663122147.6911447,"logger":"tls.handshake","msg":"choosing certificate","identifier":"bot.127.0.0.1.nip.io","num_choices":1}
caddy_1    | {"level":"debug","ts":1663122147.6913388,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"bot.127.0.0.1.nip.io","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122147.6914988,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"expiration":1663155366,"hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122147.6942685,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"web:3000","total_upstreams":1}
caddy_1    | {"level":"debug","ts":1663122147.7962182,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"caddy"}
caddy_1    | {"level":"debug","ts":1663122147.7962415,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*"}
caddy_1    | {"level":"debug","ts":1663122147.7962444,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":"caddy"}
caddy_1    | {"level":"debug","ts":1663122147.7962477,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"caddy","remote":"172.21.0.6:50984","identifier":"caddy","cipher_suites":[4866,4867,4865,49199,49195,49200,49196,158,49191,103,49192,107,163,159,52393,52392,52394,49327,49325,49315,49311,49245,49249,49239,49235,162,49326,49324,49314,49310,49244,49248,49238,49234,49188,106,49187,64,49162,49172,57,56,49161,49171,51,50,157,49313,49309,49233,156,49312,49308,49232,61,60,53,47,255],"cert_cache_fill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
caddy_1    | {"level":"debug","ts":1663122147.7962894,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.21.0.6:50984: no certificate available for 'caddy'"}
caddy_1    | {"level":"debug","ts":1663122147.8304703,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"web:3000","duration":0.136091333,"request":{"remote_ip":"172.21.0.1","remote_port":"62562","proto":"HTTP/2.0","method":"GET","host":"bot.127.0.0.1.nip.io","uri":"/signup","headers":{"Accept-Language":["en-AU,en;q=0.7"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["bot.127.0.0.1.nip.io"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["cross-site"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["172.21.0.1"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Cache-Control":["max-age=0"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"bot.127.0.0.1.nip.io"}},"headers":{"Content-Type":["text/html; charset=utf-8"],"Vary":["Accept-Encoding"],"Content-Encoding":["gzip"],"Connection":["keep-alive"],"Keep-Alive":["timeout=5"],"Date":["Wed, 14 Sep 2022 02:22:27 GMT"]},"status":500}
caddy_1    | {"level":"debug","ts":1663122147.8534358,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"web:3000","total_upstreams":1}
caddy_1    | {"level":"debug","ts":1663122147.860412,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"web:3000","duration":0.006924917,"request":{"remote_ip":"172.21.0.1","remote_port":"62562","proto":"HTTP/2.0","method":"GET","host":"bot.127.0.0.1.nip.io","uri":"/favicon.ico","headers":{"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["no-cors"],"Referer":["https://bot.127.0.0.1.nip.io/signup"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"X-Forwarded-For":["172.21.0.1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Accept-Language":["en-AU,en;q=0.7"],"X-Forwarded-Host":["bot.127.0.0.1.nip.io"],"Sec-Fetch-Dest":["image"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"bot.127.0.0.1.nip.io"}},"headers":{"Accept-Ranges":["bytes"],"Content-Encoding":["gzip"],"Keep-Alive":["timeout=5"],"Cache-Control":["public, max-age=3600"],"Etag":["W/\"423e-18313800ab0\""],"Date":["Wed, 14 Sep 2022 02:22:27 GMT"],"Connection":["keep-alive"],"Vary":["Accept-Encoding"],"Last-Modified":["Tue, 06 Sep 2022 15:51:42 GMT"],"Content-Type":["image/x-icon"]},"status":200}
caddy_1    | {"level":"debug","ts":1663122986.481453,"logger":"tls.handshake","msg":"choosing certificate","identifier":"bot.127.0.0.1.nip.io","num_choices":1}
caddy_1    | {"level":"debug","ts":1663122986.4817808,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"bot.127.0.0.1.nip.io","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122986.4817975,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"expiration":1663155366,"hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122986.4878454,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.21.0.1:62568: remote error: tls: unknown certificate"}
caddy_1    | {"level":"debug","ts":1663122986.4913292,"logger":"tls.handshake","msg":"choosing certificate","identifier":"bot.127.0.0.1.nip.io","num_choices":1}
caddy_1    | {"level":"debug","ts":1663122986.4913647,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"bot.127.0.0.1.nip.io","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122986.4915278,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"expiration":1663155366,"hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663122986.4976053,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"web:3000","total_upstreams":1}
caddy_1    | {"level":"debug","ts":1663122986.5097258,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"web:3000","duration":0.012050333,"request":{"remote_ip":"172.21.0.1","remote_port":"62570","proto":"HTTP/2.0","method":"GET","host":"bot.127.0.0.1.nip.io","uri":"/favicon.ico","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["bot.127.0.0.1.nip.io"],"Sec-Fetch-Mode":["no-cors"],"X-Forwarded-For":["172.21.0.1"],"Sec-Fetch-Dest":["image"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-AU,en;q=0.9,ru;q=0.8"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Gpc":["1"],"Sec-Fetch-Site":["cross-site"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"bot.127.0.0.1.nip.io"}},"headers":{"Cache-Control":["public, max-age=3600"],"Content-Encoding":["gzip"],"Keep-Alive":["timeout=5"],"Accept-Ranges":["bytes"],"Last-Modified":["Tue, 06 Sep 2022 15:51:42 GMT"],"Etag":["W/\"423e-18313800ab0\""],"Date":["Wed, 14 Sep 2022 02:36:26 GMT"],"Connection":["keep-alive"],"Content-Type":["image/x-icon"],"Vary":["Accept-Encoding"]},"status":200}
caddy_1    | {"level":"debug","ts":1663123063.6596324,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.21.0.1:62574: remote error: tls: unknown certificate"}
caddy_1    | {"level":"debug","ts":1663123063.6643908,"logger":"tls.handshake","msg":"choosing certificate","identifier":"bot.127.0.0.1.nip.io","num_choices":1}
caddy_1    | {"level":"debug","ts":1663123063.6644871,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"bot.127.0.0.1.nip.io","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"issuer_key":"local","hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663123063.6645117,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["bot.127.0.0.1.nip.io"],"managed":true,"expiration":1663155366,"hash":"0ef46b5984c6479b13203f4ffa24af98d83fcc47ebb9c844898379a04af487e7"}
caddy_1    | {"level":"debug","ts":1663123063.6716216,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"web:3000","total_upstreams":1}
caddy_1    | {"level":"debug","ts":1663123063.7725623,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"caddy"}
caddy_1    | {"level":"debug","ts":1663123063.7725844,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*"}
caddy_1    | {"level":"debug","ts":1663123063.7725878,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","sni":"caddy"}
caddy_1    | {"level":"debug","ts":1663123063.7725909,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"caddy","remote":"172.21.0.6:50998","identifier":"caddy","cipher_suites":[4866,4867,4865,49199,49195,49200,49196,158,49191,103,49192,107,163,159,52393,52392,52394,49327,49325,49315,49311,49245,49249,49239,49235,162,49326,49324,49314,49310,49244,49248,49238,49234,49188,106,49187,64,49162,49172,57,56,49161,49171,51,50,157,49313,49309,49233,156,49312,49308,49232,61,60,53,47,255],"cert_cache_fill":0.0001,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
caddy_1    | {"level":"debug","ts":1663123063.7727156,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.21.0.6:50998: no certificate available for 'caddy'"}
caddy_1    | {"level":"debug","ts":1663123063.8098497,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"web:3000","duration":0.138129333,"request":{"remote_ip":"172.21.0.1","remote_port":"62576","proto":"HTTP/2.0","method":"GET","host":"bot.127.0.0.1.nip.io","uri":"/signup","headers":{"Sec-Fetch-Site":["cross-site"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-AU,en;q=0.7"],"Accept-Encoding":["gzip, deflate, br"],"Cache-Control":["max-age=0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Gpc":["1"],"Sec-Fetch-User":["?1"],"X-Forwarded-For":["172.21.0.1"],"X-Forwarded-Host":["bot.127.0.0.1.nip.io"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"bot.127.0.0.1.nip.io"}},"headers":{"Date":["Wed, 14 Sep 2022 02:37:43 GMT"],"Keep-Alive":["timeout=5"],"Content-Type":["text/html; charset=utf-8"],"Content-Encoding":["gzip"],"Connection":["keep-alive"],"Vary":["Accept-Encoding"]},"status":500}
caddy_1    | {"level":"debug","ts":1663123063.8343945,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"web:3000","total_upstreams":1}
caddy_1    | {"level":"debug","ts":1663123063.8380742,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"web:3000","duration":0.003623458,"request":{"remote_ip":"172.21.0.1","remote_port":"62576","proto":"HTTP/2.0","method":"GET","host":"bot.127.0.0.1.nip.io","uri":"/favicon.ico","headers":{"X-Forwarded-For":["172.21.0.1"],"X-Forwarded-Host":["bot.127.0.0.1.nip.io"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["no-cors"],"Accept-Language":["en-AU,en;q=0.7"],"Sec-Fetch-Dest":["image"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Gpc":["1"],"Referer":["https://bot.127.0.0.1.nip.io/signup"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"bot.127.0.0.1.nip.io"}},"headers":{"Last-Modified":["Tue, 06 Sep 2022 15:51:42 GMT"],"Content-Type":["image/x-icon"],"Keep-Alive":["timeout=5"],"Connection":["keep-alive"],"Vary":["Accept-Encoding"],"Content-Encoding":["gzip"],"Date":["Wed, 14 Sep 2022 02:37:43 GMT"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=3600"],"Etag":["W/\"423e-18313800ab0\""]},"status":200}

Could be a few things.

That means this client rejected the certificate because it didn’t trust it. The client doesn’t have the certificate’s signer (or root) in their trust store. (Common for self-signed or internal certificates, like for IP addresses or internal hostnames.) Trust needs to be configured on the client.

This means a client is connecting to your server with the ServerName set to caddy, but that hostname isn’t in Caddy’s configuration so it can’t get a certificate for it and serve it up.

The TLS handshake is failing, so no connection is established, so no HTTP requests can even be started.

Hopefully that gives you some clues for troubleshooting!

1 Like

Problem solved by just adding caddy to the list of hosts:

{$WEBSITE_HOST:bot.127.0.0.1.nip.io}, caddy {...}

Thank you very much. As it often happens, a fresh look at a problem helped.

1 Like