Docker Applications that I am trying to proxy give me "We’re having trouble finding that site."

Help
1

1. The problem I’m having:

Caddy is only reverse-proxying select docker containers and I don’t know why. Is there something wrong with my certificates that I do not realize? So far I am able to proxy homarr and portainer without issue, but not applications I run in a different stack. I have made sure they are using the same docker network as well.

2. Error messages and/or full log output:

For some context, I was having issues with DNS resolvers. I still am and have not found a solution, so I’m just getting certificates one domain at a time for now. It was working without issue. Here’s a link to the previous problem if anyone wants to take a crack at it or thinks it could be related.

Near the end I get a warning about my firewall (still not sure what that could be) but ultimately the certificates get validated.

{"level":"info","ts":1703859358.235878,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["auth.4tress.xyz","authelia.4tress.xyz","docker.4tress.xyz","home.4tress.xyz"]}
{"level":"info","ts":1703859358.2359967,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1703859358.2363653,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1703859358.2364037,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1703859358.2364683,"logger":"tls.obtain","msg":"acquiring lock","identifier":"authelia.4tress.xyz"}
{"level":"info","ts":1703859358.2375038,"logger":"tls.obtain","msg":"lock acquired","identifier":"authelia.4tress.xyz"}
{"level":"info","ts":1703859358.2376359,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"authelia.4tress.xyz"}
{"level":"info","ts":1703859358.2384279,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["authelia.4tress.xyz"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1703859358.2384388,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["authelia.4tress.xyz"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1703859358.2385433,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
{"level":"info","ts":1703859358.7117724,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"authelia.4tress.xyz","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1703859369.1422465,"logger":"http.acme_client","msg":"challenge failed","identifier":"authelia.4tress.xyz","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"68.13.176.197: Fetching http://authelia.4tress.xyz/.well-known/acme-challenge/dOMtI8Gy3imWKwbyZotxf23VBMHShd0AT0PVZ1M6hEA: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
{"level":"error","ts":1703859369.1423311,"logger":"http.acme_client","msg":"validating authorization","identifier":"authelia.4tress.xyz","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"68.13.176.197: Fetching http://authelia.4tress.xyz/.well-known/acme-challenge/dOMtI8Gy3imWKwbyZotxf23VBMHShd0AT0PVZ1M6hEA: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1487202206/232752889646","attempt":1,"max_attempts":3}
{"level":"info","ts":1703859370.404195,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"authelia.4tress.xyz","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1703859370.5505674,"logger":"tls","msg":"served key authentication certificate","server_name":"authelia.4tress.xyz","challenge":"tls-alpn-01","remote":"3.129.58.108:46626","distributed":false}
{"level":"info","ts":1703859370.6795526,"logger":"tls","msg":"served key authentication certificate","server_name":"authelia.4tress.xyz","challenge":"tls-alpn-01","remote":"23.178.112.209:34395","distributed":false}
{"level":"info","ts":1703859370.7393312,"logger":"tls","msg":"served key authentication certificate","server_name":"authelia.4tress.xyz","challenge":"tls-alpn-01","remote":"54.212.46.28:45214","distributed":false}
{"level":"info","ts":1703859371.0376031,"logger":"http.acme_client","msg":"authorization finalized","identifier":"authelia.4tress.xyz","authz_status":"valid"}
{"level":"info","ts":1703859371.0376143,"logger":"http.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/1487202206/232752927046"}
{"level":"info","ts":1703859371.9260838,"logger":"http.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/041493425b64aac7eded7930166240f825b9"}
{"level":"info","ts":1703859371.926424,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"authelia.4tress.xyz"}
{"level":"info","ts":1703859371.9265113,"logger":"tls.obtain","msg":"releasing lock","identifier":"authelia.4tress.xyz"}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

Caddyfile:

{
	#acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

(internal) {
	tls internal {
		on_demand
	}
}

(tls_config) {
	tls {
		dns googleclouddns {
			gcp_project {env.GCP_PROJECT}
			gcp_application_default {env.GCP_APPLICATION_DEFAULT}
		}
	}
}

(secure) {
	forward_auth {args.0} https://auth.4tress.xyz {
		uri /api/verify?rd=https://auth.4tress.xyz
		copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
		header_up Host {upstream_hostport}
	}
}

docker.4tress.xyz {
	reverse_proxy portainer:9000
}

home.4tress.xyz {
	reverse_proxy homarr:7575
}

auth.4tress.xyz, authelia.4tress.xyz {
	reverse_proxy authelia:9091
}

Docker Compose:

version: "3.7"

services:
  caddy:
    image: mycaddy/4tressxyz
    container_name: caddy
    restart: unless-stopped
    build:
      context: ./build
      dockerfile: dockerfile
    cap_add:
      - NET_ADMIN
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - $PWD/Caddyfile:/etc/caddy/Caddyfile 
      - $PWD/site:/srv
      - $PWD/data:/data
      - $PWD/config:/config
      
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:latest
    volumes:
      - $PWD/portainer-data:/data
      - /mount/disk:/mount/disk
      - /var/run/docker.sock:/var/run/docker.sock
    #restart: unless-stopped

networks:
 default:
  name: dockernet
  external: true

My other stacks that I am trying to proxy is Authelia:

version: "3.7"

services:
  authelia:
    image: authelia/authelia
    container_name: authelia
    restart: unless-stopped
    volumes:
      - $PWD/data/config:/config
      - $PWD/data/config/configuration.yaml:/config/configuration.yaml
    environment:
      AUTHELIA_JWT_SECRET_FILE: /config/secrets/JWT_SECRET
      AUTHELIA_SESSION_SECRET_FILE: /config/secrets/SESSION_SECRET
      AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE: /config/secrets/SMTP_PASSWORD
      AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE: /config/secrets/STORAGE_ENCRYPTION_KEY
      AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE: /config/secrets/STORAGE_PASSWORD
      AUTHELIA_SESSION_REDIS_PASSWORD_FILE: /config/secrets/REDIS_PASSWORD
    depends_on:
      - database
      - redis
  
  database:
    container_name: authpg
    image: postgres:15
    restart: unless-stopped
    volumes:
      - $PWD/data/postgres:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: "authelia"
      POSTGRES_PASSWORD: "[snip]"
  
  redis:
    container_name: authredis
    image: redis:7
    restart: unless-stopped
    command: "redis-server --save 60 1 --loglevel warning --requirepass [snip]"
    volumes:
      - $PWD/data/redis:/data

networks:
 default:
  name: dockernet
  external: true

5. Links to relevant resources:

I am following this guide for Authelia

Edit:
For clarification, this issue isn’t only with Authelia. Another troubleshooting step I have tried is exposing ports in docker compose (Authelia included, I exposed port 9091 in one iteration) and that has not been fruitful either.

2

If I try connecting to that domain, I get:

$ curl -v auth.4tress.xyz
*   Trying 68.13.176.197:80...

In other words, I’m not getting a response from your server.

Are you certain that ports 80 and 443 are open and forwarded?

If this is a home server, are you sure that your ISP isn’t blocking incoming connections on those ports? Are you sure that your IP address is uniquely pointing to your network, and your ISP isn’t using CGNAT?

3

It is a home server. It appears that my ISP filters port 80. Port 80 and 443 are forwarded according to my asus router. Trace routing my public ip produces a single hop so I think that means I’m not using CGNAT?

4

In that case, the HTTP challenge won’t be possible for you to use. You can turn off that challenge if that’s the case. Caddy can still issue certs using port 443, but it does mean HTTP requests will never work so you’ll always have to type https:// in your browser when connecting to your server.

But even so, it seems like port 443 is also blocked:

$ curl -v https://auth.4tress.xyz
*   Trying 68.13.176.197:443...
* connect to 68.13.176.197 port 443 failed: Timed out
* Failed to connect to auth.4tress.xyz port 443 after 21080 ms: Couldn't connect to server
5

I don’t know what could’ve changed between the past few days and today to cause my 443 port to be blocked like this. On Debian, 443 is open, 443 is also forwarded according to my router, and even when my firewall is turned off inside my router I am having no results. Further more, docker.4tress.xyz works, not auth.4tress.xyz? I am so confused lol

6 
$ curl -v https://auth.4tress.xyz
*   Trying 68.13.176.197:443...
* Connected to auth.4tress.xyz (68.13.176.197) port 443

It now works for me :man_shrugging: you must’ve changed something.

7

That’s crazy. I guess I’ll update this topic if I ever figure this out, nothing has changed on my end.

8

Is there any reason why some sites work and not others?

9

It looks like Caddy only has a Let’s Encrypt staging certificate (i.e. not trusted) for your auth domain. Your logs should show details about what happened.

Since the HTTP challenge is impossible in your case, you could add this to each site to disable it:

tls {
	issuer acme {
		disable_http_challenge
	}
	issuer zerossl {
		disable_http_challenge
	}
}
10

So I tried that, and there was no change on my end. So finally, I turned on my VPN. And I was able to connect to auth.4tress.xyz without issue. So the issue is my network, I just do not know what and I frustrated. I checked my router’s firewall, I checked debian’s firewall, I checked Proxmox, I checked Windows, I am trying to learn nmap, I have to laugh because I still don’t know why 443 is filtered like this.

11

Okay, I fixed it. Restarting my router for some reason resolved the issue.

Just kidding, nothing has been fixed. It worked for five seconds and then it stopped again. Carry on.

12

Ah, probably because your home router doesn’t support NAT hairpinning, so it doesn’t know how to route packets destined for your WAN IP (which your domain resolves to) back into your network, so it gets dropped.

The typical solution is to run a DNS server inside your home network which resolves the domain to your server’s LAN IP instead (i.e. 192.168.x.x) so that your router knows what to do with those connections.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.