Timeout during connect (likely firewall problem)

kkrrp1 · January 27, 2023, 8:41am

1. Caddy version:

from docker-compose exec caddy caddy version:no configuration file provided: not found
From “inspect” v2.6.2

2. How I installed, and run Caddy:

I can run Nextcloud and access it just fine via docker run --sig-proxy=false --name nextcloud-aio-mastercontainer --restart always --publish 80:80 --publish 8080:8080 --publish 8443:8443 --volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config --volume //var/run/docker.sock:/var/run/docker.sock:ro nextcloud/all-in-one:latest

However when I try to add caddy I’m unable to access at all.

a. System environment:

Windows 11 on Docker-desktop v4.16.2

b. Command:

docker-compose up -d

c. Service/unit/compose file:

services:
  caddy:
    image: caddy:alpine
    restart: unless-stopped
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs:/certs
      - ./config:/config
      - ./data:/data
      - ./sites:/srv
    network_mode: "host"

  nextcloud:
    image: nextcloud/all-in-one:latest
    restart: unless-stopped
    container_name: nextcloud-aio-mastercontainer
    ports:
      - "8080:8080"
    environment:
      - APACHE_PORT=11000
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - //var/run/docker.sock://var/run/docker.sock:ro
    depends_on:
      - caddy

volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer

d. My complete Caddy config:

https://nextcloud.weme.wtf:443 {
    header Strict-Transport-Security max-age=31536000;
    reverse_proxy localhost:11000
}

3. The problem I’m having:

It’s acting like my firewalls blocking the ports for acme-challenge
It’s saying there’s no config?
I don’t even see 80 and 443 being listened on.

4. Error messages and/or full log output:

2023-01-26 21:27:39 {"level":"error","ts":1674793659.4830575,"logger":"http.acme_client","msg":"validating authorization","identifier":"nextcloud.weme.wtf","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"24.10.145.50: Fetching http://nextcloud.weme.wtf/.well-known/acme-challenge/mZQ1Nq81Neti9C5bn-3cU5aUFtNzYWJr7jRwyaERnoQ: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/84890263/6816273823","attempt":2,"max_attempts":3}
2023-01-26 21:27:39 {"level":"error","ts":1674793659.4830747,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nextcloud.weme.wtf","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:connection - 24.10.145.50: Fetching http://nextcloud.weme.wtf/.well-known/acme-challenge/mZQ1Nq81Neti9C5bn-3cU5aUFtNzYWJr7jRwyaERnoQ: Timeout during connect (likely firewall problem)"}
2023-01-26 21:27:44 {"level":"info","ts":1674793664.0261712,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"nextcloud.weme.wtf","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
2023-01-26 21:27:59 {"level":"error","ts":1674793679.8731074,"logger":"http.acme_client","msg":"challenge failed","identifier":"nextcloud.weme.wtf","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
2023-01-26 21:27:59 {"level":"error","ts":1674793679.8731427,"logger":"http.acme_client","msg":"validating authorization","identifier":"nextcloud.weme.wtf","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/xAoex9kRbZX8TKUOzsOxFg","attempt":1,"max_attempts":3}
2023-01-26 21:27:59 {"level":"error","ts":1674793679.8731935,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nextcloud.weme.wtf","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
2023-01-26 21:27:59 {"level":"error","ts":1674793679.873224,"logger":"tls.obtain","msg":"will retry","error":"[nextcloud.weme.wtf] Obtain: [nextcloud.weme.wtf] solving challenge: nextcloud.weme.wtf: [nextcloud.weme.wtf] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":4,"retrying_in":300,"elapsed":493.483120303,"max_duration":2592000}

{"level":"info","ts":1674793189.8594553,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
{"level":"warn","ts":1674793189.8601122,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/Caddyfile","line":2}
{"level":"info","ts":1674793189.8611803,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"warn","ts":1674793189.8612688,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
{"level":"warn","ts":1674793189.8612738,"logger":"http","msg":"automatic HTTP->HTTPS redirects are disabled","server_name":"srv1"}
{"level":"warn","ts":1674793189.8613467,"logger":"tls","msg":"YOUR SERVER MAY BE VULNERABLE TO ABUSE: on-demand TLS is enabled, but no protections are in place","docs":"https://caddyserver.com/docs/automatic-https#on-demand-tls"}
{"level":"info","ts":1674793189.8613973,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1674793189.8613966,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000730380"}
{"level":"info","ts":1674793189.8614073,"logger":"http","msg":"enabling HTTP/3 listener","addr":":8443"}
{"level":"info","ts":1674793189.8614342,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/mnt/docker-aio-config/caddy/"}
{"level":"info","ts":1674793189.8614912,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1674793189.8614545,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}
{"level":"info","ts":1674793189.8615794,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"error","ts":1674793189.8615954,"msg":"unable to create folder for config autosave","dir":"/root/.config/caddy","error":"mkdir /root/.config: permission denied"}
{"level":"info","ts":1674793189.861602,"msg":"serving initial configuration"}
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.0.2. Set the 'ServerName' directive globally to suppress this message
[Fri Jan 27 04:19:49.868783 2023] [ssl:warn] [pid 117] AH01906: 192.168.0.2:8080:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jan 27 04:19:49.868822 2023] [ssl:warn] [pid 117] AH01909: 192.168.0.2:8080:0 server certificate does NOT include an ID which matches the server name

5. What I already tried:

Tried removing header Strict-Transport-Security max-age=31536000; as I’ve seen configs with it not there
Tried cocker-compose.yml: line 23: - /var/run/docker.sock:/var/run/docker.sock:ro however this way Nextcloud just kept restarting over and over again. Online someone mentioned to add // before var which got passed that.
removed docker-compose.yml: line 30 as someone’s config in the examples secion on nextcloud all-in-one github said theirs worked and it didn’t have that line under “volumes”
Cleared out the containers/images/volumes between changes.
Rebooted/reinstalled docker.
Ruled out router port-forwarding/firewall by swapping the port-forwarding back to my other server nextcloud-snap instance. which was still working with https.
Ruled out windows 11 firewall by first resetting FW to defaults and reading rules then just disabling it disabling it
pwsh: netstat -a -b doesn’t show listening on port 80 or 443

Separate (mostly unrelated) issue but my first attempt at revers proxy for nextcloud was with nextcloud-snap on ubuntu (reddit story below). total fail. Honestly i’m exhausted so any help would be appreciated. I’m sold on docker/caddy though if I can get this working.

6. Links to relevant resources:

github.com

nextcloud/all-in-one/blob/main/reverse-proxy.md

# Reverse Proxy Documentation

**Please note:** Publishing the AIO interface with a valid certificate to the public internet is **not** the goal of this documentation! Instead, the main goal is to publish Nextcloud with a valid certificate to the public internet which is **not** running inside the mastercontainer but in a different container! If you need a valid certificate for the AIO interface, see [point 5](#5-optional-get-a-valid-certificate-for-the-aio-interface).

In order to run Nextcloud behind a reverse proxy, you need to specify the port that the Apache container shall use, add a specific config to your reverse proxy and modify the startup command a bit. All examples below will use port `11000` as example Apache port which will be exposed on the host. Modify it to your needings.

**Attention:** The process to run Nextcloud behind a reverse proxy consists of at least steps 1, 2 and 4:
1. **Configure the reverse proxy! See [point 1](#1-add-this-to-your-reverse-proxy-config)**
1. **Use the in this document provided startup command! See [point 2](#2-use-this-startup-command)**
1. Optional: If the reverse proxy is installed on the same host, you should limit the apache container to only listen on localhost. See [point 3](#3-if-the-reverse-proxy-is-installed-on-the-same-host-you-should-configure-the-apache-container-to-only-listen-on-localhost)
1. **Open the AIO interface. See [point 4](#4-open-the-aio-interface)**
1. Optional: Get a valid certificate for the AIO interface! See [point 5](#5-optional-get-a-valid-certificate-for-the-aio-interface)
1. Optional: How to debug things? See [point 6](#6-how-to-debug-things)

## 1. Add this to your reverse proxy config

**Please note:** Since the Apache container gets spawned by the mastercontainer, there is **NO** way to provide custom docker labels or custom environmental variables for the Apache container. So please do not attempt to do this because you will fail! Only the documented way will work!

### Apache

This file has been truncated. show original

francislavoie · January 27, 2023, 9:09am

Are you sure you have ports 80 and 443 forwarded on your router, and allowed in your firewall (including Windows’ firewall)?

Where do you see that? Not sure I follow.

Docker might interact weirdly with WSL on Windows. That’s more of a Docker question than a Caddy one. I can’t really answer that, I don’t use Docker on Windows.

Why would NextCloud need to interact with the Docker API? That doesn’t seem right.

What line is that? I don’t see line numbers here.

This is weird. Caddy shouldn’t be trying to write to /root/.config, it should be trying to write to /config.

I just tried to load https://nextcloud.weme.wtf myself, and from the HTTP headers, it doesn’t look like it’s reaching Caddy at all, it’s directly serving NextCloud with its Apache server.

You set Caddy to use host networking, but you didn’t bind anything to port 11000 on the host. You only bound port 8080. So I don’t think that’ll connect to anything.

Why are you using host for Caddy? You could just bind ports 80 and 443 instead. Then to proxy to NextCloud, you’d probably use reverse_proxy nextcloud:8080

kkrrp1 · January 28, 2023, 12:33am

Thanks for your help!

Are you sure you have ports 80 and 443 forwarded on your router, and allowed in your firewall (including Windows’ firewall)?
&
Docker might interact weirdly with WSL on Windows. That’s more of a Docker question than a Caddy one. I can’t really answer that, I don’t use Docker on Windows.

You’re right I thought It was weird the [Example here] only showed a ports: line under nextcloud. so seems to be a Docker thing.

To make quadruple sure It’s not FW or windows, I left out nextcloud. This worked just fine.
docker-compose.yml

services:
  caddy:
    image: caddy:alpine
    restart: unless-stopped
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs:/certs
      - ./config:/config
      - ./data:/data
      - ./sites:/srv
#    network_mode: "host"
    ports:
      - "80:80"
      - "443:443"

Caddyfile

https://nextcloud.weme.wtf:443 {
    header Strict-Transport-Security max-age=31536000;
    respond "nextcloud.weme.wtf test"
}
https://weme.wtf:443 {
    header Strict-Transport-Security max-age=31536000;
    respond "weme.wtf test"
}

Where do you see that? Not sure I follow.

The caddy logs show:

2023-01-27 17:02:53 {"level":"warn","ts":1674864173.204812,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}

I ran caddy fmt from the caddy container terminal I saw it added a second tab so I added that in my code. It still gave the error after restart and also running cat /etc/caddy/Caddyfile then caddy fmt again confirmed formatting it made no changes…

Why would NextCloud need to interact with the Docker API? That doesn’t seem right.

Both commenting this line out and using /var instead of //var yields this error in nextcloud-aio-mastercontainer

2023-01-27 13:46:38 Docker socket is not available. Cannot continue.

This is weird. Caddy shouldn’t be trying to write to /root/.config, it should be trying to write to /config.

Beets me. /root/ may be how docker see’s this command on Windows WSL. I can’t imagine why it would add a period before it though but running cat /etc/caddy/Caddyfile shows it’s getting there fine.

I just tried to load https://nextcloud.weme.wtf myself, and from the HTTP headers, it doesn’t look like it’s reaching Caddy at all, it’s directly serving NextCloud with its Apache server.

Yeah I moved my port forwarding back to my old Nextcloud instance on a different machine while I’m not actively working on it. I’ll leave it alone this time in case that helps.

You set Caddy to use host networking, but you didn’t bind anything to port 11000 on the host. You only bound port 8080. So I don’t think that’ll connect to anything.

Why are you using host for Caddy? You could just bind ports 80 and 443 instead. Then to proxy to NextCloud, you’d probably use reverse_proxy nextcloud:8080

The [[Nextcloud/all-in-one]] github reverse-proxy docs say:

Please note: The above configuration will only work if your reverse proxy is running directly on the host that is running the docker daemon. If the reverse proxy is running in a docker container, you can use the --network host option (or network_mode: host for docker-compose) when starting the reverse proxy container in order to connect the reverse proxy container to the host network.A

Hmmm but I just found docker windows limitation:

The host networking driver only works on Linux hosts, and is not supported on Docker Desktop for Mac, Docker Desktop for Windows, or Docker EE for Windows Server.

kkrrp1 · January 28, 2023, 12:53am

OMG I just found it.

If the reverse proxy is running in a docker container, you can use the --network host option (or network_mode: host for docker-compose) when starting the reverse proxy container in order to connect the reverse proxy container to the host network. If that is not an option for you, you can alternatively instead of localhost use the ip-address that is displayed after running the following command on the host OS: ip a | grep "scope global" | head -1 | awk '{print $2}' | sed 's|/.*||' (the command only works on Linux)

Apparently I’m one that that’s not an option for.

I ran it and pull the ip address. I’m pretty confused since that’s not my IP address. I guess WSL has a different IP than windows and I have some studying on that to do on that.

Here’s my working config
Caddyfile

https://nextcloud.weme.wtf:443 {
        header Strict-Transport-Security max-age=31536000;
        reverse_proxy <WSL IP Address>:11000
}
https://weme.wtf:443 {
        header Strict-Transport-Security max-age=31536000;
        respond "weme.wtf test"
}

docker-compose.yml

services:
  caddy:
    image: caddy:alpine
    restart: unless-stopped
    container_name: caddy
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs:/certs
      - ./config:/config
      - ./data:/data
      - ./sites:/srv
#    network_mode: "host" incompatible with docker-desktop for windows
    ports:
      - "80:80"
      - "443:443"

  nextcloud:
    image: nextcloud/all-in-one:latest
    restart: unless-stopped
    container_name: nextcloud-aio-mastercontainer
    ports:
      - "8080:8080"
    environment:
      - APACHE_PORT=11000
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - //var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - caddy

volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer

Thanks for pointing out things to look at. Glad I can get some other services going.

francislavoie · January 28, 2023, 4:03am

kkrrp1:

Why would NextCloud need to interact with the Docker API? That doesn’t seem right.

Both commenting this line out and using /var instead of //var yields this error in nextcloud-aio-mastercontainer
2023-01-27 13:46:38 Docker socket is not available. Cannot continue.

Well then. Looks like the all-in-one container has that requirement.

I’m not sure I like that approach. It’s an interesting idea, but it really seems like it overly complicates some things that would otherwise be simple.

That’s pretty silly. They should be adding the Apache container to a Docker network that Caddy can be added to, so proxying can happen by container/service name.

I’d suggest you bring feedback to them about this

kkrrp1:

Where do you see that? Not sure I follow.

The caddy logs show:
2023-01-27 17:02:53 {"level":"warn","ts":1674864173.204812,"msg":"Caddyfile input is not formatted; run the 'caddy fmt' command to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
I ran caddy fmt from the caddy container terminal I saw it added a second tab so I added that in my code. It still gave the error after restart and also running cat /etc/caddy/Caddyfile then caddy fmt again confirmed formatting it made no changes…

Ah, that’s just a warning, not an error.

But you can run caddy fmt --overwrite to have it overwrite the file in-place. That should get rid of the warning. There’s probably some other differences in the file that you didn’t notice.

szaimen · February 6, 2023, 10:34am

Hi, just to clarify things:

AIO actually creates its own docker network which the caddy container could be added to.

However we’ve built a domain validation into the project which makes sure that the domain and port-forwarding and co. is correctly configured. For that to work, the mastercontainer initially spawns a domaincheck container (which is much smaller in size than the apache container and only has this one purpose) that uses the chosen apache-port. However since it has a different container_name, you would need to point caddys reverse proxy at the domaincheck container, but this would break the domain validation since after the domaincheck is passed, the apache container gets spawned which has a different container_name. (Using the same name for both containers would complicate the internal logic by a lot and would rather feel like a hack).

So to make sure that the domain validation works correctly, we require this exact setup.

Also it comes with the added benefit that this even works when the reverse proxy is not running on the same host or in the host network.

One possible workaround would be simply skipping the domain validation and pointing caddys reverse proxy directly to the apache container (after adding caddy to the correct docker network; also note that the apache container does not exists initially, only after entering the domain and starting the containers in the aio interface) which however would obviously skip the domain validation.

I hope that makes sense to you!

francislavoie · February 7, 2023, 1:32am

Thanks for following up @szaimen, that’s interesting!

I still think that feels very over-engineered though. Way too many moving parts for me to be comfortable with it.

system · March 9, 2023, 1:32am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.