Reverse proxy not working in virtualbox. Same config works on raspberry pi

Help
1

1. Caddy version (caddy version):

docker-compose.yml requests “latest”

2. How I run Caddy:

Docker Compose

a. System environment:

OSX > Vagrant > Virtualbox > Ubuntu (hashicorp/bionic64) > Docker-Compose

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic

$ docker --version
Docker version 20.10.7, build f0df350

$ docker-compose --version
docker-compose version 1.24.1, build 4667896b

b. Command:

docker-compose up -d

c. Service/unit/compose file:

# docker-compose.yml

version: "3.7"

services:

    caddy:
        image: caddy
        restart: unless-stopped
        ports:
            - "80:80"
            - "443:443"
        volumes:
            - $PWD/Caddyfile:/etc/caddy/Caddyfile
            - $PWD/site:/srv
            - caddy_data:/data
            - caddy_config:/config

    jellyfin:
        restart: always
        image: jellyfin/jellyfin:latest
        user: 1000:1000 # should be owner of volumes
        network_mode: "host"
        restart: "unless-stopped"
        volumes:
            - /srv/jellyfin/config:/config
            - /srv/jellyfin/cache:/cache
            - /media/Music:/media

volumes:
    caddy_data:
    caddy_config:

d. My complete Caddyfile or JSON config:

{
    email somebody@somewhere.com
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    #debug
}

x.duckdns.org {
    reverse_proxy 192.168.1.x:8096
}

3. The problem I’m having:

unable to get a successful reverse proxy going.

4. Error messages and/or full log output:

stage server output: acme-staging-v02
debug is off. there seemed to be enough base errors to start troubleshooting

$ docker-compose logs -f --tail 500 caddy
Attaching to vagrant_caddy_1
caddy_1     | {"level":"info","ts":1625883647.1926813,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_1     | {"level":"info","ts":1625883647.2015343,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy_1     | {"level":"info","ts":1625883647.2149746,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1     | {"level":"info","ts":1625883647.215157,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1     | {"level":"info","ts":1625883647.2157936,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["x.duckdns.org"]}
caddy_1     | {"level":"info","ts":1625883647.2162442,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1     | {"level":"info","ts":1625883647.216494,"msg":"serving initial configuration"}
caddy_1     | {"level":"info","ts":1625883647.2166073,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1     | {"level":"info","ts":1625883647.2173343,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1     | {"level":"info","ts":1625883647.217439,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000d96c0"}
caddy_1     | {"level":"info","ts":1625883647.2215476,"logger":"tls.obtain","msg":"acquiring lock","identifier":"x.duckdns.org"}
caddy_1     | {"level":"info","ts":1625883647.232989,"logger":"tls.obtain","msg":"lock acquired","identifier":"x.duckdns.org"}
caddy_1     | {"level":"info","ts":1625883647.3102748,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["x.duckdns.org"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"caddy@gearslip.com"}
caddy_1     | {"level":"info","ts":1625883647.3105738,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["x.duckdns.org"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"caddy@gearslip.com"}
caddy_1     | {"level":"info","ts":1625883648.4414392,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"x.duckdns.org","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1625883659.2972374,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"x.duckdns.org","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Timeout during connect (likely firewall problem)"}
caddy_1     | {"level":"error","ts":1625883659.2974432,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"x.duckdns.org","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97837002","attempt":1,"max_attempts":3}
caddy_1     | {"level":"info","ts":1625883660.4364653,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"x.duckdns.org","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1625883670.7935524,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"x.duckdns.org","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Fetching http://x.duckdns.org/.well-known/acme-challenge/CnLC-Q7bLEmU8_P-AeQVTXYiJom9bCDGOL-idvdtaVw: Timeout during connect (likely firewall problem)"}
caddy_1     | {"level":"error","ts":1625883670.7937672,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"x.duckdns.org","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Fetching http://x.duckdns.org/.well-known/acme-challenge/CnLC-Q7bLEmU8_P-AeQVTXYiJom9bCDGOL-idvdtaVw: Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97837123","attempt":2,"max_attempts":3}
caddy_1     | {"level":"error","ts":1625883672.3087118,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[x.duckdns.org] solving challenges: x.duckdns.org: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97837257) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy_1     | {"level":"error","ts":1625883672.3090603,"logger":"tls.obtain","msg":"will retry","error":"[x.duckdns.org] Obtain: [x.duckdns.org] solving challenges: x.duckdns.org: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97837257) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":25.075907144,"max_duration":2592000}
caddy_1     | {"level":"info","ts":1625883733.0823252,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"x.duckdns.org","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1625883743.3607962,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"x.duckdns.org","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Fetching http://x.duckdns.org/.well-known/acme-challenge/nvyV1A_sY8L_Yt3rC8DjeMUfbNi8ZePD_a_2vbkZhJM: Timeout during connect (likely firewall problem)"}
caddy_1     | {"level":"error","ts":1625883743.3610396,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"x.duckdns.org","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Fetching http://x.duckdns.org/.well-known/acme-challenge/nvyV1A_sY8L_Yt3rC8DjeMUfbNi8ZePD_a_2vbkZhJM: Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97837891","attempt":1,"max_attempts":3}
caddy_1     | {"level":"info","ts":1625883744.499879,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"x.duckdns.org","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
caddy_1     | {"level":"error","ts":1625883755.6387696,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"x.duckdns.org","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Timeout during connect (likely firewall problem)"}
caddy_1     | {"level":"error","ts":1625883755.6389778,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"x.duckdns.org","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97838025","attempt":2,"max_attempts":3}
caddy_1     | {"level":"error","ts":1625883756.8411572,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[x.duckdns.org] solving challenges: x.duckdns.org: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97838164) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy_1     | {"level":"error","ts":1625883756.8414094,"logger":"tls.obtain","msg":"will retry","error":"[x.duckdns.org] Obtain: [x.duckdns.org] solving challenges: x.duckdns.org: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/20209807/97838164) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":109.608256304,"max_duration":2592000}

5. What I already tried:

The exact same caddy + duckdns + jellyfin reverse proxy setup runs fine on a raspberry pi (rpi).

Decided to copy (edit: meant “recreate”. did not just copy the instance over) it over to VM on a desktop machine on the same LAN.

  1. The rpi instance is shutdown to avoid potential conflicts.

  2. Router port redirect and Caddyfile reverse proxy are both pointing at the correct IP address.

  3. All the familiar ports seem to be open.

[another mechine on same LAN] $ nc -zvw3 192.168.1.x 80 443 8096 2019
Connection to 192.168.1.x 80 port [tcp/http] succeeded!
Connection to 192.168.1.x 443 port [tcp/https] succeeded!
Connection to 192.168.1.x 8096 port [tcp/*] succeeded!
nc: connect to 192.168.1.x port 2019 (tcp) failed: Connection refused
--

  1. Port 2019, listed in the logs above, is closed, but since it’s also closed on the working rpi instance, so I’m assuming (there’s that word) that’s expected.

  2. Browsing the the VM’s jellyfin instance via IP address from other machines on the LAN works fine.

  3. Curl tests:

# http - success:
$ curl -v http://x.duckdns.org
    ...
* Connected to x.duckdns.org (x.x.x.x) port 80 (#0)
    ...
* Connection #0 to host x.duckdns.org left intact

# https. - error:
$ curl -v https://x.duckdns.org
    ...
* connect to x.x.x.x port 443 failed: Connection refused
* Failed to connect to x.duckdns.org port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to x.duckdns.org port 443: Connection refused

Not recalling anything else at the moment.

6. Links to relevant resources:

I did a lot searches with various combinations of keywords (ie: caddy reverse proxy vagrant virtualbox, etc), but nothing really bore fruit.

Since the same setup on the rpi runs fine, I’m guessing it’s has something to do with the vagrant + virtualbox + docker combo.

Any thoughts or clues greatly appreciated. Thanks!

2

You can run docker-compose exec caddy caddy version to find the actual version. It’s often relevant.

Try not using host networking, and instead reverse_proxy to the container directly.

    reverse_proxy jellyfin:8096

Looks like Let’s Encrypt can’t reach your server. Make sure your port forwarding and firewalls are configured to allow traffic on ports 80 and 443 through to your VM, for connections from the internet.

3

Caddy Version: v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

I can’t seem to hit “jellyfin:8096”. Should I be able to? Is there a command to get the container’s hostname?

Thanks!

4

Using the default networking mode (not host mode), you should be able to connect containers to eachother by their service name. This doesn’t work from outside the container network, like from the host machine, because this is Docker’s built-in DNS server which resolves those names to the container IPs.

5

Success!

Removing network_mode: “host” and pointing at the jellyfin:8096 got me most of the way there. Then, I realized (thanks for the link) that I also needed to open port 8096 for jellyfin in the docker compose file.

Thanks for all your guidance. Cheers!

1 Like
6

You can use expose: for that to only expose it to other containers in the same network, rather than ports: which publishes it to the host.

:tada:

7

Nice tip. Thanks, again.

8

This topic was automatically closed after 30 days. New replies are no longer allowed.