Cannot start Caddy via Docker

zung102 · January 1, 2023, 12:00am

1. Output of `caddy version`:

Caddy:2

2. How I run Caddy:

sudo docker run -d -p 80:80 -p 443:443 --name caddy -v /etc/Caddyfile:/etc/caddy/Caddyfile -v /etc/caddy:/root/.local/share/caddy --restart on-failure caddy:2

a. System environment:

WSL2 with Ubuntu20.4
Docker Desktop

b. Command:

sudo docker run  -d -p 80:80 -p 443:443 --name caddy -v /etc/Caddyfile:/etc/caddy/Caddyfile -v /etc/caddy:/root/.local/share/caddy --restart on-failure caddy:2

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane. -->

d. My complete Caddy config:

``dzung.duckdns.org {
encode gzip

The negotiation endpoint is also proxied to Rocket

reverse_proxy /notifications/hub/negotiate 0.0.0.0:80

Notifications redirected to the websockets server

reverse_proxy /notifications/hub 0.0.0.0:3012

Send all other traffic to the regular Vaultwarden endpoint

reverse_proxy 0.0.0.0:80
}

Paste your config here, replacing this text.
Use caddy fmt to make it readable.
DO NOT REDACT anything except credentials.
LEAVE DOMAIN NAMES INTACT.
Make sure the backticks stay on their own lines.

The Caddyfile resides at /etc and etc/caddy/



### 3. The problem I'm having:
<!-- What are you trying to do, and what isn't working? Please describe the issue thoroughly enough so that anyone can reproduce the exact behavior you're seeing. Be as specific as possible. DO NOT USE WEB BROWSERS; use "curl -v" instead. -->

Caddy container was created but failed to start

### 4. Error messages and/or full log output:
<!--
    Please DO NOT REDACT any information except credentials. Leave domain names intact!
    Please DO NOT POST TRUNCATED LOG LINES as systemd is notorious for this.
    Please USE THIS COMMAND TO VIEW LOGS with systemd:
        $ journalctl -u caddy --no-pager | less +G`
    Please DO NOT USE WEB BROWSERS. Use `curl -v` instead.
    Please ENABLE DEBUG MODE FIRST by adding "debug" to the global options of your Caddyfile.
-->
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/641d0cc536e6a36b2d065081e4109cffb48a2a3427f15e2c0be7cb25a6934b25" to rootfs at "/etc/caddy/Caddyfile": mount /run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu-20.04/641d0cc536e6a36b2d065081e4109cffb48a2a3427f15e2c0be7cb25a6934b25:/etc/caddy/Caddyfile (via /proc/self/fd/14), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

Paste logs/commands/output here.
USE THE PREVIEW PANE TO MAKE SURE IT LOOKS NICELY FORMATTED.



### 5. What I already tried:
<!-- Show us what effort you've put in to solving the problem. Be specific -- people are volunteering their time to help you! Low effort posts are not likely to get good answers! -->



### 6. Links to relevant resources:

zung102 · January 1, 2023, 12:28am

I did not fill out the resources that I have used. I was installing Docker Vaultwarden …and referring to section Configure Caddy as Reverse Proxy…

How to Self-Host the vaultwarden Password Manager | Linode - https://www.linode.com/

zung102 · January 1, 2023, 2:49am

I have solved this problem. It was due to an existing /etc/Caddyfile as an directory. The Caddy container now started OK.

However, I cannot reach this url https//dzung.duckdns.org but can reach http://dzung.duckdns.org OK

What did I do wrong?

gyfer · January 1, 2023, 10:04am

Please post your Caddyfile config to further examine your issue
Example:

dzung.duckns.org {
    root * /usr/local/www
    file_server
}

One wild guess is you instruct Caddy to use HTTP. You CANNOT reach HTTPS if you do the following:

dzung.duckns.org:80 {
    ....
}

http://dzung.duckns.org {
    ....
}

Cheers

zung102 · January 1, 2023, 3:18pm

dzung.duckdns.org {
	encode gzip

	# The negotiation endpoint is also proxied to Rocket
	reverse_proxy /notifications/hub/negotiate 0.0.0.0:80

	# Notifications redirected to the websockets server
	reverse_proxy /notifications/hub 0.0.0.0:3012

	# Send all other traffic to the regular Vaultwarden endpoint
	reverse_proxy 0.0.0.0:8008
}

Also tried …
reverse_proxy 0.0.0.0:80
reverse_proxy 192.168.0.146:80 which is local address where vaultwarden resides

gyfer · January 1, 2023, 4:27pm

you reverse_proxy to 0.0.0.0 ??
can you even ping it ?

zung102 · January 1, 2023, 4:54pm

I was following installation instructions from various source. However, I have tried the following without success. …
reverse_proxy localhost:8008
reverse_proxy 192.168.0.146:8008
reverse_proxy :8008

gyfer · January 1, 2023, 5:08pm

If your Vaultwarden server IP address is 192.168.0.146
and if you can access it through HTTP , example http://192.168.0.146:8008
then

dzung.duckdns.org {
    reverse_proxy * http://192.168.0.146:8008
}

if you can only access through HTTPS , example https://192.168.0.146:8008
then

dzung.duckdns.org {
    reverse_proxy * https://192.168.0.146:8008 {
        header_up Host {http.reverse_proxy.upstream.hostport}
        transport http {
            tls
            tls_insecure_skip_verify
        }
   }
}

zung102 · January 1, 2023, 5:35pm

My case is of the first option. I applied it to Caddyfile, restarted the Caddy container and it still timed out for https://dzung.duckdns.org

zung102 · January 1, 2023, 10:24pm

Below is the info from the log …
2023-01-01 17:15:22 {“level”:“error”,“ts”:1672611322.5395856,“logger”:“http.acme_client”,“msg”:“validating authorization”,“identifier”:“dzung.duckdns.org”,“problem”:{“type”:“urn:ietf:params:acme:error:connection”,“title”:“”,“detail”:“72.140.211.179: Timeout during connect (likely firewall problem)”,“instance”:“”,“subproblems”:[]},“order”:“https://acme-v02.api.letsencrypt.org/acme/order/897366557/156313414517",“attempt”:1,"max_attempts”:3}
2023-01-01 17:15:23 {“level”:“info”,“ts”:1672611323.813485,“logger”:“http.acme_client”,“msg”:“trying to solve challenge”,“identifier”:“dzung.duckdns.org”,“challenge_type”:“http-01”,“ca”:“https://acme-v02.api.letsencrypt.org/directory”}
2023-01-01 17:15:34 {“level”:“error”,“ts”:1672611334.1061301,“logger”:“http.acme_client”,“msg”:“challenge failed”,“identifier”:“dzung.duckdns.org”,“challenge_type”:“http-01”,“problem”:{“type”:“urn:ietf:params:acme:error:connection”,“title”:“”,“detail”:“72.140.211.179: Fetching http://dzung.duckdns.org/.well-known/acme-challenge/t5ymhZiVKHK6uAEqH9k3VFA2Gf-0n1b4G9vN4B3N548: Timeout during connect (likely firewall problem)”,“instance”:“”,“subproblems”:[]}}
2023-01-01 17:15:34 {“level”:“error”,“ts”:1672611334.1063597,“logger”:“http.acme_client”,“msg”:“validating authorization”,“identifier”:“dzung.duckdns.org”,“problem”:{“type”:“urn:ietf:params:acme:error:connection”,“title”:“”,“detail”:“72.140.211.179: Fetching http://dzung.duckdns.org/.well-known/acme-challenge/t5ymhZiVKHK6uAEqH9k3VFA2Gf-0n1b4G9vN4B3N548: Timeout during connect (likely firewall problem)”,“instance”:“”,“subproblems”:[]},“order”:“https://acme-v02.api.letsencrypt.org/acme/order/897366557/156313443267",“attempt”:2,"max_attempts”:3}
2023-01-01 17:15:34 {“level”:“error”,“ts”:1672611334.1078453,“logger”:“tls.obtain”,“msg”:“could not get certificate from issuer”,“identifier”:“dzung.duckdns.org”,“issuer”:“acme-v02.api.letsencrypt.org-directory”,“error”:“HTTP 400 urn:ietf:params:acme:error:connection - 72.140.211.179: Fetching http://dzung.duckdns.org/.well-known/acme-challenge/t5ymhZiVKHK6uAEqH9k3VFA2Gf-0n1b4G9vN4B3N548: Timeout during connect (likely firewall problem)”}

A firewall problem? from WSL2? if so how to correct it?

matt · January 1, 2023, 11:34pm

(Thank you for helping, @gyfer!)

Usually a firewall problem means that a router or firewall is misconfigured and is not allowing the connection on port 80 or port 443 through to Caddy.

zung102 · January 1, 2023, 11:48pm

Do you see any problem below …
To Action From

80/tcp ALLOW IN Anywhere
443/tcp ALLOW IN Anywhere
22/tcp (OpenSSH) ALLOW IN Anywhere
443/tcp ALLOW IN 192.168.0.146
80/tcp (v6) ALLOW IN Anywhere (v6)
443/tcp (v6) ALLOW IN Anywhere (v6)
22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)

gyfer · January 2, 2023, 4:23am

If your Vaultwarden server IP address is 192.168.0.146, your firewall settings for PORT 80 and 443 should forward it to your Caddy IP (not 192.168.0.146) , and let Caddy do it’s reverse_proxy task.

You should accept ALL connection FROM WAN side for 80/tcp, 443/tcp, 80/tcp(v6), 443/tcp(v6) to your Caddy server / service. I am not sure and not comfortable with “ALLOW IN Anywhere”

gyfer · January 2, 2023, 4:41am

I just saw your are using WSL2. I now understand why you are sturggling.
WSL2 networks are all isolated by default. You have to open up and port the port manually, as mentioned here in Microsoft documentation ( Comparing WSL Versions | Microsoft Learn )

I went thought the same issue when I trying to run chia-blockchain through WSL2.
Here is some info : WSL Port Forwarding - DEV Community 👩‍💻👨‍💻
Script here : GitHub - snowphone/WSL2-port-forwarding-guide: Forwards Windows ports into WSL2 at startup (without logging-in)

This person show a quick but less secure way.

Hope above help.
Cheers.

zung102 · January 5, 2023, 2:54am

Thank you for pointing to me this great resource about WSL port forwarding need. However, I have applied this but the problem remained. I have no experience about Caddy and according to the log Caddy has encountered issues related to validating authorization timeout, or various challenge failures, could not get certificate from issuer

zung102 · January 5, 2023, 5:43pm

I have tried another way: manually obtaining the SSL certificate via acme.sh for DSN Duckdsn.org and specified it in Caddyfile as below …
dzung.duckdns.org:443 {
encode gzip
tls internal {$HOME/.acme.sh/dzung.duckdns.org/dzung.duckdns.org.cer} {$H> reverse_proxy * 192.168.0.146:8080

}

now there is no error in the Caddy log but I still could not access vaultwarden by HTTPS …timeout still

Wonder if you can detect any error here

gyfer · January 6, 2023, 3:44am

We need to rule out other possibilities to understand what is the problem.
Let’s put caddy config aside, let’s start with something basic.

dzung.duckdns.org {
    tls internal
    respond "Hello World!"
}

Apply above caddy config, can you access the page from host computer ?
How about other device at local network ?
If not, are you able to Ping it ?
What happen if you try to access https://127.0.0.1 and the host IP in local network ?
When you ping dzung.duckdns.org , what IP do you get ?
Is it same as your public IP which you able to find out and obtain here https://www.wtfismyip.com/

zung102 · January 6, 2023, 4:52am

I ran Caddy with your suggested ‘something basic’ I can access vaultwarden using localhost:8080 or dzung.duckdns.org:8080 or 192.168.0.146:8080 (i.e. no HTPPS) from another PC on the LAN. Pinging dzung.duckdns.org does return the public IP address as set by ISP

localhost:443 or 127.0.0.1:443 resulted in “Client sent an HTTP request to an HTTPS server.”

https://127.0.0.1:443 resulted in

The connection for this site is not secure.
ERR_SSL_PROTOCOL_ERROR

gyfer · January 6, 2023, 7:26am

https://x.x.x.x is standard on 443 port, you only need to specify if other than 443.
same goes to http://x.x.x.x:80
hence, vaultwarden is at hostip:8080 , with HTTP protocol., in your case http://127.0.0.1:8080

Because you are using locally issue certificate, your browser will prompt ERR_CERT_AUTH_INVALID “Certificate Invalid” when you try to access it. You can click proceed to continue
But not ERR_SSL_PROTOCOL_ERROR

If you are getting ERR_SSL_PROTOCOL_ERROR , something is fishy here. We need to check to make sure it come from the Caddy Docker , not other service.

Try to do curl --insecure https://localhost -v , and curl http://localhost:8080 -v

And proper curl response will something like below

zung102 · January 6, 2023, 2:35pm

zung@Dzungabc:/etc$ curl --insecure https://localhost -v

Trying 127.0.0.1:443…
TCP_NODELAY set
Connected to localhost (127.0.0.1) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS alert, internal error (592):
error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

zung@Dzungabc:/etc$ curl http://localhost:8080 -v

Trying 127.0.0.1:8080…
TCP_NODELAY set
Connected to localhost (127.0.0.1) port 8080 (#0)

GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.68.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< cache-control: public, max-age=600
< expires: Fri, 06 Jan 2023 14:42:33 GMT
< server: Rocket
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
< referrer-policy: same-origin
< x-xss-protection: 0
< content-security-policy: default-src ‘self’; base-uri ‘self’; form-action ‘self’; object-src ‘self’ blob:; script-src ‘self’; style-src ‘self’ ‘unsafe-inline’; child-src ‘self’ https://.duosecurity.com https://.duofederal.com; frame-src ‘self’ https://.duosecurity.com https://.duofederal.com; frame-ancestors ‘self’ chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src ‘self’ data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src ‘self’ https://api.pwnedpasswords.com https://2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ ;
< content-length: 1240
< date: Fri, 06 Jan 2023 14:32:33 GMT
<
<!doctype html>Vaultwarden Web Vault

by the way port forwarding as seen below …

PS C:\Users\user1> netsh interface portproxy show v4tov4

Listen on ipv4: Connect to ipv4:

Address Port Address Port

192.168.0.146 80 172.22.221.184 80
192.168.0.146 443 172.22.221.184 443
192.168.0.146 8080 172.22.221.184 8080
192.168.0.146 8443 172.22.221.184 8443