Cloudflare, subdomains and Caddy

prathampatel · July 21, 2022, 5:11am

1. Caddy version (`caddy version`):

I am using the caddy:2-alpine container image.

podman exec -it caddy-vishveshwaraya sh
/srv # caddy version
v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

a. System environment:

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="36 (Server Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server

$ podman --version
podman version 4.1.1

$ systemctl --version
systemd 250 (v250.8-1.fc36)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

$ ip addr | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 10.0.0.19/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
    inet6 fe80::6d41:a002:a508:f7f0/64 scope link noprefixroute

b. Command:

(none? since I’m using a compose file… the compose file runs the command caddy run --config /etc/caddy/Caddyfile)

c. Service/unit/compose file:

version: 3.3

services:

    reverse-proxy:
        image: caddy:2-alpine
        container_name: caddy-vishveshwaraya
        command: caddy run --config /etc/caddy/Caddyfile
        restart: always
        ports:
            - 8008:80
            - 8443:443
        volumes:
            - /containers/volumes/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
            - /containers/volumes/caddy/site:/srv:Z
            - /containers/volumes/caddy/caddy_data:/data:Z
            - /containers/volumes/caddy/caddy_config:/config:Z
            - /containers/volumes/caddy/ssl:/etc/ssl:Z

d. My complete Caddyfile or JSON config:

thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy http://localhost:8020
}

www.thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy http://localhost:8020
}

git.thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy http://localhost:8010
}

What have I done until now

Purchase my domain from Google Domains
Get nameservers from Cloudlare
Enter cloudflare NS into my settings for the domain (Google Domains)
Wait for a while
In Cloudflare, add DNS record for my domain:
Type: A, Name @, Content: 103.112.224.205, Proxy Status: DNS Only, TTL: auto
Type: CNAME, Name: www, Content: @, Proxy Status: DNS Only, TTL: auto
Type: CNAME, Name: git, Content: @, Proxy Status: DNS Only, TTL: auto
In Cloudflare, goto SSL/TLS > Overview, set Encryption mode to Full. (not the strict one)
In Cloudflare, goto SSL/TLS > Origin Server > Create Certificate, use default values for ‘Private Key type’, ‘Hostnames’ and Certificate expiry.
Copy cert to /containers/volumes/caddy/ssl/certs/certificate.pem and key to /containers/volumes/caddy/ssl/private/key.pem.
chmod 700 /containers/volumes/caddy/ssl/private
chmod 600 /containers/volumes/caddy/ssl/private/key.pem
(just like ~/.ssh)

Internal config

Router:
Forward external port 80 to internal 10.0.0.19:8008
Forward external port 8443 to internal 10.0.0.19:8443

On my Server:
Have two containers running:

gitea:

        ports:
            - 8010:3000 #web UI
            - 8011:2222

nextcloud:

        ports:
            - 8020:80

3. The problem I’m having:

When I access www.thefossguy.com from the web browser, I get the following entry in my Caddy server log:

{"level":"error","ts":1658376374.4418187,"logger":"http.log.error","msg":"dial tcp [::1]:8020: connect: connection refused","request":{"remote_ip":"10.89.0.106","remote_port":"39116","proto":"HTTP/2.0","method":"GET","host":"www.thefossguy.com","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-GB,en;q=0.5"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"www.thefossguy.com"}},"duration":0.003173477,"status":502,"err_id":"utejac14x","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

But when I run the command curl -v www.thefossguy.com, I get no such entry in the log. Here is the output (along with https:// as well)

$ curl -v www.thefossguy.com
*   Trying 103.112.224.205:80...
* Connected to www.thefossguy.com (103.112.224.205) port 80 (#0)
> GET / HTTP/1.1
> Host: www.thefossguy.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://www.thefossguy.com/
< Server: Caddy
< Date: Thu, 21 Jul 2022 04:08:32 GMT
< Content-Length: 0
<
* Closing connection 0

$ curl -v https://www.thefossguy.com
*   Trying 103.112.224.205:443...
* Connected to www.thefossguy.com (103.112.224.205) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

When I access git.thefossguy.com, Firefox gives an error saying We can’t connect to the server at git.thefossguy.com.. Even curl errors out. But I can dig it…

$ dig git.thefossguy.com @1.1.1.1

; <<>> DiG 9.10.6 <<>> git.thefossguy.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61777
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;git.thefossguy.com.            IN      A

;; ANSWER SECTION:
git.thefossguy.com.     300     IN      CNAME   thefossguy.com.
thefossguy.com.         300     IN      A       103.112.224.205

;; Query time: 244 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Jul 21 09:39:50 IST 2022
;; MSG SIZE  rcvd: 77

$ curl -vvv git.thefossguy.com
* Could not resolve host: git.thefossguy.com
* Closing connection 0
curl: (6) Could not resolve host: git.thefossguy.com

$ curl -vvv https://git.thefossguy.com
* Could not resolve host: git.thefossguy.com
* Closing connection 0
curl: (6) Could not resolve host: git.thefossguy.com

Initial log (before accessing the domain)

{"level":"info","ts":1658376190.6972106,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
{"level":"info","ts":1658376190.7044237,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1658376190.7055142,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400047bb90"}
{"level":"warn","ts":1658376190.7063463,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [cloudflare origin certificate *.thefossguy.com thefossguy.com]: no URL to issuing certificate"}
{"level":"info","ts":1658376190.706652,"logger":"http","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"www.thefossguy.com","server_name":"srv0"}
{"level":"info","ts":1658376190.7069805,"logger":"http","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"git.thefossguy.com","server_name":"srv0"}
{"level":"info","ts":1658376190.7072499,"logger":"http","msg":"skipping automatic certificate management because one or more matching certificates are already loaded","domain":"thefossguy.com","server_name":"srv0"}
{"level":"info","ts":1658376190.7074814,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1658376190.7092395,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1658376190.7107055,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1658376190.7107725,"msg":"serving initial configuration"}
{"level":"info","ts":1658376190.7173502,"logger":"tls","msg":"finished cleaning storage units"}

5. What I already tried:

I tried changing Proxy status from ‘DNS Only’ to ‘Proxied’ and tried accessing domains.

For thefossguy.com, www.thefossguy.com and git.thefossguy.com, I get Error 502 from Cloudflare with an error message that says The web server reported a bad gateway error..

Below are my logs, in the order of accessing thefossguy, www, and git (from the brower).

{"level":"error","ts":1658379982.3300018,"logger":"http.log.error","msg":"dial tcp [::1]:8020: connect: connection refused","request":{"remote_ip":"10.89.0.111","remote_port":"38600","proto":"HTTP/2.0","method":"GET","host":"thefossguy.com","uri":"/","headers":{"Cf-Ray":["72e16828191c89b0-SIN"],"Sec-Fetch-Mode":["navigate"],"Cf-Connecting-Ip":["103.112.224.205"],"Sec-Fetch-Site":["none"],"Cookie":[],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-GB,en;q=0.5"],"X-Forwarded-For":["103.112.224.205"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Upgrade-Insecure-Requests":["1"],"Cf-Ipcountry":["IN"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"thefossguy.com"}},"duration":0.002376939,"status":502,"err_id":"9tstx21wb","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1658379991.286322,"logger":"http.log.error","msg":"dial tcp [::1]:8020: connect: connection refused","request":{"remote_ip":"10.89.0.111","remote_port":"34722","proto":"HTTP/2.0","method":"GET","host":"www.thefossguy.com","uri":"/","headers":{"X-Forwarded-For":["103.112.224.205"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-GB,en;q=0.5"],"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Cf-Connecting-Ip":["103.112.224.205"],"Accept-Encoding":["gzip"],"Cf-Ray":["72e168602be989b0-SIN"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Sec-Fetch-Dest":["document"],"Cf-Ipcountry":["IN"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"www.thefossguy.com"}},"duration":0.003317578,"status":502,"err_id":"5ni1mdw4g","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1658379994.3847785,"logger":"http.log.error","msg":"dial tcp [::1]:8010: connect: connection refused","request":{"remote_ip":"10.89.0.111","remote_port":"34736","proto":"HTTP/2.0","method":"GET","host":"git.thefossguy.com","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Firefox/102.0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Cf-Connecting-Ip":["103.112.224.205"],"Cf-Ipcountry":["IN"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Cf-Ray":["72e168737f3189b0-SIN"],"Accept-Language":["en-GB,en;q=0.5"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["103.112.224.205"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"git.thefossguy.com"}},"duration":0.007401204,"status":502,"err_id":"25ncajf2i","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}

I removed reverse_proxy and instead, added respond like so:

thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        respond "Hello Nextcloud (main)!"
}

www.thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        respond "Hello Nextcloud (www)!"
}

git.thefossguy.com:443 {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        respond "Hello Git!"
}

With this change, I get the correct reponse for each subdomain. But when I revert back to using reverse_proxy, I keep getting 502 from Cloudflare.

I’m totally new to the web-deploy side of things (except for local ones). I can’t, for the life of me, figure out what is wrong and/or what needs to be done.

Thank you in advance!

francislavoie · July 21, 2022, 3:21pm

localhost will only reach services running inside the same container. If you want to proxy to other containers, use its service name and the internal port of that service. For example: reverse_proxy gitea:3000

You don’t need to specify :443 here, that’s the default when Caddy is serving HTTPS.

Did you mean 443 -> 8443?

prathampatel · July 21, 2022, 4:12pm

Ah it works! THANK YOU A LOT!!!

Yes, I meant 443 -> 8443. Sorry, it was a typo.

EDIT: Solution is quoted below.

I changed the “mapped” port and localhost to the name I used in container_name and their real name.

Now, my Caddyfile looks like this:

thefossguy.com {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy nextcloud-aarav:80
}

www.thefossguy.com {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy nextcloud-aarav:80
}

git.thefossguy.com {
        tls /etc/ssl/certs/certificate.pem /etc/ssl/private/key.pem
        reverse_proxy gitea-aatman:3000
}

Note that gitea-aatman and nextcloud-aarav are the strings I used as container_name in the respective compose scripts.

system · August 20, 2022, 5:12am

This topic was automatically closed after 30 days. New replies are no longer allowed.