File server not automatically forwarding http to https

1. The problem I’m having:

I’ve noticed that my domain doesn’t forward my website automatically to the https:// domain when I just input the domain without http at all. I expect that when I put in domain.org, it should automatically be sent to https://r.domain.org/. When I use curl -vL domain.org the output is actually correct here’s a snippet showing it

*   Trying X.X.X.X:80...
* Connected to r.domain.org (X.X.X.X) port 80 (#0)
> GET / HTTP/1.1
> Host: r.domain.org
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://r.domain.org/
< Server: Caddy
< Date: Mon, 18 Sep 2023 11:45:49 GMT
< Content-Length: 0

But on many browsers, if I put http first or if I put nothing first it doesn’t redirect properly.

2. Error messages and/or full log output:

I couldn’t figure out how to enable debug mode, or the logs. I used

log {
        output file /home/user/Code/caddy_log_output.txt
    }

inside the domain I was interested in, but it didn’t generate the file.

3. Caddy version:

v2.7.4

4. How I installed and ran Caddy:

I believe I installed in using snap

a. System environment:

I’ve been running it on Ubuntu 22.04.3 LTS

b. Command:

I normally just let systemd run it as a service.

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/caddy run --config /home/user/Code/Caddyfile
ExecReload=/usr/bin/caddy reload --config /home/user/Code/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

n.domain.org {
        reverse_proxy :11000
}

j.domain.org {
        reverse_proxy :8096
}

o.domain.org {
        reverse_proxy :5000
}

r.domain.org {
        root * /home/user/Code/ryan.dikdan.info
        file_server
        log {
                output file /home/user/Code/caddy_log_output.txt
        }
}

Hi, welcome.

When I run curl -vL domain.org I get:

*   Trying 65.254.244.180:80...
* Connected to domain.org (65.254.244.180) port 80
> GET / HTTP/1.1
> Host: domain.org
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Mon, 18 Sep 2023 15:36:38 GMT
< Content-Type: text/html; charset=iso-8859-1
< Content-Length: 206
< Connection: keep-alive
< Server: Apache
< Content-Security-Policy: frame-ancestors 'self'  https://*.weeblycloud.com  https://*.sitelock.com  https://*.mojomarketplace.com  http://*.ipage.com  http://*.yourhostingaccount.com  https://*.ecwid.com  https://platform.cloud.coveo.com  https://search.cloud.coveo.com
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: frame-ancestors 'self'  https://*.weeblycloud.com  https://*.sitelock.com  https://*.mojomarketplace.com  http://*.ipage.com  http://*.yourhostingaccount.com  https://*.ecwid.com  https://platform.cloud.coveo.com  https://search.cloud.coveo.com
< X-Frame-Options: SAMEORIGIN
< Location: http://www.domain\.com/
< 
* Ignoring the response-body
* Connection #0 to host domain.org left intact
* Issue another request to this URL: 'http://www.domain\.com/'
*   Trying 13.57.130.190:80...
* Connected to www.domain\.com (13.57.130.190) port 80
> GET / HTTP/1.1
> Host: www.domain\.com
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Mon, 18 Sep 2023 15:36:38 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
< Location: https://www.domain\.com/
< 
* Ignoring the response-body
* Connection #1 to host www.domain\.com left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://www.domain\.com/'
*   Trying 18.221.195.49:443...
* Connected to www.domain\.com (18.221.195.49) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.domain\.com
*  start date: Aug 28 00:00:00 2023 GMT
*  expire date: Aug 27 23:59:59 2024 GMT
*  subjectAltName: host "www.domain\.com" matched cert's "*.domain\.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.domain\.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.domain\.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: www.domain\.com
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/2 200 
< server: nginx
< date: Mon, 18 Sep 2023 15:36:39 GMT
< content-type: text/html;charset=utf-8
< content-length: 135866
< x-dispatcher: 02
< x-vhost: publish
< x-content-type-options: nosniff
< accept-ranges: bytes
< vary: Accept-Encoding,User-Agent
< x-frame-options: SAMEORIGIN
< 

and so I’m not seeing Caddy anywhere in this, and am unable to reproduce the issue to help.

The other possibility is that you redacted the domain names against the forum rules. Unfortunately because the domain names are redacted it’s impossible for us to help you more than guessing. The actual configuration, commands, and domain names matter.

I expect that when I put in domain.org, it should automatically be sent to https://r.domain.org/. When I use curl -vL domain.org the output is actually correct here’s a snippet showing it

* Connected to r.domain.org (X.X.X.X) port 80 (#0)

Sorry; this doesn’t quite make sense to me either. The config you posted doesn’t have domain.org in it at all, and I’m not sure why curl would be trying to contact r.domain.org when you ask for domain.org.


Please follow the instructions in the help template so we can help you.

Ah, I didn’t see that rule. The domain I’m trying to have forward properly is r5.website.com . Sorry about that. I censored things just so I don’t get scraped, but I see the utility in sharing it.

Thanks. Can you provide your config then?

Oh right.

n.website.com {
        reverse_proxy :11000
}

j.website.com {
        reverse_proxy :8096
}

o.website.com {
        reverse_proxy :5000
}

r5.website.com {
        root * /home/user/Code/website
        file_server
        log {
                output file /home/user/Code/caddy_log_output.txt
        }
}

I guess I can edit these things out when I find a solution.

Thanks – when I use curl on those domains, it just says “Trying …” and then hangs. I suspect the route to the host is messed up, or the network (firewall? port forwarding?) is misconfigured. But I’m not seeing the same behavior you’re reporting.

Does it work when you try them with https://

Make sure you have port 80 open and forwarded.

They are both forwarded, I don’t think they are closed on my ubuntu machine since I can access all of these domains, it’s just that if I don’t include the https:// some browsers get hung up.

Well, the problem resides somewhere between the public internet and your server, not with Caddy itself. You need to make sure nothing blocks port 80. Check with your ISP to see if they block that port (some ISPs do).

Gotcha. It does appear that port 80 is blocked even after I ran sudo ufw allow 80/tcp which implies to me that the ISP is blocking it, but not blocking 443 since https:// works fine. Thanks for your guidance!

But this doesn’t explain why curl worked fine. It seems that curl can get the 308 Permanent Redirect status code and then accurately get redirected to the https:// domain. Why can’t browsers do the same? Perhaps caddy isn’t working well with the browsers then?

It entirely depends on where the request is coming from and which machines it gets routed through. If you tried curl from the machine Caddy is running on, then there’s not going to be a networking issue because it’s connecting to itself.

$ curl -v http://jellyfin.dikdan.info                                                                              
*   Trying 69.112.209.11:80...
^C

$ curl -v https://jellyfin.dikdan.info                                                                             
*   Trying 69.112.209.11:443...
* Connected to jellyfin.dikdan.info (69.112.209.11) port 443 (#0)

:point_up: it fails from my machine, therefore it’s a problem between the internet and your server, e.g. your router, or ISP.

FYI, domains are public information. Cert issuance is tracked in public certificate transparency logs. Redacting them from your posts only wastes our time and yours.

1 Like

Thanks for the update, and in the future I’d recommend you inform people of this type of information in a less rude way (“wastes our time and yours”), lest you dissuade more people from learning and enjoying themselves online, but I guess this is just the general attitude of forums…

Also I’d appreciate if you’d remove my domain.

In my opinion Francis’ tone isn’t rude, he’s just stating the facts. (Maybe this is a cultural difference. ) It is true that spending time redacting was wasted because we couldn’t help because of that extra effort, and the original config still needed to be posted.

lest you dissuade more people from learning and enjoying themselves online, but I guess this is just the general attitude of forums…

Part of the problem is we don’t have enough helpers, so we have to get through each question very efficiently. It’s not ideal, but it’s difficult to help everyone with the current ratio of askers to helpers.

I could also understand the frustration. The help template that you filled out has very loud notices to not redact information like domain names, yet we still see this on probably most topics that get posted, so I think it’s understandable that there’s even less patience when people go out of their way to disregard the rules.

Also I’d appreciate if you’d remove my domain.

You should be able to edit your posts – if not, maybe it’s not allowed for very brand new users, but even if you do, you can still see the edit history; but even if that didn’t exist, bots probably archived/cached these posts and we can’t do anything about that. And even IF bots didn’t capture this page, the domain names are public information anyway so not posting them here doesn’t keep them secret or anything :man_shrugging:

Anyway, to address your question:

I don’t know. I still can’t make sense of your curl command because the domain you gave it doesn’t match its output.

Neither Francis nor I can reproduce the behavior. To us, it looks like port 80 is blocked.

Caddy works fine with browsers, it’s something about the network on your end. Maybe your ISP. Maybe your firewall. Maybe your port forwarding. Maybe your kernel. There’s a lot of things it could be.

You can check this yourself by doing the request from your phone’s cellular network, if your server is also at home.

2 Likes

Thanks for the thorough reply. I can understand the frustrations with limited number of people and everyone under the sun not knowing how it all works. I really appreciate all your effort, I’ll try to web redirect the http to some https or something using dynu.

In terms of requesting not to have my server posted publicly, while domains are public knowledge, I simply don’t want people from the forum or just from google, finding this page and clicking there. I’m not at all an expert in this field, but I do appreciate that the lesser known a site is, the safer it might be.

Thanks again.

1 Like

Thanks for understanding.

That’s fine; but people clicking there from this forum topic is also kind of orthogonal to everything. Bots are going to do that anyway. People scan CT logs, IPs, and ports, and will do that anyway, they don’t need a forum post to discover something. (The whole IPv4 internet can’t be scanned in a few hours IIRC). Just FYI :slight_smile: You are not any safer by redacting domain names here.

Gotcha, thanks for the information about domains. I know this isn’t a caddy issue then, but in the case where the ISP is blocking 80 (and not 443), is there a way to forward the http to https perhaps at the DNS level? Or any other ways to get around this mild inconvenience?

SRV records can specify ports to use, but otherwise DNS doesn’t deal with ports AFAIK. And SRV records are only useful for systems that support those, like cluster orchestrators. Regular web browsers and ACME CAs won’t, I’m afraid.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.