Caddy on QNAP - set up reverse proxy

Help
1

Hello!

I am completely new to Caddy and I’m finding myself very confused, I think what I am trying to achieve is relatively simple for someone with the right knowledge so hopefully someone out there can help me out.

I have a QNAP NAS and my goal is to access NZBGet, Radarr & Sonarr remotely and over SSL. From what I have read, using a reverse proxy is the easiest way to achieve this on QNAP. I am able to access my NAS home page remotely via SSL and it’s set up via DDNS.

So I can type mynas.myurl:443 for example and it will resolve to my IP and display my nas login. I can forward any port and substitute that port in the url to go to that page, anything non-ssl works fine but when attempting to use SSL, I would need to configure each app individually and it looks like trouble on QNAP.

My understanding is that I can use a Caddyfile so that I can type mynas.myurl/nzbget and then I could be directed there with SSL.

My first problem is my understanding of where do I find or where do I put the Caddyfile, my default installation of Caddy is “/share/CACHEDEV1_DATA/.qpkg/Caddy” but there is no caddyfile. Any documentation I have seen states that the Caddyfile can go anywhere but I don’t quite understand how that can be possible.

Secondly, as you can probably tell, I am very new to this and I don’t understand even after reading the caddyfile tutorial it just goes over my head and I find myself no closer to understanding anything over the first line of the caddyfile.

I would appreciate any insight, thank you :slight_smile:

2

Hi @mupet0000, welcome to the Caddy community!

You said you’ve read over the Caddyfile tutorial, were you referring to this page?

https://caddyserver.com/tutorial/caddyfile

That’s really the best documentation we have for beginners. If you’re having trouble understanding it, I suggest taking it one part at a time - you can post questions you have about each part as you go.

To quote the Caddyfile Primer, the reason this works:

If the Caddyfile is in a different location or has a different name, tell Caddy where it is:

caddy -conf ../path/to/Caddyfile

Basically, you can put the Caddyfile wherever you like as long as you tell Caddy where to find it by specifying the file path using the -conf flag.

3

Thank you for your response. I believe I have understood how to do what I want, but I’ve run into some issues.

I created a simple Caddyfile as follows:
my_url_here_com
proxy /radarr localhost:1234

I placed the caddyfile in the directory of my choice and ran the command but this is the result:

[~] # caddy -conf …/share/CACHEDEV1_DATA/.qpkg/Caddy/Caddyfile
Activating privacy features… 2019/05/10 22:30:42 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/05/10 22:30:43 [INFO] [myurlhere] AuthURL: letsencrypturlhere
2019/05/10 22:30:43 [INFO] [myurlhere] acme: use tls-alpn-01 solver
2019/05/10 22:30:43 [INFO] [myurlhere] acme: Trying to solve TLS-ALPN-01
2019/05/10 22:31:29 [INFO] Unable to deactivated authorizations: letsencrypturlhere
2019/05/10 22:31:29 [myurlhere] failed to obtain certificate: acme: Error → One or more domains had a problem:
[myurlhere] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Cannot negotiate ALPN protocol “acme-tls/1” for tls-alpn-01 challenge,

I’m not sure what’s going wrong and I’m wondering if you could point me in the correct direction.

EDIT:
I think the above error was caused by my NAS having port 443 already binded. I disabled the NAS’s built in SSL and this allowed caddy to get the SSL certificate validated.

This time the command ended in:

2019/05/10 23:39:51 [INFO] [myurlhere] The server validated our request
2019/05/10 23:39:51 [INFO] [myurlhere] acme: Validations succeeded; requesting certificates
2019/05/10 23:39:53 [INFO] [myurlhere] Server responded with a certificate.
done.
2019/05/10 23:39:53 Listen: listen tcp :80: bind: address already in use

When attempting to visit myurl/radarr it’s a “The requested URL /radarr was not found on this server.”.
When attempting to visit myurl I am not redirected to an SSL page.
When attempting to visit https://myurl I am given “ERR_CONNECTION_REFUSED”

So I changed the http-port to 85 and the port to 2019 and got this:

Serving HTTP on port 2019
http://:2019

WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with ulimit -n 8192.

I watched a video on YouTube where the caddyfile was literally one line which was the url and in 30 seconds the guy had his SSL domain loaded. I cannot understand why I am facing so many errors when I’m trying to achieve the exact same thing :confused:

I don’t know where to go from here.

4

For Automatic HTTPS, Caddy needs to be able to bind ports 80 and 443.

This makes me think that Apache is running on port 80, not Caddy. Caddy would redirect you, as you noticed.

Caddy’s not running at all, so you don’t get a response from HTTPS on port 443, the OS just refuses connection.

How exactly did you go about this?

5

You were right, Apache had port 80 so I reconfigured the NAS to use alternative ports to free up the default ports for Caddy and it is now working and reverse proxy things like /radarr are functioning correctly with SSL too so that’s fantastic!

I am facing one last hurdle which is now that when I go to myurl, instead of being presented with the default webpage that it usually serves up, I get a (the page is served in SSL though):

https://myurl/redirect.html?count=0.36205010057744524
with the content:
404 Not Found

Normal behaviour without caddy is:

  1. myurlhere (port 80, but now 86) goes to /home/Qthttpd/index.html
  2. it changes to webserver port 8080 and redir to /home/httpd/index.html
  3. then to /home/httpd/redirect.html
  4. finally to /home/httpd/cgi-bin/Qts.cgi and the homepage is displayed

If I tell caddy the root is /home/httpd it gives me the same error but it also downloads qts.cgi through chrome.

I have configured the NAS to use port 86 for normal http and port 446 for SSL. When I attempt to visit myurl:86 I am automatically changed to HTTPS with:

ERR_SSL_PROTOCOL_ERROR

If I go to myurl:446 I get served my page with a default cert issued from my NAS and chrome says:

You cannot visit myurlhere right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

If I can get the normal home page to load up then everything is configured and I’m done. What do you think I need to check to fix this?

For reference, here is my current caddyfile which I have built using an example file:

myurlhere
root /home/Qhttpd
gzip

header / {
X-Content-Type-Options nosniff
X-XSS-Protection “1; mode=block”
Strict-Transport-Security “max-age=31536000; includeSubDomains; preload”
-Server
}

proxy /radarr http://127.0.0.1:7878 { # https://radarr.video/
transparent
header_upstream X-Forwarded-Host {host}
}

proxy /sonarr http://127.0.0.1:8989 { # https://sonarr.tv/
transparent
header_upstream X-Forwarded-Host {host}
}

proxy /nzbget http://127.0.0.1:6789 { # http://nzbget.net/
transparent
header_upstream X-Forwarded-Host {host}
}

}

My current solution is to turn on the NAS built in LetsEncrypt cert and browse to myurl:446 which allows me to visit the nas home page via ssl without going through caddy. Not sure why going through caddy is breaking the home page.

Sorry for the barrage of information but I’m trying to provide everything I know so that you have all the correct information and also if anyone else faces the same issue they are able to reference this thread :slight_smile:

6

More information is never bad!

This is a side effect of this:

header / {
  X-Content-Type-Options nosniff
  X-XSS-Protection “1; mode=block”
  Strict-Transport-Security “max-age=31536000; includeSubDomains; preload”
  -Server
}

Specifically, the Strict-Transport-Security. You’ve now visited myurl hosted by Caddy, your browser received that header, and now until 31,536,000 seconds (365 days) have passed since your last visit, your browser won’t allow non-HTTPS access to that URL.

That said, you’re on a non-standard HTTP port, so it doesn’t change the port when it attempts HTTPS, it just tries it on 86, which you’re serving HTTP from. Hence the protocol error - your browser is trying to negotiate HTTPS and your HTTP server is understandably confused by that.

Also a side effect of Strict-Transport-Security. You cannot use an untrusted certificate with HSTS. You will not be able to use myurl to access any website that doesn’t have valid HTTPS, at least until the HSTS header wears off for myurl in your browser.

Suggestion: Don’t navigate to myurl:86 or myurl:446. Instead, have Caddy proxy to your NAS, e.g. nas.myurl with a proxy to localhost:86 or similar as required, and use Caddy’s valid HTTPS with your HSTS header.

7

Thank you very much!

All appears to be working with HTTPS once I added this to my caddyfile:

proxy / 127.0.0.1:8080 {
transparent
header_upstream X-Forwarded-Host {host}
}

I originally tried with port 86 but it seems that the nas’s web portal automatically redirects from your web port to the system port which defaults as 8080, with caddy directing to port 86 and then the nas redirecting me to port 8080 I lost SSL. But with caddy going straight to 8080 SSL is fully working.

It looks like I have achieved exactly what I set out to so I am very grateful for your help as I really couldn’t have done it without you!

Lastly I have a question about security. I originally had that code that was causing issues in my caddyfile because it’s from a sample caddyfile and it supposedly increases security. Obviously I don’t want to be insecure in any way so I’m wondering if I should add anything into my caddyfile or be aware of anything?

8

Good to hear Caddy’s working for you!

Not particularly! Caddy is secure by default.

Many people believe it’s necessary to implement a battery of HTTP security headers, like the ones you had above. There’s plenty of guides out there on the common headers and what they do (such as this one: Hardening Your HTTP Security Headers - KeyCDN).

In my opinion, these aren’t necessary most of the time. In cases like Strict-Transport-Security, they can actually be harmful if you don’t have a full understanding of how it works and when you should (or shouldn’t) use it. Others like Content-Security-Policy can also cause lots of headaches for site owners that haven’t taken the time to fully understand the usage.

The oddball there is -Server, which some people believe reduces the likelihood a malicious actor can identify the software you use to serve your website, and therefore make it harder for them to exploit it. I’m not particularly convinced that it’s very effective, though.

I don’t generally run any of these security measures on my own sites, except occasionally Content-Security-Policy - but I don’t configure that on the web server, I specify it in a <meta> tag in the HTML document.

2 Likes
9

It’s been smooth sailing up until recently. Nothing has changed on my end but for whatever reason, Caddy is no longer working. Recently I noticed that when attempting to visit my urls I am getting connection refused and connection closed errors.

I noticed there was an update for Caddy from 1.0 to 1.01 on my NAS and installed that. Same issue. When the command is run I get this output:

caddy -conf /share/CACHEDEV1_DATA/.qpkg/Caddy/Caddyfile
2019/07/21 18:12:15 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:12:16 [INFO] [myurlhere] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:12:16 [INFO] [myurlhere] acme: Could not find solver for: tls-alpn-01
2019/07/21 18:12:16 [INFO] [myurlhere] acme: use http-01 solver
2019/07/21 18:12:16 [INFO] [myurlhere] acme: Trying to solve HTTP-01
2019/07/21 18:12:43 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:12:44 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:12:45 [INFO] [myurlhere] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:12:45 [INFO] [myurlhere] acme: Could not find solver for: tls-alpn-01
2019/07/21 18:12:45 [INFO] [myurlhere] acme: use http-01 solver
2019/07/21 18:12:45 [INFO] [myurlhere] acme: Trying to solve HTTP-01
2019/07/21 18:13:00 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:01 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:13:02 [INFO] [myurlhere] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:02 [INFO] [myurlhere] acme: Could not find solver for: tls-alpn-01
2019/07/21 18:13:02 [INFO] [myurlhere] acme: use http-01 solver
2019/07/21 18:13:02 [INFO] [myurlhere] acme: Trying to solve HTTP-01
2019/07/21 18:13:16 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:17 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:13:18 [INFO] [myurlhere] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:18 [INFO] [myurlhere] acme: use tls-alpn-01 solver
2019/07/21 18:13:18 [INFO] [myurlhere] acme: Trying to solve TLS-ALPN-01
2019/07/21 18:13:31 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:32 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:13:33 [INFO] [myurlhere] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:13:33 [INFO] [myurlhere] acme: use tls-alpn-01 solver
2019/07/21 18:13:33 [INFO] [myurlhere] acme: Trying to solve TLS-ALPN-01
2019/07/21 18:14:00 [INFO] Unable to deactivated authorizations: https://acme-v02.api.letsencrypt.org/acme/authz/urlhere
2019/07/21 18:14:01 [INFO] [myurlhere] acme: Obtaining bundled SAN certificate
2019/07/21 18:14:02 failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/, url:

It appears to keep trying and failing until I reach the rate limit. My caddyfile has not changed at all. All I noticed is that I had a firmware update, the NAS rebooted and since then Caddy hasn’t been working. I am unsure what to investigate to fix this.

Looking at one of the letsencrypt logs:

{
  "identifier": {
    "type": "dns",
    "value": "myurlhere"
  },
  "status": "invalid",
  "expires": "2019-07-28T17:12:15Z",
  "challenges": [
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/urlhere",
      "token": "tokenhere"
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/urlhere",
      "token": "tokenhere"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "Fetching http://myurlhere/.well-known/acme-challenge/urlhere: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/urlhere",
      "token": "tokenhere",
      "validationRecord": [
        {
          "url": "http://myurlhere/.well-known/acme-challenge/urlhere",
          "hostname": "myurlhere",
          "port": "80",
          "addressesResolved": [
            "correctIPhere"
          ],
          "addressUsed": "correctIPhere"
        }
      ]
    }
  ]
}

I have checked:
The NAS is not configured to use port 80 or 443, left for caddy
DDNS is configured and still working

Not sure what to check next as everything seems right and it was previously working for around 2 months straight with this exact configuration.

10

Hi @mupet0000, it’s strange that we’re not seeing this error from the LetsEncrypt logs shown in your Caddy output.

That said, the guts of the error from LE’s end is Timeout during connect (likely firewall problem) - so it looks like, to them, your Caddy isn’t accessible.

You can try testing whether Caddy is still externally accessible by running caddy -port 80 "status 200 /" on your server and then running curl -I http://[your url] on a computer outside your network (tethering a laptop to your phone works well for this).

11

Does the NAS have the same internal IP address? Are you sure packets are being forwarded properly after the reboot?

12

After running caddy -port 80 “status 200 /” I get this result:

$ curl -I http://myurl HTTP/1.1 504 Gateway Time-out Server: WebProxy/1.0 Pre-Alpha Date: Wed, 24 Jul 2019 21:22:14 GMT Content-Length: 0 Connection: keep-alive

It looks like Caddy is unable to get out onto the internet, maybe it’s unable to forward ports since the the NAS update, I am unsure. Is it possible to get a debug log output from Caddy to see if everything appears to be working as it should, or would I get an error in the normal log if it was unsuccessful at using the port it needs?

Nothing else has changed. My DDNS is serving pages on other ports that are unrelated to Caddy with no issues just as it was previously.

Nothing else is using port 80 or 443 so there should be no reason why Caddy can’t take those ports.

13

Server: WebProxy/1.0 Pre-Alpha

This part is concerning.

It means your port forwards aren’t reaching Caddy. They’re forwarding to another proxy server, and this proxy server can’t find Caddy.

14

Ah I see. So it’s either going to be malware or that new qnap update has done something stupid. I think I’ll contact qnap and ask them as I’m relatively confident my NAS isn’t infected but I will double check.

There must be some network tools I can use to find out what program is running this proxy. I will do some further investigating. Once I’m able to eliminate whatever it is, I assume Caddy will begin working again.

I’ll post again once I have figured it out. Thanks for the help, it’s very much appreciated!

1 Like
15

So after a long back and forth with QNAP on this issue, it appears to be resolved and Caddy is now working again. QNAP were not able to identify the cause or really help me at all, but I noticed some things which led me to resolving the issue.

First of all, I eliminated some programs that were running on my NAS that I did not need running, one of them was Qapache. It was not configured to use port 80 nor did it serve anything (on port 80), other than possibly that weird 503 error, but once Qapache stopped running, port 80 went back to being like any other closed port and refused connections.

Note that Qapache was able to reach the internet on port 80 in order to give a 503 error, without me forwarding any ports for it. Just like Caddy was working on port 80 and 443 without me forwarding any ports for it.

Once port 80 was back in that state I thought that Caddy would begin working again as it could now pick up the unused port. Nope, I still had the exact same output from Caddy. After some thinking I manually port forwarded 80 and 443 using the QNAP router config program and Caddy began working. Remember that I specifically had to disable port 80 and 443 from being forwarded previously as this caused Caddy to fail on the previous QTS update.

Some kind of behaviour has changed since this QTS update that’s for sure, QNAP have no idea of course but luckily with a bit of trial and error it’s sorted.

I hope my trial and error can help anyone facing similar issues with Caddy on QNAP.
I’ll see you at the next QTS update :stuck_out_tongue:

2 Likes
16

Hi,

I am a QNAP and Caddy user. Not sure I can really help you here as it seems you already went far beyond what I did on my Nas (i.e. use caddy for a lot of apps, but in http only).

However I write you to tell that, each time the Caddy package is updatesd, you have to reconfigure the Caddy start script to make it point to the right config file (and use the right ports, if you set them therein).

Finally, I am curious to know how you set up caddy in https, including QTS access (i.e. the Qnap administration GUI), as I always failed to do so.

As it may be OT here, I suggest you to open a post on Qnap community forum.

Thanks!

17

Hi giopas,

At the moment my QNAP+Caddy setup is working 100% correctly due to the reasons that I have specified in my previous post. I am aware that with the Caddy update on QTS that the config file was set back to default, but my issue began before the Caddy update and remained after the update. My issue was caused by a QTS update, seperate from the Caddy update. I appreciate you trying to help though.

I pretty much learned how to setup Caddy in https via this thread with much thanks going to Whitestrake for answering all of my questions. I was a complete noob when I started this thread and he helped me from not having any understanding at all to making it work 100%.

The great thing about Caddy is that it does all the HTTPS stuff by itself and you just need to setup the caddyfile pretty much. As long as your caddyfile is correct and your ports are forwarded, caddy does the rest.

My caddyfile is as follows:


	mynashere.myqnapcloud.com
	root /home/Qhttpd/
	gzip	


	proxy /radarr http://localhost:7878 {				# https://radarr.video/
        	transparent
        	header_upstream X-Forwarded-Host {host}
        	}
	
	proxy /sonarr http://localhost:8989 {			# https://sonarr.tv/
			transparent
			header_upstream X-Forwarded-Host {host}
			}

	proxy /nzbget http://localhost:6789 {			# http://nzbget.net/
			transparent
			header_upstream X-Forwarded-Host {host}
			}

	proxy / localhost:8080 {		# blank
			transparent
			header_upstream X-Forwarded-Host {host}
			}




}

The last part is the part that forwards the main QTS admin home page. Then inside my Caddy.sh file I have this line which runs Caddy with my caddyfile at every boot:

caddy -conf /share/CACHEDEV1_DATA/.qpkg/Caddy/Caddyfile -agree -log $QPKG_ROOT/var/logs/caddy.log -pidfile /var/run/caddy.pid &

Using this configuration along with the QTS settings of:

myQNAPCloud, Auto Router Configuration - Port 80 and Port 443 TCP forwarded

I also have NAS Web & Web Server, Multimedia Station forwarded to 8080 and 86. 8080 is the Nas Web that QTS forwards to, with caddy it will become HTTPS. 86 is my HTTP backup that can work without caddy.
You don’t need to use port 86, but you will need to go into Control Panel, Applications, Web Server & change the port from 80, otherwise caddy cannot use port 80. Also disable HTTPS there and also in System, General Settings. Caddy will manage HTTPS so you need to turn off QNAPs.

DDNS enabled using myqnapcloud url
SSL Certificate disabled and OFF (Caddy will manage this)

These are the settings I am using to successfully use QNAP+Caddy. Note that for other apps like Radarr, Sonarr etc, you need to configure them to have an URL base like /radarr so that it’s 192.0.0:1234/radarr rather than just 192.0.0:1234

I hope this helps.

3 Likes
18

Hi @mupet0000,

Thanks for your explanations!

If you have time, I would ask you advise then to move from HTTP to HTTPS reverse proxy in my configuration.

I try to be as exhaustive as possible:

Current router set up

External port 80 forwarded to internal port 8963 # this is used for HTTP reverse proxy with Caddy
External port 443 forwarded to internal port 443 # this is currently used to access Qnap QTS in HTTPS
IP static
myqnapcloud DDNS enabled

QTS configuration

QTS System port (HTTPS only): 443
Web Server: not enabled
Virtual Host: not enabled
WebDAV: enabled on HTTP port 5000, not enabled on HTTPS port
SSL purchased certificate on qnap.mydomain.com loaded # but I would happily use the one managed by Caddy

Caddy file

(gzipconf) {
	gzip {
		ext *
		level 7
		min_length 1
	}
}

(addheader) {
	header / {
		Strict-Transport-Security "max-age=31536000;"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		X-Frame-Options "DENY"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
	}
}

http://webdav.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.1:5000 {
		keepalive 32
		transparent
		websocket
	}
}

http://sonarr.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.30:8989 {
		keepalive 32
		transparent
		websocket
	}
}

http://transmission.mydomain.com:8963 {
	import gzipconf
	import addheader
	proxy / http://10.0.0.30:49091 {
		keepalive 32
		transparent
		websocket
	}
}

Caddy conf
caddy -conf /share/homes/user/script/caddy.conf -agree -port 20015 -root /home/Qthttpd -log $QPKG_root/var/logs/caddy.log -pidfile /var/run/caddy.pid &

As you can see, I can access to the different services simply using a subdomain, and I can access to QTS with https://qnap.mydomain.com (as any https calls are now taken by QTS).

Starting from this configuration, how could I bring everything to HTTPS?

Thanks for your hints!

Best regards,

giopas

19

Hi,

I am finally trying to do the jump to HTTP but I have a problem.

To summarize: I can now access to QTS via HTTP only on port 8080 and disabled the Webserver. So port 443 and 80 should be free (those are also been forwarded on the router to port 8964 and 8963 for Caddy).

I have then configured Caddy to point to the config file, which looks as follows:

sub.domain.com {
proxy / https://10.1.1.1:8080 {
keepalive 32
transparent
websocket
}
}

But the problem, when I start Caddy is the following:

Activating privacy features… 2019/09/25 21:38:22 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:23 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:24 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:26 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:27 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:28 [INFO] [sub.domain.com] acme: Obtaining bundled SAN certificate

2019/09/25 21:38:30 failed to obtain certificate: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url:

What am I doing wrong/missing?

20

You’ve been rate limited.

The link has more info on rate limits.