Let's Encrypt Not Working for me in V2

osuhickeys · May 17, 2020, 1:01pm

1. Caddy version (`caddy version`):

latest official Caddy v2 Container

2. How I run Caddy:

Docker start caddy

a. System environment:

Microsoft Windows [Version 10.0.18363.836]
Docker version 19.03.8, build afacb8b

b. Command:

N/A

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

3. The problem I’m having:

Let’s encrypt emailed ma saying certificates are expiring. Sites are not available external to my network any longer.

I recently upgraded from V1 to V2. I copied the certs from data\caddy\certificates V1 to the same folder in v2. It looks like they have not been updated since I switched to V2. I tried deleting the files and sites are strill nto accessible externally.

4. Error messages and/or full log output:

5. What I already tried:

Reviewed documentation and searched the forum, but I am not seeing anything in the logs or my searches to solve the problem and I am not sure what the problem actually is.

6. Links to relevant resources:

N/A

francislavoie · May 17, 2020, 3:16pm

For next time, it’s easier to follow and comment on your Caddyfile if you put it in the forums. You can use ``` on lines before and after your config to give it code formatting.

As a side note, you can use the caddy fmt command to clean up the indentation of your Caddyfile, it’ll make it much easier to read. Actually, I’ll do that for you right now:

################################################################################################
# Global Options Block																		   #
################################################################################################
#{
#	acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
#	tls user@domain.net        # Email for Let's Encrypt Verification
#}
################################################################################################
# Airsonic Admin subdomain code block example 			   https://github.com/airsonic/airsonic#
################################################################################################
https://jukebox.domain.net/ http://local.airsonic/ {
	encode gzip
	log {
		output file /logs/airsonic.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:4040 {
		header_up Host {http.reverse_proxy.upstream.hostport}
	}
}
##############################################################################################
# Calibre subdomain code block example 							   https://calibre-ebook.com/#
##############################################################################################
library1.domain.net http://local.calibre {
	encode gzip
	log {
		output file /logs/calibre.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:7070
}
##############################################################################################
# Calibre-gui subdomain code block example	    https://github.com/JiriS97/docker-calibre-gui#
##############################################################################################
library-gui.domain.net http://local.calibre-gui {
	encode gzip
	log {
		output file /logs/calibre-gui.log
		format single_field common_log
	}
	#basicauth {
	#	domain JDJhJDEwJEUzN0NxUFBzbnA4dEZHQUh0TmpHV3VnZVgzSlZtVXU1TU9lMkR1dWJWNEdmUlFQTU92cHJD
	#}
	reverse_proxy 192.168.1.103:5800
}
##############################################################################################
# Calibre-web subdomain code block example			  https://github.com/janeczku/calibre-web#
##############################################################################################
library2.domain.net http://local.calibre-web {
	encode gzip
	log {
		output file /logs/calibre-web.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8083
}
##############################################################################################
# Deluge Admin subdomain code block example						   http://deluge-torrent.org/#
##############################################################################################
flood.domain.net http://local.deluge {
	encode gzip
	log {
		output file /logs/deluge.log
		format single_field common_log
	}
	reverse_proxy http://192.168.1.103:8112
}
##############################################################################################
# FireDaemon Fusion subdomain code block example											 #
##############################################################################################
services.domain.net http://local.fusion {
	encode gzip
	log {
		output file /logs/fusion.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:20604
}
##############################################################################################
# Grafana subdomain code block example						    	   	 https://grafana.com/#
##############################################################################################
performance.domain.net http://local.performance {
	encode gzip
	log {
		output file /logs/grafana.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:3000
}
##############################################################################################
# InfluxDB subdomain code block example														 #
##############################################################################################
metrics.domain.net {
	encode gzip
	log {
		output file /logs/influxdb.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8086
}
##############################################################################################
# Jackett subdomain code block example					   https://github.com/Jackett/Jackett#
##############################################################################################
tsearches.domain.net http://local.jackett {
	encode gzip
	log {
		output file /logs/jackett.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:9117
}
##############################################################################################
# LazyLibrarian subdomain code block example		https://github.com/DobyTang/LazyLibrarian#
##############################################################################################
books.domain.net http://local.lazylibrarian {
	encode gzip
	log {
		output file /logs/lazylibrarian.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:5299 {
		header_up Host {http.reverse_proxy.upstream.hostport}
	}
}
##############################################################################################
# Lidarr subdomain code block example							https://https://lidarr.audio/#
##############################################################################################
music2.domain.net http://local.lidarr {
	encode gzip
	log {
		output file /logs/lidarr.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8686
}
##############################################################################################
# Logarr subdomain code block example					   https://github.com/Monitorr/logarr#
##############################################################################################
logs.domain.net http://local.logarr {
	encode gzip
	log {
		output file /logs/logarr.log
		format single_field common_log
	}
	basicauth {
		domain JDJhJDEwJEUzN0NxUFBzbnA4dEZHQUh0TmpHV3VnZVgzSlZtVXU1TU9lMkR1dWJWNEdmUlFQTU92cHJD
	}
	reverse_proxy 192.168.1.103:8081
}
##############################################################################################
# Monitorr subdomain code block example					https://github.com/Monitorr/Monitorr/#
##############################################################################################
monitor.domain.net http://local.monitorr {
	encode gzip
	log {
		output file /logs/monitorr.log
		format single_field common_log
	}
	basicauth {
		domain JDJhJDEwJEUzN0NxUFBzbnA4dEZHQUh0TmpHV3VnZVgzSlZtVXU1TU9lMkR1dWJWNEdmUlFQTU92cHJD
	}
	reverse_proxy 192.168.1.103:8084
}
##############################################################################################
# Nzbhydra2 subdomain code block example			   https://github.com/theotherp/nzbhydra2#
##############################################################################################
asearches.domain.net http://local.nzbhydra2 {
	encode gzip
	log {
		output file /logs/nybhydra2.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:5076
}
##############################################################################################
# Ombi subdomain code block example							 https://github.com/tidusjar/Ombi#
##############################################################################################
requests.domain.net http://local.ombi {
	encode gzip
	log {
		output file /logs/ombi.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:3579
}
##############################################################################################
# Organizr subdomain code block example					  https://github.com/causefx/Organizr#
##############################################################################################
portal.domain.net http://local.organizr {
	encode gzip
	log {
		output file /logs/organizr.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8082
}
##############################################################################################
# Plex subdomain code block																	 #
##############################################################################################
htpc.domain.net http://local.plex {
	encode gzip
	log {
		output file /logs/plex.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:32400
}
##############################################################################################
# Portainer subdomain code block example			   https://github.com/portainer/portainer#
##############################################################################################
dockdash.domain.net http://local.portainer {
	encode gzip
	log {
		output file /logs/portainer.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:9000
}
##############################################################################################
# Radarr subdomain code block example						 https://github.com/Radarr/Radarr#
##############################################################################################
movies.domain.net http://local.radarr {
	encode gzip
	log {
		output file /logs/radarr.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:7878
}
##############################################################################################
# Sabnzbd subdomain code block example									 https://sabnzbd.org/#
##############################################################################################
downloads.domain.net http://local.sabnzbd {
	encode gzip
	log {
		output file /logs/sabnzdb.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8080
}
##############################################################################################
# Sonarr subdomain code block example									   https://sonarr.tv/#
##############################################################################################
tv.domain.net http://local.sonarr {
	encode gzip
	log {
		output file /logs/sonarr.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8989
}
##############################################################################################
# Tautulli Admin subdomain code block example			 https://github.com/Tautulli/Tautulli#
##############################################################################################
htpcstats.domain.net http://local.tautulli {
	encode gzip
	log {
		output file /logs/tautulli.log
		format single_field common_log
	}
	reverse_proxy 192.168.1.103:8181
}

So now for some feedback:

https://jukebox.domain.net/ http://local.airsonic/ {

You’ll need to remove the trailing / here, this’ll make only requests to the root of that site match. Caddy v2’s path matching is exact match, meaning that you would need a * at the end of those to match all paths… or just omit the / altogether.

The rest of your Caddyfile looks good!

Now looking at the logs…

2020/05/17 12:45:31 [ERROR] error: one or more domains had a problem:
[movies.domain.net] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from http://movies.domain.net/.well-known/acme-challenge/gKaKQf1UYHeLuV7gzBLUY1Pt3ILs-GNGGGTcm5K8hTw [71.91.198.200]: "<html>\r\n    <head><title>Document Error: Not Found</title></head>\r\n    <body>\r\n        <h2>Access Error: Not Found</h2>\r\n       ", url:
 (challenge=http-01 remaining=[tls-alpn-01])

This is the error that’s causing certificate issuance to fail. Essentially this is saying that Let’s Encrypt tried to make a request to get the ACME challenge from your domain, but instead it got a 404 from whatever server it was able to contact.

Do you have another serving on those domains already? Is your DNS pointing to an old server? You’ll need to make sure the routing is correct.

osuhickeys · May 17, 2020, 3:40pm

The domains are correctly registered with the right IP with my DNS provider. I do have things exposed externally on port 4443 instead of 443. Wondering if that is my issue. I got a new router and it listens on 443. Cannot figure out how to get it to not listen on that port.

francislavoie · May 17, 2020, 4:02pm

Let’s Encrypt will try to connect on port 80 for the HTTP challenge, so at least that needs to be reachable.

Your router probably shouldn’t be listening on port 80/443 publicly, that seems like a bad idea.

I think you’ll need to dig into the router manual or something!

osuhickeys · May 18, 2020, 11:43am

Agreed on not exposing the router to the Internet. I do not have remote management enabled. It appears to be uPnP. I have one device inside my network registering port 443 with uPnP and the router then does not let me enable port forwarding for port 443 saying it is in use. I have a ticket open with the manufacturer.

For now, I have disabled uPnP and put the Caddy host in the DMZ to ensure network connectivity is not the issue.

I also removed the Caddy V1 files I copied over. They seem to have had no effect. Looks like I may need to wait a month for the certs to expire before V2 can renew them and things might start working again then?

It has been very painful moving from V1 to V2.

Updated Log File
{“level”:“info”,“ts”:1589801726.9808369,“msg”:“using provided configuration”,“config_file”:"/etc/caddy/Caddyfile",“config_adapter”:“caddyfile”}
{“level”:“info”,“ts”:1589801727.0023038,“logger”:“admin”,“msg”:“admin endpoint started”,“address”:“tcp/localhost:2019”,“enforce_origin”:false,“origins”:[“localhost:2019”,"[::1]:2019",“127.0.0.1:2019”]}
2020/05/18 11:35:27 [INFO][cache:0xc00073e0f0] Started certificate maintenance routine
{“level”:“info”,“ts”:1589801727.0151393,“logger”:“http”,“msg”:“server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS”,“server_name”:“srv0”,“https_port”:443}
{“level”:“info”,“ts”:1589801727.0152335,“logger”:“http”,“msg”:“enabling automatic HTTP->HTTPS redirects”,“server_name”:“srv0”}
{“level”:“info”,“ts”:1589801727.015375,“logger”:“http”,“msg”:“server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server”,“server_name”:“srv1”,“http_port”:80}
{“level”:“warn”,“ts”:1589801727.0154614,“logger”:“http”,“msg”:“user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects”,“server_name”:“srv1”,“interface”:“tcp/:80”}
{“level”:“info”,“ts”:1589801727.239435,“logger”:“tls”,“msg”:“cleaned up storage units”}
{“level”:“info”,“ts”:1589801727.2395964,“logger”:“http”,“msg”:“enabling automatic TLS certificate management”,“domains”:[“performance.domain.net”,“downloads.domain.net”,“movies.domain.net”,“music2.domain.net”,“tsearches.domain.net”,“monitor.domain.net”,“asearches.domain.net”,“library1.domain.net”,“dockdash.domain.net”,“flood.domain.net”,“htpc.domain.net”,“htpcstats.domain.net”,“library2.domain.net”,“portal.domain.net”,“library-gui.domain.net”,“services.domain.net”,“metrics.domain.net”,“books.domain.net”,“logs.domain.net”,“requests.domain.net”,“jukebox.domain.net”,“tv.domain.net”]}
2020/05/18 11:35:27 [WARNING] Stapling OCSP: no OCSP stapling for [library-gui.domain.net]: parsing OCSP response: ocsp: error from server: unauthorized
2020/05/18 11:35:27 [INFO][library-gui.domain.net] Renew certificate; acquiring lock…
2020/05/18 11:35:27 [ERROR] library-gui.domain.net: renewing certificate: invalid character ‘\x00’ looking for beginning of value
{“level”:“info”,“ts”:1589801727.4025934,“msg”:“autosaved config”,“file”:"/config/caddy/autosave.json"}
{“level”:“info”,“ts”:1589801727.402619,“msg”:“serving initial configuration”}
2020/05/18 11:35:27 [INFO][metrics.domain.net] Renew certificate; acquiring lock…
2020/05/18 11:35:27 [INFO][FileStorage:/data/caddy] Lock for ‘cert_acme_metrics.domain.net_acme-v02.api.letsencrypt.org-directory’ is stale (created: 2020-05-18 11:34:13.219026337 +0000 UTC, last update: 2020-05-18 11:35:08.257035786 +0000 UTC); removing then retrying: /data/caddy/locks/cert_acme_metrics.domain.net_acme-v02.api.letsencrypt.org-directory.lock
2020/05/18 11:35:27 [INFO][metrics.domain.net] Renew: Lock acquired; proceeding…
2020/05/18 11:35:27 [INFO][metrics.domain.net] Renew: 453h2m51.577315295s remaining
2020/05/18 11:35:27 [INFO][metrics.domain.net] Waiting on rate limiter…
2020/05/18 11:35:27 [INFO][metrics.domain.net] Done waiting
2020/05/18 11:35:27 [INFO] [metrics.domain.net] acme: Obtaining bundled SAN certificate given a CSR
2020/05/18 11:35:28 [INFO] [metrics.domain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4658680243
2020/05/18 11:35:28 [INFO] [metrics.domain.net] acme: use tls-alpn-01 solver
2020/05/18 11:35:28 [INFO] [metrics.domain.net] acme: Trying to solve TLS-ALPN-01
2020/05/18 11:35:28 http: TLS handshake error from 127.0.0.1:56820: EOF
2020/05/18 11:35:35 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4658680243
2020/05/18 11:35:35 [ERROR] error: one or more domains had a problem:
[metrics.domain.net] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for metrics.domain.net - check that a DNS record exists for this domain, url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/05/18 11:35:35 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4658680243
2020/05/18 11:35:37 [INFO] [metrics.domain.net] acme: Obtaining bundled SAN certificate given a CSR
2020/05/18 11:35:38 [ERROR] acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: (challenge=http-01 remaining=[])
2020/05/18 11:35:40 [ERROR] attempt 1: [metrics.domain.net] Renew: [metrics.domain.net] acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Rate Limits - Let's Encrypt, url: - retrying in 1m0s (12.745545435s/720h0m0s elapsed)…

Whitestrake · May 19, 2020, 6:32am

Whatever domain this actually is (I assume you don’t own domain.net), LetsEncrypt tried to look it up and failed to find a registered domain (it may be misspelled in your Caddyfile?). This error will persist until the domain is actually registered or you stop Caddy from requesting certificates for a non-existent domain.

This error happens when you hit LetsEncrypt’s abuse prevention rate limits. The linked rate limits URL has information on how long this error will stay in effect. Likely the NXDOMAIN errors above have caused the failure count to reach the rate limit.

osuhickeys · May 19, 2020, 5:28pm

LOL. You are correct I do not own domain.net, but I wish I did.

There was no A record in my domain registrar for the metrics.domain.net. I have now added it and that error has disappeared. Still don’t seem to be getting any certs updated and working.

Newest Logs
https://pastebin.com/BEbQWz6u

matt · May 19, 2020, 6:33pm

Ah, that’s why we ask for the full and unredacted logs and config, because using filler/fake information won’t help us help you.

osuhickeys · May 27, 2020, 3:41pm

I see certificates that seem to be renewing now that I left caddy shit off for a week, but I am only able to hit one site which is “library1.domain.net”. None of the other are working.

Begin logs:

{"level":"info","ts":1590579496.3518152,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
2020/05/27 11:38:16 [INFO][cache:0xc0007bc190] Started certificate maintenance routine
{"level":"info","ts":1590579496.3611498,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1590579496.3611996,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1590579496.3612819,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"warn","ts":1590579496.3613198,"logger":"http","msg":"user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects","server_name":"srv1","interface":"tcp/:80"}
{"level":"info","ts":1590579496.899302,"logger":"tls","msg":"cleaned up storage units"}
{"level":"info","ts":1590579496.899417,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["monitor.domain.net","asearches.domain.net","htpcstats.domain.net","services.domain.net","movies.domain.net","music2.domain.net","downloads.domain.net","tsearches.domain.net","metrics.domain.net","tv.domain.net","library1.domain.net","books.domain.net","logs.domain.net","performance.domain.net","library2.domain.net","flood.domain.net","library-gui.domain.net","portal.domain.net","htpc.domain.net","jukebox.domain.net","dockdash.domain.net","requests.domain.net"]}
2020/05/27 11:38:17 [INFO][htpcstats.domain.net] Renew certificate; acquiring lock...
2020/05/27 11:38:17 [INFO][htpcstats.domain.net] Renew: Lock acquired; proceeding...
2020/05/27 11:38:17 [INFO][htpcstats.domain.net] Renew: 615h35m12.823512499s remaining
2020/05/27 11:38:17 [INFO][music2.domain.net] Renew certificate; acquiring lock...
2020/05/27 11:38:17 [INFO][music2.domain.net] Renew: Lock acquired; proceeding...
2020/05/27 11:38:17 [INFO][music2.domain.net] Renew: 615h35m3.62101235s remaining
2020/05/27 11:38:17 [INFO][htpcstats.domain.net] Waiting on rate limiter...
2020/05/27 11:38:17 [INFO][htpcstats.domain.net] Done waiting
2020/05/27 11:38:17 [INFO] [htpcstats.domain.net] acme: Obtaining bundled SAN certificate given a CSR
2020/05/27 11:38:17 [INFO] [music2.domain.net] acme: Obtaining bundled SAN certificate given a CSR
2020/05/27 11:38:17 [INFO][music2.domain.net] Waiting on rate limiter...
2020/05/27 11:38:17 [INFO][music2.domain.net] Done waiting
2020/05/27 11:38:17 [INFO] [music2.domain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4843671383
2020/05/27 11:38:17 [INFO] [music2.domain.net] acme: Could not find solver for: tls-alpn-01
2020/05/27 11:38:17 [INFO] [music2.domain.net] acme: use http-01 solver
2020/05/27 11:38:17 [INFO] [music2.domain.net] acme: Trying to solve HTTP-01
2020/05/27 11:38:17 [INFO] [htpcstats.domain.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4843671386
2020/05/27 11:38:17 [INFO] [htpcstats.domain.net] acme: Could not find solver for: tls-alpn-01
2020/05/27 11:38:17 [INFO] [htpcstats.domain.net] acme: use http-01 solver
2020/05/27 11:38:17 [INFO] [htpcstats.domain.net] acme: Trying to solve HTTP-01
2020/05/27 11:38:18 [INFO][htpcstats.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.2293239,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44452","host":"htpcstats.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc HTTP/1.1\" 200 87","duration":0.0016447,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][music2.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.2681785,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44456","host":"music2.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY HTTP/1.1\" 200 87","duration":0.002766199,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][music2.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.3039744,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44460","host":"music2.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY HTTP/1.1\" 200 87","duration":0.001189299,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][htpcstats.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.3754811,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44464","host":"htpcstats.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc HTTP/1.1\" 200 87","duration":0.001371699,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
{"level":"info","ts":1590579498.4187613,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1590579498.4188256,"msg":"serving initial configuration"}
2020/05/27 11:38:18 [INFO][music2.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.4329178,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44468","host":"music2.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY HTTP/1.1\" 200 87","duration":0.0007275,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][htpcstats.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.4357517,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44472","host":"htpcstats.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc HTTP/1.1\" 200 87","duration":0.000603599,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][music2.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.5561109,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44476","host":"music2.domain.net","headers":{"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/dAdRKK91tKWq7_fOBVjnfXPbbQ_9L2yVWzBo2rH5LqY HTTP/1.1\" 200 87","duration":0.0009391,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:18 [INFO][htpcstats.domain.net] Served key authentication (HTTP challenge)
{"level":"info","ts":1590579498.5688448,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44480","host":"htpcstats.domain.net","headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"],"Accept":["*/*"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:38:18 +0000] \"GET /.well-known/acme-challenge/U0uQzrPfFfed9p_RlkDsixfOLUZouB1yL9OLThRx3mc HTTP/1.1\" 200 87","duration":0.0007472,"size":87,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain"]}}
2020/05/27 11:38:20 [INFO] [htpcstats.domain.net] The server validated our request
2020/05/27 11:38:20 [INFO] [htpcstats.domain.net] acme: Validations succeeded; requesting certificates
2020/05/27 11:38:21 [INFO] [htpcstats.domain.net] Server responded with a certificate.
2020/05/27 11:38:21 [INFO][htpcstats.domain.net] Certificate renewed successfully
2020/05/27 11:38:21 [INFO][htpcstats.domain.net] Renew: Releasing lock
2020/05/27 11:38:21 [INFO] Reloading managed certificate for [htpcstats.domain.net]
2020/05/27 11:38:21 [INFO] Replaced certificate in cache for [htpcstats.domain.net] (new expiration date: 2020-08-25 10:38:22)
2020/05/27 11:38:22 [INFO] [music2.domain.net] The server validated our request
2020/05/27 11:38:22 [INFO] [music2.domain.net] acme: Validations succeeded; requesting certificates
2020/05/27 11:38:22 [INFO] [music2.domain.net] Server responded with a certificate.
2020/05/27 11:38:22 [INFO][music2.domain.net] Certificate renewed successfully
2020/05/27 11:38:22 [INFO][music2.domain.net] Renew: Releasing lock
2020/05/27 11:38:22 [INFO] Reloading managed certificate for [music2.domain.net]
2020/05/27 11:38:22 [INFO] Replaced certificate in cache for [music2.domain.net] (new expiration date: 2020-08-25 10:38:24)
{"level":"info","ts":1590579610.4808161,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:44984","host":"music2.domain.net","headers":{"Accept-Encoding":["gzip, deflate"],"Accept":["*/*"],"Connection":["keep-alive"],"User-Agent":["python-requests/2.23.0"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:40:10 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.001163801,"size":0,"status":308,"resp_headers":{"Location":["https://music2.domain.net/"],"Connection":["close"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1590579622.5801861,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:45014","host":"htpcstats.domain.net","headers":{"Pragma":["no-cache"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.71 Safari/537.36"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"Referer":["http://www.bing.com/search?q=amazon"],"Cache-Control":["no-cache"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate"],"Connection":["keep-alive"]}},"common_log":"172.17.0.1 - - [27/May/2020:11:40:22 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0000265,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://htpcstats.domain.net/"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590584866.2924383,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/en/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:58806","host":"jukebox.domain.net","headers":{"Accept":["*/*"],"Referer":["http://jukebox.domain.net/en/wp-login.php"],"Range":["bytes=0-1048576"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0"]}},"common_log":"172.17.0.1 - - [27/May/2020:13:07:46 +0000] \"GET /en/wp-login.php HTTP/1.1\" 308 0","duration":0.0000354,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://jukebox.domain.net/en/wp-login.php"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590585677.9585192,"logger":"http.log.access","msg":"handled request","request":{"method":"POST","uri":"/cgi-bin/mainfunction.cgi","proto":"HTTP/1.1","remote_addr":"172.17.0.1:60992","host":"127.0.0.1","headers":{"Content-Length":["89"],"Accept-Encoding":["gzip, deflate"],"Accept-Language":["en-US,en;q=0.9"],"Connection":["close"],"User-Agent":["XTC"]}},"common_log":"172.17.0.1 - - [27/May/2020:13:21:17 +0000] \"POST /cgi-bin/mainfunction.cgi HTTP/1.1\" 308 0","duration":0.0000633,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://127.0.0.1/cgi-bin/mainfunction.cgi"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590585818.5548,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/interface.php?m=downloader&client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025","proto":"HTTP/1.1","remote_addr":"172.17.0.1:32950","host":"platform.wondershare.com","headers":{"Accept":["*/*"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/4.0 (compatible; MSIE 6.00)"],"Connection":["Keep-Alive"],"Accept-Encoding":["gzip,deflate"]}},"common_log":"172.17.0.1 - - [27/May/2020:13:23:38 +0000] \"GET /interface.php?m=downloader&client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025 HTTP/1.1\" 308 0","duration":0.000071,"size":0,"status":308,"resp_headers":{"Location":["https://platform.wondershare.com/interface.php?m=downloader&client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025"],"Connection":["close"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1590585818.5624125,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/rest/v2/message/getlist/?client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025&user_type=2,2&install_times=20200409,20200409","proto":"HTTP/1.1","remote_addr":"172.17.0.1:32954","host":"platform.wondershare.com","headers":{"Accept":["*/*"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"User-Agent":["Mozilla/4.0 (compatible; MSIE 6.00)"],"Connection":["Keep-Alive"],"Accept-Encoding":["gzip,deflate"]}},"common_log":"172.17.0.1 - - [27/May/2020:13:23:38 +0000] \"GET /rest/v2/message/getlist/?client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025&user_type=2,2&install_times=20200409,20200409 HTTP/1.1\" 308 0","duration":0.0000572,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://platform.wondershare.com/rest/v2/message/getlist/?client_sign={78584E81-DF7C-4D27-AF7A-52B84DCA0C91}&product_ids=846,1025&user_type=2,2&install_times=20200409,20200409"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590586731.3015347,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/news/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:35384","host":"jukebox.domain.net","headers":{"Range":["bytes=0-1048576"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0"],"Accept":["*/*"],"Referer":["http://jukebox.domain.net/news/wp-login.php"]}},"common_log":"172.17.0.1 - - [27/May/2020:13:38:51 +0000] \"GET /news/wp-login.php HTTP/1.1\" 308 0","duration":0.0001329,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://jukebox.domain.net/news/wp-login.php"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590588773.0332494,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:40622","host":"71.91.198.200:80","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0"],"Accept":["*/*"],"Connection":["keep-alive"]}},"common_log":"172.17.0.1 - - [27/May/2020:14:12:53 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0000918,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://71.91.198.200/"],"Content-Type":[]}}
{"level":"info","ts":1590590463.332659,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:45036","host":"performance.domain.net","headers":{"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:14:41:03 +0000] \"GET /wp-login.php HTTP/1.1\" 308 0","duration":0.0000328,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://performance.domain.net/wp-login.php"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590590511.3698792,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:45100","host":"71.91.198.200","headers":{"User-Agent":["Cloud mapping experiment. Contact research@pdrlabs.net"],"Accept":["*/*"],"Accept-Encoding":["gzip"]}},"common_log":"172.17.0.1 - - [27/May/2020:14:41:51 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0000267,"size":0,"status":308,"resp_headers":{"Location":["https://71.91.198.200/"],"Connection":["close"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1590590526.1672707,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/clientaccesspolicy.xml","proto":"HTTP/1.1","remote_addr":"172.17.0.1:45154","host":"71.91.198.200","headers":{"Content-Length":["0"],"User-Agent":["Cloud mapping experiment. Contact research@pdrlabs.net"],"Accept-Encoding":["identity"]}},"common_log":"172.17.0.1 - - [27/May/2020:14:42:06 +0000] \"GET /clientaccesspolicy.xml HTTP/1.1\" 308 0","duration":0.0000977,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://71.91.198.200/clientaccesspolicy.xml"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590592029.088053,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:49086","host":"asearches.domain.net","headers":{"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:07:09 +0000] \"GET /wp-login.php HTTP/1.1\" 308 0","duration":0.0000278,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://asearches.domain.net/wp-login.php"],"Content-Type":[]}}
{"level":"info","ts":1590592311.919389,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:49822","host":"71.91.198.200:80","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"],"Content-Length":["0"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:11:51 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0000508,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://71.91.198.200/"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590592407.424156,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:49956","host":"requests.domain.net","headers":{"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:13:27 +0000] \"GET /wp-login.php HTTP/1.1\" 308 0","duration":0.0000865,"size":0,"status":308,"resp_headers":{"Location":["https://requests.domain.net/wp-login.php"],"Connection":["close"],"Content-Type":[],"Server":["Caddy"]}}
{"level":"info","ts":1590592525.9088075,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:50502","host":"71.91.198.200:80","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7"],"Content-Length":["0"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:15:25 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0000285,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://71.91.198.200/"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590592666.8669362,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/wp-login.php","proto":"HTTP/1.1","remote_addr":"172.17.0.1:50686","host":"htpcstats.domain.net","headers":{"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"],"Accept-Encoding":["gzip"],"Connection":["close"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:17:46 +0000] \"GET /wp-login.php HTTP/1.1\" 308 0","duration":0.0000913,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://htpcstats.domain.net/wp-login.php"],"Connection":["close"],"Content-Type":[]}}
{"level":"info","ts":1590593110.3769667,"logger":"http.log.access","msg":"handled request","request":{"method":"GET","uri":"/","proto":"HTTP/1.1","remote_addr":"172.17.0.1:51978","host":"71.91.198.200","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"]}},"common_log":"172.17.0.1 - - [27/May/2020:15:25:10 +0000] \"GET / HTTP/1.1\" 308 0","duration":0.0001088,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Location":["https://71.91.198.200/"],"Connection":["close"],"Content-Type":[]}}```

inxsible · May 27, 2020, 3:59pm

@osuhickeys, you really need to get into the habit of using code blocks for logs and code snippets. I am trying to fix some reverse proxy issues of my own, and just browsing for similar posts, but good grief, it’s difficult to follow your posts due to lack of formatting.

osuhickeys · May 27, 2020, 4:53pm

Sorry about that. I gues I am not sure what you mean. Are you talking about using the Blockquote icon for logs and code snippets?

The log format is what caddy is spitting out by default. Not quite sure how to get caddy to format it differently to make it more readable.

francislavoie · May 27, 2020, 5:06pm

I explained how to do it way up here ^

osuhickeys · May 27, 2020, 10:16pm

I was able to update my last post with logs. Definitely much easier to read. Older posts won’t let me edit them.

I also tried to use the caddy fmt command. It seems to work fine to the console, but the --overwrite option to write back to the file does not appear to be working at least on Windows 10.

I am using caddy version:
v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

I updated to the latest version of Docker today, rebooted the host, and removed the Caddy v2 container and re-added it, and everything seems to be working now. Not sure what the root cause was. It was added via a script so nothing changed. Maybe the image had become corrupted somehow or there was a bud in Docker I stumbled across.

REALLY appreciate everyone’s help on this!

system · June 16, 2020, 1:01pm

This topic was automatically closed after 30 days. New replies are no longer allowed.