Help understanding log files

Help
1

1. The problem I’m having:

No problem really, just need help understanding the errors I’m seeing in my log files.

2. Error messages and/or full log output:

caddy  | {"level":"info","ts":1697837817.767656,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy  | {"level":"warn","ts":1697837817.7705762,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy  | {"level":"info","ts":1697837817.7803497,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy  | {"level":"info","ts":1697837817.7811875,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00035ff80"}
caddy  | {"level":"info","ts":1697837817.7812967,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy  | {"level":"info","ts":1697837817.7813187,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy  | {"level":"info","ts":1697837817.7865067,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy  | {"level":"info","ts":1697837817.7869754,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy  | {"level":"info","ts":1697837817.7895164,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy  | {"level":"info","ts":1697837817.7896059,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy  | {"level":"info","ts":1697837817.7896128,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["dnd.longpants.ca","request.longpants.ca"]}
caddy  | {"level":"info","ts":1697837817.8000422,"logger":"tls","msg":"finished cleaning storage units"}
caddy  | {"level":"info","ts":1697837817.8035674,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy  | {"level":"info","ts":1697837817.8035948,"msg":"serving initial configuration"}
caddy  | {"level":"error","ts":1697837822.3766484,"logger":"http.log.error","msg":"dial tcp 192.168.1.101:30000: connect: connection refused","request":{"remote_ip":"216.245.221.92","remote_port":"38194","client_ip":"216.245.221.92","proto":"HTTP/1.1","method":"GET","host":"dnd.longpants.ca","uri":"/","headers":{"Cache-Control":["no-cache"],"Referer":["https://dnd.longpants.ca"],"Accept-Encoding":[""],"User-Agent":["Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.8"],"Connection":["close"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"dnd.longpants.ca"}},"duration":0.000564683,"status":502,"err_id":"qk09quiiz","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
caddy  | {"level":"error","ts":1697837828.5615337,"logger":"http.log.error","msg":"read tcp 172.23.0.2:57534->192.168.1.101:5055: read: connection reset by peer","request":{"remote_ip":"192.168.1.1","remote_port":"42750","client_ip":"192.168.1.1","proto":"HTTP/1.1","method":"GET","host":"request.longpants.ca","uri":"/","headers":{"Connection":["close"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Uptime-Kuma/1.23.3"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"request.longpants.ca"}},"duration":0.000752057,"status":502,"err_id":"m8w2233bu","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
caddy  | {"level":"error","ts":1697877486.3307326,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"192.168.1.101:30000","duration":0.007199816,"request":{"remote_ip":"36.99.136.129","remote_port":"47347","client_ip":"36.99.136.129","proto":"HTTP/1.1","method":"GET","host":"dnd.longpants.ca","uri":"/scripts/prosemirror.js","headers":{"Referer":["https://dnd.longpants.ca/join"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"],"Accept":["*/*"],"X-Forwarded-For":["36.99.136.129"],"X-Forwarded-Proto":["https"],"Accept-Language":["en"],"Cookie":[],"X-Forwarded-Host":["dnd.longpants.ca"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"dnd.longpants.ca"}},"error":"writing: write tcp 172.23.0.2:443->36.99.136.129:47347: write: broken pipe"}
caddy  | {"level":"error","ts":1697877486.3740113,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"192.168.1.101:30000","duration":0.027851921,"request":{"remote_ip":"36.99.136.129","remote_port":"47345","client_ip":"36.99.136.129","proto":"HTTP/1.1","method":"GET","host":"dnd.longpants.ca","uri":"/scripts/commons.js","headers":{"Cookie":[],"Accept-Encoding":["gzip"],"X-Forwarded-Host":["dnd.longpants.ca"],"Accept-Language":["en"],"Referer":["https://dnd.longpants.ca/join"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"],"X-Forwarded-For":["36.99.136.129"],"X-Forwarded-Proto":["https"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"dnd.longpants.ca"}},"error":"writing: write tcp 172.23.0.2:443->36.99.136.129:47345: write: connection reset by peer"}
caddy  | {"level":"error","ts":1697877486.380231,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"192.168.1.101:30000","duration":0.010220778,"request":{"remote_ip":"36.99.136.129","remote_port":"47348","client_ip":"36.99.136.129","proto":"HTTP/1.1","method":"GET","host":"dnd.longpants.ca","uri":"/scripts/foundry.js","headers":{"Accept-Language":["en"],"X-Forwarded-For":["36.99.136.129"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["dnd.longpants.ca"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"],"Accept":["*/*"],"Cookie":[],"Referer":["https://dnd.longpants.ca/join"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"dnd.longpants.ca"}},"error":"writing: write tcp 172.23.0.2:443->36.99.136.129:47348: write: broken pipe"}
caddy  | {"level":"error","ts":1697877486.418957,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"192.168.1.101:30000","duration":0.010701425,"request":{"remote_ip":"36.99.136.128","remote_port":"59315","client_ip":"36.99.136.128","proto":"HTTP/1.1","method":"GET","host":"dnd.longpants.ca","uri":"/scripts/pixi.min.js","headers":{"Accept-Language":["en"],"Referer":["https://dnd.longpants.ca/join"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"],"Accept":["*/*"],"X-Forwarded-Host":["dnd.longpants.ca"],"Cookie":[],"X-Forwarded-For":["36.99.136.128"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"dnd.longpants.ca"}},"error":"writing: write tcp 172.23.0.2:443->36.99.136.128:59315: write: broken pipe"}

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

Using docker compose, and this official container: Docker

a. System environment:

Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-87-generic x86_64)

b. Command:

sudo docker compose up -d

c. Service/unit/compose file:

  caddy:
    image: caddy:latest
    container_name: caddy
    restart: unless-stopped
    cap_add:
      - NET_ADMIN
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - /home/plexmini/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/plexmini/docker/caddy/caddy_data:/data
      - /home/plexmini/docker/caddy/caddy_config:/config

volumes:
  caddy_data:
    external: true
  caddy_config:

d. My complete Caddy config:

request.longpants.ca {
tls myemail@gmail.com
reverse_proxy 192.168.1.101:5055
}
dnd.longpants.ca {
tls myemail@gmail.com
reverse_proxy 192.168.1.101:30000
}
1 Like
2

When running in Docker, 192.168.x.x may not reach what you expect it to, because containers are in a virtual Docker network.

What machine is at 192.168.1.101?

1 Like
3

The reverse proxies are working. Caddy is serving both sites/services just fine. I was just looking for help interpreting the errors I’m seeing in the log files.

I see unknown IP addresses making requests, with errors like “broken pipe, connection reset, aborting with incomplete response”

To answer your question though, 192.168.1.101 is the host server that caddy (and the services) is running on, via Docker.

4

Are you sure? Because Caddy is saying it couldn’t connect. Maybe you didn’t reach Caddy at all. Make a request with curl -v, you should see Server: Caddy if it actually hits Caddy.

You should probably proxy to host.docker.internal instead then, which is a special DNS name that points to the Docker host.

You might need to add this to your config to make it work depending on your Docker version (under the caddy service):

    extra_hosts:
      - host.docker.internal:host-gateway

(Saw you edited to add this after). Ah, then that’s probably just some bots/crawlers hitting your server and closing the connection before Caddy wrote the full response (especially for those aborting with incomplete response logs). That’s “normal” as long as clients you don’t care about are doing weird things.

1 Like
5

curl -v shows both sites being served by caddy with no obvious (to me) errors. I guess that connection refused entry in the logs is old and to be ignored. I was just curious about the activity from the extra IP’s.

Considering my Caddyfile is very basic with just 2 simple reverse proxies, is there anything I need to do on Caddys end to strengthen my security or are these bots/crawlers active no matter what?

I’ll investigate using the host.docker.internal commands, but it’s working as is right now so I’m tempted to just leave it.

6

No, the bots are harmless. It’s just noise in your logs.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.