Site stops responding when busy

1. Caddy version (caddy version):

v2.4.5

2. How I run Caddy:

Through docker-compose

a. System environment:

VPS:

Ubuntu 18.04 Server

Docker (looks like this could do with an update…):

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

b. Command:

sudo docker-compose -f docker-compose-master.yml up -d

c. Service/unit/compose file:

docker-compose-master.yml:

version: "3.1"
services:

  caddy:
    image: caddy
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "24m"
    container_name: gb-caddy
    working_dir: /application
    environment:
      - HOST=getblogged.net
    volumes:
      - .:/application
      - ./phpdocker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - ./phpdocker/caddy/data:/data
      - ./phpdocker/caddy/config:/config
      - /var/log/caddy/:/var/log/caddy/
    depends_on:
      - php-fpm
      - php-fpm-ws
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped

  mysql:
    image: mysql:5.6
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "24m"
    container_name: gb-mysql
    working_dir: /application
    volumes:
      - ./phpdocker/db-import:/docker-entrypoint-initdb.d
      - ./phpdocker/db:/var/lib/mysql
      - .:/application
    environment:
      - MYSQL_ROOT_PASSWORD=<removed>
      - MYSQL_DATABASE=<removed>
      - MYSQL_USER=<removed>
      - MYSQL_PASSWORD=<removed>
    ports:
      - "8002:3306"

  php-fpm:
    build: phpdocker/php-fpm
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "24m"
    container_name: gb-php-fpm
    working_dir: /application
    volumes:
      - .:/application
      - ./phpdocker/php-fpm/php-ini-overrides.ini:/etc/php/7.3/fpm/conf.d/99-overrides.ini
    environment:
      - DOCKER_MYSQL_HOST=gb-mysql

  php-fpm-ws:
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "24m"
    build: phpdocker/php-fpm
    container_name: gb-php-fpm-ws
    working_dir: /application
    restart: always
    volumes:
      - .:/application
      - ./phpdocker/php-fpm/php-ini-overrides.ini:/etc/php/7.3/fpm/conf.d/99-overrides.ini
    environment:
      - DOCKER_MYSQL_HOST=gb-mysql
    command: php public_html/getblogged/app/chat/php-socket.php

  php-fpm-setup:
    build: phpdocker/php-fpm-setup
    logging:
      driver: "json-file"
      options:
        max-file: "5"
        max-size: "24m"
    container_name: gb-php-fpm-setup
    working_dir: /application
    volumes:
      - .:/application
      - ./phpdocker/php-fpm/php-ini-overrides.ini:/etc/php/7.3/fpm/conf.d/99-overrides.ini
    environment:
      - DOCKER_MYSQL_HOST=gb-mysql

d. My complete Caddyfile or JSON config:

Caddyfile:

{
	debug
}

(app) {
  root * /application/public_html/getblogged/app
  encode gzip zstd

  #CORS
  header {
    Access-Control-Allow-Origin https://www.getblogged.net
  }

  @ws {
    header Connection *Upgrade*
    header Upgrade websocket
  }
  reverse_proxy @ws php-fpm-ws:8090

  php_fastcgi php-fpm:9000
  file_server

  log {
    output file /var/log/caddy/app.access.log {
      roll_size 100mb
      roll_keep 30
      roll_keep_for 720h
    }
    format json
    level DEBUG
  }
}

(portal) {
  root * /application/public_html/getblogged/portal
  encode gzip zstd

  @ws {
    header Connection *Upgrade*
    header Upgrade websocket
  }
  reverse_proxy @ws php-fpm-ws:8090

  php_fastcgi php-fpm:9000
  file_server

  log {
    output file /var/log/caddy/portal.access.log {
      roll_size 100mb
      roll_keep 30
      roll_keep_for 720h
    }
    format json
    level DEBUG
  }
}

app.{$HOST} {
  #tls internal
  import app
}

portal.{$HOST} {
  #tls internal
  import portal
}

:80 {
  redir https://{host}{uri}
}

:443 {
  tls {
    on_demand
  }
  import app
}

3. The problem I’m having:

Sporadically, seemingly when the site is busy the site will stop responding. The browser just waits for it to load before timing out.

We’ve been having this issue for a few months now where the site will go down between 2 to 10 times a week. We were initially using a LetsEncrypt container with NGINX and switch to Caddy so we could take advantage of the on demand TLS and have been having this issues since.

I’ve tried to capture as much debug output as possible but as this only happens on our production server I am in a rush to get it back up.

Any help would be extremely appreciated. Been pulling my hair out for a few months now on this one!

curl -v output (clientside):

*   Trying 51.68.212.198:443...
* TCP_NODELAY set
* Connected to app.getblogged.net (51.68.212.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.getblogged.net
*  start date: Dec 17 12:31:18 2021 GMT
*  expire date: Mar 17 12:31:17 2022 GMT
*  subjectAltName: host "app.getblogged.net" matched cert's "app.getblogged.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56501e0a2e30)
> GET / HTTP/2
> Host: app.getblogged.net
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!

curl -v output on VPS when down:

root@vps784967:~# curl -v https://app.getblogged.net/
*   Trying 51.68.212.198...
* TCP_NODELAY set
* connect to 51.68.212.198 port 443 failed: Connection refused
* Failed to connect to app.getblogged.net port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to app.getblogged.net port 443: Connection refused

curl -v output on VPS when working fine:

root@vps784967:~# curl -v https://app.getblogged.net/
*   Trying 51.68.212.198...
* TCP_NODELAY set
* Connected to app.getblogged.net (51.68.212.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.getblogged.net
*  start date: Dec 17 12:31:18 2021 GMT
*  expire date: Mar 17 12:31:17 2022 GMT
*  subjectAltName: host "app.getblogged.net" matched cert's "app.getblogged.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x555a6af0e600)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET / HTTP/2
> Host: app.getblogged.net
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 302 
< access-control-allow-origin: https://www.getblogged.net
< cache-control: no-store, no-cache, must-revalidate
< content-type: text/html; charset=UTF-8
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< location: /login
< pragma: no-cache
< server: Caddy
< set-cookie: PHPSESSID=dvql9h5gpmo59493pegk9evrul; path=/; domain=.app.getblogged.net
< status: 302 Found
< content-length: 0
< date: Thu, 10 Feb 2022 17:02:14 GMT
< 
* Connection #0 to host app.getblogged.net left intact

4. Error messages and/or full log output:

We don’t actually get any errors. Here are the last log entries though before it stopped responding:

caddy container log:

{"log":"{\"level\":\"debug\",\"ts\":1644496405.8380249,\"logger\":\"http.handlers.reverse_proxy\",\"msg\":\"upstream roundtrip\",\"upstream\":\"php-fpm-ws:8090\",\"request\":{\"remote_addr\":\"97.70.160.221:50566\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"ws.getblogged.net\",\"uri\":\"/chat/php-socket.php\",\"headers\":{\"Sec-Websocket-Version\":[\"13\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Sec-Websocket-Key\":[\"91piOtRuFiK1mQCwmCfopw==\"],\"Pragma\":[\"no-cache\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Origin\":[\"https://app.getblogged.net\"],\"Cookie\":[\"_fbp=fb.1.1644492997118.1007065267; _ga=GA1.2.1201346913.1644492992; _gid=GA1.2.657030684.1644492992; __hstc=32639585.ac4910a52502a3374020a12569f1a7ab.1643285198506.1643551792117.1643925912344.3; hubspotutk=ac4910a52502a3374020a12569f1a7ab; _hjSessionUser_764671=eyJpZCI6ImUyODgyNzc5LWI2NTctNWFjZS04Y2ZlLTFhNzEyZGVkMWE3ZSIsImNyZWF0ZWQiOjE2NDMyODUxOTczMTUsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22\"],\"X-Forwarded-Proto\":[\"https\"],\"User-Agent\":[\"Mozilla/5.0 (iPhone; CPU iPhone OS 15_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Mobile/15E148 Safari/604.1\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-For\":[\"97.70.160.221\"],\"Accept\":[\"*/*\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"proto_mutual\":true,\"server_name\":\"ws.getblogged.net\"}},\"headers\":{\"Upgrade\":[\"websocket\"],\"Connection\":[\"Upgrade\"],\"Websocket-Origin\":[\"ws.localhost\"],\"Websocket-Location\":[\"wss://ws.localhost/chat/php-socket.php\"],\"Sec-Websocket-Accept\":[\"TicBEHxZ9T7X3tSpTh4F+KpDV84=\"]},\"status\":101}\n","stream":"stderr","time":"2022-02-10T12:33:25.838684474Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.8381214,\"logger\":\"http.handlers.reverse_proxy\",\"msg\":\"upgrading connection\",\"upstream\":\"php-fpm-ws:8090\",\"request\":{\"remote_addr\":\"97.70.160.221:50566\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"ws.getblogged.net\",\"uri\":\"/chat/php-socket.php\",\"headers\":{\"Sec-Websocket-Version\":[\"13\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Sec-Websocket-Key\":[\"91piOtRuFiK1mQCwmCfopw==\"],\"Pragma\":[\"no-cache\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Origin\":[\"https://app.getblogged.net\"],\"Cookie\":[\"_fbp=fb.1.1644492997118.1007065267; _ga=GA1.2.1201346913.1644492992; _gid=GA1.2.657030684.1644492992; __hstc=32639585.ac4910a52502a3374020a12569f1a7ab.1643285198506.1643551792117.1643925912344.3; hubspotutk=ac4910a52502a3374020a12569f1a7ab; _hjSessionUser_764671=eyJpZCI6ImUyODgyNzc5LWI2NTctNWFjZS04Y2ZlLTFhNzEyZGVkMWE3ZSIsImNyZWF0ZWQiOjE2NDMyODUxOTczMTUsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22\"],\"X-Forwarded-Proto\":[\"https\"],\"User-Agent\":[\"Mozilla/5.0 (iPhone; CPU iPhone OS 15_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Mobile/15E148 Safari/604.1\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-For\":[\"97.70.160.221\"],\"Accept\":[\"*/*\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"proto_mutual\":true,\"server_name\":\"ws.getblogged.net\"}}}\n","stream":"stderr","time":"2022-02-10T12:33:25.873362351Z"}

app.access.log

{"level":"info","ts":1644496457.6839786,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"176.24.66.159:56493","proto":"HTTP/2.0","method":"GET","host":"app.getblogged.net","uri":"/res/js/affiliate.js?v=1","headers":{"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Mobile/15E148 Safari/604.1"],"Accept-Language":["en-GB,en;q=0.9"],"Referer":["https://www.getblogged.net/"],"Cookie":["_ga=GA1.2.451233662.1644227918; _gid=GA1.2.2116029319.1644227918; _fbp=fb.1.1644227918287.1124069725; password_cookie_token=87ea99723cb20510881be6957a597d2b; remember_me=1; __hstc=32639585.6fa8383c6b059284785202527ca247c0.1644227919109.1644339483749.1644404403135.5; hubspotutk=6fa8383c6b059284785202527ca247c0; _hjSessionUser_764671=eyJpZCI6IjAyZjcyNTEyLTFjMzItNWYxMS04OGRlLWNmZmU2OGY3NTQxMiIsImNyZWF0ZWQiOjE2NDQyMjc5MTgwMTAsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22; tk_r3d=%22%22; banner-merchandising-oneTime-37=true"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"app.getblogged.net"}},"common_log":"176.24.66.159 - - [10/Feb/2022:12:34:17 +0000] \"GET /res/js/affiliate.js?v=1 HTTP/2.0\" 200 537","user_id":"","duration":0.023586972,"size":537,"status":200,"resp_headers":{"Content-Type":["application/javascript"],"Last-Modified":["Wed, 09 Jun 2021 21:01:34 GMT"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"Server":["Caddy"],"Access-Control-Allow-Origin":["https://www.getblogged.net"],"Etag":["\"qugd2m11f\""]}}
{"level":"info","ts":1644496457.6882086,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"176.24.66.159:56493","proto":"HTTP/2.0","method":"GET","host":"app.getblogged.net","uri":"/res/js/external/register.js?date=2022-02-27b","headers":{"Referer":["https://www.getblogged.net/"],"Cookie":["_ga=GA1.2.451233662.1644227918; _gid=GA1.2.2116029319.1644227918; _fbp=fb.1.1644227918287.1124069725; password_cookie_token=87ea99723cb20510881be6957a597d2b; remember_me=1; __hstc=32639585.6fa8383c6b059284785202527ca247c0.1644227919109.1644339483749.1644404403135.5; hubspotutk=6fa8383c6b059284785202527ca247c0; _hjSessionUser_764671=eyJpZCI6IjAyZjcyNTEyLTFjMzItNWYxMS04OGRlLWNmZmU2OGY3NTQxMiIsImNyZWF0ZWQiOjE2NDQyMjc5MTgwMTAsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22; tk_r3d=%22%22; banner-merchandising-oneTime-37=true"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate, br"],"User-Agent":["Mozilla/5.0 (iPhone; CPU iPhone OS 15_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Mobile/15E148 Safari/604.1"],"Accept-Language":["en-GB,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"app.getblogged.net"}},"common_log":"176.24.66.159 - - [10/Feb/2022:12:34:17 +0000] \"GET /res/js/external/register.js?date=2022-02-27b HTTP/2.0\" 200 3602","user_id":"","duration":0.032281172,"size":3602,"status":200,"resp_headers":{"Server":["Caddy"],"Access-Control-Allow-Origin":["https://www.getblogged.net"],"Etag":["\"r6d8qn9zc\""],"Content-Type":["application/javascript"],"Last-Modified":["Thu, 27 Jan 2022 11:18:23 GMT"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"]}}

5. What I already tried:

As this is an on-going development, we expected the issue to stem from our codebase. We removed large portions of slow code and did find some instances where the database was being queried for way too much data at once (we were thinking that we were having memory issues perhaps). We’ve addressed all of these issues and added logging for each query and result which does not appear to have had an effect (though the site does run better when it’s up!).

I saw in a thread yesterday ( Server not reachable after a few days - #26 by mdathersajjad ) where they are having a similar issue with the same result that php-fpm may be to blame. I restarted php-fpm on it’s own to no effect.

Restarted caddy container on it’s own too to no effect.

We doubled the server CPU and RAM wondering if this would help at all but it appears to have had no effect.

I believe I’ve enabled debug logging globally for all containers (let me know if there is anything else needed)

It only seems to come backup when I do a full docker-compose restart.

6. Links to relevant resources:

n/a

I’m suspecting you’re hitting the maximum number of open file descriptors on your system. Find the number of open files by the process:

https://www.cyberciti.biz/tips/linux-procfs-file-descriptors.html

You might have to change it if it’s too low. Figuring out the right number depends on your system capacity.

Yeah, that was my thought too. I’m also confused as to how there’s only 2 log entries total. If something was wrong in Caddy, there’d be an error message.

I agree it’s probably a file descriptor issue. I bet when you have too many websocket connections, you run out of file descriptors so your server stops accepting any new connections past that point.

In addition to the above comments, a few things I noticed in your config (not directly related to your reported problem):

Be careful with this, it looks like you’re not using an ask endpoint (global options), which means your site is open for abuse by an attacker, who could force your server to continually issue certificates for an infinite amount of domains.

Just as an aside, this is a bit of a misnomer, you’re not using php-fpm for this, you’re using PHP’s CLI program. FPM is “FastCGI Process Manager”. You’re overriding the container’s command with just php which means it’s not running php-fpm at all.

You shouldn’t need this, Caddy sets up HTTP->HTTPS redirects automatically.

Access logs never output anything DEBUG level, only INFO and ERROR currently, so setting the level here doesn’t have any effect.

1 Like

Yeah, that was my thought too.

Thank you both for responding. Can I ask why you suspect this please?
I had heard of this being an issue for someone else as well and did actually check this when it went down earlier with sysctl fs.file-nr. The number was below 2000 and we have 750000 maximum apparently. Not sure if this is false reporting though so I will increase it just in case.

1 Like

The forum wouldn’t let me post that many characters so I clipped it to the last two entries as that’s all that would fit.

@francislavoie thank you for all of the feedback, I will action these.

1 Like

I’ve bumped your trust level up. Maybe try posting more logs now? But if it’s a lot of logs, can you post your full logs to a pastebin or something please? I’d love to look into this more.

Does this perchance coincide with config reloads at all? Or just when it’s running, undisturbed, with no changes to configuration? Currently tracking a similar report but it’s specific to TLS handshakes and config reloads only. And it doesn’t happen in browsers, only curl or other command-line tools. So it sounds like a separate issue, but I’m curious nonetheless. Seems very rare in both cases.

Thank you Matt!
There is a bigger chunk of the Caddy log from docker below. Please let me know if any other logs may help:

{"log":"{\"level\":\"debug\",\"ts\":1644496269.5146444,\"logger\":\"http.reverse_proxy.transport.fastcgi\",\"msg\":\"roundtrip\",\"request\":{\"remote_addr\":\"85.115.54.202:32512\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"index.php?_ga=2.32243390.121785154.1644491272-1172054233.1625576632\",\"headers\":{\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Sec-Fetch-Dest\":[\"document\"],\"Client-Ip\":[\"176.24.66.159\"],\"X-Forwarded-For\":[\"176.24.66.159, 85.115.54.202\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Sec-Fetch-Site\":[\"same-site\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Via\":[\"1.1 hosted.websense 77j\"],\"Sec-Ch-Ua\":[\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"98\\\", \\\"Google Chrome\\\";v=\\\"98\\\"\"],\"Sec-Fetch-User\":[\"?1\"],\"Referer\":[\"https://www.getblogged.net/\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"Cookie\":[\"_ga=GA1.2.1172054233.1625576632; _hjid=bc01f9e4-e533-4750-8b8b-6cdfd64c2d49; _fbp=fb.1.1625576633556.1493868548; PHPSESSID=e976u9sva2qm4i1u1dbk91289h; remember_me=1; _hcfnl_funnel_uid=YPBPEGDwTxBGiWiBSneV1A==; _hcfnl_incognito=0; _hcfnl_fpr=1cdcd09ce74a3bf5df9f6b8521163773; tk_or=%22https%3A%2F%2Fapp.getblogged.net%2F%22; _hjSessionUser_2205995=eyJpZCI6ImQ0ZmExY2E5LTc2MjAtNTFlZi1iMjgwLWFlOTVmMjEyMjQ4NiIsImNyZWF0ZWQiOjE2MzczMjI4NTk2MzQsImV4aXN0aW5nIjp0cnVlfQ==; _hcfnl_hash=bd9a621a-b499-43b6-a225-af919f980d99; hubspotutk=0d29891b2001bd966e846d4ea919c3fb; __hssrc=1; _hjSessionUser_764671=eyJpZCI6IjBkNWNjMzhmLTZiZDctNTMwMS05NzQ2LWEwYzc1NDQwMTdhNyIsImNyZWF0ZWQiOjE2NDE4Mjk1ODMzMTMsImV4aXN0aW5nIjp0cnVlfQ==; tk_r3d=%22https%3A%2F%2Fwww.google.com%2F%22; tk_lr=%22%22; _gid=GA1.2.121785154.1644491272; __hstc=32639585.0d29891b2001bd966e846d4ea919c3fb.1641390550721.1644491272099.1644496184501.14; _hjSession_764671=eyJpZCI6IjdkMjliNGM0LWNhNmEtNDVhZi05MmZhLTgyZTFkN2ZmZTlmOCIsImNyZWF0ZWQiOjE2NDQ0OTYxODU3MTQsImluU2FtcGxlIjp0cnVlfQ==; _hjAbsoluteSessionInProgress=0; password_cookie_token=5f0af0eba5c01f8fd4329e41bde6b737; __hssc=32639585.2.1644496184501; _dc_gtm_UA-10569405-11=1\"],\"X-Forwarded-Proto\":[\"https\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"dial\":\"php-fpm:9000\",\"env\":{\"AUTH_TYPE\":\"\",\"CONTENT_LENGTH\":\"\",\"CONTENT_TYPE\":\"\",\"DOCUMENT_ROOT\":\"/application/public_html/getblogged/app\",\"DOCUMENT_URI\":\"index.php\",\"GATEWAY_INTERFACE\":\"CGI/1.1\",\"HTTPS\":\"on\",\"HTTP_ACCEPT\":\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\"HTTP_ACCEPT_ENCODING\":\"gzip, deflate\",\"HTTP_ACCEPT_LANGUAGE\":\"en-GB,en-US;q=0.9,en;q=0.8\",\"HTTP_CLIENT_IP\":\"176.24.66.159\",\"HTTP_COOKIE\":\"_ga=GA1.2.1172054233.1625576632; _hjid=bc01f9e4-e533-4750-8b8b-6cdfd64c2d49; _fbp=fb.1.1625576633556.1493868548; PHPSESSID=e976u9sva2qm4i1u1dbk91289h; remember_me=1; _hcfnl_funnel_uid=YPBPEGDwTxBGiWiBSneV1A==; _hcfnl_incognito=0; _hcfnl_fpr=1cdcd09ce74a3bf5df9f6b8521163773; tk_or=%22https%3A%2F%2Fapp.getblogged.net%2F%22; _hjSessionUser_2205995=eyJpZCI6ImQ0ZmExY2E5LTc2MjAtNTFlZi1iMjgwLWFlOTVmMjEyMjQ4NiIsImNyZWF0ZWQiOjE2MzczMjI4NTk2MzQsImV4aXN0aW5nIjp0cnVlfQ==; _hcfnl_hash=bd9a621a-b499-43b6-a225-af919f980d99; hubspotutk=0d29891b2001bd966e846d4ea919c3fb; __hssrc=1; _hjSessionUser_764671=eyJpZCI6IjBkNWNjMzhmLTZiZDctNTMwMS05NzQ2LWEwYzc1NDQwMTdhNyIsImNyZWF0ZWQiOjE2NDE4Mjk1ODMzMTMsImV4aXN0aW5nIjp0cnVlfQ==; tk_r3d=%22https%3A%2F%2Fwww.google.com%2F%22; tk_lr=%22%22; _gid=GA1.2.121785154.1644491272; __hstc=32639585.0d29891b2001bd966e846d4ea919c3fb.1641390550721.1644491272099.1644496184501.14; _hjSession_764671=eyJpZCI6IjdkMjliNGM0LWNhNmEtNDVhZi05MmZhLTgyZTFkN2ZmZTlmOCIsImNyZWF0ZWQiOjE2NDQ0OTYxODU3MTQsImluU2FtcGxlIjp0cnVlfQ==; _hjAbsoluteSessionInProgress=0; password_cookie_token=5f0af0eba5c01f8fd4329e41bde6b737; __hssc=32639585.2.1644496184501; _dc_gtm_UA-10569405-11=1\",\"HTTP_HOST\":\"app.getblogged.net\",\"HTTP_REFERER\":\"https://www.getblogged.net/\",\"HTTP_SEC_CH_UA\":\"\\\" Not A;Brand\\\";v=\\\"99\\\", \\\"Chromium\\\";v=\\\"98\\\", \\\"Google Chrome\\\";v=\\\"98\\\"\",\"HTTP_SEC_CH_UA_MOBILE\":\"?0\",\"HTTP_SEC_CH_UA_PLATFORM\":\"\\\"Windows\\\"\",\"HTTP_SEC_FETCH_DEST\":\"document\",\"HTTP_SEC_FETCH_MODE\":\"navigate\",\"HTTP_SEC_FETCH_SITE\":\"same-site\",\"HTTP_SEC_FETCH_USER\":\"?1\",\"HTTP_UPGRADE_INSECURE_REQUESTS\":\"1\",\"HTTP_USER_AGENT\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36\",\"HTTP_VIA\":\"1.1 hosted.websense 77j\",\"HTTP_X_FORWARDED_FOR\":\"176.24.66.159, 85.115.54.202\",\"HTTP_X_FORWARDED_PROTO\":\"https\",\"PATH_INFO\":\"\",\"QUERY_STRING\":\"_ga=2.32243390.121785154.1644491272-1172054233.1625576632\",\"REMOTE_ADDR\":\"85.115.54.202\",\"REMOTE_HOST\":\"85.115.54.202\",\"REMOTE_IDENT\":\"\",\"REMOTE_PORT\":\"32512\",\"REMOTE_USER\":\"\",\"REQUEST_METHOD\":\"GET\",\"REQUEST_SCHEME\":\"https\",\"REQUEST_URI\":\"/login?_ga=2.32243390.121785154.1644491272-1172054233.1625576632\",\"SCRIPT_FILENAME\":\"/application/public_html/getblogged/app/index.php\",\"SCRIPT_NAME\":\"/index.php\",\"SERVER_NAME\":\"app.getblogged.net\",\"SERVER_PROTOCOL\":\"HTTP/1.1\",\"SERVER_SOFTWARE\":\"Caddy/v2.4.5\",\"SSL_CIPHER\":\"TLS_AES_128_GCM_SHA256\",\"SSL_PROTOCOL\":\"TLSv1.3\"}}\n","stream":"stderr","time":"2022-02-10T12:31:09.515362152Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496270.4008703,\"logger\":\"tls.handshake\",\"msg\":\"choosing certificate\",\"identifier\":\"app.getblogged.net\",\"num_choices\":1}\n","stream":"stderr","time":"2022-02-10T12:31:10.401955574Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496270.4010084,\"logger\":\"tls.handshake\",\"msg\":\"default certificate selection results\",\"identifier\":\"app.getblogged.net\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"issuer_key\":\"acme-v02.api.letsencrypt.org-directory\",\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:31:10.402088444Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496270.4010336,\"logger\":\"tls.handshake\",\"msg\":\"matched certificate in cache\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"expiration\":1647520277,\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:31:10.402112781Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496270.4063487,\"logger\":\"http.handlers.file_server\",\"msg\":\"sanitized path join\",\"site_root\":\"/application/public_html/getblogged/app\",\"request_path\":\"/service-worker.js\",\"result\":\"/application/public_html/getblogged/app/service-worker.js\"}\n","stream":"stderr","time":"2022-02-10T12:31:10.407050954Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496270.40639,\"logger\":\"http.handlers.file_server\",\"msg\":\"opening file\",\"filename\":\"/application/public_html/getblogged/app/service-worker.js\"}\n","stream":"stderr","time":"2022-02-10T12:31:10.40709998Z"}
{"log":"{\"level\":\"info\",\"ts\":1644496270.4065704,\"logger\":\"http.log.access.log0\",\"msg\":\"handled request\",\"request\":{\"remote_addr\":\"85.115.54.202:40131\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"/service-worker.js\",\"headers\":{\"Via\":[\"1.1 hosted.websense 49j\"],\"Client-Ip\":[\"176.24.66.159\"],\"Sec-Fetch-Dest\":[\"serviceworker\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"Service-Worker\":[\"script\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36\"],\"Cookie\":[\"_ga=GA1.2.1172054233.1625576632; _hjid=bc01f9e4-e533-4750-8b8b-6cdfd64c2d49; _fbp=fb.1.1625576633556.1493868548; PHPSESSID=e976u9sva2qm4i1u1dbk91289h; remember_me=1; _hcfnl_funnel_uid=YPBPEGDwTxBGiWiBSneV1A==; _hcfnl_incognito=0; _hcfnl_fpr=1cdcd09ce74a3bf5df9f6b8521163773; tk_or=%22https%3A%2F%2Fapp.getblogged.net%2F%22; _hjSessionUser_2205995=eyJpZCI6ImQ0ZmExY2E5LTc2MjAtNTFlZi1iMjgwLWFlOTVmMjEyMjQ4NiIsImNyZWF0ZWQiOjE2MzczMjI4NTk2MzQsImV4aXN0aW5nIjp0cnVlfQ==; _hcfnl_hash=bd9a621a-b499-43b6-a225-af919f980d99; hubspotutk=0d29891b2001bd966e846d4ea919c3fb; __hssrc=1; _hjSessionUser_764671=eyJpZCI6IjBkNWNjMzhmLTZiZDctNTMwMS05NzQ2LWEwYzc1NDQwMTdhNyIsImNyZWF0ZWQiOjE2NDE4Mjk1ODMzMTMsImV4aXN0aW5nIjp0cnVlfQ==; tk_r3d=%22https%3A%2F%2Fwww.google.com%2F%22; tk_lr=%22%22; _gid=GA1.2.121785154.1644491272; __hstc=32639585.0d29891b2001bd966e846d4ea919c3fb.1641390550721.1644491272099.1644496184501.14; _hjSession_764671=eyJpZCI6IjdkMjliNGM0LWNhNmEtNDVhZi05MmZhLTgyZTFkN2ZmZTlmOCIsImNyZWF0ZWQiOjE2NDQ0OTYxODU3MTQsImluU2FtcGxlIjp0cnVlfQ==; _hjAbsoluteSessionInProgress=0; password_cookie_token=5f0af0eba5c01f8fd4329e41bde6b737; __hssc=32639585.2.1644496184501; _dc_gtm_UA-10569405-11=1\"],\"If-Modified-Since\":[\"Mon, 18 Oct 2021 14:27:23 GMT\"],\"X-Forwarded-For\":[\"176.24.66.159\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"If-None-Match\":[\"\\\"r16g5n1vm\\\"\"],\"Referer\":[\"https://app.getblogged.net/service-worker.js\"],\"Cache-Control\":[\"max-age=0\"],\"Accept\":[\"*/*\"],\"Sec-Fetch-Mode\":[\"same-origin\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"common_log\":\"85.115.54.202 - - [10/Feb/2022:12:31:10 +0000] \\\"GET /service-worker.js HTTP/1.1\\\" 304 0\",\"user_id\":\"\",\"duration\":0.000493312,\"size\":0,\"status\":304,\"resp_headers\":{\"Server\":[\"Caddy\"],\"Access-Control-Allow-Origin\":[\"https://www.getblogged.net\"],\"Etag\":[\"\\\"r16g5n1vm\\\"\"]}}\n","stream":"stderr","time":"2022-02-10T12:31:10.40710994Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496274.401307,\"logger\":\"tls.handshake\",\"msg\":\"choosing certificate\",\"identifier\":\"app.getblogged.net\",\"num_choices\":1}\n","stream":"stderr","time":"2022-02-10T12:31:14.403185459Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496274.4013975,\"logger\":\"tls.handshake\",\"msg\":\"default certificate selection results\",\"identifier\":\"app.getblogged.net\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"issuer_key\":\"acme-v02.api.letsencrypt.org-directory\",\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:31:14.403373934Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496274.4014103,\"logger\":\"tls.handshake\",\"msg\":\"matched certificate in cache\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"expiration\":1647520277,\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:31:14.403385663Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496274.4341307,\"logger\":\"http.handlers.file_server\",\"msg\":\"sanitized path join\",\"site_root\":\"/application/public_html/getblogged/app\",\"request_path\":\"/service-worker.js\",\"result\":\"/application/public_html/getblogged/app/service-worker.js\"}\n","stream":"stderr","time":"2022-02-10T12:31:14.434727143Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496274.4341726,\"logger\":\"http.handlers.file_server\",\"msg\":\"opening file\",\"filename\":\"/application/public_html/getblogged/app/service-worker.js\"}\n","stream":"stderr","time":"2022-02-10T12:31:14.434760318Z"}
{"log":"{\"level\":\"info\",\"ts\":1644496274.4343014,\"logger\":\"http.log.access.log0\",\"msg\":\"handled request\",\"request\":{\"remote_addr\":\"85.115.54.202:55006\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"/service-worker.js\",\"headers\":{\"Sec-Fetch-Mode\":[\"same-origin\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36\"],\"Sec-Fetch-Site\":[\"same-origin\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Cookie\":[\"_ga=GA1.2.1172054233.1625576632; _hjid=bc01f9e4-e533-4750-8b8b-6cdfd64c2d49; _fbp=fb.1.1625576633556.1493868548; PHPSESSID=e976u9sva2qm4i1u1dbk91289h; remember_me=1; _hcfnl_funnel_uid=YPBPEGDwTxBGiWiBSneV1A==; _hcfnl_incognito=0; _hcfnl_fpr=1cdcd09ce74a3bf5df9f6b8521163773; tk_or=%22https%3A%2F%2Fapp.getblogged.net%2F%22; _hjSessionUser_2205995=eyJpZCI6ImQ0ZmExY2E5LTc2MjAtNTFlZi1iMjgwLWFlOTVmMjEyMjQ4NiIsImNyZWF0ZWQiOjE2MzczMjI4NTk2MzQsImV4aXN0aW5nIjp0cnVlfQ==; _hcfnl_hash=bd9a621a-b499-43b6-a225-af919f980d99; hubspotutk=0d29891b2001bd966e846d4ea919c3fb; __hssrc=1; _hjSessionUser_764671=eyJpZCI6IjBkNWNjMzhmLTZiZDctNTMwMS05NzQ2LWEwYzc1NDQwMTdhNyIsImNyZWF0ZWQiOjE2NDE4Mjk1ODMzMTMsImV4aXN0aW5nIjp0cnVlfQ==; tk_r3d=%22https%3A%2F%2Fwww.google.com%2F%22; tk_lr=%22%22; _gid=GA1.2.121785154.1644491272; __hstc=32639585.0d29891b2001bd966e846d4ea919c3fb.1641390550721.1644491272099.1644496184501.14; _hjSession_764671=eyJpZCI6IjdkMjliNGM0LWNhNmEtNDVhZi05MmZhLTgyZTFkN2ZmZTlmOCIsImNyZWF0ZWQiOjE2NDQ0OTYxODU3MTQsImluU2FtcGxlIjp0cnVlfQ==; _hjAbsoluteSessionInProgress=0; password_cookie_token=5f0af0eba5c01f8fd4329e41bde6b737; __hssc=32639585.2.1644496184501; _dc_gtm_UA-10569405-11=1\"],\"If-Modified-Since\":[\"Mon, 18 Oct 2021 14:27:23 GMT\"],\"Via\":[\"1.1 hosted.websense 04j\"],\"Client-Ip\":[\"176.24.66.159\"],\"Cache-Control\":[\"max-age=0\"],\"Service-Worker\":[\"script\"],\"Accept\":[\"*/*\"],\"Sec-Fetch-Dest\":[\"serviceworker\"],\"Referer\":[\"https://app.getblogged.net/service-worker.js\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"If-None-Match\":[\"\\\"r16g5n1vm\\\"\"],\"X-Forwarded-For\":[\"176.24.66.159\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"common_log\":\"85.115.54.202 - - [10/Feb/2022:12:31:14 +0000] \\\"GET /service-worker.js HTTP/1.1\\\" 304 0\",\"user_id\":\"\",\"duration\":0.000397451,\"size\":0,\"status\":304,\"resp_headers\":{\"Server\":[\"Caddy\"],\"Access-Control-Allow-Origin\":[\"https://www.getblogged.net\"],\"Etag\":[\"\\\"r16g5n1vm\\\"\"]}}\n","stream":"stderr","time":"2022-02-10T12:31:14.43476727Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496350.8179564,\"logger\":\"http.handlers.rewrite\",\"msg\":\"rewrote request\",\"request\":{\"remote_addr\":\"81.110.207.113:59106\",\"proto\":\"HTTP/2.0\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"/\",\"headers\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\"],\"Accept-Language\":[\"en-GB,en-US;q=0.7,en;q=0.3\"],\"Cookie\":[\"PHPSESSID=8051jiopf1qjmqhb0354mig2f7\"],\"Sec-Fetch-User\":[\"?1\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"]},\"tls\":{\"resumed\":true,\"version\":772,\"cipher_suite\":4865,\"proto\":\"h2\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"method\":\"GET\",\"uri\":\"/index.php\"}\n","stream":"stderr","time":"2022-02-10T12:32:30.824334856Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496350.8186092,\"logger\":\"http.reverse_proxy.transport.fastcgi\",\"msg\":\"roundtrip\",\"request\":{\"remote_addr\":\"81.110.207.113:59106\",\"proto\":\"HTTP/2.0\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"/index.php\",\"headers\":{\"X-Forwarded-For\":[\"81.110.207.113\"],\"X-Forwarded-Proto\":[\"https\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\"],\"Te\":[\"trailers\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Dnt\":[\"1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"Sec-Fetch-User\":[\"?1\"],\"Accept-Language\":[\"en-GB,en-US;q=0.7,en;q=0.3\"],\"Cookie\":[\"PHPSESSID=8051jiopf1qjmqhb0354mig2f7\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Site\":[\"none\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0\"],\"Accept-Encoding\":[\"gzip, deflate, br\"]},\"tls\":{\"resumed\":true,\"version\":772,\"cipher_suite\":4865,\"proto\":\"h2\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"dial\":\"php-fpm:9000\",\"env\":{\"AUTH_TYPE\":\"\",\"CONTENT_LENGTH\":\"\",\"CONTENT_TYPE\":\"\",\"DOCUMENT_ROOT\":\"/application/public_html/getblogged/app\",\"DOCUMENT_URI\":\"/index.php\",\"GATEWAY_INTERFACE\":\"CGI/1.1\",\"HTTPS\":\"on\",\"HTTP_ACCEPT\":\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\",\"HTTP_ACCEPT_ENCODING\":\"gzip, deflate, br\",\"HTTP_ACCEPT_LANGUAGE\":\"en-GB,en-US;q=0.7,en;q=0.3\",\"HTTP_COOKIE\":\"PHPSESSID=8051jiopf1qjmqhb0354mig2f7\",\"HTTP_DNT\":\"1\",\"HTTP_HOST\":\"app.getblogged.net\",\"HTTP_SEC_FETCH_DEST\":\"document\",\"HTTP_SEC_FETCH_MODE\":\"navigate\",\"HTTP_SEC_FETCH_SITE\":\"none\",\"HTTP_SEC_FETCH_USER\":\"?1\",\"HTTP_TE\":\"trailers\",\"HTTP_UPGRADE_INSECURE_REQUESTS\":\"1\",\"HTTP_USER_AGENT\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0\",\"HTTP_X_FORWARDED_FOR\":\"81.110.207.113\",\"HTTP_X_FORWARDED_PROTO\":\"https\",\"PATH_INFO\":\"\",\"QUERY_STRING\":\"\",\"REMOTE_ADDR\":\"81.110.207.113\",\"REMOTE_HOST\":\"81.110.207.113\",\"REMOTE_IDENT\":\"\",\"REMOTE_PORT\":\"59106\",\"REMOTE_USER\":\"\",\"REQUEST_METHOD\":\"GET\",\"REQUEST_SCHEME\":\"https\",\"REQUEST_URI\":\"/\",\"SCRIPT_FILENAME\":\"/application/public_html/getblogged/app/index.php\",\"SCRIPT_NAME\":\"/index.php\",\"SERVER_NAME\":\"app.getblogged.net\",\"SERVER_PROTOCOL\":\"HTTP/2.0\",\"SERVER_SOFTWARE\":\"Caddy/v2.4.5\",\"SSL_CIPHER\":\"TLS_AES_128_GCM_SHA256\",\"SSL_PROTOCOL\":\"TLSv1.3\"}}\n","stream":"stderr","time":"2022-02-10T12:32:30.824519046Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496365.7543225,\"logger\":\"tls.handshake\",\"msg\":\"choosing certificate\",\"identifier\":\"app.getblogged.net\",\"num_choices\":1}\n","stream":"stderr","time":"2022-02-10T12:32:45.756478813Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496365.7545192,\"logger\":\"tls.handshake\",\"msg\":\"default certificate selection results\",\"identifier\":\"app.getblogged.net\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"issuer_key\":\"acme-v02.api.letsencrypt.org-directory\",\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:32:45.756534492Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496365.754561,\"logger\":\"tls.handshake\",\"msg\":\"matched certificate in cache\",\"subjects\":[\"app.getblogged.net\"],\"managed\":true,\"expiration\":1647520277,\"hash\":\"4d9887ada99241030a5e740faaabd526fc3c821e19c8b4bd1ea2963e71093ecf\"}\n","stream":"stderr","time":"2022-02-10T12:32:45.75654482Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496365.7849662,\"logger\":\"http.handlers.rewrite\",\"msg\":\"rewrote request\",\"request\":{\"remote_addr\":\"81.19.210.221:46440\",\"proto\":\"HTTP/2.0\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"/marketplace/view/u86033f7c-6302-46fd-81d8-b23b5d568649/?proposal=241786\",\"headers\":{\"Sec-Ch-Ua\":[\"\\\" Not;A Brand\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"97\\\", \\\"Chromium\\\";v=\\\"97\\\"\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Sec-Fetch-Site\":[\"cross-site\"],\"Sec-Fetch-User\":[\"?1\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"],\"Cookie\":[\"tk_lr=%22https%3A%2F%2Fwww.google.com%2F%22; tk_or=%22https%3A%2F%2Fwww.google.com%2F%22; _ga=GA1.2.1137858084.1643973678; _fbp=fb.1.1643973678274.516029474; _hjSessionUser_764671=eyJpZCI6ImU0NjI2NGU0LWI4MDQtNWYyYi1iMTM4LTEzZDJhNDAyOGRmMCIsImNyZWF0ZWQiOjE2NDM5NzM2NzgzMDMsImV4aXN0aW5nIjpmYWxzZX0=; __hstc=32639585.37ed074f269d80d064e3e491f576d48a.1643973678480.1643973678480.1643973678480.1; hubspotutk=37ed074f269d80d064e3e491f576d48a\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"h2\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"method\":\"GET\",\"uri\":\"index.php?proposal=241786\"}\n","stream":"stderr","time":"2022-02-10T12:32:45.78779636Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496365.7853694,\"logger\":\"http.reverse_proxy.transport.fastcgi\",\"msg\":\"roundtrip\",\"request\":{\"remote_addr\":\"81.19.210.221:46440\",\"proto\":\"HTTP/2.0\",\"method\":\"GET\",\"host\":\"app.getblogged.net\",\"uri\":\"index.php?proposal=241786\",\"headers\":{\"Sec-Fetch-User\":[\"?1\"],\"Sec-Ch-Ua\":[\"\\\" Not;A Brand\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"97\\\", \\\"Chromium\\\";v=\\\"97\\\"\"],\"X-Forwarded-Proto\":[\"https\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Ch-Ua-Mobile\":[\"?0\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Cookie\":[\"tk_lr=%22https%3A%2F%2Fwww.google.com%2F%22; tk_or=%22https%3A%2F%2Fwww.google.com%2F%22; _ga=GA1.2.1137858084.1643973678; _fbp=fb.1.1643973678274.516029474; _hjSessionUser_764671=eyJpZCI6ImU0NjI2NGU0LWI4MDQtNWYyYi1iMTM4LTEzZDJhNDAyOGRmMCIsImNyZWF0ZWQiOjE2NDM5NzM2NzgzMDMsImV4aXN0aW5nIjpmYWxzZX0=; __hstc=32639585.37ed074f269d80d064e3e491f576d48a.1643973678480.1643973678480.1643973678480.1; hubspotutk=37ed074f269d80d064e3e491f576d48a\"],\"X-Forwarded-For\":[\"81.19.210.221\"],\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Sec-Fetch-Site\":[\"cross-site\"],\"Sec-Fetch-Dest\":[\"document\"],\"User-Agent\":[\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36\"],\"Sec-Ch-Ua-Platform\":[\"\\\"Windows\\\"\"],\"Accept-Language\":[\"en-GB,en-US;q=0.9,en;q=0.8\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"h2\",\"proto_mutual\":true,\"server_name\":\"app.getblogged.net\"}},\"dial\":\"php-fpm:9000\",\"env\":{\"AUTH_TYPE\":\"\",\"CONTENT_LENGTH\":\"\",\"CONTENT_TYPE\":\"\",\"DOCUMENT_ROOT\":\"/application/public_html/getblogged/app\",\"DOCUMENT_URI\":\"index.php\",\"GATEWAY_INTERFACE\":\"CGI/1.1\",\"HTTPS\":\"on\",\"HTTP_ACCEPT\":\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\"HTTP_ACCEPT_ENCODING\":\"gzip, deflate, br\",\"HTTP_ACCEPT_LANGUAGE\":\"en-GB,en-US;q=0.9,en;q=0.8\",\"HTTP_COOKIE\":\"tk_lr=%22https%3A%2F%2Fwww.google.com%2F%22; tk_or=%22https%3A%2F%2Fwww.google.com%2F%22; _ga=GA1.2.1137858084.1643973678; _fbp=fb.1.1643973678274.516029474; _hjSessionUser_764671=eyJpZCI6ImU0NjI2NGU0LWI4MDQtNWYyYi1iMTM4LTEzZDJhNDAyOGRmMCIsImNyZWF0ZWQiOjE2NDM5NzM2NzgzMDMsImV4aXN0aW5nIjpmYWxzZX0=; __hstc=32639585.37ed074f269d80d064e3e491f576d48a.1643973678480.1643973678480.1643973678480.1; hubspotutk=37ed074f269d80d064e3e491f576d48a\",\"HTTP_HOST\":\"app.getblogged.net\",\"HTTP_SEC_CH_UA\":\"\\\" Not;A Brand\\\";v=\\\"99\\\", \\\"Google Chrome\\\";v=\\\"97\\\", \\\"Chromium\\\";v=\\\"97\\\"\",\"HTTP_SEC_CH_UA_MOBILE\":\"?0\",\"HTTP_SEC_CH_UA_PLATFORM\":\"\\\"Windows\\\"\",\"HTTP_SEC_FETCH_DEST\":\"document\",\"HTTP_SEC_FETCH_MODE\":\"navigate\",\"HTTP_SEC_FETCH_SITE\":\"cross-site\",\"HTTP_SEC_FETCH_USER\":\"?1\",\"HTTP_UPGRADE_INSECURE_REQUESTS\":\"1\",\"HTTP_USER_AGENT\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36\",\"HTTP_X_FORWARDED_FOR\":\"81.19.210.221\",\"HTTP_X_FORWARDED_PROTO\":\"https\",\"PATH_INFO\":\"\",\"QUERY_STRING\":\"proposal=241786\",\"REMOTE_ADDR\":\"81.19.210.221\",\"REMOTE_HOST\":\"81.19.210.221\",\"REMOTE_IDENT\":\"\",\"REMOTE_PORT\":\"46440\",\"REMOTE_USER\":\"\",\"REQUEST_METHOD\":\"GET\",\"REQUEST_SCHEME\":\"https\",\"REQUEST_URI\":\"/marketplace/view/u86033f7c-6302-46fd-81d8-b23b5d568649/?proposal=241786\",\"SCRIPT_FILENAME\":\"/application/public_html/getblogged/app/index.php\",\"SCRIPT_NAME\":\"/index.php\",\"SERVER_NAME\":\"app.getblogged.net\",\"SERVER_PROTOCOL\":\"HTTP/2.0\",\"SERVER_SOFTWARE\":\"Caddy/v2.4.5\",\"SSL_CIPHER\":\"TLS_AES_128_GCM_SHA256\",\"SSL_PROTOCOL\":\"TLSv1.3\"}}\n","stream":"stderr","time":"2022-02-10T12:32:45.787855854Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.6614919,\"logger\":\"tls.handshake\",\"msg\":\"choosing certificate\",\"identifier\":\"ws.getblogged.net\",\"num_choices\":1}\n","stream":"stderr","time":"2022-02-10T12:33:25.663849683Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.661915,\"logger\":\"tls.handshake\",\"msg\":\"default certificate selection results\",\"identifier\":\"ws.getblogged.net\",\"subjects\":[\"ws.getblogged.net\"],\"managed\":true,\"issuer_key\":\"acme-v02.api.letsencrypt.org-directory\",\"hash\":\"8a12faf93455f6215a30fe876c94eb67f1002ab7d6768c1d391eb1b7735b4535\"}\n","stream":"stderr","time":"2022-02-10T12:33:25.664089001Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.6620045,\"logger\":\"tls.handshake\",\"msg\":\"matched certificate in cache\",\"subjects\":[\"ws.getblogged.net\"],\"managed\":true,\"expiration\":1647520251,\"hash\":\"8a12faf93455f6215a30fe876c94eb67f1002ab7d6768c1d391eb1b7735b4535\"}\n","stream":"stderr","time":"2022-02-10T12:33:25.664103492Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.8380249,\"logger\":\"http.handlers.reverse_proxy\",\"msg\":\"upstream roundtrip\",\"upstream\":\"php-fpm-ws:8090\",\"request\":{\"remote_addr\":\"97.70.160.221:50566\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"ws.getblogged.net\",\"uri\":\"/chat/php-socket.php\",\"headers\":{\"Sec-Websocket-Version\":[\"13\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Sec-Websocket-Key\":[\"91piOtRuFiK1mQCwmCfopw==\"],\"Pragma\":[\"no-cache\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Origin\":[\"https://app.getblogged.net\"],\"Cookie\":[\"_fbp=fb.1.1644492997118.1007065267; _ga=GA1.2.1201346913.1644492992; _gid=GA1.2.657030684.1644492992; __hstc=32639585.ac4910a52502a3374020a12569f1a7ab.1643285198506.1643551792117.1643925912344.3; hubspotutk=ac4910a52502a3374020a12569f1a7ab; _hjSessionUser_764671=eyJpZCI6ImUyODgyNzc5LWI2NTctNWFjZS04Y2ZlLTFhNzEyZGVkMWE3ZSIsImNyZWF0ZWQiOjE2NDMyODUxOTczMTUsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22\"],\"X-Forwarded-Proto\":[\"https\"],\"User-Agent\":[\"Mozilla/5.0 (iPhone; CPU iPhone OS 15_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Mobile/15E148 Safari/604.1\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-For\":[\"97.70.160.221\"],\"Accept\":[\"*/*\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"proto_mutual\":true,\"server_name\":\"ws.getblogged.net\"}},\"headers\":{\"Upgrade\":[\"websocket\"],\"Connection\":[\"Upgrade\"],\"Websocket-Origin\":[\"ws.localhost\"],\"Websocket-Location\":[\"wss://ws.localhost/chat/php-socket.php\"],\"Sec-Websocket-Accept\":[\"TicBEHxZ9T7X3tSpTh4F+KpDV84=\"]},\"status\":101}\n","stream":"stderr","time":"2022-02-10T12:33:25.838684474Z"}
{"log":"{\"level\":\"debug\",\"ts\":1644496405.8381214,\"logger\":\"http.handlers.reverse_proxy\",\"msg\":\"upgrading connection\",\"upstream\":\"php-fpm-ws:8090\",\"request\":{\"remote_addr\":\"97.70.160.221:50566\",\"proto\":\"HTTP/1.1\",\"method\":\"GET\",\"host\":\"ws.getblogged.net\",\"uri\":\"/chat/php-socket.php\",\"headers\":{\"Sec-Websocket-Version\":[\"13\"],\"Sec-Websocket-Extensions\":[\"permessage-deflate\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Sec-Websocket-Key\":[\"91piOtRuFiK1mQCwmCfopw==\"],\"Pragma\":[\"no-cache\"],\"Accept-Language\":[\"en-US,en;q=0.9\"],\"Origin\":[\"https://app.getblogged.net\"],\"Cookie\":[\"_fbp=fb.1.1644492997118.1007065267; _ga=GA1.2.1201346913.1644492992; _gid=GA1.2.657030684.1644492992; __hstc=32639585.ac4910a52502a3374020a12569f1a7ab.1643285198506.1643551792117.1643925912344.3; hubspotutk=ac4910a52502a3374020a12569f1a7ab; _hjSessionUser_764671=eyJpZCI6ImUyODgyNzc5LWI2NTctNWFjZS04Y2ZlLTFhNzEyZGVkMWE3ZSIsImNyZWF0ZWQiOjE2NDMyODUxOTczMTUsImV4aXN0aW5nIjp0cnVlfQ==; tk_lr=%22%22; tk_or=%22%22\"],\"X-Forwarded-Proto\":[\"https\"],\"User-Agent\":[\"Mozilla/5.0 (iPhone; CPU iPhone OS 15_2_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Mobile/15E148 Safari/604.1\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"Upgrade\"],\"Upgrade\":[\"websocket\"],\"X-Forwarded-For\":[\"97.70.160.221\"],\"Accept\":[\"*/*\"]},\"tls\":{\"resumed\":false,\"version\":772,\"cipher_suite\":4865,\"proto\":\"http/1.1\",\"proto_mutual\":true,\"server_name\":\"ws.getblogged.net\"}}}\n","stream":"stderr","time":"2022-02-10T12:33:25.873362351Z"}

Hi Francis, I wanted to get back to your feedback specifically as it was quite interesting:

… who could force your server to continually issue certificates for an infinite amount of domains.

I do have a ticket lined up for addressing this but I do appreciate you highlighting it.

You’re overriding the container’s command with just php which means it’s not running php-fpm at all.

That was a good spot! Thanks for pointing it out.

You shouldn’t need this, Caddy sets up HTTP->HTTPS redirects automatically.

I think we had an issue with that initially not redirecting but I will remove it off production and try again.

Access logs never output anything DEBUG level, only INFO and ERROR currently, so setting the level here doesn’t have any effect.

I thought this was from the docs but I must have read it wrong. If I want to record the maximum amount should I set it to INFO?

Upping the file descriptor limit does not appear to have worked unfortunately. The site has gone down again and this is the terminal output:

root@vps784967:~# sysctl fs.file-nr
fs.file-nr = 1888       0       5000000

Running curl on the server itself got the following result (it hanged at the end)

root@vps784967:~# curl -v https://app.getblogged.net/
*   Trying 51.68.212.198...
* TCP_NODELAY set
* Connected to app.getblogged.net (51.68.212.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.getblogged.net
*  start date: Dec 17 12:31:18 2021 GMT
*  expire date: Mar 17 12:31:17 2022 GMT
*  subjectAltName: host "app.getblogged.net" matched cert's "app.getblogged.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55c823193620)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET / HTTP/2
> Host: app.getblogged.net
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):

There’s a difference between runtime logs (stdout/stderr logs) and access logs (per-site, the log directive).

Access logs in particular only emit logs at INFO and ERROR levels, and the default level for the writer is INFO, so it will write all the access logs by default. If you wanted to only see ERROR access logs and not INFO, then you could set the log level to WARN or ERROR which would exclude INFO.

By setting the debug global option, you’re making all the runtime logs get written, by lowering the log level to also include DEBUG logs.

Bah. I don’t know what the issue is then. It looks like it does reach Caddy and attempts to initiate a TLS handshake, but then it just gives up for some reason. That’s really odd.

I think whatever command you’re using to see your logs is causing Docker to write its own logs in JSON format, which wraps Caddy’s logs which are already JSON inside of that, causing all the logs to get escaped etc. It’s hard to read like that. Maybe you could adjust the command to not do that.

The last log you have there looks to be a websocket upgrade, I’m not sure if any assumptions can be made due to that, but worth mentioning.

1 Like

Thanks, that’s very interesting. And frustrating! (For both you and me, sorry for the trouble.)

Are there any Caddy logs that correspond to your specific TLS handshake from any of your curl -v commands while troubleshooting? (The “last log entries” you posted above look like they come from other, unrelated requests.) Because I do see stuff like “choosing certificate” in the expanded logs (thanks for posting those) which is what I would expect to see if it was a bug in our Caddy/CertMagic code.

Otherwise, the fact that it’s just hanging leaves to believe it might be a lower-level networking issue? Did you say these hangs occur without any modification of the configs? Do only TLS handshakes hang or do all connections hang (including plaintext HTTP requests)?

Hi Matt,
Sorry I missed your question, was a bit of a hectic night!

Are there any Caddy logs that correspond to your specific TLS handshake from any of your curl -v commands while troubleshooting?

I’m not too sure where I would look for this? Below is another curl -v output while the site was not responding if that helps.

MacBook-Pro-2:~ benjaminhitchens$ curl -v https://app.getblogged.net/
*  Trying 51.68.212.198...
* TCP_NODELAY set
* Connected to app.getblogged.net (51.68.212.198) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
 CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=app.getblogged.net
* start date: Dec 17 12:31:18 2021 GMT
* expire date: Mar 17 12:31:17 2022 GMT
* subjectAltName: host "app.getblogged.net" matched cert's "app.getblogged.net"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7faabe80e600)
> GET / HTTP/2
> Host: app.getblogged.net
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!


Did you say these hangs occur without any modification of the configs? Do only TLS handshakes hang or do all connections hang (including plaintext HTTP requests)?

If I understand you correctly no manual modifications are being made. I’m able to login via ssh and access the database which is outside of Caddy, it just seems to be the connection to the site which is going through Caddy that is hanging (I think this is what you mean by TLS handshakes?)

Just for reference, below you can see how many times the site went down lastnight. Each time it came back we were manually restarting it. At about 1AM I updated Docker and some other misc bits of the OS, applied the suggested file descriptor increase and rebooted the machine. It went down one more time immediately after but seemed to be OK up until half an hour ago.

It pains me to say this but I may have to setup a script on the server to monitor the site and restart docker each time it goes down automatically. This will log users out unfortunately but I’m not sure what else to do at the moment.

Just your regular Caddy logs. (Ensure debug mode is enabled, i.e. debug-level logging.) Not really interested in the request/access logs, but rather the normal Caddy log output (stderr by default).

Why’s that?

Looks like the TLS handshake completed, and the HTTP request was sent, but then the response was never received.

What’s the goroutine dump like while this happens? You can get one from http://localhost:2019/debug/pprof from the machine where the Caddy instance is running.

Just your regular Caddy logs.

Ah, I thought that’s what the log I had was. I’ll double check the docs and report back.

Why’s that?

Restarting caddy individually does not fix the issue but restarting any of the other containers individually doesn’t either, so we have to restart all of them which I believe clears the php sessions as well.

What’s the goroutine dump like while this happens?

I will report back when it goes again!

Oh, right – the container log in your case. You did post that above, but I want to see the logs pertaining to just your specific requests that time out.

This leads me to believe it’s a bug in the PHP application/infrastructure.

Thanks, let me know when you have more info!

Hi Matt, just looking into this in preparation. I’ve made my restart script and was thinking of saving the output here automatically. This is not loading anything for me though:

wget http://localhost:2019/debug/pprof                                                                                                                                                                                                                        
--2022-02-12 02:27:36--  http://localhost:2019/debug/pprof                                                                                                                                                                                                                      
Resolving localhost (localhost)... 127.0.0.1                                                                                                                                                                                                                                    
Connecting to localhost (localhost)|127.0.0.1|:2019... failed: Connection refused.

I can’t seem to find any documentation on this specifically either. Are you able to point me to some please?

1 Like

Since you’re running in Docker, Caddy will only accept connections coming from inside the Caddy container, since it listens on localhost:2019. You can configure the admin endpoint to :2019 to allow connections from anything, but make sure not to expose that port publicly, otherwise some bad actor could change your server’s config.

1 Like

Yep, and to open the admin port up, you’d want to do something like this in your global options: admin :2019 (but again, be careful who you let access that port into Docker).

Ah of course! :man_facepalming: Long week…
Thanks guys! Will report back soon.

1 Like