Error "TLS alert, internal error (592)" (disappeared after Caddy restart)

Help
1

1. Caddy version (caddy version):

2.3.0

2. How I run Caddy:

a. System environment:

Ubuntu 18.04.5 LTS
systemd
baremetal
installed package from deb repo https://dl.cloudsmith.io/public/caddy/stable/deb/debian

b. Command:

systemctl start caddy.service

c. Service/unit/compose file:

cat /lib/systemd/system/caddy.service
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{

    #Bind admin endpoint on all interfaces
    admin 0.0.0.0:2019

    on_demand_tls {
        ask http://localhost:8000/api/cname/validate
    }

    # Workaround to prevent on_demand being enabled on all domains
    # see https://github.com/caddyserver/caddy/pull/4128
    email me@company.co
}


# Redirect any accesses using an IP instead of a domain name
# hackerone report: https://hackerone.com/reports/723126
51.91.128.49:80,
91.121.37.140:80,
147.135.229.201:80
{
    redir https://app.lemlist.com{uri}
}

(cloudflare_matchers) {
    remote_ip 103.21.244.0/22
    remote_ip 103.22.200.0/22
    remote_ip 103.31.4.0/22
    remote_ip 104.16.0.0/13
    remote_ip 104.24.0.0/14
    remote_ip 108.162.192.0/18
    remote_ip 131.0.72.0/22
    remote_ip 141.101.64.0/18
    remote_ip 162.158.0.0/15
    remote_ip 172.64.0.0/13
    remote_ip 173.245.48.0/20
    remote_ip 188.114.96.0/20
    remote_ip 190.93.240.0/20
    remote_ip 197.234.240.0/22
    remote_ip 198.41.128.0/17
    remote_ip 2400:cb00::/32
    remote_ip 2606:4700::/32
    remote_ip 2803:f800::/32
    remote_ip 2405:b500::/32
    remote_ip 2405:8100::/32
    remote_ip 2a06:98c0::/29
    remote_ip 2c0f:f248::/32
}

(front_servers) {
    to localhost:42001
    to localhost:42002
    to localhost:42003
    to localhost:42004
    to localhost:42005
    to localhost:42006
    to localhost:42007
    to localhost:42008

    health_path /api/ping
    health_interval 1s
    health_timeout 5s
}

#
# Main application
#

app.lemlist.com
{
    log {
        level INFO
        output file /var/log/caddy/front.log {
                roll_size     100MiB
        }
        format console
    }

    # route keyword is mandatory to force Caddy to keep our
    #   directives order instead of using the Caddy builtin order
    route {

        request_body {
            max_size 20MB
        }

        @from_cloudflare {
            import cloudflare_matchers
        }

        @not_from_cloudflare {
            not {
                import cloudflare_matchers
            }
        }

        ##############################################
        # pass all api lempire (e, i) requests to one
        #   dedicated Meteor app process
        ##############################################
        @api_e_i {
            #path /campaigns/*
            path /api/lempire/*
        }

        header @api_e_i {
            X-Cache-Debug "app.api_e_i"
            Server ""
        }

        reverse_proxy @api_e_i {
            to localhost:8000
        }

        ##############################################
        # pass all api requests to Meteor api process
        ##############################################
        @api {
            #path /campaigns/*
            path /api/*
        }

        header @api {
            X-Cache-Debug "app.api"
            Server ""
        }

        reverse_proxy @api {
            to localhost:42100
        }

        ##############################################
        # Caches
        ##############################################
        # must be after /api routes or https://app.lemlist.com/api/froala/files/*.jpg will not be served by meteor
        @cache_app {
            path *.svg *.woff2 *.eot *.ttf *.woff *.jpg *.jpeg *.png *.gif *.ico *.html
        }

        header @cache_app {
            # 1d expiration
            Cache-Control max-age=86400
            X-Cache-Debug "app.cache"
            Server ""
        }

        file_server @cache_app {
            root /opt/lemlist/app/programs/web.browser/app
        }

        @cache_app2 {
            path_regexp ^/lib/.*\.(map|js|css)$
        }

        header @cache_app2 {
            # 1d expiration
            Cache-Control max-age=86400
            X-Cache-Debug "app.cache2"
            Server ""
        }

        file_server @cache_app2 {
            root /opt/lemlist/app/programs/web.browser/app
        }


        @cache_version {
            path_regexp ^/[0-9a-z]+\.(js|css|map).*
        }

        header @cache_version {
            # 1d expiration
            Cache-Control max-age=86400
            X-Cache-Debug "app.cache_version"
            Server ""
        }

        file_server @cache_version {
            root /opt/lemlist/web.browser
        }


        ##############################################
        # Anything else goes to Meteor fronts
        ##############################################

        header * {
            X-Cache-Debug "app.main"
            Server ""
        }

        reverse_proxy @from_cloudflare {
            header_up X-Real-IP {http.request.header.CF-Connecting-IP}
            header_up X-Forwarded-For {http.request.header.CF-Connecting-IP}

            import front_servers

            lb_policy   header CF-Connecting-IP
        }

        reverse_proxy @not_from_cloudflare {

            import front_servers

            lb_policy   ip_hash
        }

        handle * {
            respond 404
        }
    }

    handle_errors {
        @errors {
            expression {http.error.status_code} == '502'
        }
        rewrite @errors /{http.error.status_code}.html
        file_server {
            root /var/www
        }
    }

    #dev only : to work on laptop localhost
    #tls internal
}

#
# Custom domains
#
(custom_domains) {
    log {
        level INFO
        output file /var/log/caddy/custom_{args.0}.log {
            roll_size     100MiB
        }
        format console
    }

    route {
        #######################################
        # render a special page when on root so the user can see if
        #   its custom domain is correctly setted up
        #######################################
        header / {
            X-Cache-Debug "custom.cname_ok"
            Server ""
        }

        rewrite / /lemlist_cname_ok.html

        file_server /lemlist_cname_ok.html {
            root /var/www
        }

        #######################################
        # pass all requests "track open/click" to track
        #######################################
        @track {
            path_regexp ^/api/(t/c|track/open|reply)/
        }

        header @track {
            X-Cache-Debug "custom.track"
            Server ""
        }

        reverse_proxy @track {
            to localhost:42160
        }

        #######################################
        # pass all requests "images + pages" to images
        #######################################

        @images {
            path /*
        }

        header @images {
            X-Cache-Debug "custom.images"
            Server ""
        }

        reverse_proxy @images {
            to localhost:42140
        }
        handle * {
            respond 404
        }

    }
}

# WARNING : DO NOT add any other site address than :443 here
:443
{
    import custom_domains on_demand

    tls {
        on_demand

        issuer acme {
            email me@company.co
            dir https://acme.zerossl.com/v2/DV90
            eab foo bar
        }

        issuer acme {
            email me@company.co
            dir https://acme-v02.api.letsencrypt.org/directory
        }
    }
}

pages.lemlist.com,
zr3.lemlst.org,
lemlst.org,
lemlst.eu,
lmlst.com
{
    import custom_domains predefined
}

# WARNING : DO NOT add any other site address than :80 here
:80
{
    import custom_domains port80
}

#
# public direct access to service
#
(public_passthrough) {
    {args.0}
    {
        log {
            level INFO
            output file /var/log/caddy/public_passthrough/{args.0}.log {
                roll_size     100MiB
            }
            format console
        }

        route {
            request_body {
                max_size 20MB
            }


            header * {
                X-Cache-Debug "{args.0}.main"
                Server ""
            }

            reverse_proxy * {
                to localhost:{args.1}
            }
        }
    }
}
import public_passthrough api.lemlist.com 42100
import public_passthrough img.lemlist.com 42150
# https://front.lempire.co -> admin
import public_passthrough front.lempire.co 8000


#
# private direct access to services
#
(private_passthrough) {
    front.lempire.co:{args.0}
    {
        log {
            level INFO
            output file /var/log/caddy/private_passthrough/{args.0}.log {
                roll_size     100MiB
            }
            format console
        }

        route {
            @cache_version {
                path_regexp ^/[0-9a-z]+\.(js|css|map).*
            }

            header @cache_version {
                # 1d expiration
                Cache-Control max-age=86400
                X-Cache-Debug "direct.cache_version"
                Server ""
            }

            file_server @cache_version {
                root /opt/lemlist/web.browser
            }

            header * {
                X-Cache-Debug "direct.main"
                Server ""
            }

            reverse_proxy * {
                to localhost:{args.1}
            }
        }

        basicauth * {
            foo bar
        }
    }
}

# Fronts
import private_passthrough 43001 42001
import private_passthrough 43002 42002
import private_passthrough 43003 42003
import private_passthrough 43004 42004
import private_passthrough 43005 42005
import private_passthrough 43006 42006
import private_passthrough 43007 42007
import private_passthrough 43008 42008

# API
import private_passthrough 43100 42100
# Cron
import private_passthrough 43130 42130
# Images
import private_passthrough 43140 42140
# Images Front
import private_passthrough 43150 42150
# Track
import private_passthrough 43160 42160

3. The problem I’m having:

Hello,

I got this TLS issue with only one domain

curl -k -vvv https://img.lemlist.com
*   Trying 147.135.229.201...
* TCP_NODELAY set
* Connected to img.lemlist.com (147.135.229.201) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, internal error (592):
* error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error

At the same time I made a curl to api.lemlist.com with no issue at all
After a restart the issue disappeared.

Service was up since 2021-04-20 18:16:01
Service was restarted the 2021-05-06 14:58:29

Does it ring a bell to you ?

4. Error messages and/or full log output:

Unfortunately I found nothing in syslog, so I got no idea when issue started.
In Caddy journalctl I only found this line related to the domain:

May  5 04:03:57 front caddy[7314]: {"level":"info","ts":1620180237.058937,"logger":"tls.cache.maintenance","msg":"advancing OCSP staple","identifiers":["img.lemlist.com"],"from":1620480028,"to":1620645628}

5. What I already tried:

Googled, found some sni related topics, but nothing very helpful
Reload caddy service
Restart caddy service

6. Links to relevant resources:

2

Enabling debug mode in your Caddyfile will reveal what the internal error was. I’m guessing Caddy didn’t have a certificate for img.lemlist.com because it doesn’t appear as a site address in your Caddyfile.

3

Thank for your quick reply
img.lemlist.com is actually a site address, declared using import.

(public_passthrough) {
    {args.0}
    {
        log {
            level INFO
            output file /var/log/caddy/public_passthrough/{args.0}.log {
                roll_size     100MiB
            }
            format console
        }

        route {
            request_body {
                max_size 20MB
            }


            header * {
                X-Cache-Debug "{args.0}.main"
                Server ""
            }

            reverse_proxy * {
                to localhost:{args.1}
            }
        }
    }
}
import public_passthrough api.lemlist.com 42100
import public_passthrough img.lemlist.com 42150

But I agree the Caddyfile is quite huge, and not easy to read.

From the service log we can see Caddy was checking for the certificate TTL a few hours earlier, and said it was ok. I just checked, and the expiration date is in 1.5 month (22 of june) and was not renewed or generated when I restarted Caddy.

Honestly this post is kind of a “message in a bottle” :slight_smile: In case anyone got (or will get) same issue. As it occurred only once since we use Caddy (4 months), and this may never happen again.

a couple of last questions :

I’m not sure, is it safe to enable debug mode in production ?
BTW do you know were the error code 592 came from ? It can be a lead to my investigations.

4

Yeah, all it does is emit more logs (sets the log level to debug). You can turn it off then clear out the journal afterwards if you have concerns about any of the data in them.

5

This topic was automatically closed after 30 days. New replies are no longer allowed.