Caddy service exits with code 0 in Cloud Run

Help
1

1. Output of caddy version:

Using the caddy:alpine docker image.

2. How I run Caddy:

I run it in a Cloud Run environment, using the following Cloud Build yaml file:

steps:
  # Obtain cached images for Container Registry
  # - id: "Caching (1/2) pull build image"
  - id: "Pull Cache: build"
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
    - '-c'
    - |
      docker pull us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:latest || true
  - id: "Update Image: latest"
    name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args:
    - '-c'
    - |
      docker build \
      -t us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:$COMMIT_SHA \
      -t us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:latest \
      --cache-from us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:latest . \
      && \
      docker push us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:latest \
      && \
      docker push us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:$COMMIT_SHA
    dir: staging
  # Deploy container image to Cloud Run
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args:
    - 'run'
    - 'deploy'
    - '${_SERVICE_NAME}'
    - '--image'
    - 'us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:$COMMIT_SHA'
    - '--region'
    - 'us-central1'
images:
    - 'us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:$COMMIT_SHA'
    - 'us-central1-docker.pkg.dev/$PROJECT_ID/geyser-docker-repo/${_SERVICE_NAME}:latest'
timeout: 1800s

My complete Caddy config:

{
    debug
    auto_https off
}

auth.staging.geyser.fund {
        reverse_proxy https://api.staging.geyser.fund
}

3. The problem I’m having:

The Cloud Build deploy succeeds, but the Caddy server exits almost immediately after starting up, with no apparent error in the logs.

4. Error messages and/or full log output:

2022-08-15T20:35:14.160609Z{address: tcp/localhost:2019, enforce_origin: false, level: info, logger: admin, msg: admin endpoint started, origins: […], ts: 1660595714.1604772}
2022-08-15T20:35:14.161042Z{level: warn, logger: http, msg: automatic HTTPS is completely disabled for server, server_name: srv0, ts: 1660595714.1609643}
2022-08-15T20:35:14.161447Z{address: [::]:443, http3: false, level: debug, logger: http, msg: starting server loop, tls: true, ts: 1660595714.1613967}
2022-08-15T20:35:14.162288Z{file: /config/caddy/autosave.json, level: info, msg: autosaved config (load with --resume flag), ts: 1660595714.1622367}
2022-08-15T20:35:14.162303Z{level: info, msg: serving initial configuration, ts: 1660595714.1622536}
2022-08-15T20:35:14.163170Z{cache: 0xc0002541c0, level: info, logger: tls.cache.maintenance, msg: started background certificate maintenance, ts: 1660595714.1631124}
2022-08-15T20:35:14.163526Z{description: FileStorage:/data/caddy, level: info, logger: tls, msg: cleaning storage unit, ts: 1660595714.1634383}
2022-08-15T20:35:14.163877Z{level: info, logger: tls, msg: finished cleaning storage units, ts: 1660595714.163831}
2022-08-15T20:35:14.184574Z{level: debug, logger: http.stdlib, msg: http: TLS handshake error from 169.254.1.1:62304: EOF, ts: 1660595714.1844828}
2022-08-15T20:35:14.184817Z{level: debug, logger: http.stdlib, msg: http: TLS handshake error from 169.254.1.1:40745: EOF, ts: 1660595714.184773}
2022-08-15T20:35:14.215556Z{level: debug, logger: http.stdlib, msg: http: TLS handshake error from 169.254.1.1:51555: EOF, ts: 1660595714.2153747}
2022-08-15T20:35:14.215575Z{level: debug, logger: http.stdlib, msg: http: TLS handshake error from 169.254.1.1:27691: EOF, ts: 1660595714.215505}
2022-08-15T20:35:17.601078ZCloud Rungeyser-caddy-lb-00019-vog {@type: type.googleapis.com/google.cloud.audit.AuditLog, resourceName: namespaces/geyser-mvp-staging/revisions/geyser-caddy-lb-00019-vog, response: {…}, serviceName: run.googleapis.com, status: {…}}
2022-08-15T20:35:18.002849ZCloud Rungeyser-caddy-lb {@type: type.googleapis.com/google.cloud.audit.AuditLog, resourceName: namespaces/geyser-mvp-staging/services/geyser-caddy-lb, response: {…}, serviceName: run.googleapis.com, status: {…}}
2022-08-15T20:36:12.695256Z{level: info, msg: shutting down apps, then terminating, signal: SIGTERM, ts: 1660595772.6951368}
2022-08-15T20:36:12.695278Z{level: warn, msg: exiting; byeee!! đź‘‹, signal: SIGTERM, ts: 1660595772.6951723}
2022-08-15T20:36:12.697730Z{cache: 0xc0004e8d90, level: info, logger: tls.cache.maintenance, msg: stopped background certificate maintenance, ts: 1660595772.6976595}
2022-08-15T20:36:12.699824Z{address: tcp/localhost:2019, level: info, logger: admin, msg: stopped previous server, ts: 1660595772.6997597}
2022-08-15T20:36:12.699841Z{exit_code: 0, level: info, msg: shutdown complete, signal: SIGTERM, ts: 1660595772.699786}

Thank you in advance for any help!

2

Hi :wave:

Gcloud Run already terminates tls for you, if I am not mistaken.
auto_https off will still make Caddy serve your vhosts over https:// and :443:.
You will have to prefix your vhost with http:// to force http:// and :80.

E.g.

http://auth.staging.geyser.fund {

# instead of
auth.staging.geyser.fund {

Also, I don’t know how sophisticated Gcloud Run’s automatic port discovery is, so you may need to add --port=80 to your gcloud run deploy, or instruct Caddy to use :8080 (default Cloud Run port):

http://auth.staging.geyser.fund:8080 {
1 Like
3

Hi IndeedNotJames!

Thanks for your response. I will give this solution a try when I get some time. Could you indicate which part of the logs is hinting at this issue? I would also appreciate any additional explanation on the reason Caddy shuts down in that case.

Again, thanks for your time!

4

Ye, sorry, my response was written in a bit of a rush.

Essentially:

Caddy received SIGTERM from something, presumably CloudRun.
SIGTERM is basically your good ol’ CTRL + C in a terminal.
Or, for example, if you run Caddy in a Docker Container, then a docker stop $container would yield the same POSIX signal.

POSIX signal handling seem to be currently missing in the Caddy documentation, but I think I saw the lack of signal handling documentation mentioned on the private slack lately. Not sure though :thinking:

The relevant code is in

Anyway…

From

you can see that Caddy is listening on tcp/localhost:2019 (admin endpoint) and [::]:443 (tls: true → https://).

[::] is the IPv6 representation for “listen on all IPv4 and IPv6 interfaces” fyi

CloudRun terminates https:// for you, that’s mentioned somewhere in the CloudRun docs.

Your debug logs suggest that something is trying to connect from 169.254.1.1:

Such TLS handshake errors may happen due to a simple “is the port open”-check made by CloudRun.
Those aren’t error you need to worry about :innocent: (and will be hidden as soon as you remove the debug from the global options in your Caddyfile)

Which brings me to:
I just assumed that, not only does your Caddy

but it’s also just straight up not serving any requests when running.

If it is serving requests just fine, then some assumptions I made about CloudRun might be veeery wrong and your question might just have been about the usual lifecycle of a CloudRun container instead :exploding_head:

2 Likes
5

Amazing detailed answer! That’s helped me understand the issue and logs a lot better.

Regarding this:

I think it’s fair to assume Cloud Run is not shutting down the container due to idling as I’ve instructed it to keep at least one instance running at all times.

I’ve had to move forward with the Cloud Load Balancing for now, but I will come back to this issue at a later stage and try getting it to work. I can report back on it then!

Hopefully in the meantime that helps some other community members :slight_smile:

1 Like
6

This topic was automatically closed after 30 days. New replies are no longer allowed.