Where is the TLS `.pem` certificate and key on remote EC2 server

1. The problem I’m having:

I am working on my AWS EC2 server and I can’t find the .pem certificate and key files that Caddy uses to create an TLS connection.

I want to set up a WebSocket server on the same EC2 instance using Caddy.
I am using uWebSockets to enable TLS on the uWebSocket server, uWebSockets’ config needs the .pem file for both the TLS certificate and key.

For example, here is my SSLConfig:

// Read SSL/TLS certificate and private key
export const SSL_OPTIONS = {
  key_file_name: 'misc/local-key.pem',
  cert_file_name: 'misc/local-cert.pem',
  passphrase: '1234'  // Optional: Remove if no passphrase is set
}

And here is my uWebSockets server file:

/* src/server.ts */
// Externals
import uWS, { 
  App, 
  WebSocket, 
  WebSocketBehavior 
} from 'uWebSockets.js'
// Locals
import { 
  PORT,
  SSL_OPTIONS,
  IDLE_TIMEOUT, 
  MAX_LIFETIME, 
  MAX_BACKPRESSURE, 
  MAX_PAYLOAD_LENGTH, 
} from './utils'



const wsBehavior: WebSocketBehavior<any> = {
  // What permessage- deflate compression to use `uWS.DISABLED`, 
  // `uWS.SHARED_COMPRESSOR` or any of the `uWS.DEDICATED_COMPRESSOR_xxxKB`.
  // Defaults to `uWS.DISABLED`.
  compression: uWS.SHARED_COMPRESSOR,
  // Maximum number of minutes a WebSocket may be connected before being closed 
  // by the server. 0 disables the feature.
  maxLifetime: MAX_LIFETIME,
  // Maximum length of allowed backpressure per socket when publishing or 
  // sending messages. Slow receivers with too high backpressure will be skipped
  // until they catch up or timeout. Defaults to 64 * 1024
  maxBackpressure: MAX_BACKPRESSURE,
  // Maximum length of incoming message
  maxPayloadLength: MAX_PAYLOAD_LENGTH,
  // Maximum idle time before a connection is closed (in seconds). Disable by 
  // using 0. Defaults to 120.
  idleTimeout: IDLE_TIMEOUT,

  // Method called on each new WebSocket connection
  open: (ws) => {
    console.log('A WebSocket connected')
    ws.subscribe('broadcast')
  },

  // Method called whenever a message is received
  message: async (ws, message: ArrayBuffer, isBinary: boolean) => {
    const msg = Buffer.from(message).toString('utf8')
    console.log(`Received message: ${msg}`)

    // Send the message to your Next.js API route
    try {
      const apiEndpoint = 'https://localhost:3000/api/v1/social-rating/game/wss/local/players'
      const response = await fetch(apiEndpoint, {
        method: 'POST',
        body: msg
      })

      const json = await response.json()

      // Log the response from the Next.js API route
      console.log(`API Response: ${response}`)

      if (response.status === 200) {
        console.log(`API Response: ${response}`)
        ws.publish('broadcast', msg, isBinary)
      }
    } catch (error: any) {
      const errorMessage = 'Failed to send message to API route: '
      console.error(errorMessage, error)
    }
  },

  // Method called when a connection is closed
  close: (ws, code: number, message: ArrayBuffer) => {
    const msg = Buffer.from(message).toString()
    console.log(`WebSocket closed with code: ${code}, message: ${ msg }`)
  }
}

// Main app
const app = uWS./*SSL*/App(SSL_OPTIONS)
  .ws('/*', wsBehavior)
  .get('/*', (res, req) => {
    res.writeStatus('200 OK').end('WebSocket server is running')
  })
  .listen(PORT, (token: any) => {
    if (token) {
      console.log(`Server is listening on port ${PORT}`)
    } else {
      console.log('Failed to listen to port')
    }
  })

2. Error messages and/or full log output:

Upon reviewing the Caddy logs for tls, I see:

tls cleaning storage unit {"storage": "FileStorage:/root/.local/share/caddy"}

Here’s a screenshot of the log for clarity:

I tried to access that file path but the EC2 instance won’t let me.

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

  • Installed by running:
    caddy upgrade
    
  • Ran caddy by running
    sudo caddy run --config /etc/caddy/Caddyfile
    

a. System environment:

  • OS:
    • Amazon Linux 2023
  • AMI ID:
    • ami-00beae93a2d981137
  • Size:
    • t2.micro
  • Docker version:
    • Docker version 25.0.5, build 5dc9bcc

b. Command:

  1. To search for directories with the caddy keyword, I ran:

    sudo find / -type d -name "caddy"
    
  2. To start the Caddy server, I ran the usual:

    sudo caddy run --config /etc/caddy/Caddyfile
    

c. Service/unit/compose file:

d. My complete Caddy config:

canpersonalitychange.com {
        encode gzip
        header {
                Strict-Transport-Security "max-age=31536000;"
                Access-Control-Allow-Origin "*"
        }
        reverse_proxy localhost:3000
}

You can just proxy websockets through Caddy. That way Caddy terminates all TLS for you.

Probably because it’s owned by root and you’re trying with a non-root user?

You should probably be running Caddy as a systemd service though. Install with these instructions:

Then follow this to use it:

1 Like

How do I proxy websockets through Caddy?

Just to be clear, I want to ensure that my WebSocket connection has TLS enabled.

Here’s what my Caddyfile looks like now:

 https://example.com  {
         encode gzip
         header {
                 Strict-Transport-Security "max-age=31536000;"
                 Access-Control-Allow-Origin "*"
         }

         # Main application
         reverse_proxy localhost:3000

         # WebSocket
         reverse_proxy localhost:3001
 }

Do I need to make any other changes to proxy the WebSocket server through Caddy and ensure the WebSocket uses TLS?

Or is this it and I’m done?

Caddy’s reverse_proxy supports Websockets.

If you want to serve both on the same port, you need to use request matchers to split the traffic. In this case, using the request headers to detect a websocket connection:

example.com {
	# Websockets
	@ws `header({'Connection': '*Upgrade*', 'Upgrade': 'websocket'})`
	reverse_proxy @ws localhost:3001
  
	# App
	reverse_proxy localhost:3000
}

That example is taken from here Request matchers (Caddyfile) — Caddy Documentation

Nope. Just use wss:// in your client.

1 Like

So, within my EC2 instance, when I start my Next.js app by running its Docker container, which I do with the following command:

docker run -it -p 3000:3000 <IMAGE_ID>

and then start the Caddy server (which is NOT being run within in a container), I experience no issues; I can see the web app being served on canpersonalitychange.com.

This is what my Caddyfile looks like (which is located at /etc/caddy/Caddyfile:

canpersonalitychange.com {
        # App
        reverse_proxy localhost:3000

        # WebSocket
        @ws `header({'Connection': '*Upgrade*', 'Upgrade': 'websocket'})`
        reverse_proxy @ws localhost:3001
}

Within the same EC2 instance but on another screen (I am using screen so that I can switch between multiple services and programs on the same EC2 instance), I start the uWebSocket server by running its Docker container.

To do that I run:

docker run -it -p 3001:3001 <IMAGE_ID>

Then, I try to connect to the websocket Docker container by running:

npx wcat -c wss://0.0.0.0:3001/

I get an unhelpful Node error:

node:events:497
      throw er; // Unhandled 'error' event
      ^

Error: ENOENT: no such file or directory, open '-c'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '-c'
}

Node.js v20.17.0

I get also get this error when trying to connect to ws://localhost:3001/ too.

How do I connect to the websocket if it’s in a Docker container?

My expectation was that connecting to the websocket server would be as simple as serving the Docker container of the Next.js server that holds my web app.

Shouldn’t it be as simple as starting the Next.js server on localhost:3000?
Or am I missing something?

You’re not connecting through Caddy at all by doing this.

I don’t know anything about wcat so I can’t help with that. You probably need to do ws://127.0.0.1:3001 instead to prove that it works over unencrypted websockets.

If you want to connect through Caddy, with TLS, you would do:

npx wcat -c wss://canpersonalitychange.com
1 Like

Sorry, the unhelpful Node error above is from me using the wrong package, wcat, instead of the correct package, wscat.

Anyway, I fixed that and also was able to locate the TLS certificate and private key files (within /var/lib/caddy/.local/share/caddy/certificates/) and now, the persistent issue is that any attempts to connect to the uWebSocket hang until they timeout.

I moved the uWebSocket from the same EC2 instance that the Next web app is being served on and to its own EC2 instance. This second EC2 instance that the uWebSocket server is on is also being served with Caddy. It is at https://schuage.com. You can try to connect to it with wscat -c wss://schuage.com:3001, but again, the ‘timing out’ issue persists, so you will see that the attempt to connect times out.

I opened a GitHub issue with the uWebSocket.js repository so that I can hopefully get a better understanding at what this ‘timing out’ issue might be.

Fixed the issue. See the GitHub issue

Wow, they were incredibly rude. I’m so sorry.

I was busy today so I wasn’t able to take a look earlier, but glad you figured it out. But yeah, TLS is between the browser and Caddy, then Caddy’s reverse_proxy does an unencrypted connection to your backend. So yeah, no need to set up TLS on the backend, no need to touch Caddy’s certs at all.

4 Likes

Straight up abusive IMO. :broken_heart: I’ve permanently blocked those two from the caddyserver org on GitHub.

Thanks for helping. :100:

2 Likes

Yeah, they were acting sort of like trolls.

The maintainer even deleted the original issue, so I thought I’d archive it with this post here and share the screenshots of their condescending responses in the discussion:





They appear to be a lot more focused on their capital gains made via cryptocurrency than helping out their own users, especially since most of uWebSockets’ revenue comes from " BitMEX, Bitfinex, […], Coinbase, [and] Bitwyre"

I worked in that crypto-industry before and I eventually had enough of their elitist demeanor all the time, so I left.

3 Likes

Thanks for the record keeping. I also saved a screenshot of the original thread to my computer in case things ever arise again.

2 Likes

I want to say thank you to @francislavoie for joining on that GitHub-issue-thread and thank you a ton for all the help you gave me with this issue!

And thank you @matt for authoring Caddy! It makes things a lot simpler to manage, even if I have added documentation to my university project for both Caddy and NGINX :smiley:

4 Likes

Thanks for the kind words :slight_smile:

2 Likes