Localhost with https for synapse

Help
1

1. The problem I’m having:

My problem is with an appservice for matrix. I need to connect to my synapse from the same machine it’s hosted on with this bot, and I have used caddy for my public facing synapse instance. The issue arises when I try to connect. I first got an SSL timeout error, prompting a 2 day long quest to find out if it was an issue with the appservice, turns out it was my machine!
I tried connecting via localhost, which worked, however, I need to localhost over https because it must specify content length.
Now, I tried to open my public facing domain in the browser, time out. I tried to connect to https://localhost, also timeout, and regular localhost gives me an SSL error.
Curling any of them does, well, nothing at all.
So I think the best solution for the appservice is to use localhost over https, and I can’t figure that out. Unless I’m missing something, how would I do this? I couldn’t find any other forum posts about how to set this up exactly and I don’t really have a caddy specific error, I think at least, it’s just something I can’t find how to do.

2. Error messages and/or full log output:

Timeout when loading my domain on the server in a browser or when curling, and I can't figure out how I localhost on https

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

I followed this tutorial exactly and I haven’t changed anything else except for my matrix config in minor and superficial ways

→

a. System environment:

Ubuntu 22.04.3 LTS
Without docker

b. Command:

Systemd service

c. Service/unit/compose file:

d. My complete Caddy config:

The way he did it in the video

5. Links to relevant resources:

https://www.youtube.com/watch?v=YMj6nb-cm38 (how I installed)

2

I should also note the timestamp where he explains the caddy setup is 10:43 and linked here

3

Content length and HTTPS don’t have anything to do with eachother. I’m not sure what you mean by that.

You said you tried multiple different things. What exactly did you try? Share your actual config.

Show the actual curl commands you’re running and their output. Use curl -v for more details.

4

My apologies
I found out most of the configs I tried weren’t working because I was using the wrong terms (like “tls self_signed”)
Here is my current and in use config:

kc01.com {
  reverse_proxy /_matrix/* localhost:8008
  reverse_proxy /_synapse/client/* localhost:8008
  reverse_proxy localhost:8008
}

kc01.com:8448 {
  reverse_proxy localhost:8008
}

127.0.0.1 {
  tls internal
  reverse_proxy localhost:8008
}

This boots, and in the browser I can do to https://127.0.0.1, however it does that “are you sure you want to proceed”, I can actually access it now whereas before it was a dead timeout or SSL error.
Here is the output of curl:

curl -v https://127.0.0.1
*   Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I get a very similar error when I try to run the matrix appservice as below:

node scripts/seed.js
This could take up to 30 seconds. Please be patient.
✅ Configuration looks good...
✅ Database is ready...
✅ Emojis are ready...

/home/user/out-of-your-element/node_modules/node-fetch/lib/index.js:1505
            reject(new FetchError(request to ${request.url} failed, reason: ${err.message}, 'system', err));
                   ^
FetchError: request to https://127.0.0.1/_matrix/media/v3/upload failed, reason: unable to get local issuer certificate
    at ClientRequest.<anonymous> (/home/user/out-of-your-element/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:526:35)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:514:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY',
  code: 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY'
}

Node.js v20.8.1

I tried following a forum and using mkcert to generate local keys
It did nothing that I could see and above commands had the same output as before

The developer of the appservice bot I’m using told me the synapse codebase has problems with it and I found this relating to it

5

You’ll need to run sudo caddy trust to install Caddy’s root CA cert to your system’s trust store, to get rid of the trust error.

If these are all using the same port, you can remove the first two lines because they all proxy to the same place.

6

I tried this and it seems to work, but the browser still has the same certificate errors as before, and so does the appservice when I try to run it
Here is the new output of curl:

curl -v https://127.0.0.1
*   Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Oct 18 11:22:35 2023 GMT
*  expire date: Oct 18 23:22:35 2023 GMT
*  subjectAltName: host "127.0.0.1" matched cert's IP address!
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55ac3a36be90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: 127.0.0.1
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 302 
< alt-svc: h3=":443"; ma=2592000
< content-type: text/html; charset=utf-8
< date: Wed, 18 Oct 2023 16:14:03 GMT
< location: /_matrix/static
< server: Caddy
< server: Synapse/1.89.0
< content-length: 208
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):

<html>
    <head>
        <meta http-equiv="refresh" content="0;URL=/_matrix/static">
    </head>
    <body bgcolor="#FFFFFF" text="#000000">
    <a href="/_matrix/static">click here</a>
    </body>
</html>
* Connection #0 to host 127.0.0.1 left intact

Interesting, these were originally from the matrix documentation

7

Yeah, I mean, can’t really expect other projects to understand Caddy best-practices. They wrote something that worked for them and documented it as such. But it can be simplified.

Is this all on the same machine? Browsers have their own trust stores, so Caddy’s root CA cert needs to be installed in the browser’s trust store as well. Where is this “appservice” running? Is it in Docker or something? If it uses the same machine’s trust store, it should work fine. I can’t really help debug that though.

Again, I’m not sure why you need to connect to Caddy over HTTPS from the same machine. Why can’t you use HTTP? Set up Caddy to listen on a different port (like 8080 or something) and don’t expose that port outside of that machine, and have your app connect to that port over HTTP.

8

Yes, all the same machine. The appservice is running natively with no docker.
The developer of the bot told me I should really use https, and the bot really does not like http because of the synapse code base being not up to scratch

How would I do this?

9

Are you sure they don’t just mean requests from the outside world should be HTTPS? It’s very much common practice to use a server which terminates TLS, then proxies over HTTP.

Just a site block like :8080 with no hostname.

10

I asked and the developer said the following:
it needs to go via the reverse proxy because the reverse proxy will add the necessary content-length header to help Synapse

Okay, I’ll give that one a go

11

I don’t see what Content-Length has to do with HTTPS. Those are entirely separate concepts.

But either way, I agree using a reverse proxy (like Caddy) to terminate TLS is a great idea. And that means you don’t need to proxy from Caddy to your app over HTTPS, because the vulnerable part of the connection (between the client and your server) was secured by TLS connection to Caddy.

12

Okay I think I understand now, TLS and HTTPS are not the one in the same and can be used separately
Also, if I am listening locally on 8080, what URL do I tell the bot to use? http://localhost:8080 ?

13

Kinda, but not really. HTTPS is HTTP+TLS.

But yes, TLS can be used to encrypt other kinds of traffic than only HTTP (like SMTP for mail, LDAP for connecting to Active Directory and such, etc.)

Yeah.

14

Okay I see

I tried this, and got an error

/home/user/out-of-your-element/node_modules/node-fetch/lib/index.js:1622
                            reject(new FetchError('Cannot follow redirect with body being a readable stream', 'unsupported-redirect'));
                                   ^
FetchError: Cannot follow redirect with body being a readable stream
15

What’s your Caddyfile at this point? Enable the debug global option, what do you see in Caddy’s logs?

16

My caddy file now is like this:

kc01.com {
  reverse_proxy /_matrix/* localhost:8008
  reverse_proxy /_synapse/client/* localhost:8008
  reverse_proxy localhost:8008
}

kc01.com:8448 {
  reverse_proxy localhost:8008
}


:8080 {
  reverse_proxy localhost:8008
  tls internal
}

I tried using http://localhost:8080 with the bot and got this error:

/home/user/out-of-your-element/node_modules/node-fetch/lib/index.js:273
                return Body.Promise.reject(new FetchError(invalid json response body at ${_this2.url} reason: ${err.message}, 'invalid-json'));
                                           ^
FetchError: invalid json response body at http://localhost:8080/_matrix/media/v3/upload reason: Unexpected token 'C', "Client sen"... is not valid JSON
    at /home/user/out-of-your-element/node_modules/node-fetch/lib/index.js:273:32
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.mreq (/home/user/out-of-your-element/matrix/mreq.js:40:15)
    at async Object._actuallyUploadDiscordFileToMxc (/home/user/out-of-your-element/matrix/file.js:68:15)
    at async /home/user/out-of-your-element/matrix/file.js:52:16
    at async /home/user/out-of-your-element/scripts/seed.js:110:20 {
  type: 'invalid-json'
}

When curling:

curl -v http://localhost:8080
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< 
Client sent an HTTP request to an HTTPS server.
* Closing connection 0

I tried then curling with https:

curl -v https://localhost:8080
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Unknown (21):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

I also tried the same config with localhost instead of 8080, and localhost on other ports with the inside of the server block the same, which resulted in the TLS error or content length error from before

How do I do that?

17

Remove this.

The point of what I was suggesting was that you turn off TLS for this one server that is only accessible locally.

With that, you’re forcing TLS to be on, but then you make an HTTP request to a server which is expecting HTTPS requests (i.e. HTTP+TLS) so it fails.

18

Okay I removed the TLS line from my config
The bot gives me the content length error now

node scripts/seed.js
This could take up to 30 seconds. Please be patient.
✅ Configuration looks good...
✅ Database is ready...
✅ Emojis are ready...
/home/user/out-of-your-element/matrix/mreq.js:42
	if (!res.ok || root.errcode) throw new MatrixServerError(root, opts)
	                                   ^

MatrixServerError: Request must specify a Content-Length
    at Object.mreq (/home/user/out-of-your-element/matrix/mreq.js:42:37)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object._actuallyUploadDiscordFileToMxc (/home/user/out-of-your-element/matrix/file.js:68:15)
    at async /home/user/out-of-your-element/matrix/file.js:52:16
    at async /home/user/out-of-your-element/scripts/seed.js:110:20 {
  data: {
    errcode: 'M_UNKNOWN',
    error: 'Request must specify a Content-Length'
  },
  errcode: 'M_UNKNOWN',
  opts: {
    method: 'POST',
    body: PassThrough {
      _readableState: ReadableState {
        state: 325654,
        highWaterMark: 16384,
        buffer: BufferList { head: null, tail: null, length: 0 },
        length: 0,
        pipes: [],
        flowing: false,
        errored: null,
        defaultEncoding: 'utf8',
        awaitDrainWriters: null,
        decoder: null,
        encoding: null,
        [Symbol(kPaused)]: true
      },
      _events: [Object: null prototype] {
        prefinish: [Function: prefinish],
        error: [ [Function (anonymous)], [Function (anonymous)] ]
      },
      _eventsCount: 2,
      _maxListeners: undefined,
      _writableState: WritableState {
        state: 948198,
        highWaterMark: 16384,
        defaultEncoding: 'utf8',
        length: 0,
        corked: 0,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        afterWriteTickInfo: null,
        buffered: [],
        bufferedIndex: 0,
        pendingcb: 0,
        errored: null,
        [Symbol(kOnFinished)]: []
      },
      allowHalfOpen: true,
      [Symbol(kCapture)]: false,
      [Symbol(kCallback)]: null
    },
    headers: {
      Authorization: 'Bearer ECA3AAA27DF2F1F019B7C04386908DD314C01E9A8A739D3A44F03BA8BF487431',
      'Content-Type': 'image/png'
    }
  }
}

Node.js v20.8.1

This happens when I curl the url:

curl -v http://localhost:8080
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Content-Length: 208
< Content-Type: text/html; charset=utf-8
< Date: Sat, 21 Oct 2023 00:45:45 GMT
< Location: /_matrix/static
< Server: Caddy
< Server: Synapse/1.89.0
< 

<html>
    <head>
        <meta http-equiv="refresh" content="0;URL=/_matrix/static">
    </head>
    <body bgcolor="#FFFFFF" text="#000000">
    <a href="/_matrix/static">click here</a>
    </body>
</html>
* Connection #0 to host localhost left intact

How do I see the logs?
I tried adding a block with

log {
   output /etc/caddy/caddy.log
}

And it said unrecognized directive output

19

That seems like a bug in your bot. It should be specifying a Content-Length in its request.

You made an HTTP GET request here, but your bot made an HTTP POST. Not the same.

But anyway, it does show that you reached your app.

See Keep Caddy Running — Caddy Documentation

log is a directive, it can’t go top-level, otherwise Caddy thinks log is a domain you’re trying to configure and output is a directive in that site… but that makes no sense, because output is not a directive, it’s an option for the log directive.

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.