Trying to find where TCP requests turn into HTTP requests in code

I’ve been putzing around in the code base of v2 for a few days getting a feel for how it’s put together (the module system is pretty cool!)

One thing I can’t find is where an Accept()'ed connection parses the incoming data and turns it into an HTTP request. I’m thinking this is handed off to the stdlib, but I just can’t seem to find that.

I’m talking about the reverse proxy here, although this question sort of pertains to all of Caddy v2

Another way to word that question is: Where’s the very top of the middleware/module stack where ServeHTTP is first called with a generated *http.Request object?

I can’t seem to find where a server calls the equivalent to ListenAndServe(), although it looks like it’s part of the listen.go and listeners.go files. Listeners are generated dynamically and then they Listen() but I’m having trouble finding the Serve() equivalent that might pass a handler and get back a *http.Request object.

This is mostly just to satisfy my curiosity :smiley: I’d love any help clarifying what I’m missing (there’s likely a gap in my knowledge).

2 Likes

A noble pursuit!

If you go to https://pkg.go.dev and look up the net/http package, you’ll find the Serve() method on Server. Click on the method name and you’ll be taken into the code:

https://cs.opensource.google/go/go/+/go1.19.1:src/net/http/server.go;l=3040

Here is where connections are accepted from the listener:

https://cs.opensource.google/go/go/+/refs/tags/go1.19.1:src/net/http/server.go;l=3070

serve() is called in a goroutine before accepting the next connection:

https://cs.opensource.google/go/go/+/refs/tags/go1.19.1:src/net/http/server.go;drc=ffdfa9ff41a83fe1265c48b93381a8fa302cf480;l=1830

Here the stdlib diverges between HTTP/1 and HTTP/2. Let’s follow HTTP/1 since it’s simpler:

Here is where the HTTP/1 request is read from the wire:

https://cs.opensource.google/go/go/+/refs/tags/go1.19.1:src/net/http/server.go;l=1904;drc=ffdfa9ff41a83fe1265c48b93381a8fa302cf480

As you can see it’s in a for loop, this is because multiple requests can come over the same connection sequentially.

Anyway, you can click through that to see how it works.

In Caddy? Right here is the entry point of HTTP requests:

In Go, this is called here:

https://cs.opensource.google/go/go/+/refs/tags/go1.19.1:src/net/http/server.go;l=1979;drc=ffdfa9ff41a83fe1265c48b93381a8fa302cf480

4 Likes

Hey Matt - thanks for taking the time to answer (and so thoroughly with stdlib examples!)

Yep, I’m asking about in Caddy specifically. I did find that in server.go, so I think I’m just missing the steps in my mental model of how it’s put together (both in the stdlib http server, and also in Caddy’s inner workings).

I see what you mean in stdlib: serverHandler{c.server}.ServeHTTP(w, w.req) - that’s grabbing the Server/Handler and calling ServeHTTP against it. Make sense!

I think that finally put it together then, I was missing that Caddy’s server object wrapped http.Server. Finding the App’s Start() method lead me to find srv.server.Serve().

(Of course Caddy is doing a lot of other work but I think this is the core of what I was trying to find in this case).

Thanks again!

2 Likes

Fancy seeing you here @fideloper, I learned a lot from your Docker content some years ago, which was right at the same time as I was learning about Caddy and started getting involved with the project :blush: I keep seeing your name pop up in my composer updates (memorable username, not gonna lie). So just wanted to say thanks for what you do :+1:

2 Likes

haha that’s great, hi!

My Laravel package was brought into the core of Laravel so you’ll see of it there in composer - it was fun seeing it there but I’ll enjoy not maintaining that package :joy:

2 Likes

Yep, I saw that! I happened to have implemented Caddy’s trusted_proxies functionality semi-recently: reverse_proxy (Caddyfile directive) — Caddy Documentation

1 Like

The Internet makes the world smol.

2 Likes

oh cool!

trusted proxies is deceptively hard/weird (or maybe just annoying!) with making sure you get the right IP from a list of IP’s and so many people using hosting environments with changing or unknown IP’s :joy:

1 Like

Absolutely. Adam Pritchard wrote about it in depth recently:

https://adam-p.ca/blog/2022/03/x-forwarded-for/

2 Likes

This topic was automatically closed after 60 days. New replies are no longer allowed.