Proxy to my shinobi.video service asking to download gzip

Namekal · August 29, 2017, 5:46pm

Hi, I’m new to this as i just installed this last night, and was able to reverse proxy all my usual services just fine. I decided to add another site/service to proxy over (shinobi). I know this site uses websockets somehow so I added websocket to it.

I have tried this with and without gzip, and I have different results between the two. Without gzip it just shows the login form without any css styling or pictures, and trying to login gives me an internal server error.

You can see that I have the proxy through evolved.site/shinobi and surv.evolved.site to see if it changed anything by trying to access both ways. When trying to access either, it will ask to download the gzip of the site.

A friend of mine said its possible there’s some sort of MIME-type issue thinking it should download instead of read out/print (or whatever you would call it).

I’m just at a loss now since some of the documentation confuses me because i haven’t worked with webserver directives/arguments/etc…

Here is my Caddyfile:

evolved.site {
    root /var/www
    proxy /shinobi cent1:8080 {
          transparent
          websocket
    }
    gzip
    tls {
       dns cloudflare
    }
}



#Subdomains -------

surv.evolved.site {
gzip
proxy / cent1:8080 {
        transparent
        }
}

sab.evolved.site {
gzip
proxy / flareon:8081 {
        transparent
        }
}

couch.evolved.site {
gzip
proxy / fed1:5050 {
        transparent
        }
}


sonarr.evolved.site {
gzip
proxy / fed1:8989/ {
        transparent
        }
}

graf.evolved.site {
gzip
proxy / netmon:3000 {
        transparent
        }
}

astei · August 29, 2017, 8:52pm

Yes, when I try to access surv.evolved.site, I do get a gzip file, but it is in fact plain text. It looks as if your web server (shinobi) is reporting the wrong MIME type.

Perhaps you can try stripping out the backend server’s incorrect MIME using a Caddy proxy directive like so:

    proxy /shinobi cent1:8080 {
          transparent
          websocket
          header_downstream -Content-Type ""
    }

Namekal · August 29, 2017, 9:24pm

Thanks for checking. I apologize if trying to access evolved.site/shinobi wasn’t working, as i had removed it in the meantime but left surv.evolved.site in the file. I’m glad you tried surv.evolved.site to find the same result as me.

I tried what you said and added the header_downstream line. The issue still stands after changing and restarting Caddy.

Whitestrake · August 30, 2017, 12:06am

Wouldn’t stripping the content type leave the browser completely without context, defaulting to plain text anyway? What happens to any other kind of content, like images?

I think the only solution really is to ensure that the back end server is reporting the correct MIME type. I don’t think Caddy is the right place in the stack to try to inspect and correct problems of this nature.

matt · August 30, 2017, 4:27am

Right; it was a good experiment to try, but I think this is a bug in the backend they’ll have to fix.

slightfoot · August 30, 2017, 8:00am

Well, I’d suggest the problem is a simple one. I think the backend service (your cent1:8080) is doing client inspection to determine what to send back. This is perfectly normal for content encoding. The client would normally send a Accept-Encoding header. Which would tell the service that the content can be served using gzip. My thought would be that since this header is not in the request the backend service webserver who has been given a gzip response has no other option but to replace the Content-Type header with application/x-gzip.

So, try changing your Caddyfile to this.

surv.evolved.site {
	proxy / cent1:8080 {
		header_upstream Accept-Encoding {>Accept-Encoding}
		header_upstream Accept {>Accept}
		transparent
	}
}

Question: Is shinobi video a CCTV type IPCamera thingy?

P.S. @matt it might be worth noting that transparent doesn’t map the all the request headers across. Kinda feels like it should? or at least we should have a preset for that? Maybe just one for all the Accept headers?
https://github.com/mholt/caddy/blob/master/caddyhttp/proxy/upstream.go#L369

slightfoot · August 30, 2017, 8:16am

I just performed a test to realise all the request headers are sent over. Of course. Need a coffee. Anyway, perhaps removing the accept headers will help.

localhost:80 {
	proxy / localhost:8080 {
		header_upstream -Accept-Encoding
		header_upstream -Accept-Language
		header_upstream -Accept
		transparent
	}
}

Tested with:

package main

import (
	"net/http"
	"fmt"
	"log"
)

func main() {
	http.HandleFunc("/", rootFunc)
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func rootFunc(w http.ResponseWriter, r *http.Request) {
	w.Header().Add("Content-Type", "text/plain")
	for k, v := range r.Header {
		fmt.Fprintf(w, "%v = %v\n", k, v)
	}
}

@matt I noticed Accept-Encoding = [gzip] on the output. All other accept headers have been removed. I think this is being added someplace because Caddy itself can accept gzip content from the upstream. Perhaps this is a bug in of itself, probably unrelated to this topic.

Namekal · August 30, 2017, 2:34pm

Yes it is.

I just tried the suggestions on both of your both posts and having no change in result, it is still asking to download.

Current file (omitting non-related subdomains):

evolved.site {
    root /var/www
    proxy /shinobi cent1:8080 {
        transparent
        header_upstream -Accept-Encoding
        header_upstream -Accept-Language
        header_upstream -Accept
          }
    gzip
    tls {
       dns cloudflare
    }
    log /var/log/caddy/access.log
    errors /var/log/caddy/errors.log
}



#Subdomains -------

surv.evolved.site {
gzip
proxy / cent1:8080 {
        header_upstream Accept-Encoding {>Accept-Encoding}
        header_upstream Accept {>Accept}
        transparent
        }
    log /var/log/caddy/access.surv.log
    errors /var/log/caddy/errors.surv.log
}

I loaded up cent1:8080 locally to show the page info if this helps any:

Namekal · September 25, 2017, 7:20pm

So I’ve found that a couple people got theirs to work behind nginx and apache. How would I translate the following into the Caddyfile?

Apache user:

If someone tries to setup apache mod_proxy with Shinobi: I found the following apache2.4 config to work:
ProxyRequests On
    ProxyPass /socket.io/ ws://localhost:8080/socket.io/
    ProxyPassReverse /socket.io/ ws://localhost:8080/socket.io/

    ProxyPass "/" "http://localhost:8080/"
    ProxyPassReverse "/" "http://localhost:8080/"

nginx user 1:

server   {
    listen 443 ssl;
    server_name my.domain;

    ssl_certificate /config/keys/letsencrypt/fullchain.pem;
    ssl_certificate_key /config/keys/letsencrypt/privkey.pem;
    ssl_dhparam /config/nginx/dhparams.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
      ssl_ciphers removed
      ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
      ssl_session_cache shared:SSL:10m;
      ssl_session_tickets off; # Requires nginx >= 1.5.9
      ssl_stapling on; # Requires nginx >= 1.3.7
      ssl_stapling_verify on; # Requires nginx => 1.3.7
      add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
      add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
    
   location / {
        proxy_pass http://10.0.0.10:8083/;
        proxy_redirect off;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-NginX-Proxy true;
        proxy_set_header X-Real-IP $remote_addr;
      }
}

nginx user 2:

Hey all, just wanted to share a solution to a problem I was having that led me here. I had the same issue as @alistek with getting Shinobi to render behind nginix. It would work just fine when hitting it on the local server directly but using a reverse proxy caused the browser to render the html as text. The fix was to add “add_header X-Frame-Options “SAMEORIGIN”;” to the nginx config. So my final nginx config for shinobi looks like this:
location / {
             try_files $uri/ @nodejs;
       }
       location @nodejs {
               add_header X-Frame-Options "SAMEORIGIN";
               proxy_pass       http://<host>:<port>;
               proxy_http_version 1.1;
               proxy_set_header Upgrade $http_upgrade;
               proxy_set_header Connection 'upgrade';
               proxy_set_header Host $host;
               proxy_cache_bypass $http_upgrade;

matt · September 25, 2017, 9:23pm

Probably something like this (the key is the transparent and websocket presets):

proxy / cent1:8080 {
    transparent
    websocket
}

Not sure what your paths and backend addresses would be but I’ll leave that to you to decide.

Namekal · September 26, 2017, 5:31pm

@matt I’m sorry but I don’t understand, that was already tried up by the 3rd post. I’m testing two ways with different options while trying to figure this out, https://surv.evolved.site and https://evolved.site/shinobi

This is my current Caddyfile:

evolved.site {
    root /var/www
    proxy /shinobi http://cent1:8080 {
        transparent
        websocket
        header_upstream X-Robots-Tag "noindex, nofollow, nosnippet, noarchive"
        header_upstream X-Frame-Options "SAMEORIGIN"
          }
    gzip
    tls {
       dns cloudflare
    }
#       header / {
#       # Enable HTTP Strict Transport Security (HSTS) to force clients to always
#       # connect via HTTPS (do not use if only testing)
#       Strict-Transport-Security "max-age=31536000;"
#       # Enable cross-site filter (XSS) and tell browser to block detected attacks
#       X-XSS-Protection "1; mode=block"
#       # Prevent some browsers from MIME-sniffing a response away from the declared Content-Type
#       X-Content-Type-Options "nosniff"
#       # Disallow the site to be rendered within a frame (clickjacking protection)
#       X-Frame-Options "DENY"
#}
    log /var/log/caddy/access.log
    errors /var/log/caddy/errors.log
}



#Subdomains -------

surv.evolved.site {
gzip
proxy / cent1:8080 {
        transparent
        websocket
        }
log /var/log/caddy/access.surv.log
errors /var/log/caddy/errors.surv.log
}

Whitestrake · September 27, 2017, 12:18am

The Apache configuration is incredibly simple. The equivalent is proxy / http://localhost:8080/, I believe.

The first nginx configuration translates literally to the Caddyfile:

my.domain {
  header / Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
  header / X-Robots-Tag "noindex, nofollow, nosnippet, noarchive"
  proxy / http://10.0.0.10:8083/ {
    header_upstream X-Forwarded-For {remote}
    header_upstream Host {host}
    header_upstream Upgrade {>Upgrade}
    header_upstream Connection {>Connection}
    header_upstream X-NginX-Proxy true
    header_upstream X-Real-IP {remote}
  }
}

The proxy block is functionally identical to the following:

  proxy / http://10.0.0.10:8083/ {
    websocket
    transparent
    header_upstream X-NginX-Proxy true
  }

The only difference between them is that the simplified version also sends X-Forwarded-Proto.

The second nginx configuration translates approximately to the Caddyfile:

rewrite {
  to {path}/ /nodejs
}
proxy /nodejs http://<host>:<port> {
  without /nodejs
  header_upstream Upgrade {>Upgrade}
  header_upstream Connection "upgrade"
  header_upstream Host {host}
}

The proxy block here simplifies to:

proxy /nodejs http://<host>:<port> {
  without /nodejs
  websocket
  header_upstream Host {host} # or just put "transparent" here, which sets Host
}

nginx user 2 also states that including X-Frame-Options is the fix. Your config sets header_upstream X-Frame-Options "SAMEORIGIN", but I suspect that’s not effective; X-Frame-Options is meant to inform browsers rendering the page, not the server. Try using header_downstream X-Frame-Options "SAMEORIGIN" instead, to send that header to the client.

https://caddyserver.com/docs/proxy#presets

matt · September 27, 2017, 2:03am

Doesn’t the first proxy block example also simplify significantly with the transparent preset? (Edit: Nvm, I misread the post.)

Namekal · September 27, 2017, 3:10am

Alright! There’s success!
This seemed to have done it, simplified from the first nginx user config as @matt said

surv.evolved.site {
	header / Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
	header / X-Robots-Tag "noindex, nofollow, nosnippet, noarchive"
	proxy / http://cent1:8080/ {
		websocket
		transparent
		header_upstream X-NginX-Proxy true
	}
}

Namekal · September 27, 2017, 3:23am

Well now oddly enough…I’m wondering if it was something the site had in its code before.( possibly related note: I also had issues with viewing the site via android, it was just freezing on login) because I noticed the difference was that it did not have gzip listed. when i first tried everything, i tried without gzip and it didnt like to show any images and gave me errors. I had recently reinstalled/updated shinobi and that fixed the android viewing issue, which it may also was the fix for this.

One by one i commented out lines in that “successful” code… leaving me with

surv.evolved.site {
#	header / Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
#	header / X-Robots-Tag "noindex, nofollow, nosnippet, noarchive"
	 proxy / http://cent1:8080/ {
		websocket
		transparent
#		header_upstream X-NginX-Proxy true
	}
}

Ultimately just being

surv.evolved.site {
	proxy / http://cent1:8080/ {
		websocket
		transparent
	}
}

This kinda makes me feel like garbage in you guys taking whatever time out of your day to help. Sorry and thank you all.

Whitestrake · September 27, 2017, 3:26am

You’re welcome! I come here because I want to, no need to feel bad about it

matt · September 27, 2017, 4:17am

Don’t overthink the Caddyfile. It’s simpler than you might expect.

system · December 26, 2017, 4:17am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.