SOLVED: Jenkins Behind Caddy 2 Does Not Work

Help
1

1. Caddy version (caddy version):

2.1.1

2. How I run Caddy:

I run caddy via docker. See below docker compose file

a. System environment:

alpine linux image in ubuntu linux host

b. Command:

My docker compose file entrypoint looks as follows

caddy run --config "/etc/caddy/Caddyfile" --adapter "caddyfile"

c. Service/unit/compose file:

Docker compose file

version: "2.1"
services:
  jenkins:
    image: jenkins/jenkins:2.235.3-lts-centos7
    container_name: jenkins-container
    # environment:
    #   - JENKINS_OPTS="--prefix=/"
    volumes:
      - /home/ubuntu/app/data/jenkins/home:/var/jenkins_home
    expose:
      - 8080
      - 50000
    ports:
      - 8080:8080
      - 50000:50000
    restart: unless-stopped
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: "10m"
  caddy:
    image: caddy:2.1.1-alpine
    container_name: caddy-container
    command: caddy run --config "/etc/caddy/Caddyfile" --adapter "caddyfile"
    volumes:      
      - /home/ubuntu/app/data/caddy/data:/data
      - /home/ubuntu/app/data/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/ubuntu/app/data/caddy/config:/config
      - /home/ubuntu/app/data/caddy/www:/usr/share/caddy
    expose:
      - 80
      - 443
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: "10m"

d. My complete Caddyfile or JSON config:

{
    # email to use on Let's Encrypt
    email <my email here>
    
    # Uncomment for debug
    #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    #debug
}

# Add gzip compression to requests
(webconf) {
  encode gzip
}

# Add forward headers to requests
(theheaders) {
    header_up X-Forwarded-Ssl off
    header_up Host {host}
    header_up X-Real-IP {remote}
    header_up X-Forwarded-For {remote}
    header_up X-Forwarded-Port {server_port}
    header_up X-Forwarded-Proto http
    header_up X-Url-Scheme {scheme}
    header_up X-Forwarded-Host {host}
}

jenkins.app.example.com {
  reverse_proxy http://jenkins-container:8080 {
    import theheaders
  }

  import webconf
}

3. The problem I’m having:

When accessing jenkins this way through caddy, Jenkins returns a http 500 error. HTTP ERROR 500 java.lang.RuntimeException: java.lang.NumberFormatException: For input string: “” with the following stack trace

URI:	/
STATUS:	500
MESSAGE:	java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
SERVLET:	-
CAUSED BY:	java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
CAUSED BY:	java.lang.NumberFormatException: For input string: ""
Caused by:
java.lang.RuntimeException: java.lang.NumberFormatException: For input string: ""
	at org.eclipse.jetty.server.ForwardedRequestCustomizer.customize(ForwardedRequestCustomizer.java:390)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:379)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: ""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:592)
	at java.lang.Integer.parseInt(Integer.java:615)
	at org.eclipse.jetty.server.ForwardedRequestCustomizer$Forwarded.handlePort(ForwardedRequestCustomizer.java:654)
	at org.eclipse.jetty.server.ForwardedRequestCustomizer.customize(ForwardedRequestCustomizer.java:385)
	... 15 more

4. Error messages and/or full log output:

Accessing jenkins directly via port 8080 works. However jenkins displays the following message with the following url

It appears that your reverse proxy set up is broken.
[https://wiki.jenkins.io/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken](https://wiki.jenkins.io/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken)

5. What I already tried:

See above

6. Links to relevant resources:

https://wiki.jenkins.io/display/JENKINS/Jenkins+says+my+reverse+proxy+setup+is+broken

2

I’m pretty certain some of these are the issue based on the java error mentioning ForwardedRequestCustomizer.

In Caddy v2, the X-Forwarded-For and X-Forwarded-Proto are set for you automatically and intelligently. In your case though, since you seem to override Proto with http, you can leave that one in. Generally, it’s best to let the upstream applications know that TLS is terminated though. I don’t know how Jenkins behaves with that.

I think {server_port} might not be set, so it might be best to avoid X-Forwarded-Port altogether.

1 Like
3

Yep server_port was causing the exception. I ended up removing all the header_up statements and it just worked. Thank you.

1 Like
4

This topic was automatically closed after 30 days. New replies are no longer allowed.