I'm trying to connect a new jenkins slave node using websocket, launching the agent from slave node the response to wss call is 400 Bad request, that is receipt from agent as an "handshake error".
From my low level tests and analysis I found that incoming calls in Apache reverse proxy in HTTP/1.1 are downgraded to HTTP/1.0 when forwarded to destination node and then upgraded to HTTP/1.1 when response returned to caller. I didn't found any configuration to avoid this.
Detailed description
Architecture
- Apache Reverse Proxy, version 2.4.41, installed in a RHEL7.3, IP
192.168.1.2
Apache it is behind a public firewall and only port 443 is allowed, https it is offloaded in apache virtualhost. The communications between Apache and Jenkins Master node is in http on port 8080 - Jenkins Master node version 2.263.1, installed in a
RHEL7.6, Openjdk11, IP 192.168.1.10:8080
(NO "It appears that your reverse proxy setup is broken" message is showed in jenkins) - Jenkins Slave node, agent installed in a Windows server 2016, Openjdk11, IP
192.168.1.11
(Master and Slave nodes are in same subnet and ideally don't need to be connected passing through public network. I didn't found a way about this, it is allowed only for jnlp by setting hudson.TcpSlaveAgentListener.hostName system property, reference: https://stackoverflow.com/a/39965700 and https://stackoverflow.com/a/39136802)
Reverse Proxy Configuration
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerName off
ProxyRequests Off
AllowEncodedSlashes NoDecode
<Location /jenkins>
ProxyPreserveHost On
ProxyPass http://192.168.1.10:8080/jenkins nocanon
ProxyPassReverse http://192.168.1.10:8080/jenkins
ProxyPassReverse https://my.public.fqdn.com/jenkins
RequestHeader set X-Forwarded-Host "my.public.fqdn.com"
</Location>
<Location /jenkins/wsagents>
### Configurations tested and commented, not working ###
#SetEnv force-no-vary 1
#SetEnv force-proxy-request-1.0 1
#SetEnv proxy-nokeepalive 1
#RequestHeader unset Expect early
ProxyPreserveHost On
ProxyPass ws://192.168.1.10:8080/jenkins/wsagents
ProxyPassReverse ws://192.168.1.10:8080/jenkins/wsagents
</Location>
Behaviour
Created in jenkins a new Node from "Manage Nodes and Cloud" and selected "Launch Agent by connecting it to the master" from "Launch method", and selected "Use Websocket",
when I try to launch the agent from slave node I experienced an 400 Bad request response falling in handshake error:
java -jar agent.jar -jnlpUrl https://my.public.fqdn.com/jenkins/computer/Slave%20Windows/slave-agent.jnlp -secret @secret-file -workDir "E:\JenkinsSlave"
Sending handshake request:
> GET wss://my.public.fqdn.com/jenkins/wsagents/
> Connection: Upgrade
> Host: my.public.fqdn.com
> Node-Name: Slave Windows
> Origin: my.public.fqdn.com
> Sec-WebSocket-Key: FFGODSDcF0TTP4q/usk9Bw==
> Sec-WebSocket-Version: 13
> Secret-Key: 123123123123
> Upgrade: websocket
> X-Remoting-Capability: rO0ABXNyABpod4ucmVtbpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAf4=
2021-01-14T22:48:44.776+0100 FINE io.jenkins.remoting.shaded.org.glassfish.tyrus.core.DebugContext flush: < Session 36d3c733-37f3-4f6e-bea1-920e8cf6f3da [128 ms]: Received handshake response:
< 400
< Cache-Control: must-revalidate, no-cache, no-store
< Content-Length: 533
< Content-Type: text/html;charset=iso-8859-1
< Cookie: 52d4f682f884de63b52ae34622c7f3968acfc365d02327e2eec34f1f8e1
< Server: Jetty(9.4.33.v20201020)
< X-Content-Type-Options: nosniff
< X-Remoting-Capability: rO0ABXNyABpod4ucmVtbpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAf4=
< X-Remoting-Minimum-Version: 3.14
SEVERE hudson.remoting.jnlp.Main$CuiListener error: Handshake error.
io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake error.
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:674)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
at hudson.remoting.Engine.runWebSocket(Engine.java:628)
at hudson.remoting.Engine.run(Engine.java:470)
Caused by: io.jenkins.remoting.shaded.org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 400.
In Jenkins master node logs I found (System log Level set to ALL):
WARNING o.e.j.w.s.WebSocketServerFactory#isUpgradeRequest: Not a 'HTTP/1.1' request (was [HTTP/1.0])
The question is
Why apache reverse proxy switches HTTP/1.1 to HTTP/1.0 for websocket communications to backend, and how to fix it?
I tried many configurations in apache with no luck (i.e. force-no-vary)
Reference
Apache responses with http/1.0 even if request is http/1.1
Jenkins: How to configure Jenkins behind Nginx reverse proxy for JNLP slaves to connect
