apiserver proxy removes leading slash from path

[VasylSamoilov]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Apiserver proxy removes leading slash from path that is send to destination service or pod, effectively breaking applications.

Example tshark output:

Hypertext Transfer Protocol
GET test HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET test HTTP/1.1\r\n]
[GET test HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI: test
Request Version: HTTP/1.1
Host: localhost:8080\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36\r\n
Accept-Encoding: gzip, deflate, br\r\n
Accept-Language: en-US,en;q=0.8,uk;q=0.6,ru;q=0.4\r\n
Cache-Control: no-cache\r\n
Connection: Upgrade\r\n
Origin: http://localhost:8080\r\n
Pragma: no-cache\r\n
Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits\r\n
Sec-Websocket-Key: rtBOKvTHtRAbTpz9vhyAng==\r\n
Sec-Websocket-Version: 13\r\n
Upgrade: websocket\r\n
X-Forwarded-For: 127.0.0.1\r\n
\r\n
[Full request URI: http://localhost:8080test]
[HTTP request 1/1]

What you expected to happen:

Proxied URLs should begin with slash.
Example:
GET /test HTTP/1.1\r\n

How to reproduce it (as minimally and precisely as possible):

Use this guideline to construct proxy URL: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#discovering-builtin-services

This request will not work (404 not found):
curl http://localhost:8080/api/v1/namespaces/default/services/testservice:http/proxy/test
backend result (wrong): GET test HTTP/1.1\r\n

Request with double slash will work:
curl http://localhost:8080/api/v1/namespaces/default/services/testservice:http/proxy//test
backend result (correct): GET /test HTTP/1.1\r\n

Also old-style proxy URL format works too:
curl http://localhost:8080/api/v1/proxy/namespaces/default/services/testservice:http/test
backend result (correct): GET /test HTTP/1.1\r\n

Anything else we need to know?:
First observed with 1.7 kubernetes release

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.3", GitCommit:"6a4203eb4b5f59ac7a602e0c83023d15d991fd58", GitTreeState:"clean", BuildDate:"2017-08-23T22:58:52Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.5-dirty", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"dirty", BuildDate:"2017-09-04T14:44:32Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:
  Bare metal

• OS (e.g. from /etc/os-release):
  CoreOS 1409.6.0

• Kernel (e.g. uname -a):
  4.11.9-coreos

• Install tools:

• Others:

[k8s-github-robot]

@VasylSamoilov
There are no sig labels on this issue. Please add a sig label by:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <label>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. You can find the group list here and label list here.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

[VasylSamoilov]

@kubernetes/sig-api-machinery-bugs

[k8s-ci-robot]

@VasylSamoilov: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-bugs.

In response to this:

@kubernetes/sig-api-machinery-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

bisected to 6875e95

the stringification of the proxied location looks like it was normalizing the path and prepending a slash

that commit switched to copying the incoming request (which was being manually constructed from the parsed subresource path) as-is, which left the path missing a leading /

[liggitt]

also, this appears to be limited to upgrade requests

[liggitt]

and actually, the upgrade issue was introduced in 715d5d9#diff-1fbb2e342b0473af45e93e3e26031e80L226

[VasylSamoilov]

I can confirm that my initial report was incorrect, only upgrade requests are affected.

[VasylSamoilov]

Correct curl to reproduce the issue is to make websocket upgrade request:
curl -H "Upgrade: websocket" -H "Sec-Websocket-Version: 13" -H "Sec-Websocket-Extensions: permessage-deflate; client_max_window_bits" -H "Sec-Websocket-Key: rtBOKvTHtRAbTpz9vhyAng==" http://localhost:8080/api/v1/namespaces/default/services/testservice:http/proxy/test