RBAC error when accessing profiles-deployment from centraldashboard

[jan-stanek]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.8

Describe your issue

Since we installed kubeflow to new k8s cluster (k8s 1.29), central dashboard is not able to load list of namespaces because of the:

GET /kfam/v1/role/clusteradmin?user=... HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "-" "e7e45281-1c67-9a4e-b556-16324a1f9c49" "profiles-kfam.kubeflow:8081" "-" inbound|8081|| - ... ... - -

Policies seems to be ok, so I compared istio trace logs with older cluster where it works ok and it looks like custom headers are missing:

Not working:

2024-06-19T11:25:19.484856Z	trace	envoy connection	[C23] writing 143 bytes, end_stream false
2024-06-19T11:25:19.484859Z	trace	envoy main	item added to deferred deletion list (size=4)
2024-06-19T11:25:19.484863Z	trace	envoy main	item added to deferred deletion list (size=5)
2024-06-19T11:25:19.484871Z	debug	envoy connection	[C23] closing data_to_write=143 type=0
2024-06-19T11:25:19.484876Z	debug	envoy connection	[C23] setting delayed close timer with timeout 1000 ms
2024-06-19T11:25:19.484887Z	debug	envoy pool	[C7] response complete
2024-06-19T11:25:19.484893Z	debug	envoy pool	[C7] destroying stream: 0 remaining
2024-06-19T11:25:19.484898Z	trace	envoy http	[C7] parsed 75 bytes
2024-06-19T11:25:19.484901Z	trace	envoy main	clearing deferred deletion list (size=5)
2024-06-19T11:25:19.484914Z	trace	envoy connection	[C23] socket event: 2
2024-06-19T11:25:19.484916Z	trace	envoy connection	[C23] write ready
2024-06-19T11:25:19.484972Z	trace	envoy connection	[C23] write returns: 143
2024-06-19T11:25:19.484975Z	debug	envoy connection	[C23] write flush complete
2024-06-19T11:25:19.484977Z	debug	envoy connection	[C23] closing socket: 1
2024-06-19T11:25:19.485069Z	trace	envoy connection	[C23] raising connection event 1
2024-06-19T11:25:19.485077Z	trace	envoy conn_handler	[C23] connection on event 1
2024-06-19T11:25:19.485079Z	debug	envoy conn_handler	[C23] adding to cleanup list
2024-06-19T11:25:19.485081Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:25:19.485082Z	trace	envoy main	item added to deferred deletion list (size=2)
2024-06-19T11:25:19.485100Z	trace	envoy main	clearing deferred deletion list (size=2)
2024-06-19T11:25:20.642673Z	debug	envoy filter	original_dst: new connection accepted
2024-06-19T11:25:20.642699Z	trace	envoy filter	original_dst: set destination to ...
2024-06-19T11:25:20.642703Z	debug	envoy filter	tls inspector: new connection accepted
2024-06-19T11:25:20.642731Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.642736Z	trace	envoy filter	recv returned: 0
2024-06-19T11:25:20.642738Z	trace	envoy filter	recv return try again
2024-06-19T11:25:20.643043Z	debug	envoy filter	original_dst: new connection accepted
2024-06-19T11:25:20.643071Z	trace	envoy filter	original_dst: set destination to ...
2024-06-19T11:25:20.643076Z	debug	envoy filter	tls inspector: new connection accepted
2024-06-19T11:25:20.643107Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.643115Z	trace	envoy filter	recv returned: 0
2024-06-19T11:25:20.643116Z	trace	envoy filter	recv return try again
2024-06-19T11:25:20.643172Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.643181Z	trace	envoy filter	recv returned: 158
2024-06-19T11:25:20.643183Z	trace	envoy filter	tls inspector: recv: 158
2024-06-19T11:25:20.643189Z	debug	envoy filter	http inspector: new connection accepted
2024-06-19T11:25:20.643192Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.643195Z	trace	envoy filter	recv returned: 158
2024-06-19T11:25:20.643198Z	trace	envoy filter	http inspector: http_parser parsed 75 chars, error code: 0
2024-06-19T11:25:20.643201Z	trace	envoy filter	http inspector: done: true
2024-06-19T11:25:20.643204Z	trace	envoy filter	http inspector: set application protocol to http/1.1
2024-06-19T11:25:20.643223Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.643236Z	trace	envoy filter	recv returned: 149
2024-06-19T11:25:20.643239Z	trace	envoy filter	tls inspector: recv: 149
2024-06-19T11:25:20.643240Z	debug	envoy conn_handler	[C24] new connection from ...
2024-06-19T11:25:20.643244Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:25:20.643247Z	trace	envoy main	clearing deferred deletion list (size=1)
2024-06-19T11:25:20.643249Z	debug	envoy filter	http inspector: new connection accepted
2024-06-19T11:25:20.643251Z	trace	envoy connection	[C24] socket event: 3
2024-06-19T11:25:20.643254Z	trace	envoy connection	[C24] write ready
2024-06-19T11:25:20.643255Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:25:20.643256Z	trace	envoy connection	[C24] read ready. dispatch_buffered_data=0
2024-06-19T11:25:20.643259Z	trace	envoy filter	recv returned: 149
2024-06-19T11:25:20.643265Z	trace	envoy filter	http inspector: http_parser parsed 66 chars, error code: 0
2024-06-19T11:25:20.643275Z	trace	envoy filter	http inspector: done: true
2024-06-19T11:25:20.643279Z	trace	envoy filter	http inspector: set application protocol to http/1.1
2024-06-19T11:25:20.643265Z	trace	envoy connection	[C24] read returns: 158
2024-06-19T11:25:20.643297Z	trace	envoy connection	[C24] read error: Resource temporarily unavailable
2024-06-19T11:25:20.643300Z	trace	envoy filter	Alpn Protocol Not Found. Expected istio-peer-exchange, Got 
2024-06-19T11:25:20.643327Z	trace	envoy http	[C24] parsing 158 bytes
2024-06-19T11:25:20.643338Z	trace	envoy http	[C24] message begin
2024-06-19T11:25:20.643342Z	debug	envoy http	[C24] new stream
2024-06-19T11:25:20.643329Z	debug	envoy conn_handler	[C25] new connection from ...
2024-06-19T11:25:20.643355Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:25:20.643358Z	trace	envoy main	clearing deferred deletion list (size=1)
2024-06-19T11:25:20.643350Z	trace	envoy http	[C24] completed header: key=host value=profiles-kfam.kubeflow:8081
2024-06-19T11:25:20.643362Z	trace	envoy connection	[C25] socket event: 3
2024-06-19T11:25:20.643365Z	trace	envoy connection	[C25] write ready
2024-06-19T11:25:20.643366Z	trace	envoy http	[C24] completed header: key=accept value=application/json
2024-06-19T11:25:20.643374Z	trace	envoy http	[C24] onHeadersCompleteBase
2024-06-19T11:25:20.643378Z	trace	envoy http	[C24] completed header: key=Connection value=close
2024-06-19T11:25:20.643384Z	trace	envoy http	[C24] Server: onHeadersComplete size=3
2024-06-19T11:25:20.643367Z	trace	envoy connection	[C25] read ready. dispatch_buffered_data=0
2024-06-19T11:25:20.643390Z	trace	envoy http	[C24] message complete
2024-06-19T11:25:20.643400Z	trace	envoy connection	[C25] read returns: 149
2024-06-19T11:25:20.643407Z	debug	envoy http	[C24][S17964618679120465489] request headers complete (end_stream=true):
':authority', 'profiles-kfam.kubeflow:8081'
':path', '/kfam/v1/role/clusteradmin?user=...'
':method', 'GET'
'accept', 'application/json'
'connection', 'close'

Working:

2024-06-19T11:02:38.147378Z	trace	envoy connection	[C39] writing 143 bytes, end_stream false
2024-06-19T11:02:38.147381Z	trace	envoy main	item added to deferred deletion list (size=4)
2024-06-19T11:02:38.147384Z	trace	envoy main	item added to deferred deletion list (size=5)
2024-06-19T11:02:38.147386Z	debug	envoy connection	[C39] closing data_to_write=143 type=0
2024-06-19T11:02:38.147388Z	debug	envoy connection	[C39] setting delayed close timer with timeout 1000 ms
2024-06-19T11:02:38.147394Z	debug	envoy pool	[C5] response complete
2024-06-19T11:02:38.147397Z	debug	envoy pool	[C5] destroying stream: 0 remaining
2024-06-19T11:02:38.147400Z	trace	envoy http	[C5] parsed 75 bytes
2024-06-19T11:02:38.147402Z	trace	envoy main	clearing deferred deletion list (size=5)
2024-06-19T11:02:38.147413Z	trace	envoy connection	[C39] socket event: 2
2024-06-19T11:02:38.147415Z	trace	envoy connection	[C39] write ready
2024-06-19T11:02:38.147443Z	trace	envoy connection	[C39] write returns: 143
2024-06-19T11:02:38.147446Z	debug	envoy connection	[C39] write flush complete
2024-06-19T11:02:38.147447Z	debug	envoy connection	[C39] closing socket: 1
2024-06-19T11:02:38.147459Z	trace	envoy connection	[C39] raising connection event 1
2024-06-19T11:02:38.147463Z	trace	envoy conn_handler	[C39] connection on event 1
2024-06-19T11:02:38.147464Z	debug	envoy conn_handler	[C39] adding to cleanup list
2024-06-19T11:02:38.147465Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:02:38.147466Z	trace	envoy main	item added to deferred deletion list (size=2)
2024-06-19T11:02:38.147467Z	trace	envoy main	clearing deferred deletion list (size=2)
2024-06-19T11:02:39.235071Z	debug	envoy filter	original_dst: new connection accepted
2024-06-19T11:02:39.235107Z	trace	envoy filter	original_dst: set destination to ...
2024-06-19T11:02:39.235111Z	debug	envoy filter	tls inspector: new connection accepted
2024-06-19T11:02:39.235144Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:02:39.235151Z	trace	envoy filter	recv returned: 2200
2024-06-19T11:02:39.235153Z	trace	envoy filter	tls inspector: recv: 2200
2024-06-19T11:02:39.235171Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2024-06-19T11:02:39.235184Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.8081_._.profiles-kfam.kubeflow.svc.cluster.local
2024-06-19T11:02:39.235187Z	debug	envoy filter	http inspector: new connection accepted
2024-06-19T11:02:39.235189Z	trace	envoy filter	http inspector: cannot inspect http protocol with transport socket tls
2024-06-19T11:02:39.235250Z	debug	envoy conn_handler	[C40] new connection from ...
2024-06-19T11:02:39.235256Z	debug	envoy filter	original_dst: new connection accepted
2024-06-19T11:02:39.235271Z	trace	envoy filter	original_dst: set destination to ...
2024-06-19T11:02:39.235279Z	debug	envoy filter	tls inspector: new connection accepted
2024-06-19T11:02:39.235256Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:02:39.235287Z	trace	envoy main	clearing deferred deletion list (size=1)
2024-06-19T11:02:39.235300Z	trace	envoy filter	onFileEvent: 1
2024-06-19T11:02:39.235307Z	trace	envoy filter	recv returned: 517
2024-06-19T11:02:39.235308Z	trace	envoy filter	tls inspector: recv: 517
2024-06-19T11:02:39.235317Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2024-06-19T11:02:39.235324Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.8081_._.profiles-kfam.kubeflow.svc.cluster.local
2024-06-19T11:02:39.235327Z	debug	envoy filter	http inspector: new connection accepted
2024-06-19T11:02:39.235329Z	trace	envoy filter	http inspector: cannot inspect http protocol with transport socket tls
2024-06-19T11:02:39.235357Z	trace	envoy connection	[C40] socket event: 3
2024-06-19T11:02:39.235362Z	trace	envoy connection	[C40] write ready
2024-06-19T11:02:39.235382Z	debug	envoy conn_handler	[C41] new connection from ...
2024-06-19T11:02:39.235388Z	trace	envoy main	item added to deferred deletion list (size=1)
2024-06-19T11:02:39.235389Z	trace	envoy main	clearing deferred deletion list (size=1)
2024-06-19T11:02:39.235392Z	trace	envoy connection	[C41] socket event: 3
2024-06-19T11:02:39.235394Z	trace	envoy connection	[C41] write ready
2024-06-19T11:02:39.236409Z	trace	envoy connection	[C41] ssl error occurred while read: WANT_READ
2024-06-19T11:02:39.236420Z	trace	envoy connection	[C41] read ready. dispatch_buffered_data=0
2024-06-19T11:02:39.236409Z	trace	envoy connection	[C40] ssl error occurred while read: WANT_READ
2024-06-19T11:02:39.236428Z	trace	envoy connection	[C40] read ready. dispatch_buffered_data=0
2024-06-19T11:02:39.236422Z	trace	envoy connection	[C41] ssl error occurred while read: WANT_READ
2024-06-19T11:02:39.236439Z	trace	envoy connection	[C40] ssl error occurred while read: WANT_READ
2024-06-19T11:02:39.237968Z	trace	envoy connection	[C41] socket event: 3
2024-06-19T11:02:39.237977Z	trace	envoy connection	[C41] write ready
2024-06-19T11:02:39.238213Z	trace	envoy connection	[C41] raising connection event 2
2024-06-19T11:02:39.238223Z	trace	envoy conn_handler	[C41] connection on event 2
2024-06-19T11:02:39.238225Z	trace	envoy connection	[C41] read ready. dispatch_buffered_data=0
2024-06-19T11:02:39.238233Z	trace	envoy connection	[C41] ssl read returns: 1531
2024-06-19T11:02:39.238235Z	trace	envoy connection	[C41] ssl read returns: -1
2024-06-19T11:02:39.238236Z	trace	envoy connection	[C41] ssl error occurred while read: WANT_READ
2024-06-19T11:02:39.238240Z	trace	envoy connection	[C41] ssl read 1531 bytes
2024-06-19T11:02:39.238242Z	trace	envoy filter	Alpn Protocol Not Found. Expected istio-peer-exchange, Got http/1.1
2024-06-19T11:02:39.238267Z	trace	envoy http	[C41] parsing 1531 bytes
2024-06-19T11:02:39.238274Z	trace	envoy http	[C41] message begin
2024-06-19T11:02:39.238277Z	debug	envoy http	[C41] new stream
2024-06-19T11:02:39.238289Z	trace	envoy http	[C41] completed header: key=host value=profiles-kfam.kubeflow:8081
2024-06-19T11:02:39.238297Z	trace	envoy http	[C41] completed header: key=accept value=application/json
2024-06-19T11:02:39.238300Z	trace	envoy http	[C41] completed header: key=x-forwarded-proto value=http
2024-06-19T11:02:39.238305Z	trace	envoy http	[C41] completed header: key=x-request-id value=6a6d6561-f454-9431-ba2d-f37573c5db10
2024-06-19T11:02:39.238309Z	trace	envoy http	[C41] completed header: key=x-envoy-decorator-operation value=profiles-kfam.kubeflow.svc.cluster.local:8081/*
2024-06-19T11:02:39.238315Z	trace	envoy http	[C41] completed header: key=x-envoy-peer-metadata value=CiQKDkFQUF9DT05UQUlORVJTEhIaEGNlbnRyYWxkYXNoYm9hcmQKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzCjwKDElOU1RBTkNFX0lQUxIsGioxMC4yMzMuMTIxLjc3LDJhMDI6NTk4OjEyODo4YTAwOjoxMWM3OmU0MjYKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE2LjAKwwIKBkxBQkVMUxK4Aiq1AgoZCgNhcHASEhoQY2VudHJhbGRhc2hib2FyZAoxChthcHAua3ViZXJuZXRlcy5pby9jb21wb25lbnQSEhoQY2VudHJhbGRhc2hib2FyZAosChZhcHAua3ViZXJuZXRlcy5pby9uYW1lEhIaEGNlbnRyYWxkYXNoYm9hcmQKKQoTa3VzdG9taXplLmNvbXBvbmVudBISGhBjZW50cmFsZGFzaGJvYXJkCiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KNQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRISGhBjZW50cmFsZGFzaGJvYXJkCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKKgoETkFNRRIiGiBjZW50cmFsZGFzaGJvYXJkLThmODliNmQ1Yi1mbTk1bgoXCglOQU1FU1BBQ0USChoIa3ViZWZsb3cKVQoFT1dORVISTBpKa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2t1YmVmbG93L2RlcGxveW1lbnRzL2NlbnRyYWxkYXNoYm9hcmQKFwoRUExBVEZPUk1fTUVUQURBVEESAioACiMKDVdPUktMT0FEX05BTUUSEhoQY2VudHJhbGRhc2hib2FyZA==
2024-06-19T11:02:39.238321Z	trace	envoy http	[C41] completed header: key=x-envoy-peer-metadata-id value=sidecar~10.233.121.77~centraldashboard-8f89b6d5b-fm95n.kubeflow~kubeflow.svc.cluster.local
2024-06-19T11:02:39.238322Z	trace	envoy http	[C41] completed header: key=x-envoy-attempt-count value=1
2024-06-19T11:02:39.238325Z	trace	envoy http	[C41] completed header: key=x-b3-traceid value=3020e4bfefdc30d6028eebb92f2469ce
2024-06-19T11:02:39.238326Z	trace	envoy http	[C41] completed header: key=x-b3-spanid value=028eebb92f2469ce
2024-06-19T11:02:39.238327Z	trace	envoy http	[C41] onHeadersCompleteBase
2024-06-19T11:02:39.238328Z	trace	envoy http	[C41] completed header: key=x-b3-sampled value=1
2024-06-19T11:02:39.238331Z	trace	envoy http	[C41] Server: onHeadersComplete size=11
2024-06-19T11:02:39.238337Z	trace	envoy http	[C41] message complete
2024-06-19T11:02:39.238348Z	debug	envoy http	[C41][S12648579546682317763] request headers complete (end_stream=true):
':authority', 'profiles-kfam.kubeflow:8081'
':path', '/kfam/v1/role/clusteradmin?user=...'
':method', 'GET'
'accept', 'application/json'
'x-forwarded-proto', 'http'
'x-request-id', '6a6d6561-f454-9431-ba2d-f37573c5db10'
'x-envoy-decorator-operation', 'profiles-kfam.kubeflow.svc.cluster.local:8081/*'
'x-envoy-peer-metadata', 'CiQKDkFQUF9DT05UQUlORVJTEhIaEGNlbnRyYWxkYXNoYm9hcmQKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzCjwKDElOU1RBTkNFX0lQUxIsGioxMC4yMzMuMTIxLjc3LDJhMDI6NTk4OjEyODo4YTAwOjoxMWM3OmU0MjYKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE2LjAKwwIKBkxBQkVMUxK4Aiq1AgoZCgNhcHASEhoQY2VudHJhbGRhc2hib2FyZAoxChthcHAua3ViZXJuZXRlcy5pby9jb21wb25lbnQSEhoQY2VudHJhbGRhc2hib2FyZAosChZhcHAua3ViZXJuZXRlcy5pby9uYW1lEhIaEGNlbnRyYWxkYXNoYm9hcmQKKQoTa3VzdG9taXplLmNvbXBvbmVudBISGhBjZW50cmFsZGFzaGJvYXJkCiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KNQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRISGhBjZW50cmFsZGFzaGJvYXJkCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKKgoETkFNRRIiGiBjZW50cmFsZGFzaGJvYXJkLThmODliNmQ1Yi1mbTk1bgoXCglOQU1FU1BBQ0USChoIa3ViZWZsb3cKVQoFT1dORVISTBpKa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2t1YmVmbG93L2RlcGxveW1lbnRzL2NlbnRyYWxkYXNoYm9hcmQKFwoRUExBVEZPUk1fTUVUQURBVEESAioACiMKDVdPUktMT0FEX05BTUUSEhoQY2VudHJhbGRhc2hib2FyZA=='
'x-envoy-peer-metadata-id', 'sidecar~...~centraldashboard-8f89b6d5b-fm95n.kubeflow~kubeflow.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '3020e4bfefdc30d6028eebb92f2469ce'
'x-b3-spanid', '028eebb92f2469ce'
'x-b3-sampled', '1'

We also tried to update to current !.9 version with newer istion, but it's the same.

Steps to reproduce the issue

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

Hello, please try again with the 1.9/master branch.
1.8 is already EOL since there is 1.8.1. Since we plan to release 1.9 today, 1.8.1 will also bee soon EOL.

[jan-stanek]

I've tried 1.9.0 and it's the same.

It looks like some istio network issue, see:

2024-06-19T11:25:20.642736Z	trace	envoy filter	recv returned: 0
2024-06-19T11:25:20.642738Z	trace	envoy filter	recv return try again

Or complete diff: https://www.diffchecker.com/4gXibi2E/

[jan-stanek]

It looks like it was caused by cilium configuration, it has to be updated to work with istio: https://docs.cilium.io/en/latest/network/servicemesh/istio/

[juliusvonkohout]

Then lets close since we do not support Cilium yet.

Please continue in #2729

[juliusvonkohout]

Reopen since I want to add this link to the guidelines / readme.