RBAC error for ml-pipeline-scheduledworkflow

[andrewgdavis]

What steps did you take:

kubectl -n kubeflow logs ml-pipeline-scheduledworkflow

What happened:

Log message reported:

E0417 20:03:56.525539 1 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cronutcpd78s.1606663a49b1007e", GenerateName:"", Namespace:"kubeflow", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"ScheduledWorkflow", Namespace:"kubeflow", Name:"cronutcpd78s", UID:"b78d763b-08a3-42b6-88fb-c7be5dcabb92", APIVersion:"kubeflow.org/v1beta1", ResourceVersion:"606783885", FieldPath:""}, Reason:"Synced", Message:"Scheduled workflow synced successfull: All done", Source:v1.EventSource{Component:"scheduled-workflow-controller", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf9e4d47cd6b8a7e, ext:2591806995978068, loc:(*time.Location)(0x2040b00)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf9ea0cb1efea0f3, ext:2677324290830279, loc:(*time.Location)(0x2040b00)}}, Count:5222, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "cronutcpd78s.1606663a49b1007e" is forbidden: User "system:serviceaccount:kubeflow:ml-pipeline-scheduledworkflow" cannot patch resource "events" in API group "" in the namespace "kubeflow"' (will not retry!)

What did you expect to happen:

Correct RBAC permissions applied for serviceaccount ml-pipeline-scheduledworkflow

Environment:

Image:          gcr.io/ml-pipeline/scheduledworkflow:0.2.5
Image ID:       docker-pullable://gcr.io/ml-pipeline/scheduledworkflow@sha256:26ab7ef0d9652af904b0ada6baecb109f85d2c5bfeff9a52adeb5b1648c33533

How did you deploy Kubeflow Pipelines (KFP)?

kubectl -n kubeflow apply -k ./manifests/...

KFP version: v0.2.5

kubectl -n kubeflow get role ml-pipeline-scheduledworkflow-role -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
  creationTimestamp: "2020-02-11T23:04:52Z"
  labels:
    app: ml-pipeline-scheduledworkflow-role
    application-crd-id: kubeflow-pipelines
  name: ml-pipeline-scheduledworkflow-role
  namespace: kubeflow
  ownerReferences:
  - apiVersion: app.k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: false
    kind: Application
    name: pipeline
    uid: ef202edb-4d22-11ea-884e-06e2da35a226
  resourceVersion: "508439142"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kubeflow/roles/ml-pipeline-scheduledworkflow-role
  uid: e91f1084-4d22-11ea-884e-06e2da35a226
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - kubeflow.org
  resources:
  - scheduledworkflows
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete

Anything else you would like to add:

... curious as to why an Event is being patched, and how no one else ran into this.

/kind bug

/area backend

[andrewgdavis]

appending the following rules to the role "ml-pipeline-scheduledworkflow-role" makes the error log go away

 - apiGroups:
  - ""
  resources:
  - events
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

[Bobgy]

Thanks for reporting the issue. This is strange. I will look into it.

[Bobgy]

I tried to reproduce:

1. create a scheduled workflow
2. see scheduled workflow server's logs

Then I tried to give patch and create permissions to the role.
Now I can see some events in scheduledworkflow CR's related events like:

$ kubectl describe swf xxxx
...
Events:
  Type    Reason  Age                  From                           Message
  ----    ------  ----                 ----                           -------
  Normal  Synced  2m2s (x57 over 17m)  scheduled-workflow-controller  Scheduled workflow synced successfull: All done

These permissions don't seem to affect how it works, just for status reporting. I will fix this right now.