Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] Allow events events api #3507

Closed
wants to merge 2 commits into from
Closed

Conversation

dennybaa
Copy link

@dennybaa dennybaa commented May 8, 2024

ystem", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"arc-system", Name:"gha-rs-controller", UID:"7df82bf8-2f40-464f-bfaa-02055d3379b0", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"55357371", FieldPath:""}, Reason:"LeaderElection", Message:"gha-rs-controller-c58d4454d-dp4qv_48434b5e-ad35-43e3-8732-1a18fb5240f1 became leader", Source:v1.EventSource{Component:"gha-rs-controller-c58d4454d-dp4qv_48434b5e-ad35-43e3-8732-1a18fb5240f1", Host:""}, FirstTimestamp:time.Date(2024, time.May, 8, 17, 26, 57, 872363492, time.Local), LastTimestamp:time.Date(2024, time.May, 8, 17, 26, 57, 872363492, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"gha-rs-controller-c58d4454d-dp4qv_48434b5e-ad35-43e3-8732-1a18fb5240f1", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:arc-system:gha-rs-controller" cannot create resource "events" in API group "" in the namespace "arc-system"' (will not retry!)

@dennybaa dennybaa changed the title Update manager_cluster_role.yaml [fix] Allow events events api May 8, 2024
@nikola-jokic
Copy link
Collaborator

Hey @dennybaa,

I'm not sure why would you need event permission for the gha-runner-scale-set? There is no gha-runner-scale-set controller that emits events.

@dennybaa
Copy link
Author

dennybaa commented May 13, 2024

@nikola-jokic Hello. That's very odd. Because I've gotten this error message, after I've added the above changes it's disappeared. This is output from its logs 😕 I can only admit, that somehow during errors with the application there was something that made actually the pod emit events... In this case the issue is not legit.

@nikola-jokic
Copy link
Collaborator

Maybe something went wrong with the installation 😕
Can you please provide steps to reproduce this issue?

@dennybaa
Copy link
Author

@nikola-jokic Maybe you've got any ides?

gha-rs-controller-7867ccb597-4nfrk manager E0522 18:50:03.586677       1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"gha-rs-controller.17d1e42a5b910a64", GenerateName:"", Namespace:"arc-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"arc-system", Name:"gha-rs-controller", UID:"7df82bf8-2f40-464f-bfaa-02055d3379b0", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"69477037", FieldPath:""}, Reason:"LeaderElection", Message:"gha-rs-controller-7867ccb597-4nfrk_d6787ed3-61a0-4473-889f-e7c61db7b403 became leader", Source:v1.EventSource{Component:"gha-rs-controller-7867ccb597-4nfrk_d6787ed3-61a0-4473-889f-e7c61db7b403", Host:""}, FirstTimestamp:time.Date(2024, time.May, 22, 18, 50, 3, 580861028, time.Local), LastTimestamp:time.Date(2024, time.May, 22, 18, 50, 3, 580861028, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"gha-rs-controller-7867ccb597-4nfrk_d6787ed3-61a0-4473-889f-e7c61db7b403", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:arc-system:gha-rs-controller" cannot create resource "events" in API group "" in the namespace "arc-system"' (will not retry!)

@dennybaa
Copy link
Author

It's been just working, and now again...

@dennybaa
Copy link
Author

It doesn't complain after:

+ gha-rs-controller-7867ccb597-kb7xg › manager
gha-rs-controller-7867ccb597-kb7xg manager I0522 18:55:28.895743       1 leaderelection.go:250] attempting to acquire leader lease arc-system/gha-rs-controller...
+ gha-rs-controller-7867ccb597-h92b4 › manager
gha-rs-controller-7867ccb597-h92b4 manager I0522 18:55:25.982901       1 leaderelection.go:250] attempting to acquire leader lease arc-system/gha-rs-controller...
gha-rs-controller-7867ccb597-h92b4 manager I0522 18:55:42.238087       1 leaderelection.go:260] successfully acquired lease arc-system/gha-rs-controller

@dennybaa
Copy link
Author

More details, maybe these events are generated automatically during crash I don't know

+ gha-rs-controller-749bfb57cb-clc5h › manager
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	Update strategy set to:	{"updateStrategy": "immediate"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	AutoscalingListener logging parameters changed	{"LogLevel": "debug", "LogFormat": "text"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	Registering scale set metrics
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	starting manager	{"version": "0.9.2"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	controller-runtime.metrics	Starting metrics server
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:11Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
gha-rs-controller-749bfb57cb-clc5h manager I0522 18:59:12.021320       1 leaderelection.go:250] attempting to acquire leader lease arc-system/gha-rs-controller...
- gha-rs-controller-7867ccb597-h92b4 › manager
+ gha-rs-controller-749bfb57cb-wlmwg › manager
gha-rs-controller-7867ccb597-kb7xg manager 2024-05-22T18:59:15Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1alpha1.AutoscalingRunnerSet Informer to sync"}
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:68
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:33
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56
gha-rs-controller-7867ccb597-kb7xg manager 2024-05-22T18:59:15Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1alpha1.AutoscalingListener Informer to sync"}
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:68
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:33
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56
gha-rs-controller-7867ccb597-kb7xg manager 2024-05-22T18:59:15Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1alpha1.AutoscalingListener Informer to sync"}
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:68
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:33
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56
gha-rs-controller-7867ccb597-kb7xg manager 2024-05-22T18:59:15Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1.RoleBinding Informer to sync"}
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:68
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:33
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56
gha-rs-controller-7867ccb597-kb7xg manager 2024-05-22T18:59:15Z	ERROR	controller-runtime.source.EventHandler	failed to get informer from cache	{"error": "Timeout: failed waiting for *v1.Role Informer to sync"}
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:68
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50
gha-rs-controller-7867ccb597-kb7xg manager k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
gha-rs-controller-7867ccb597-kb7xg manager 	k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:33
gha-rs-controller-7867ccb597-kb7xg manager sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
gha-rs-controller-7867ccb597-kb7xg manager 	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/source/kind.go:56
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	Update strategy set to:	{"updateStrategy": "immediate"}
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	AutoscalingListener logging parameters changed	{"LogLevel": "debug", "LogFormat": "text"}
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	Registering scale set metrics
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	starting manager	{"version": "0.9.2"}
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	controller-runtime.metrics	Starting metrics server
gha-rs-controller-749bfb57cb-wlmwg manager 2024-05-22T18:59:15Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
gha-rs-controller-749bfb57cb-wlmwg manager I0522 18:59:15.191641       1 leaderelection.go:250] attempting to acquire leader lease arc-system/gha-rs-controller...
- gha-rs-controller-7867ccb597-kb7xg › manager
gha-rs-controller-749bfb57cb-clc5h manager I0522 18:59:30.202973       1 leaderelection.go:260] successfully acquired lease arc-system/gha-rs-controller
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	DEBUG	events	gha-rs-controller-749bfb57cb-clc5h_2bddbc20-0056-4a46-8e80-91aa3e3941a4 became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"arc-system","name":"gha-rs-controller","uid":"7df82bf8-2f40-464f-bfaa-02055d3379b0","apiVersion":"coordination.k8s.io/v1","resourceVersion":"69483705"}, "reason": "LeaderElection"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingRunnerSet"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "source": "kind source: *v1alpha1.AutoscalingListener"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting Controller	{"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunnerSet"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "source": "kind source: *v1alpha1.EphemeralRunner"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting Controller	{"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1alpha1.EphemeralRunner"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "source": "kind source: *v1.Pod"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting Controller	{"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1alpha1.AutoscalingListener"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Pod"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.ServiceAccount"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.Role"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting EventSource	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "source": "kind source: *v1.RoleBinding"}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting Controller	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener"}
gha-rs-controller-749bfb57cb-clc5h manager E0522 18:59:30.206217       1 event.go:280] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"gha-rs-controller.17d1e4ae48dfce0c", GenerateName:"", Namespace:"arc-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"arc-system", Name:"gha-rs-controller", UID:"7df82bf8-2f40-464f-bfaa-02055d3379b0", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"69483705", FieldPath:""}, Reason:"LeaderElection", Message:"gha-rs-controller-749bfb57cb-clc5h_2bddbc20-0056-4a46-8e80-91aa3e3941a4 became leader", Source:v1.EventSource{Component:"gha-rs-controller-749bfb57cb-clc5h_2bddbc20-0056-4a46-8e80-91aa3e3941a4", Host:""}, FirstTimestamp:time.Date(2024, time.May, 22, 18, 59, 30, 202938892, time.Local), LastTimestamp:time.Date(2024, time.May, 22, 18, 59, 30, 202938892, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"gha-rs-controller-749bfb57cb-clc5h_2bddbc20-0056-4a46-8e80-91aa3e3941a4", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:arc-system:gha-rs-controller" cannot create resource "events" in API group "" in the namespace "arc-system"' (will not retry!)
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting workers	{"controller": "ephemeral-runner-controller", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunner", "worker count": 1}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting workers	{"controller": "ephemeralrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "EphemeralRunnerSet", "worker count": 1}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting workers	{"controller": "autoscalingrunnerset", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingRunnerSet", "worker count": 1}
gha-rs-controller-749bfb57cb-clc5h manager 2024-05-22T18:59:30Z	INFO	Starting workers	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "worker count": 1}

@dennybaa
Copy link
Author

Noop, I can confirm, doesn't matter if it crashes or not
image

plz reopen @nikola-jokic the issue is out there

@nikola-jokic
Copy link
Collaborator

Hey @dennybaa,

I was able to reproduce this issue when CRDs are introduced after the installation. There is no need and to be honest, I wouldn't want to add events permission when we normally don't need it. The expectation is that you have CRDs installed before the controller. That is likely why some of the installations you made were successful without this error.

@dennybaa
Copy link
Author

@nikola-jokic Thanks a lot for the feedback! Less permissions better :) This one in fact doesn't bother me much.

But one small question, I've got it all running on spot instances generally everything works fine. But from time to time once in a while (let's say 2 month) scale set completely disappears (i believe i'm running with the debug off, so no details, only reinstall of the scale set and possibly restart of the controller (maybe reinstall, don't remeber) brings things back to life :(

Is there any relevant issue or maybe you are aware of?

@dennybaa dennybaa deleted the patch-1 branch May 31, 2024 17:07
@nikola-jokic
Copy link
Collaborator

Hey @dennybaa,

It might be related to this one. I cannot say without the controller log and the listener log gathered at the time when the listener went offline, but since it went offline after some time, I would guess this issue fixed with this PR fixed the problem. ☺️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants