Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error Metrics Openshift #21825

Closed
Fryguy opened this issue Apr 20, 2022 Discussed in #21824 · 1 comment · Fixed by #21829
Closed

Error Metrics Openshift #21825

Fryguy opened this issue Apr 20, 2022 Discussed in #21824 · 1 comment · Fixed by #21829
Assignees
@agrare
Labels
bug

Comments

@Fryguy
Copy link
Member

Fryguy commented Apr 20, 2022

Discussed in #21824

Originally posted by Crankz31 April 20, 2022
Hi everyone,

I saw something weird, i have built a multi-region infrastructure with multiple technologies

we have a Openshift region, i go through the logs and find an error about collecting metrics, i find that weird because on the provider view i have some metrics, i can reproduce this error by clicking on "capture metrics" or by waiting the next iteration :

[----] E, [2022-04-20T03:09:37.142445 #396679:2b0010c8f95c] ERROR -- evm: [NameError]: uninitialized constant MetricsCapture  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2022-04-20T03:09:37.142522 #396698:2aed90cc5968] ERROR -- evm: [NameError]: uninitialized constant MetricsCapture  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2022-04-20T03:09:37.142545 #396679:2b0010c8f95c] ERROR -- evm: /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:6:in `perf_capture_object'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:10:in `perf_collect_metrics'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:92:in `block in just_perf_capture'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:62:in `realtime_block'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:88:in `just_perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:37:in `perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:13:in `perf_capture_realtime'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:376:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:80:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:122:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:267:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:114:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:96:in `start'
lib/workers/bin/run_single_worker.rb:128:in `<main>'
[----] I, [2022-04-20T03:09:37.142655 #396679:2b0010c8f95c]  INFO -- evm: MIQ(MiqQueue#delivered) Message id: [3000000323741], State: [error], Delivered in [0.003994452] seconds
[----] E, [2022-04-20T03:09:37.142652 #396698:2aed90cc5968] ERROR -- evm: /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:6:in `perf_capture_object'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:10:in `perf_collect_metrics'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:92:in `block in just_perf_capture'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:62:in `realtime_block'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:88:in `just_perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:37:in `perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:13:in `perf_capture_realtime'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:376:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:80:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:122:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:267:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:114:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:96:in `start'
lib/workers/bin/run_single_worker.rb:128:in `<main>'
[----] I, [2022-04-20T03:09:37.142787 #396698:2aed90cc5968]  INFO -- evm: MIQ(MiqQueue#delivered) Message id: [3000000323742], State: [error], Delivered in [0.003839108] seconds
[----] I, [2022-04-20T03:09:37.148204 #396679:2b0010c8f95c]  INFO -- evm: MIQ(ManageIQ::Providers::Openshift::ContainerManager::MetricsCollectorWorker::Runner#get_message_via_drb) Message id: [3000000323743], MiqWorker id: [3000000000015], Zone: [default], Role: [ems_metrics_collector], Server: [], MiqTask id: [], Ident: [openshift], Target id: [], Instance id: [3000000000028], Task id: [], Command: [ContainerImage.perf_capture_realtime], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [22.324544678] seconds
[----] I, [2022-04-20T03:09:37.149078 #396698:2aed90cc5968]  INFO -- evm: MIQ(ManageIQ::Providers::Openshift::ContainerManager::MetricsCollectorWorker::Runner#get_message_via_drb) Message id: [3000000323744], MiqWorker id: [3000000000016], Zone: [default], Role: [ems_metrics_collector], Server: [], MiqTask id: [], Ident: [openshift], Target id: [], Instance id: [3000000000029], Task id: [], Command: [ContainerImage.perf_capture_realtime], Timeout: [600], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [22.321949792] seconds
[----] I, [2022-04-20T03:09:37.150136 #396679:2b0010c8f95c]  INFO -- evm: MIQ(MiqQueue#deliver) Message id: [3000000323743], Delivering...
[----] I, [2022-04-20T03:09:37.150967 #396679:2b0010c8f95c]  INFO -- evm: MIQ(ContainerImage#just_perf_capture) [realtime] Capture for ContainerImage name: [name], id: [3000000000028], start_time: [2022-04-19 00:00:00 UTC]...
[----] I, [2022-04-20T03:09:37.151122 #396698:2aed90cc5968]  INFO -- evm: MIQ(MiqQueue#deliver) Message id: [3000000323744], Delivering...
[----] I, [2022-04-20T03:09:37.151992 #396698:2aed90cc5968]  INFO -- evm: MIQ(ContainerImage#just_perf_capture) [realtime] Capture for ContainerImage name: [name], id: [3000000000029], start_time: [2022-04-19 00:00:00 UTC]...
[----] E, [2022-04-20T03:09:37.153115 #396679:2b0010c8f95c] ERROR -- evm: MIQ(MiqQueue#deliver) Message id: [3000000323743], Error: [uninitialized constant MetricsCapture]
[----] E, [2022-04-20T03:09:37.153264 #396679:2b0010c8f95c] ERROR -- evm: [NameError]: uninitialized constant MetricsCapture  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2022-04-20T03:09:37.153365 #396679:2b0010c8f95c] ERROR -- evm: /var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:6:in `perf_capture_object'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:10:in `perf_collect_metrics'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:92:in `block in just_perf_capture'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.1/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:62:in `realtime_block'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:88:in `just_perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:37:in `perf_capture'
/var/www/miq/vmdb/app/models/metric/ci_mixin/capture.rb:13:in `perf_capture_realtime'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:376:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:80:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:122:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:267:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:114:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:96:in `start'
lib/workers/bin/run_single_worker.rb:128:in `<main>'
```</div>
@Fryguy Fryguy added the bug label Apr 20, 2022
@agrare
Copy link
Member

agrare commented Apr 20, 2022

Am able to reproduce this on master, the issue is here https://github.com/ManageIQ/manageiq/blob/master/app/models/metric/ci_mixin/capture.rb#L6 for objects that aren't namespaced, thus self.class.module_parent::MetricsCapture on e.g. ::ContainerImage returns ::MetricsCapture which does not exist.

It looks like one issue is somewhere in the k8s/openshift refresh code we are hard-coding the :type to 'ContainerImage' which is bypassing the core "ensure the object isn't created without a namespace" check.

Looking to see if there are any other cases where we hit this error or if fixing the ContainerImage namespace is sufficient.

This was referenced Apr 20, 2022
Open
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@agrare agrare

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only capture metrics for supported ContainerImages agrare/manageiq
2 participants
@Fryguy @agrare