Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1954640: Support of gatherers with different periods #399

Conversation

Serhii1011010
Copy link
Contributor

@Serhii1011010 Serhii1011010 commented Apr 14, 2021

This PR mainly implements support of gatherers with different periods, but also refactores gathering logic making it more clear, easier to use and test.

Categories

  • Bugfix
  • Enhancement
  • Backporting
  • Others (CI, Infrastructure, Documentation)

Sample archive

Documentation

  • docs/insights-archive-sample/insights-operator/gathers.json # changed format little bit

Unit Tests

  • pkg/controller/periodic/periodic_test.go
  • pkg/gather/clusterconfig/clusterconfig_gatherer_test.go
  • pkg/gather/gather_test.go
  • pkg/gather/tasks_processing_test.go
  • pkg/gather/workloads/workloads_gatherer_test.go

Privacy

No new data was collected

References

https://issues.redhat.com/browse/CCXDEV-4366
https://bugzilla.redhat.com/show_bug.cgi?id=???
https://access.redhat.com/solutions/???

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 14, 2021
@Serhii1011010 Serhii1011010 marked this pull request as draft April 14, 2021 16:03
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 14, 2021
pkg/gather/interface.go Outdated Show resolved Hide resolved
@tremes
Copy link
Contributor

tremes commented Apr 15, 2021

I think it's a lot of changes in this PR. I mean it includes good ideas, but I am not sure it's the right timing to do such a big change (Maybe I would rather wait when there's no feature freeze = no bugzillas needed). I think it might also happen that we won't need this extra period gatherer (that's also the reason why I wouldn't invest such an effort right now).

I know it's still WIP, but I am missing the original workloads gatherer. I was also playing little bit with this idea and maybe something like this could be sufficient, but I am not saying that's the way we should do that.
I am also not sure I like the split of the metadata file into more files, but maybe it's OK :)

@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 15, 2021
@Serhii1011010
Copy link
Contributor Author

@tremes what's the problem with having bugzilla, btw?

I am also not sure I like the split of the metadata file into more files, but maybe it's OK :)

I'd prefer the split, but we can actually keep one file by changing logic little bit.

@tremes
Copy link
Contributor

tremes commented Apr 15, 2021

@Sergey1011010 No problem with having bugzillas. My point is that we now need a new bugzilla for every PR and if we will need some additional fixes then we need new bugzilla for each (it's always a risk with some bigger changes). I know that we didn't care much about the feature freeze in the past, but it probably makes sense not to make some bigger changes when there is this freeze.

@Serhii1011010
Copy link
Contributor Author

@tremes so do we want to postpone this?

@tremes
Copy link
Contributor

tremes commented Apr 16, 2021

@Sergey1011010 I am not sure. Maybe I am too cautious. Let's see what other team members think.

pkg/gather/gather.go Outdated Show resolved Hide resolved
pkg/gather/gather.go Outdated Show resolved Hide resolved
pkg/gather/gather.go Outdated Show resolved Hide resolved
@tremes
Copy link
Contributor

tremes commented Apr 21, 2021

I've got another idea about the extra period. We could introduce a new variable e.g dailyInterval in the periodicTrigger method (in periodic.go) and increment this variable by the "common" interval in https://github.com/openshift/insights-operator/pull/399/files#diff-dec12f0cf28a91a837b46e755b46c081c9014121cdb770b2cd8ac916af29c8deR173 and then only add condition e.g:

if dailyInterval >= 24*time.Hour {
    // extra gather with workloads here
}

I think this would be more clear.

@Serhii1011010
Copy link
Contributor Author

@tremes I think it works only as a temporarily solution, it's not very flexible

@Serhii1011010
Copy link
Contributor Author

@tremes changed the logic of periods. Now we have gatherers which are added to each archive and optional gatherers which are processed only when it's there time, periods don't need to match with interval in config. Optional gatherers have periods(time.Duration) which are now defined in the file, but can be easily moved to the config. Tomorrow I'm gonna replace some_gatherer with the real one, rebase and it will be ready.

@Serhii1011010 Serhii1011010 force-pushed the support-of-gatherers-with-different-periods branch 3 times, most recently from 1d67d27 to 1938946 Compare April 23, 2021 08:43
pkg/controller/operator.go Outdated Show resolved Hide resolved
pkg/gather/gather.go Outdated Show resolved Hide resolved
@Serhii1011010 Serhii1011010 force-pushed the support-of-gatherers-with-different-periods branch from 1938946 to a704a1a Compare April 27, 2021 14:08
@Serhii1011010 Serhii1011010 changed the title [WIP] Support of gatherers with different periods Support of gatherers with different periods Apr 27, 2021
@Serhii1011010 Serhii1011010 marked this pull request as ready for review April 27, 2021 17:43
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 10, 2021
@openshift-ci
Copy link

openshift-ci bot commented May 10, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rluders, Sergey1011010, tremes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [Sergey1011010,rluders,tremes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Serhii1011010
Copy link
Contributor Author

/retest

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

13 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@tremes
Copy link
Contributor

tremes commented May 11, 2021

/test insights-operator-e2e-tests

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@tremes
Copy link
Contributor

tremes commented May 11, 2021

/test insights-operator-e2e-tests

@rluders
Copy link
Contributor

rluders commented May 11, 2021

/retest

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

2 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link

openshift-ci bot commented May 11, 2021

@Sergey1011010: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-gcp-upgrade 04201cb link /test e2e-gcp-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@Serhii1011010
Copy link
Contributor Author

/test e2e-gcp-upgrade

@Serhii1011010
Copy link
Contributor Author

/test insights-operator-e2e-tests

@openshift-ci
Copy link

openshift-ci bot commented May 11, 2021

@Sergey1011010: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

  • /test e2e
  • /test e2e-agnostic-upgrade
  • /test images
  • /test insights-operator-e2e-tests
  • /test unit

Use /test all to run all jobs.

In response to this:

/test e2e-gcp-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-merge-robot openshift-merge-robot merged commit 7a75a38 into openshift:master May 11, 2021
@openshift-ci
Copy link

openshift-ci bot commented May 11, 2021

@Sergey1011010: All pull requests linked via external trackers have merged:

Bugzilla bug 1954640 has been moved to the MODIFIED state.

In response to this:

Bug 1954640: Support of gatherers with different periods

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants