Materialized View Update Updates #2679

sgoggins · 2024-02-08T16:35:48Z

Description

Updates logic for refreshing materialized views
Adds indexes to some materialized views that did not have them in order to allow concurrent refresh
Updates flower version to fix a time syncrhonization issue.

Updating materialized view branch with latest dev changes

Added indexes to materialized views missing indexes. Updated materialized view refresh script.

Updating materialized view refresh branch patch

Signed-off-by: Sean Goggins <outdoors@acm.org>

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

pylint

augur/tasks/github/util/util.py|72 col 4| C0415: Import outside toplevel (augur.tasks.github.util.gh_graphql_entities.GitHubRepo) (import-outside-toplevel)
augur/tasks/gitlab/events_task.py|204 col 5| W0511: TODO: Add unique key for this (fixme)
augur/tasks/gitlab/gitlab_api_handler.py|26 col 5| W0511: TODO: Add bad credentials detection that removes key (fixme)
augur/tasks/gitlab/gitlab_api_key_handler.py|76 col 8| C0415: Import outside toplevel (augur.application.db.models.WorkerOauth) (import-outside-toplevel)
augur/tasks/gitlab/gitlab_random_key_auth.py|8| R0903: Too few public methods (1/2) (too-few-public-methods)
augur/tasks/gitlab/gitlab_task_session.py|23 col 8| C0415: Import outside toplevel (augur.tasks.init.celery_app.engine) (import-outside-toplevel)
augur/tasks/gitlab/issues_task.py|42 col 12| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/tasks/gitlab/issues_task.py|91| R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
augur/tasks/gitlab/issues_task.py|229 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/issues_task.py|235 col 23| W0612: Unused variable 'page' (unused-variable)
augur/tasks/gitlab/issues_task.py|274 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/issues_task.py|305 col 8| R1704: Redefining argument with the local name 'data' (redefined-argument-from-local)
augur/tasks/gitlab/merge_request_task.py|145 col 5| W0511: TODO: Setup unique key on asignees with a value of ('cntrb_id', 'pull_request_id') and add 'cntrb_id' to assingee data (fixme)
augur/tasks/gitlab/merge_request_task.py|385 col 5| W0511: TODO: Need to add unique key with pull_request_id and cntrb_id to insert gitlab reviewers (fixme)
augur/tasks/gitlab/merge_request_task.py|1| C0114: Missing module docstring (missing-module-docstring)
augur/tasks/gitlab/merge_request_task.py|36 col 8| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/tasks/gitlab/merge_request_task.py|129 col 8| R1704: Redefining argument with the local name 'data' (redefined-argument-from-local)
augur/tasks/gitlab/merge_request_task.py|178 col 14| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|212 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/merge_request_task.py|243 col 8| R1704: Redefining argument with the local name 'data' (redefined-argument-from-local)
augur/tasks/gitlab/merge_request_task.py|280 col 14| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|312 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/merge_request_task.py|344 col 14| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|377 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/merge_request_task.py|412 col 14| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|445 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/merge_request_task.py|481 col 14| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)
augur/tasks/gitlab/merge_request_task.py|490| C0116: Missing function or method docstring (missing-function-docstring)
augur/tasks/gitlab/merge_request_task.py|503 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/gitlab/merge_request_task.py|534 col 8| W0622: Redefining built-in 'id' (redefined-builtin)
augur/tasks/start_tasks.py|24| C0413: Import "from augur.tasks.github.pull_requests.commits_model.tasks import process_pull_request_commits" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|25| C0413: Import "from augur.tasks.git.dependency_tasks.tasks import process_ossf_dependency_metrics" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|26| C0413: Import "from augur.tasks.github.traffic.tasks import collect_github_repo_clones_data" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|27| C0413: Import "from augur.tasks.gitlab.merge_request_task import collect_gitlab_merge_requests, collect_merge_request_comments, collect_merge_request_metadata, collect_merge_request_reviewers, collect_merge_request_commits, collect_merge_request_files" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|28| C0413: Import "from augur.tasks.gitlab.issues_task import collect_gitlab_issues, collect_gitlab_issue_comments" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|29| C0413: Import "from augur.tasks.gitlab.events_task import collect_gitlab_issue_events, collect_gitlab_merge_request_events" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|30| W0401: Wildcard import augur.tasks.git.facade_tasks (wildcard-import)
augur/tasks/start_tasks.py|30| C0413: Import "from augur.tasks.git.facade_tasks import *" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|31| W0401: Wildcard import augur.tasks.db.refresh_materialized_views (wildcard-import)
augur/tasks/start_tasks.py|31| C0413: Import "from augur.tasks.db.refresh_materialized_views import *" should be placed at the top of the module (wrong-import-position)
augur/tasks/start_tasks.py|99| C0116: Missing function or method docstring (missing-function-docstring)
augur/tasks/start_tasks.py|101 col 4| W0612: Unused variable 'logger' (unused-variable)
augur/tasks/start_tasks.py|170| C0116: Missing function or method docstring (missing-function-docstring)
augur/tasks/start_tasks.py|170 col 68| W0613: Unused argument 'days_until_collect_again' (unused-argument)
augur/tasks/util/collection_util.py|134| C0115: Missing class docstring (missing-class-docstring)
augur/util/repo_load_controller.py|205 col 50| W1401: Anomalous backslash in string: '.'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|205 col 55| W1401: Anomalous backslash in string: '/'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|205 col 68| W1401: Anomalous backslash in string: '-'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|205 col 74| W1401: Anomalous backslash in string: '/'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|205 col 88| W1401: Anomalous backslash in string: '-'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|206 col 50| W1401: Anomalous backslash in string: '.'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|206 col 55| W1401: Anomalous backslash in string: '/'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|206 col 69| W1401: Anomalous backslash in string: '-'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|206 col 76| W1401: Anomalous backslash in string: '/'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|206 col 89| W1401: Anomalous backslash in string: '-'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
augur/util/repo_load_controller.py|32| C0115: Missing class docstring (missing-class-docstring)
augur/util/repo_load_controller.py|166 col 24| W0212: Access to a protected member _mapping of a client class (protected-access)
augur/util/repo_load_controller.py|175 col 4| C0116: Missing function or method docstring (missing-function-docstring)
augur/util/repo_load_controller.py|196 col 4| C0116: Missing function or method docstring (missing-function-docstring)
augur/util/repo_load_controller.py|210 col 8| R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
augur/util/repo_load_controller.py|257 col 8| R1705: Unnecessary "else" after "return", remove the "else" and de-indent the code inside it (no-else-return)
augur/util/repo_load_controller.py|272 col 23| E0602: Undefined variable 'function' (undefined-variable)
augur/util/repo_load_controller.py|196 col 4| R0912: Too many branches (14/12) (too-many-branches)
tests/test_applicaton/test_db/test_models/test_augur_operations/test_user_repo.py|307 col 18| C0209: Formatting a regular string which could be an f-string (consider-using-f-string)

github-actions · 2024-02-08T16:36:37Z

augur/api/view/api.py

+                org_name, repo_name = Repo.parse_github_repo_url(url)
+                repo_git = f"https://gitlab.com/{org_name}/{repo_name}"
+
+                # TODO: gitlab ensure the whole repo git is inserted so it can be found here


[pylint] _{reported by reviewdog 🐶}
W0511: TODO: gitlab ensure the whole repo git is inserted so it can be found here (fixme)

github-actions · 2024-02-08T16:36:37Z

augur/api/view/routes.py

 import logging
+import math
 from flask import Flask, render_template, render_template_string, request, abort, jsonify, redirect, url_for, session, flash
 from sqlalchemy.orm.exc import NoResultFound
 from .utils import *


[pylint] _{reported by reviewdog 🐶}
E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)

github-actions · 2024-02-08T16:36:37Z

augur/api/view/routes.py

 import logging
+import math
 from flask import Flask, render_template, render_template_string, request, abort, jsonify, redirect, url_for, session, flash
 from sqlalchemy.orm.exc import NoResultFound
 from .utils import *


[pylint] _{reported by reviewdog 🐶}
W0401: Wildcard import utils (wildcard-import)

github-actions · 2024-02-08T16:36:37Z

augur/api/view/routes.py

@@ -37,9 +41,9 @@ def root(path=""):
 def logo(brand=None):


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

github-actions · 2024-02-08T16:36:37Z

augur/api/view/utils.py

 from pathlib import Path
 from concurrent.futures import ThreadPoolExecutor
 from flask import render_template, flash, url_for, Flask
+from .init import init_logging


[pylint] _{reported by reviewdog 🐶}
E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)

github-actions · 2024-02-08T16:36:38Z

augur/tasks/frontend.py


    print(repo_url, result)


+# TODO: Change to github specific


[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Change to github specific (fixme)

github-actions · 2024-02-08T16:36:38Z

augur/tasks/frontend.py

@@ -66,24 +74,25 @@ def add_org_repo_list(user_id, group_name, urls):



-
+# TODO: Change to github specific
 @celery.task
 def add_repo(user_id, group_name, repo_url):


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

github-actions · 2024-02-08T16:36:39Z

augur/tasks/frontend.py


    print(repo_url, result)


+# TODO: Change to github specific
 @celery.task
 def add_org(user_id, group_name, org_url):


[pylint] _{reported by reviewdog 🐶}
C0116: Missing function or method docstring (missing-function-docstring)

github-actions · 2024-02-08T16:36:39Z

augur/tasks/github/pull_requests/tasks.py


+def process_pull_requests(pull_requests, task_name, repo_id, logger, augur_db):


[pylint] _{reported by reviewdog 🐶}
R0914: Too many local variables (32/30) (too-many-locals)

github-actions · 2024-02-08T16:36:39Z

augur/tasks/github/util/github_paginator.py

@@ -154,6 +154,8 @@ class GithubApiResult(Enum):
    SECONDARY_RATE_LIMIT = 4
    RATE_LIMIT_EXCEEDED = 5
    ABUSE_MECHANISM_TRIGGERED = 6
+    # TODO: Add bad credentials detection that removes key 


[pylint] _{reported by reviewdog 🐶}
W0511: TODO: Add bad credentials detection that removes key (fixme)

Signed-off-by: Sean Goggins <seangoggins@outdoors@acm.org>

sgoggins · 2024-02-12T16:14:47Z

@ABrain7710 / @IsaacMilarky / @Ulincsys : This PR is ready for review.

IsaacMilarky

LGTM

sgoggins and others added 5 commits January 26, 2024 12:49

Merge pull request #2670 from chaoss/dev

5e86c55

Updating materialized view branch with latest dev changes

Updated materialized views to include the right logic.

54c2086

Added indexes to materialized views missing indexes. Updated materialized view refresh script.

updated file name

3667942

Merge pull request #2673 from chaoss/dev

6224cfc

Updating materialized view refresh branch patch

fixing flower time synchronization issue

f3f7fbe

Signed-off-by: Sean Goggins <outdoors@acm.org>

github-actions bot reviewed Feb 8, 2024

View reviewed changes

sgoggins changed the base branch from main to dev February 8, 2024 16:41

adding conn = op.get_bind()

f46ee05

Signed-off-by: Sean Goggins <seangoggins@outdoors@acm.org>

sgoggins assigned Ulincsys, IsaacMilarky and ABrain7710 Feb 12, 2024

sgoggins requested review from ABrain7710 and IsaacMilarky February 12, 2024 16:15

sgoggins added documentation Updates documentation database Related to Augur's unifed data model dependencies Pull requests that update a dependency file labels Feb 12, 2024

IsaacMilarky approved these changes Feb 12, 2024

View reviewed changes

sgoggins merged commit 335864c into dev Feb 12, 2024
8 checks passed

sgoggins deleted the mat-view-path-23 branch February 20, 2024 00:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Materialized View Update Updates #2679

Materialized View Update Updates #2679

sgoggins commented Feb 8, 2024

github-actions bot left a comment

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

github-actions bot Feb 8, 2024

sgoggins commented Feb 12, 2024

IsaacMilarky left a comment


		def process_pull_requests(pull_requests, task_name, repo_id, logger, augur_db):

Materialized View Update Updates #2679

Materialized View Update Updates #2679

Conversation

sgoggins commented Feb 8, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

github-actions bot Feb 8, 2024

Choose a reason for hiding this comment

sgoggins commented Feb 12, 2024

IsaacMilarky left a comment

Choose a reason for hiding this comment