Jumpstart #2902

Ulincsys · 2024-08-27T22:23:55Z

Description

Add jumpstart subroutine for more robust process management
Fix logging inconsistencies
Fix augur api start and augur collection start commands
Add Textual dependency

In preparation for changes coming in the Admin Console release, it is desirable to have a proper "process manager" in place that can communicate with external clients and control Augur processes from a central perspective. This change enables more granular control over Augur's various worker types, and in the future more features are possible.

Signed commits

Yes, I signed my commits.

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/api.py

@@ -14,15 +14,16 @@

 from augur.application.db.session import DatabaseSession


[pylint] _{reported by reviewdog 🐶}
W0611: Unused DatabaseSession imported from augur.application.db.session (unused-import)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+    """
+    Stop collection tasks if they are running, block until complete
+    """
+    processes = get_augur_processes()


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'processes' from outer scope (line 443) (redefined-outer-name)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+            stopped.append(p)
+            p.terminate()
+
+    if not len(stopped):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+
+    killed = []
+    while True:
+        for i in range(len(alive)):


[pylint] _{reported by reviewdog 🐶}
C0200: Consider using enumerate instead of iterating with range and len (consider-using-enumerate)

github-actions · 2024-08-27T22:24:31Z

augur/application/cli/backend.py

+        for i in reversed(killed):
+            alive.pop(i)
+
+        if not len(alive):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+            if command["name"] == cmd:
+                packet["cmd"] = cmd
+                for arg in command["args"]:
+                    if arg["required"] and not len(args):


[pylint] _{reported by reviewdog 🐶}
C1802: Do not use len(SEQUENCE) without comparison to determine if a sequence is empty (use-implicit-booleaness-not-len)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+
+        for command in self.spec["commands"]:
+            help_str += f'\t> [bold yellow]{command["name"]}'
+            for arg in command["args"]:


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'arg' from outer scope (line 388) (redefined-outer-name)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+                log(e)
+                log(msg)
+
+def run_app(socket_file = Path("jumpstart.sock"), ctx = None):


[pylint] _{reported by reviewdog 🐶}
W0621: Redefining name 'socket_file' from outer scope (line 387) (redefined-outer-name)

github-actions · 2024-08-27T22:24:32Z

jumpstart/tui.py

+
+import json
+import asyncio
+import logging


[pylint] _{reported by reviewdog 🐶}
W0611: Unused import logging (unused-import)

github-actions · 2024-08-27T22:24:33Z

jumpstart/tui.py

+from textual.app import App, on
+from textual.binding import Binding
+from textual.message import Message
+from textual.events import Load, Mount


[pylint] _{reported by reviewdog 🐶}
W0611: Unused Load imported from textual.events (unused-import)

sgoggins

I think @ABrain7710 needs to check the backend.py edits.

sgoggins · 2024-08-27T22:26:22Z

augur/api/view/init.py

@@ -91,4 +91,4 @@ def write_settings(current_settings):
 # Initialize logging
 def init_logging():
    global logger
-    logger = AugurLogger("augur_view", reset_logfiles=True).get_logger()
+    logger = AugurLogger("augur_view", reset_logfiles=False).get_logger()


Does this only affect view?

No, it previously would reset all log files globally. So now the log files will continue to exist after starting Augur

sgoggins · 2024-08-27T22:28:51Z

augur/application/cli/collection.py

@@ -22,14 +22,15 @@
 from augur.application.db.session import DatabaseSession
 from augur.application.logs import AugurLogger
 from augur.application.db.lib import get_value
-from augur.application.cli import test_connection, test_db_connection, with_database
+from augur.application.cli import test_connection, test_db_connection, with_database, DatabaseContext


Is this one just incorporating the latest from @ABrain7710 ?

No, the changes are unrelated

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <gogginss@missouri.edu> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

…g to understand event API limits Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

…in the data returned from the events API. It DOES look like what is returned may be a partially complte set in some cases. Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

…taException class (I think) Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

…e to the API Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

…empty repos in repo_info Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Traceback (most recent call last): File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/pool.py', line 366, in workloop put((READY, (job, i, result, inqW_fd))) File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/queues.py', line 366, in put self.send_payload(ForkingPickler.dumps(obj)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File '/home/ubuntu/github/virtualenvs/hosted/lib/python3.11/site-packages/billiard/reduction.py', line 56, in dumps cls(buf, protocol).dump(obj) billiard.pool.MaybeEncodingError: Error sending result: ''(1, <ExceptionInfo: MetadataException('\'checks\' | Additional metadata: required_output: {}')>, None)''. Reason: ''PicklingError('Can\'t pickle <class \'augur.tasks.util.metadata_exception.MetadataException\'>: it\'s not the same object as augur.tasks.util.metadata_exception.MetadataException')''. Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Signed-off-by: Sean P. Goggins <outdoors@acm.org>

Ulincsys requested a review from sgoggins as a code owner August 27, 2024 22:23

github-actions bot reviewed Aug 27, 2024

View reviewed changes

sgoggins previously approved these changes Aug 27, 2024

View reviewed changes

Ulincsys dismissed sgoggins’s stale review via c2e128d August 27, 2024 23:03

Ulincsys and others added 26 commits August 27, 2024 18:06

Pull relevant mods from admin-changes

36ae56c

Signed-off-by: Ulincsys <ulincsys@gmail.com>

initial work on jumpstart project

f0e27a3

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Implement decoding and blocking

709e771

Signed-off-by: Ulincsys <ulincsys@gmail.com>

add TUI, expand server basis

d7c6a87

Signed-off-by: Ulincsys <ulincsys@gmail.com>

update encoding

3bac827

Signed-off-by: Ulincsys <ulincsys@gmail.com>

continue work on concurrency

cf6c9ec

Signed-off-by: Ulincsys <ulincsys@gmail.com>

mvp

121ee03

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Fix improper client shutdown on some platforms

761cc50

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Add DB check before parsing start command

e920890

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Add github graphql data access'

2067d44

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Improve paginate_resource logic

140ea06

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Remove unused method

5577ce0

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Move repo_info to new graphql endpoint

06d2c06

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Migrate pr files to github_graphql_data_access

fde0ec4

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Indent client'

06bc536

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Remove strip

2e05618

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Define keys as list

ec71977

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Return data properly

034333e

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Add self

93577d7

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Fix syntax error

078bc8a

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Fix syntax error

358b146

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Fixes

a96b380

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Raise from original exception

97af39b

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Make annoying info log a debug

ecb7b57

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

updating version

46438df

Signed-off-by: Sean P. Goggins <gogginss@missouri.edu> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Address release bugs

f3cb92a

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

ABrain7710 and others added 26 commits August 27, 2024 18:06

Handle url not found in contributor breadth worker

cf99b23

Signed-off-by: Andrew Brain <andrewbrain2019@gmail.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

docker fun

44b1c62

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

failed docker config

456b0f5

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

saving failed config

07c2c38

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Docker compose script now workinggit add docker/

f0cad14

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

removed old file

6a6ba73

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

debugging a couple errors. One with a missing import, the other tryin…

56808aa

…g to understand event API limits Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

possible fix for commit messages

ca0d4b2

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

changed page_count > 300 to a warning instead of an exception

f512443

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

updated unterminated string error

457b03d

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Fixing NPM

d7d435b

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

events page count seems to top out at 1,000, not 300

21660e1

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

update to events to use correct column name

e665aec

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

update to augur event collection

b80a480

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

logging event so we know what's supposed to be there

c35f449

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

fixing pickling error caused by not consistently importing our Metada…

de320b2

…taException class (I think) Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

additional update for when issue is not an object in an event respons…

dbdb7e3

…e to the API Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Trying to fix issue when the committer count API returns no value on …

4c45d8f

…empty repos in repo_info Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

exception handling

ed50be2

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

trying to fix the empty committer count issue

f510da4

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

added some comments explaining the query_committers_count method changes

82b9159

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

Making repo GONE log message more clear.

8967212

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

formatting exception for OpenSSF Scorecard pickling error more typically

97fee7e

Signed-off-by: Sean P. Goggins <s@goggins.com> Signed-off-by: Ulincsys <ulincsys@gmail.com>

fix for missing comma, fix misspelling

ebca962

Signed-off-by: Ulincsys <ulincsys@gmail.com>

Ulincsys force-pushed the jumpstart branch from c2e128d to ebca962 Compare August 27, 2024 23:06

Merge branch 'dev' into jumpstart

491b688

Signed-off-by: Sean P. Goggins <outdoors@acm.org>

sgoggins merged commit aae2f19 into dev Aug 27, 2024
6 of 7 checks passed

Ulincsys deleted the jumpstart branch August 27, 2024 23:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jumpstart #2902

Jumpstart #2902

Ulincsys commented Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

github-actions bot Aug 27, 2024

sgoggins left a comment

sgoggins Aug 27, 2024

Ulincsys Aug 27, 2024

sgoggins Aug 27, 2024

Ulincsys Aug 27, 2024

		@@ -14,15 +14,16 @@

		from augur.application.db.session import DatabaseSession

Jumpstart #2902

Jumpstart #2902

Conversation

Ulincsys commented Aug 27, 2024

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

github-actions bot Aug 27, 2024

Choose a reason for hiding this comment

sgoggins left a comment

Choose a reason for hiding this comment

sgoggins Aug 27, 2024

Choose a reason for hiding this comment

Ulincsys Aug 27, 2024

Choose a reason for hiding this comment

sgoggins Aug 27, 2024

Choose a reason for hiding this comment

Ulincsys Aug 27, 2024

Choose a reason for hiding this comment