Make exception handling explicit #5202

holmanb · 2024-04-23T02:42:27Z

Impetus

Following the transition to the new exit code (2) for easier user introspection of cloud-init, there was a request to audit current callsites for error conditions that are logged at too high or low of a logging level. This PR starts that effort, but only scratches the surface. Much more work needs to be done auditing log levels.

Objective

This PR is the result of an audit of exceptions of type Exception. This audit had a simple goal: prevent cloud-init from swallowing exceptions. We don't want exceptions such as AttributeError, IndexError, KeyError, TypeError, ZeroDivisionError to be ignored when the exception that we expect to handle is a failed filesystem operation - this would mask important errors at runtime.

Rules

To complete the objective, a set of rules was defined and implemented. They are described in one of the commit messages:

Require a minimum of one of the following in each Exception handler:

Log a warning at level WARN or higher containing the exception type and message

Re-raise the exception without redefining its type.

By following these rules, we can expect that every Exception handler will either issue its own warning or raise to another exception which will raise a warning. By doing this, silently ignoring important classes of exceptions will not happen.

Any log of WARN would suffice for ensuring that these exceptions are logged, however standardizing this message across the codebase simplifies code auditing and makes it clear to developers when they see this message what has happened.

Methodology

Inspect the code for sources of expected exceptions and add them above the Exception handler, and adding a proper WARN log to the existing Exception exception. This allows us to flesh out any unexpected changes easily in integration testing without risking unintentional logic changes.

Reviewer notes

I recommend reading the individual commit messages while reviewing. This PR carries some risk of changing cloud-init logic, but it attempts to mitigate the risk by retaining Exception (in most cases) while making all Exception handling into a recoverable error. I believe that the risk that this poses will (after thoroughly reviewed) be outweighed by the current risk we have of unintentionally silencing unexpected exceptions.

A few things to keep in mind while reviewing:

subp.subp() throws OSError
OSError is thrown by many filesystem and operating system operations. It is used liberally in this PR.
I don't like the repetitive LOG.warning("Unhandled..."). Ideas are welcome for better messaging and maybe a helper function would make sense - maybe something in log.py?
Perhaps the rules above should probably go in a dev doc?

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

TheRealFalcon

Thanks for spearheading this effort! I left some comments inline.

Beyond those, I have a few more broad concerns.

Your LOG.warning("Unhandled exception: %s", e) rule is a bit too rigid to me. There are a bunch of places that had something like:

try:
    do_something()
except Exception as e:
    LOG.warning("oh no! %s", e)
    recover()

that turned into something like:

try:
    do_something
except ValueError as e:
    LOG.warning("oh no! %s", e)
    recover()
except Exception as e:
    LOG.warning("Unhandled exception: %s", e)
    LOG.warning("oh no! %s", e)
    recover()

I'm not sure how this is an improvement. If we're logging the exception with the old message, then why is the broad exception a problem? Sure, best practices says you should only catch exceptions as narrowly as you need to, but there are certainly use cases for catching Exception, and as long as we're not actively swallowing exceptions, I don't see the problem...especially if the exception handling is sitting at the top of a giant call stack. In those cases it makes sense to be overly broad in our handling.

To take an example from a recent change that I approved:

cloud-init/cloudinit/net/netplan.py

Line 268 in a22d853

except Exception as e:

I saw the Exception here, thought about it for a second, and then decided that it was the best course of action. For two reasons:

We're logging the exception, so any unexpected issues with still get logged.
I quite literally don't care what happens for the purposes of setting up cloud-init. The netplan API could raise a YourServerIsLiterallyOnFireException. We'll log it, bail, and then do our own v2 rendering.

In this case, what benefit is there to narrowing the exception type and having duplicate exception handlings with one having an additional warning? Is narrowing the exceptions worth the risk that we missed one or that a new one may pop up in the future?

My other major concern is logging new warnings that will cause cloud-init to exit 2. There are a number of places (random example) where we'll get a new warning where we didn't before. In that example, it even seems there was some effort in ensuring the exception can be silenced when we want it to.

Also for future reference, the bigger a PR gets, the more that is going to fall through the cracks during review. For PRs this big, I think it'd be best to move anything not strictly part of the goal to separate PRs. There's a few refactoring-type commits (json decoding, ModuleDetails) that don't really seem to belong here. Now that they're reviewed, I'm not asking for them to be moved...more just a future thing.

cloudinit/analyze/show.py

TheRealFalcon · 2024-05-01T18:20:53Z

cloudinit/cmd/clean.py

@@ -149,6 +151,7 @@ def remove_artifacts(remove_logs, remove_seed=False, remove_config=None):
    try:
        runparts(settings.CLEAN_RUNPARTS_DIR)
    except Exception as e:
+        LOG.warning("Unhandled exception: %s", e)


I'm not sure I agree with these unhandled exception rules when it comes to CLI commands. The exception is being printed/handled using the error() call below this one, and does it in a more graceful way. I think a log message on the CLI like the following will just add more noise:

2024-05-01 18:10:39,200 - clean.py[WARNING]: Unhandled exception: ...

+1 see comment below

No response needed, but leaving this open because I have changes to make here.

TheRealFalcon · 2024-05-01T18:33:34Z

cloudinit/cmd/main.py

@@ -745,8 +745,12 @@ def status_wrapper(name, args):
    else:
        try:
            status = json.loads(util.load_text_file(status_path))
-        except Exception:
-            pass
+        except OSError:


Do we actually want to warn if a file isn't found here?

Regardless, I would probably downgrade these all to info.

If this doesn't happen, then either a previous stage failed to create a status.json, or something removed/modified it/corrupted it, or a cloud-init stage is getting manually run out of order(basically the same thing as this PR).

All of these sound like "unexpected / bad things" to me - wouldn't a warning make sense?

@TheRealFalcon waiting for response on this comment

but that other PR already contains a warning, right? I guess it seems like fairly large shift from "it doesn't matter if this fails because we'll create it anyway" to "we need to warn you about this even though it has no bearing on the resulting artifact".

That's fair, I'll remove it.

cloudinit/config/cc_mounts.py

TheRealFalcon · 2024-05-01T22:06:26Z

cloudinit/sources/helpers/vultr.py

@@ -32,7 +32,7 @@ def get_metadata(
                connectivity_url_data={"url": url},
            ):
                # Fetch the metadata
-                v1 = read_metadata(url, timeout, retries, sec_between, agent)
+                v1 = _read_metadata(url, timeout, retries, sec_between, agent)


Nit: I prefer to convert from bytes to strings ASAP and I'm not sure why we needed to rename the function. Neither change is bad but the changes in the file feel superfluous to me.

I prefer to convert from bytes to strings ASAP

generally no objections to this philosophy, however in cases like this co-locating the .decode() with the json.loads() could make the exception handling simpler by including the exception handler that's already required for the json.loads() with the UnicodeDecodeError handler. That said, we aren't actually handling this exception here, so that's not an argument for keeping it this way. I can drop the changes to this file if you want.

I can drop the changes to this file if you want.

@TheRealFalcon ping - that whole commit might be unwanted given the "convert from bytes asap" philosophy. Thoughts on how to proceed?

It was a nit, so it's not a strong opinion. I'm fine keeping it this way.

cloudinit/util.py

cloudinit/stages.py

holmanb · 2024-05-04T00:28:23Z

Thanks for the review @TheRealFalcon! Great feedback all around.

Your LOG.warning("Unhandled exception: %s", e) rule is a bit too rigid to me. There are a bunch of places that had something like:

That's fair. It made auditing the code easier to implement this way, but we don't need to be so strict. I've updated the rules in the description above to say:

Require a minimum of one of the following in each Exception handler:

Log a warning at level WARN or higher containing the exception type and message

Re-raise the exception without redefine its type.

I want to keep the rule list short so that it is easy to remember, while being obvious and clear enough that it doesn't require much interpretation when trying to write or review code. Let me know what you think, suggestions are welcome.

My other major concern is logging new warnings that will cause cloud-init to exit 2. There are a number of places (random example) where we'll get a new warning where we didn't before.

I have some concerns about that too. In the past we've allowed some behavior changes that are bug fixes. I would consider properly warning (and exit 2) about a previously unreported error condition to be a bug fix. There is risk of big unknowns that crop up from this change, but I'm happy to be slow and meticulous with this PR if that will help us build confidence about its direction and consequences.

I'm not sure I agree with these unhandled exception rules when it comes to CLI commands. The exception is being printed/handled using the error() call below this one, and does it in a more graceful way. I think a log message on the CLI like the following will just add more noise:

2024-05-01 18:10:39,200 - clean.py[WARNING]: Unhandled exception: ...

Agreed, applying the rule feels clunky here and doesn't improve the end user's experience. Unfortunately, writing to stderr wouldn't be an appropriate way to log unhandled warnings outside of CLI commands. The short/obvious/clear thing breaks down when trying to draw a box around where cli commands start and end, primarily for the following two reasons:

cloud-init imports code back from cloudinit.cmd into the rest of the codebase (read_cfg_paths)
a big chunk of cloud-init's core runtime logic is implemented in cloudinit.cmd.main

These are just symptoms of the codebase's haphazardous structure. Both of these points are undesirable for several reasons, but while 1) would be trivial to fix 2) would probably take some forethought and planning. I have some thoughts on this topic for how we can better structure the core codebase and how this would benefit the project in different ways, but I'll save that discussion for later.

For now maybe we add a rule that says something like: > - for code in cloudinit/cmd/ (excluding main.py and __init__.py): Print a message to stderr containing the exception type and message.?

Maybe someday if we feel the urge to address 1) and 2) we can change the rule to say > - Code in cloudinit/cmd/: Print a message to stderr containing the exception type and message.

Thoughts?

TheRealFalcon · 2024-05-08T01:28:51Z

From a high level, I think everything you said makes sense. Thanks for laying it all out. I just want to make sure I understand. By Log a warning at level WARN or higher containing the exception type and message, you mean that we don't need a separate handler, correct? Something like this could be ok?

try:
    do_something()
except Exception as e:
    LOG.warning("oh no! %s", e)
    recover()

If so, +1. If not, let me know what I'm missing.

If we want to have hard rules around this behavior, I think:

Let's document it. Maybe our docs could use a section for such coding guidelines? (doesn't have to be this PR)
I also want to be flexible where needed. I think as long as we have a comment explaining why such a rule doesn't apply (and a reviewer agrees), we should be ok with occasionally breaking the rules.

holmanb · 2024-05-08T14:57:50Z

From a high level, I think everything you said makes sense. Thanks for laying it all out. I just want to make sure I understand. By Log a warning at level WARN or higher containing the exception type and message, you mean that we don't need a separate handler, correct? Something like this could be ok?
try:
    do_something()
except Exception as e:
    LOG.warning("oh no! %s", e)
    recover()
If so, +1. If not, let me know what I'm missing.

Yes, that would be just fine. I think we're on the same page about this.

If we want to have hard rules around this behavior, I think:
1. Let's document it. Maybe our docs could use a section for such coding guidelines? (doesn't have to be this PR)

+1 Agreed. I can do that in a follow-up PR

2. I also want to be flexible where needed. I think as long as we have a comment explaining why such a rule doesn't apply (and a reviewer agrees), we should be ok with occasionally breaking the rules.

+1 I think the rules are best followed in most cases, but exceptions should be allowable. Even in this PR I break this rule in hotplug_hook.py. In that case it follows the spirit of the rule by eventually raising something if an unknown exception is caught, and it's probably not safe to reasonably assume that we know all possible exceptions that might be raised from that call site. Even if we got the exceptions right in this PR, future changes to any code called in that site might break things.

I do think that this PR a larger hurdle in risk of behavior changes than future code which follows these rules will have. That is because in future PRs, we won't have to worry so much about existing runtime exceptions which silently behave in a way that is currently expected by the user since any of those that exist in theory should get fleshed out by this PR.

holmanb · 2024-05-14T13:06:50Z

Pre-merge requirements:

Eliminate unnecessary LOG.warning("Unhandled exception",) -> @holmanb
Wait until after the 24.2 release conflicts again -> @holmanb

Follow-up:

Document the rules

github-actions · 2024-05-29T00:22:03Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

holmanb · 2024-05-29T22:23:59Z

@TheRealFalcon I forgot to mention that I'd also be okay with patching this out of older releases to mitigate the risk of this change. If we do see new bugs shake out of cloud-init in Oracular as a result of this, the resulting bug fixes will still benefit old releases.

TheRealFalcon

I made another pass here. I know you still have the Eliminate unnecessary LOG.warning("Unhandled exception",) piece left to do but figured this was looking at anyway.

I forgot to mention that I'd also be okay with patching this out of older releases to mitigate the risk of this change

I think that's overkill for the entire PR. These changes are spread far enough that I can foresee hitting quilt conflicts all the time if we patched it all out. That said, it might be worth separating the risky ones and patching those. In this review, I added a Rule exception or quilt patch. comment to lines where I think we either need to make an exception to the rule or add a quilt patch for existing releases for that particular line.

cloudinit/config/cc_growpart.py

cloudinit/cmd/main.py

TheRealFalcon · 2024-05-30T19:29:38Z

cloudinit/config/cc_ssh_import_id.py

-            LOG.debug(
-                "User %s is not correctly configured for ssh_import_id", user
-            )
+            util.logexc(LOG, "Unhandled configuration for user %s", user)


Valid warning, but we should patch it as to not accidentally break deployments that have an invalid import id but otherwise work.

cloudinit/net/__init__.py

TheRealFalcon · 2024-05-30T21:00:56Z

cloudinit/util.py

+            except IndexError:
+                continue
+            except Exception as e:
+                LOG.warning("Unhandled exception: %s", e)


Rule exception or quilt patch.

TheRealFalcon · 2024-05-30T21:01:07Z

cloudinit/util.py

+            except IndexError:
+                continue
+            except Exception as e:
+                LOG.warning("Unhandled exception: %s", e)


Rule exception or quilt patch.

TheRealFalcon · 2024-05-30T21:01:18Z

cloudinit/util.py

+    except ValueError:
+        ts = "??"
+    except Exception as e:
+        LOG.warning("Unhandled exception: %s", e)


Rule exception or quilt patch.

TheRealFalcon · 2024-05-30T21:02:07Z

cloudinit/util.py

-        except Exception:
-            pass
+        except Exception as e:
+            LOG.warning("Unhandled exception: %s", e)


quilt patch

TheRealFalcon · 2024-05-30T21:03:17Z

tests/unittests/config/test_cc_bootcmd.py

@@ -100,24 +100,6 @@ def test_handler_creates_and_runs_bootcmd_script_with_instance_id(self):
            my_id + " iid-datasource-none\n", util.load_text_file(out_file)
        )

-    def test_handler_runs_bootcmd_script_with_error(self):


Why is this no longer needed? The log is gone but we should still get the exception.

github-actions · 2024-07-03T00:22:27Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

holmanb · 2024-07-31T22:41:04Z

setting to wip label, blocked on #5573

github-actions · 2024-08-15T00:22:32Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

github-actions · 2024-09-04T00:24:36Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

From this helper's docstring: Ubuntu cloud images previously included a 'eth0.cfg' that had hard coded content. That file would interfere with the rendered configuration if it was present. This code is no longer relevant due to the following: 1) Ubuntu uses netplan 2) This appears like an Ubuntu-specific workaround 3) Debian cloud images use netplan

Remove unnecessary arguments.

Since Python 3.6, json.loads() is capable of receiving arguments of type str, bytes, and bytearray. Cloud-init's code historically manually handled type conversion. This boilerplate is no longer necessary. Add docstring and typing to util.load_json().

Cloud-init's codebase makes extensive use of exception handlers which catch Exception and then do something based on the assumed exception type. This is bad practice and makes it possible for exceptions to be unexpectedly ignored. To remedy this, this commit updates cloud-init to make `Exception` always be an unknown exception which therefore should always produce a log of level `WARN` or higher. Require a minimum of one of the following in each Exception handler: - LOG.warning("Unhandled exception: %s", e) - util.logexc() - with a log level of WARN or higher - re-raise the exception without setting exception type when re-raising[1] Make cloud-init's code adhere to the above set of rules with the following changes: - Limit the scope where Exception is handled to only unexpected error paths. - Add new handlers for more specific exception types. - Add warning logs or increase log level to WARN in some cases. - Add typing where exceptions are returned. - Replace various Exception handlers with ProcessExecutionError. - Remove handling that does nothing. - Remove unused code which handles Exception. [1] This would likely reduce the scope of the exception and make it possible to mask an Exception.

In Python 3.3+, IOError is an alias of OSError[1]. Some call sites use one or the other or both. Standardize on the OSError. [1] https://docs.python.org/3/library/exceptions.html#IOError

github-actions · 2024-10-05T00:25:46Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

github-actions · 2024-10-22T00:26:35Z

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.)

holmanb force-pushed the holmanb/stricter-exception-handling branch 2 times, most recently from 7cee29f to 6a845c4 Compare April 23, 2024 03:04

holmanb changed the title ~~Stricter exception handling~~ Make exception handling explicit Apr 23, 2024

holmanb force-pushed the holmanb/stricter-exception-handling branch 2 times, most recently from dfaeb56 to 0888048 Compare April 23, 2024 18:44

holmanb marked this pull request as draft April 26, 2024 00:50

holmanb force-pushed the holmanb/stricter-exception-handling branch from 0888048 to 12ffead Compare April 26, 2024 01:34

holmanb marked this pull request as ready for review April 26, 2024 01:36

holmanb force-pushed the holmanb/stricter-exception-handling branch 2 times, most recently from fe33823 to 8e41a0e Compare April 27, 2024 00:13

holmanb mentioned this pull request Apr 30, 2024

fix(ec2): Fix broken uuid match with other-endianness #5236

Merged

5 tasks

holmanb force-pushed the holmanb/stricter-exception-handling branch 4 times, most recently from 15ac27c to 2864b51 Compare April 30, 2024 21:48

TheRealFalcon self-assigned this May 1, 2024

TheRealFalcon requested changes May 2, 2024

View reviewed changes

holmanb force-pushed the holmanb/stricter-exception-handling branch from 4a24b71 to 3376535 Compare May 3, 2024 23:18

github-actions bot added the stale-pr Pull request is stale; will be auto-closed soon label May 29, 2024

holmanb removed the stale-pr Pull request is stale; will be auto-closed soon label May 29, 2024

TheRealFalcon requested changes May 30, 2024

View reviewed changes

dermotbradley mentioned this pull request Jun 2, 2024

fix: cc_user_groups incorrectly assumes "useradd" never locks password field #5355

Merged

5 tasks

holmanb added this to the cloud-init-24.3 milestone Jun 18, 2024

holmanb added the wip Work in progress, do not land label Jul 31, 2024

github-actions bot added the stale-pr Pull request is stale; will be auto-closed soon label Aug 15, 2024

holmanb removed the stale-pr Pull request is stale; will be auto-closed soon label Aug 15, 2024

holmanb modified the milestones: cloud-init-24.3, cloud-init-24.4 Aug 20, 2024

github-actions bot added the stale-pr Pull request is stale; will be auto-closed soon label Sep 4, 2024

github-actions bot closed this Sep 12, 2024

holmanb reopened this Sep 12, 2024

holmanb removed the stale-pr Pull request is stale; will be auto-closed soon label Sep 12, 2024

holmanb added 11 commits September 27, 2024 10:50

fix(gce): Warn when key publishing fails

48781ce

chore(bootcmd): Eliminate redundant exception handling

f5622b7

refactor(helpers): Remove unused conditional codepath

9f866e8

refactor(net): Simplify read_sys_net exception handling

324926c

Remove unnecessary arguments.

chore: Remove redundant end-of-module exception handling

3b65a27

chore(json): Simplify json loading

eec8f50

Since Python 3.6, json.loads() is capable of receiving arguments of type str, bytes, and bytearray. Cloud-init's code historically manually handled type conversion. This boilerplate is no longer necessary. Add docstring and typing to util.load_json().

fixup! chore: Make exception handling more explicit

02b775d

chore(IOError): Update IOError to OSError

5294901

In Python 3.3+, IOError is an alias of OSError[1]. Some call sites use one or the other or both. Standardize on the OSError. [1] https://docs.python.org/3/library/exceptions.html#IOError

chore(exception): Replace errno with proper exceptions

1786f9d

github-actions bot added the stale-pr Pull request is stale; will be auto-closed soon label Oct 5, 2024

holmanb force-pushed the holmanb/stricter-exception-handling branch from d68eb04 to 1786f9d Compare October 7, 2024 14:08

holmanb removed the stale-pr Pull request is stale; will be auto-closed soon label Oct 7, 2024

github-actions bot added the stale-pr Pull request is stale; will be auto-closed soon label Oct 22, 2024

github-actions bot closed this Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make exception handling explicit #5202

Make exception handling explicit #5202

holmanb commented Apr 23, 2024 •

edited

Loading

TheRealFalcon left a comment

TheRealFalcon May 1, 2024

holmanb May 4, 2024

holmanb May 14, 2024

TheRealFalcon May 1, 2024

holmanb May 3, 2024

holmanb May 14, 2024

TheRealFalcon May 14, 2024

holmanb Jun 11, 2024

TheRealFalcon May 1, 2024

holmanb May 3, 2024

holmanb May 14, 2024 •

edited

Loading

TheRealFalcon May 14, 2024

holmanb commented May 4, 2024 •

edited

Loading

TheRealFalcon commented May 8, 2024

holmanb commented May 8, 2024

holmanb commented May 14, 2024

github-actions bot commented May 29, 2024

holmanb commented May 29, 2024

TheRealFalcon left a comment

TheRealFalcon May 30, 2024

TheRealFalcon May 30, 2024

TheRealFalcon May 30, 2024

TheRealFalcon May 30, 2024

TheRealFalcon May 30, 2024

TheRealFalcon May 30, 2024

github-actions bot commented Jul 3, 2024

holmanb commented Jul 31, 2024

github-actions bot commented Aug 15, 2024

github-actions bot commented Sep 4, 2024

github-actions bot commented Oct 5, 2024

github-actions bot commented Oct 22, 2024

Make exception handling explicit #5202

Make exception handling explicit #5202

Conversation

holmanb commented Apr 23, 2024 • edited Loading

Impetus

Objective

Rules

Methodology

Reviewer notes

Merge type

TheRealFalcon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holmanb May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holmanb commented May 4, 2024 • edited Loading

TheRealFalcon commented May 8, 2024

holmanb commented May 8, 2024

holmanb commented May 14, 2024

github-actions bot commented May 29, 2024

holmanb commented May 29, 2024

TheRealFalcon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 3, 2024

holmanb commented Jul 31, 2024

github-actions bot commented Aug 15, 2024

github-actions bot commented Sep 4, 2024

github-actions bot commented Oct 5, 2024

github-actions bot commented Oct 22, 2024

holmanb commented Apr 23, 2024 •

edited

Loading

holmanb May 14, 2024 •

edited

Loading

holmanb commented May 4, 2024 •

edited

Loading