Terminate partition when FASTER refuses to checkpoint for over a minute #301

davidmrdavid · 2023-08-28T18:27:28Z

Related: Azure/azure-functions-durable-extension#2534

~~This is a WIP, rough draft of adding logs to the checkpointing state machine so that we can better identify any bugs on this codepath.~~

Update:

This PR contains a fix that terminates a partition when FASTER refuses to checkpoint for over a minute. In addition to this fix, we're adding more logs to the entire checkpoint state machine for easier debuggability moving forward. This PR is meant to be merged.

davidmrdavid · 2023-08-28T18:28:53Z

This is not to be merged as-is, it is for a private package. If this logs prove to be useful there, we can look to format them in a way that is amenable for the prod-ready package.

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

davidmrdavid · 2023-08-31T23:18:33Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+        async Task RunCheckpointingStateMachine()
+        {
+            // handle progression of checkpointing state machine:  none -> pendingCompaction -> pendingIndexCheckpoint ->  pendingStoreCheckpoint -> none)
+            if (this.pendingStoreCheckpoint != null)


I just moved the checkpoint state machine logic from Process onto this own method. But I realize that may make this PR harder to review. If it helps, I can undo this change and delay it to a separate PR.

It's fine. It means i can't compare line by line but I think I can spot the differences since I looked at this code a lot in the last few days.

davidmrdavid · 2023-08-31T23:19:11Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+        void StartCheckpointOrFailOnTimeout(Func<bool> checkpointRoutine, string messageOnError)
+        {
+            var checkpointStarted = checkpointRoutine.Invoke();
+            if (checkpointStarted)
+            {
+                this.timeOfFirstRefusedCheckpoint = null; 
+            }
+            else
+            {
+                // track start of FASTER refusal to start checkpoint
+                var currentTime = DateTime.UtcNow;
+                this.timeOfFirstRefusedCheckpoint ??= currentTime;
+
+                // if the refusal to checkpoint started over a minute ago, terminate partition
+                TimeSpan duration = currentTime - this.timeOfFirstRefusedCheckpoint.Value;
+                if (duration > TimeSpan.FromMinutes(1))
+                {
+                    messageOnError += $" FASTER first refused to checkpoint at '{this.timeOfFirstRefusedCheckpoint}'. Duration of refusal = {duration}";
+                    Exception err = new Exception(messageOnError);
+                    this.partition.ErrorHandler.HandleError(nameof(StartCheckpointOrFailOnTimeout), messageOnError, err, terminatePartition: true, reportAsWarning: false);
+                }
+            }
+        }


This is the fix

Got it. seems right.

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

davidmrdavid · 2023-08-31T23:20:21Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                if (duration > TimeSpan.FromMinutes(1))
+                {
+                    messageOnError += $" FASTER first refused to checkpoint at '{this.timeOfFirstRefusedCheckpoint}'. Duration of refusal = {duration}";
+                    Exception err = new Exception(messageOnError);


I wasn't sure if we had more fine grained Exception classes for this kind of error, so I just used a generic Exception constructor

You don't need to pass an exception, it is optional (only meant to be used if you caught one). Just pass null.

Good to know. I'll do that and also update the HandleError signature to flag that the Exception can be null.

sebastianburckhardt

First round of comments with minor suggestions.

sebastianburckhardt · 2023-08-31T23:15:43Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+            }
+        }
+
+        async Task RunCheckpointingStateMachine()


You can make this a ValueTask to save an allocation on the common path (where the method completes synchronously)

sebastianburckhardt · 2023-08-31T23:23:46Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+            {
+                if (this.pendingIndexCheckpoint.IsCompleted == true)
+                {
+                    this.traceHelper.FasterProgress("Checkpointing state machine: trying to start store checkpoint");


suggested wording: pendingIndexCheckpoint has completed

applied in next commit

sebastianburckhardt · 2023-08-31T23:25:43Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+            {
+                if (this.pendingStoreCheckpoint.IsCompleted == true)
+                {
+                    this.traceHelper.FasterProgress("Checkpointing state machine: store checkpoint no longer pending");


suggested wording: pendingStoreCheckpoint has completed

Yeah I recognize the phrasing here and in other similar logs was weird. I just wanted to avoid implying that every time this log was emitted, that the pendingStoreCheckpoint completed again, which isn't necessarily the case.

But now I think it's too minor of a concern. I'll take your suggestion.

sebastianburckhardt · 2023-08-31T23:28:23Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                }
+                else
+                {
+                    this.traceHelper.FasterProgress("Checkpointing state machine: index checkpoint still pending");


Same as above; may make more sense to print this info as part of the LogCheckpointStats

sebastianburckhardt · 2023-08-31T23:29:17Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                }
+                else
+                {
+                    this.traceHelper.FasterProgress("Checkpointing state machine: store checkpoint still pending");


This will again spam the logs. It may make more sense to print this info as part of the LogCheckpointStats

sebastianburckhardt · 2023-08-31T23:36:07Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+        async Task RunCheckpointingStateMachine()
+        {
+            // handle progression of checkpointing state machine:  none -> pendingCompaction -> pendingIndexCheckpoint ->  pendingStoreCheckpoint -> none)
+            if (this.pendingStoreCheckpoint != null)


It's fine. It means i can't compare line by line but I think I can spot the differences since I looked at this code a lot in the last few days.

sebastianburckhardt · 2023-08-31T23:37:17Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                if (duration > TimeSpan.FromMinutes(1))
+                {
+                    messageOnError += $" FASTER first refused to checkpoint at '{this.timeOfFirstRefusedCheckpoint}'. Duration of refusal = {duration}";
+                    Exception err = new Exception(messageOnError);


You don't need to pass an exception, it is optional (only meant to be used if you caught one). Just pass null.

sebastianburckhardt · 2023-08-31T23:40:35Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                TimeSpan duration = currentTime - this.timeOfFirstRefusedCheckpoint.Value;
+                if (duration > TimeSpan.FromMinutes(1))
+                {
+                    messageOnError += $" FASTER first refused to checkpoint at '{this.timeOfFirstRefusedCheckpoint}'. Duration of refusal = {duration}";


nit: string is starting with an extra space
I dont' think you need to add quotes around a time.

This is actually on purpose. Since we're adding new text to the error, I need to separate it with a space. I'm adding a dot (".") in there to make this clearer.

sebastianburckhardt · 2023-08-31T23:41:43Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+        void StartCheckpointOrFailOnTimeout(Func<bool> checkpointRoutine, string messageOnError)
+        {
+            var checkpointStarted = checkpointRoutine.Invoke();
+            if (checkpointStarted)
+            {
+                this.timeOfFirstRefusedCheckpoint = null; 
+            }
+            else
+            {
+                // track start of FASTER refusal to start checkpoint
+                var currentTime = DateTime.UtcNow;
+                this.timeOfFirstRefusedCheckpoint ??= currentTime;
+
+                // if the refusal to checkpoint started over a minute ago, terminate partition
+                TimeSpan duration = currentTime - this.timeOfFirstRefusedCheckpoint.Value;
+                if (duration > TimeSpan.FromMinutes(1))
+                {
+                    messageOnError += $" FASTER first refused to checkpoint at '{this.timeOfFirstRefusedCheckpoint}'. Duration of refusal = {duration}";
+                    Exception err = new Exception(messageOnError);
+                    this.partition.ErrorHandler.HandleError(nameof(StartCheckpointOrFailOnTimeout), messageOnError, err, terminatePartition: true, reportAsWarning: false);
+                }
+            }
+        }


Got it. seems right.

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

davidmrdavid · 2023-09-05T16:23:55Z

@sebastianburckhardt: I'm unable to respond to your comments to state that I've applied your feedback, so here's me stating just that. I think all your feedback has been addressed.

cgillum · 2023-09-05T17:03:50Z

If this PR isn't meant to be merged, can we switch it to being a draft PR? We do this enough that I wonder if a new label might also be helpful.

davidmrdavid · 2023-09-05T18:04:31Z

If this PR isn't meant to be merged, can we switch it to being a draft PR? We do this enough that I wonder if a new label might also be helpful.

For visibility: we discussed this offline and clarified that this PR is meant to be merged. I also updated the description to make this clearer.

…rite into dajusto/add-checkpointing-logs

davidmrdavid · 2023-09-26T20:25:10Z

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

@@ -499,20 +503,39 @@ public override async Task<long> RunCompactionAsync(long target)
                    target - this.Log.BeginAddress,
                    this.GetElapsedCompactionMilliseconds());

+                var timeoutTask = Task.Delay(TimeSpan.FromMinutes(10));


Note: 10 minutes was determined to be a safe timeout window using various kusto queries to determine the expected duration of compaction. But it could be too conservative

Conservative is probably o.k. as a starting point.

sebastianburckhardt

Mostly pretty good but there are a few things that need fixing I think.

sebastianburckhardt · 2023-10-10T17:47:19Z

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

@@ -499,20 +503,39 @@ public override async Task<long> RunCompactionAsync(long target)
                    target - this.Log.BeginAddress,
                    this.GetElapsedCompactionMilliseconds());

+                var timeoutTask = Task.Delay(TimeSpan.FromMinutes(10));


Conservative is probably o.k. as a starting point.

sebastianburckhardt · 2023-10-10T17:48:58Z

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

+                    this.partition.ErrorHandler.HandleError(nameof(RunCompactionAsync), exceptionMessage, e: null, terminatePartition: true, reportAsWarning: true);
+
+                    // we need resolve the task to ensure the 'finally' block is executed which frees up another thread to start compating
+                    tcs.SetException(new OperationCanceledException(exceptionMessage));


I think you should use TrySetException here since the timeout will still happen after the compaction completed successfully and SetException then throws an exception.

Similarly, you need to change the tcs.SetResult (1x) and tcs.SetException (2x) further below to tcs.TrySetResult and tcs.TrySetException.

Applied in latest commit: df76d16

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

sebastianburckhardt · 2023-10-10T18:37:52Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+        void LogCheckpointStats()
+        {
+            long inputQueuePositionLag = this.GetInputQueuePositionLag();
+            this.CheckpointDue(out CheckpointTrigger trigger, out long? compactUntil);


CheckpointDue has several side effects, so I think it is not safe to insert an extra call to it from this location. I would just remove it. Your are already printing a LOT of tracing even without the 'trigger' information, and one can still look at the other traces to see the triggers that take place.

Removed in my latest commit: df76d16

sebastianburckhardt · 2023-10-10T19:49:32Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                this.pendingCheckpointTrigger = trigger;
+
+                var isCompactionSkipped = !compactUntil.HasValue;
+                this.pendingCompaction = isCompactionSkipped ? Task.FromResult((long?)null) : this.RunCompactionAsync(compactUntil.Value);


by skipping the call to RunCompactionAsync you are also eliminating the Notify() (see comment below). However, I think that one is still needed. The point of Notify is to tell the storeworker that there is more work to do, i.e. that it should run another iteration. That is necessary here because the next run of the storeworker is going to advance the state machine immediately.

In my latest commit, I move the Notify method out here so it is still called in both places. Does that seem like a fair refactoring?

Not quite, unfortunately. When doing the compaction, the Notify needs to be called after the compaction completes. With your change, it gets called after the compaction is started, which is too soon.

(the point of Notify is to tell the storeworker that it is possible to move the state machine forward, which is true only if (a) we skipped compaction, or (b) compaction is complete)

sebastianburckhardt · 2023-10-10T19:56:30Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                var isCompactionSkipped = !compactUntil.HasValue;
+                this.pendingCompaction = isCompactionSkipped ? Task.FromResult((long?)null) : this.RunCompactionAsync(compactUntil.Value);
+
+                if (isCompactionSkipped)
+                {
+                    this.traceHelper.FasterProgress($"RunCompactionAsync was skipped because a compaction range was not provided by trigger = {trigger}.");
+                }


I think this log message is fully redundant with the one 8 lines earlier.

sebastianburckhardt · 2023-10-10T19:57:46Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

-            }
-
+            target = await this.store.RunCompactionAsync(target);
+            this.partition.Settings.TestHooks?.CheckpointInjector?.CompactionComplete(this.partition.ErrorHandler);
            this.Notify();


this.Notify should still be called even if compaction is skipped (see earlier comment)

Address in earlier comment ,good catch!

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

sebastianburckhardt · 2023-10-12T20:17:25Z

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs

+                this.pendingCheckpointTrigger = trigger;
+
+                var isCompactionSkipped = !compactUntil.HasValue;
+                this.pendingCompaction = isCompactionSkipped ? Task.FromResult((long?)null) : this.RunCompactionAsync(compactUntil.Value);


Not quite, unfortunately. When doing the compaction, the Notify needs to be called after the compaction completes. With your change, it gets called after the compaction is started, which is too soon.

(the point of Notify is to tell the storeworker that it is possible to move the state machine forward, which is true only if (a) we skipped compaction, or (b) compaction is complete)

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>

…icrosoft/durabletask-netherite into dajusto/add-checkpointing-logs

davidmrdavid · 2023-10-12T21:38:25Z

@sebastianburckhardt: I see now why you wrote RunCompactionAsync in the way you did. I've reverted my changes to it in this commit: 9158006

…rite into dajusto/add-checkpointing-logs

…icrosoft/durabletask-netherite into dajusto/add-checkpointing-logs

sebastianburckhardt · 2023-10-19T18:50:58Z

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

@@ -504,7 +504,7 @@ public override async Task<long> RunCompactionAsync(long target)
                    this.GetElapsedCompactionMilliseconds());

                var tokenSource = new CancellationTokenSource();
-                var timeoutTask = Task.Delay(TimeSpan.FromMinutes(10), tokenSource.Token); //.ContinueWith(_ => tokenSource.Dispose());
+                var timeoutTask = Task.Delay(TimeSpan.FromMinutes(10), tokenSource.Token).ContinueWith(_ => tokenSource.Dispose());


I don't think this is the right way to dispose the timer task.

The problem is that you want to dispose it after the timeout completes AND after line 526. Otherwise line 526 may throw an exception.

That is why I suggested to put the ContinueWith on line 529.

* Update GH automation (#303) * initial commit (#290) * Revert "initial commit (#290)" (#314) This reverts commit 3a2d193. * Bump Azure.Identity from 1.7.0 to 1.10.2 in /samples/TokenCredentialDTFx (#323) Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.7.0 to 1.10.2. - [Release notes](https://github.com/Azure/azure-sdk-for-net/releases) - [Commits](Azure/azure-sdk-for-net@Azure.Identity_1.7.0...Azure.Identity_1.10.2) --- updated-dependencies: - dependency-name: Azure.Identity dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Terminate partition when FASTER refuses to checkpoint for over a minute (#301) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Varshitha Bachu <vabachu@microsoft.com> Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update CreationRequestReceived.cs * remove the faster-alternate data store and fix broken deserialization. (#308) * fix and instrument commitlog serialization and deserialization. (#305) * fix discard of packets so the next packet to expect is updated correctly (#302) * New EventHubs performance tests (#178) * rebase, and remove changes to scale monitor * add hello cities test that prints the history of a nested orchestration * implement a watchdog that terminates CompletePending quickly if it hangs (#318) * add unique id to scale monitor constructor (#316) * fix bugs from PR #302 (#315) * fix cache size reporting (#321) * added a comment * update durable task package references to 2.15.1 and 2.12.0 (#317) * update DT and DF package references * update to latest * sync dev w/ main (#324) * Update GH automation (#303) * initial commit (#290) * Revert "initial commit (#290)" (#314) This reverts commit 3a2d193. * Bump Azure.Identity from 1.7.0 to 1.10.2 in /samples/TokenCredentialDTFx (#323) Bumps [Azure.Identity](https://github.com/Azure/azure-sdk-for-net) from 1.7.0 to 1.10.2. - [Release notes](https://github.com/Azure/azure-sdk-for-net/releases) - [Commits](Azure/azure-sdk-for-net@Azure.Identity_1.7.0...Azure.Identity_1.10.2) --- updated-dependencies: - dependency-name: Azure.Identity dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Terminate partition when FASTER refuses to checkpoint for over a minute (#301) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Varshitha Bachu <vabachu@microsoft.com> Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com> Co-authored-by: David Justo <david.justo.1996@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

add state machine logs

1337a5e

sebastianburckhardt reviewed Aug 28, 2023

View reviewed changes

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs Outdated Show resolved Hide resolved

sebastianburckhardt reviewed Aug 28, 2023

View reviewed changes

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs Outdated Show resolved Hide resolved

davidmrdavid added 2 commits August 28, 2023 12:53

add trace after pendingStoreCheckpoint invocation

4d4b152

removed duplicate 1.1.1 for 1.1.2

878b70f

davidmrdavid requested a review from sebastianburckhardt August 28, 2023 19:56

add version suffix

e511ccf

davidmrdavid mentioned this pull request Aug 28, 2023

Orchestration getting stuck while getting the lock Azure/azure-functions-durable-extension#2534

Closed

davidmrdavid added 2 commits August 28, 2023 16:26

remove noisy log

cfed6c2

add timeout fix and minor refactoring

2745a83

davidmrdavid commented Aug 31, 2023

View reviewed changes

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs Outdated Show resolved Hide resolved

davidmrdavid added 2 commits August 31, 2023 16:12

remove unused imports

d36f2bb

minor refactoring

9577cd7

davidmrdavid commented Aug 31, 2023

View reviewed changes

src/DurableTask.Netherite/StorageLayer/Faster/StoreWorker.cs Show resolved Hide resolved

davidmrdavid commented Aug 31, 2023

View reviewed changes

davidmrdavid changed the title ~~[WIP] Adding checkpointing state machine logs~~ [WIP] Terminate partition when FASTER refuses to checkpoint for over a minute Aug 31, 2023

sebastianburckhardt reviewed Aug 31, 2023

View reviewed changes

davidmrdavid added 4 commits September 1, 2023 11:00

apply feedback

887c8fa

rev version suffix

576996d

apply feedback

294a448

improve formatting of logs

7599f0f

davidmrdavid marked this pull request as ready for review September 5, 2023 16:20

davidmrdavid changed the title ~~[WIP] Terminate partition when FASTER refuses to checkpoint for over a minute~~ Terminate partition when FASTER refuses to checkpoint for over a minute Sep 5, 2023

davidmrdavid requested a review from sebastianburckhardt September 5, 2023 16:23

davidmrdavid added 5 commits September 11, 2023 12:52

rev worker pkg version

adb7452

remove 'abort thread'

f73d898

Merge branch 'main' of https://github.com/microsoft/durabletask-nethe…

ce8db2b

…rite into dajusto/add-checkpointing-logs

remove typos and version rev

f40fc36

remove versionSuffix

14af672

davidmrdavid commented Sep 26, 2023

View reviewed changes

sebastianburckhardt added this to the 1.4.1 milestone Sep 27, 2023

sebastianburckhardt requested changes Oct 10, 2023

View reviewed changes

sebastianburckhardt removed this from the 1.4.1 milestone Oct 11, 2023

reply to feedback

df76d16

sebastianburckhardt requested changes Oct 12, 2023

View reviewed changes

davidmrdavid and others added 3 commits October 12, 2023 14:30

Update src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

d74fbee

Co-authored-by: Sebastian Burckhardt <sburckha@microsoft.com>

revert RunCompactionChanges

9158006

Merge branch 'dajusto/add-checkpointing-logs' of https://github.com/m…

025bfc3

…icrosoft/durabletask-netherite into dajusto/add-checkpointing-logs

davidmrdavid requested a review from sebastianburckhardt October 12, 2023 21:38

sebastianburckhardt approved these changes Oct 17, 2023

View reviewed changes

davidmrdavid added 7 commits October 17, 2023 15:01

fix build err

799693d

Merge branch 'main' of https://github.com/microsoft/durabletask-nethe…

5819b99

…rite into dajusto/add-checkpointing-logs

add continuewith to timeout task

8585bf6

quick test: remove disposal

013035c

call RunCompactionAsync with long?

1a9cd79

Merge branch 'dajusto/add-checkpointing-logs' of https://github.com/m…

143c7e8

…icrosoft/durabletask-netherite into dajusto/add-checkpointing-logs

add back disposal of timertask

bbb35e7

sebastianburckhardt requested changes Oct 19, 2023

View reviewed changes

dispose timeout task after cancellation

f9543ea

sebastianburckhardt approved these changes Oct 19, 2023

View reviewed changes

davidmrdavid merged commit ba15eed into main Oct 19, 2023
2 checks passed

Terminate partition when FASTER refuses to checkpoint for over a minute #301

Terminate partition when FASTER refuses to checkpoint for over a minute #301

Conversation

davidmrdavid commented Aug 28, 2023 • edited Loading

davidmrdavid commented Aug 28, 2023

davidmrdavid Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebastianburckhardt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidmrdavid commented Sep 5, 2023

cgillum commented Sep 5, 2023

davidmrdavid commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebastianburckhardt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidmrdavid commented Oct 12, 2023

Choose a reason for hiding this comment

davidmrdavid commented Aug 28, 2023 •

edited

Loading

davidmrdavid Aug 31, 2023 •

edited

Loading