Attempt to make the monarch more thread safe. #11189

Rosefield · 2021-09-09T21:50:25Z

Summary of the Pull Request

References

PR Checklist

Closes #xxx
CLA signed. If not, go over here and sign the CLA
Tests added/passed
Documentation updated. If checked, please file a pull request on our docs repo and link it here: #xxx
Schema updated.
I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Detailed Description of the Pull Request / Additional comments

While testing the save/quit features a number of issues were found that were caused by poor synchronization on the monarch, resulting in various unexpected crashes. Because this uses std collections, and I didn't see any builtin winrt multithreaded containers I went with the somewhat heavy-handed mutex approach.

e.g.

This also makes it so that on quit peasants don't try to become the monarch, and the monarch closes their peasant last to prevent elections from happening.

Validation Steps Performed

Create many windows (hold down ctrl-shift-n) then use the quit action from peasants/the monarch to make sure everything closes properly.

github-actions · 2021-09-09T21:53:27Z

@check-spelling-bot Report

Unrecognized words, please review:

lookekup

Previously acknowledged words that are now absent

SPACEBAR Unregister xIcon yIcon

To accept these unrecognized words as correct (and remove the previously acknowledged and now absent words), run the following commands

... in a clone of the git@github.com:Rosefield/terminal.git repository
on the bug/make-monarch-more-thread-safe branch:

update_files() {
perl -e '
my @expect_files=qw('".github/actions/spelling/expect/alphabet.txt
.github/actions/spelling/expect/expect.txt
.github/actions/spelling/expect/web.txt"');
@ARGV=@expect_files;
my @stale=qw('"$patch_remove"');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
next if /^(?:$re)(?:(?:\r|\n)*$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spelling/expect/54ed2955885065a041dbcb56762a5ee1f8ac4302.txt";
use File::Path qw(make_path);
use File::Basename qw(dirname);
make_path (dirname($new_expect_file));
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"$patch_add"');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a)."-".$a cmp lc($b)."-".$b} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;
system("git", "add", $new_expect_file);
'
}

comment_json=$(mktemp)
curl -L -s -S \
  --header "Content-Type: application/json" \
  "https://api.github.com/repos/microsoft/terminal/issues/comments/916463668" > "$comment_json"
comment_body=$(mktemp)
jq -r .body < "$comment_json" > $comment_body
rm $comment_json

patch_remove=$(perl -ne 'next unless s{^</summary>(.*)</details>$}{$1}; print' < "$comment_body")
  

patch_add=$(perl -e '$/=undef;
$_=<>;
s{<details>.*}{}s;
s{^#.*}{};
s{\n##.*}{};
s{(?:^|\n)\s*\*}{}g;
s{\s+}{ }g;
print' < "$comment_body")
  
update_files
rm $comment_body
git add -u

✏️ Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

⚠️ The command is written for posix shells. You can copy the contents of each perl command excluding the outer ' marks and dropping any '"/"' quotation mark pairs into a file and then run perl file.pl from the root of the repository to run the code. Alternatively, you can manually insert the items...

If the listed items are:

... misspelled, then please correct them instead of using the command.
... names, please add them to .github/actions/spelling/allow/names.txt.
... APIs, you can add them to a file in .github/actions/spelling/allow/.
... just things you're using, please add them to an appropriate file in .github/actions/spelling/expect/.
... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spelling/patterns/.

See the README.md in each directory for more information.

🔬 You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. 😉

🗜️ If you see a bunch of garbage

If it relates to a ...

well-formed pattern

See if there's a pattern that would match it.

If not, try writing one and adding it to a patterns/{file}.txt.

Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

Note that patterns can't match multiline strings.

binary-ish string

Please add a file path to the excludes.txt file instead of just accepting the garbage.

File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

Rosefield · 2021-09-09T21:53:48Z

Since it doesn't seem like I can assign reviews, @zadjii-msft @DHowett @lhecker @leonMSFT I would appreciate plenty of eyes and advice on this.

src/cascadia/Remoting/WindowManager.cpp

lhecker · 2021-09-09T22:29:45Z

src/cascadia/Remoting/Monarch.h

            {
-                try
+                std::shared_lock lock{ _peasantsMutex };


If you hold this lock for the entire duration of the function you don't need to lock it again below.

I deliberately wanted to separate the read only vs the rw portions of the lock to reduce contention if there was nothing to be removed.

The lock is largely uncontented and as such there shouldn't be any noticeable advantage to using read-only locking. Personally I'd just acquire an exclusive lock in all cases. But I don't mind the current code either.

lhecker · 2021-09-09T22:30:37Z

src/cascadia/Remoting/Monarch.h

        std::unordered_map<uint64_t, winrt::Microsoft::Terminal::Remoting::IPeasant> _peasants;
+        std::shared_mutex _peasantsMutex{};


Consider using

til::shared_mutex<std::unordered_map<uint64_t, IPeasant>> _peasants

I did originally consider it, but disregarded because the first iteration of this had a single std::recursive_mutex _mutex that was used for locking both _peasants and _mruPeasants. Of course the current iteration does not have that, and I was able to separate out (and downgrade to) a std::shared_mutex.

Once we are confident that this locking strategy is what we want I can take another look at using til::shared_mutex, although I don't look forward to updating all of the remoting tests that rely on friend access.

If possibly I would like this refactor to be deferred for now. I took a shot at doing so, but it proved to be a bit more difficult than I hoped.

lhecker · 2021-09-09T22:32:00Z

src/cascadia/Remoting/Monarch.h


        std::vector<Remoting::WindowActivatedArgs> _mruPeasants;
+        std::recursive_mutex _mruPeasantsMutex{};


Is there ever a situation where this mutex is locked recursively?

_getMostRecentPeasantID calls (through a couple hops) _clearOldMruEntries

Oh right... That's unfortunate - recursive mutexes are really annoying (it uses critical sections brrrr).
I wonder if it would simplify the code if we just slapped a single mutex to the beginning of all public methods of this class and made all private methods non-thread-safe. It would get rid of the recursion and the atomics at least... The Monarch class has an extremely slow "throughput" (API calls per second) right?
Hmm... 🤔

In normal circumstances, yes. The problem is with loading / quitting where there are potentially "many" (O(#windows)) calls, and of course that is where I found these problems in the first place.

Yea, I', not terribly worried about the perf here, there's not really going to be that many calls coming through here per second, and they're pretty much all user-input driven. Plus, O(N) where N=number of windows? That's gonna be pretty small unless you're actively trying to spawn 100's of windows per second and close them quickly.

I can take a look if there is an easy refactor to get rid of the recursive mutex for a shared_mutex, but I think it is still better to have the more granular locking in general.

Without rewriting _getMostRecentPeasantID and _getPeasant the easiest safe thing I can think of is to pass a std::unique_lock<...>* to _getPeasant and _clearOldMruEntries as an argument and if that is null then lock in _clearOldMruEntries. I'm not a huge fan of that approach because it is error prone, but functional.

I made _getPeasant take an optional clearMruPeasantOnFailure boolean, which is set to false in _getMostRecentPeasantID. I then updated _getMostRecentPeasantID to handle its own cleanup, and switched the recursive mutex to a shared mutex.

Spent more time trying to make sure we can't deadlock. I am ~90% confident, but have not proved, that the current version of the code cannot deadlock. Currently _getMostRecentPeasantID (and transitively anything that calls it) is the only method that attempts to hold both locks at the same time. There should not be the case where someone else holds a unique lock on _peasantsMutex and is waiting to get a lock on _mruPeasantsMutex which would deadlock with _getMostRecentPeasantID.

lhecker · 2021-09-09T22:35:48Z

src/cascadia/Remoting/Monarch.cpp

+        // If we are quitting we don't care about maintaining our list of
+        // peasants anymore, and don't need to notify the host that something
+        // changed.
+        if (_quitting.load(std::memory_order_acquire))


Why does this need to sync the memory with Monarch::_handleQuitAll?
Is there a reason _quitting is needed? Why would a listener care if it got notified during shutdown?

I don't think it is necessary, but since each peasant calls this on close I don't want them waiting for the unique_lock until after the _handleQuitAll finishes. This would also trigger _GetNumberOfPeasants through a couple of event hops and that would be a bunch of needless churn and lock contention.

src/cascadia/Remoting/Monarch.cpp

…r code we expect a dead peasant to have.

Rosefield · 2021-09-10T03:58:55Z

As a sanity check I tried #11083 with these changes merged locally, and was able to successfully open / save / quit / reload at least 34 windows multiple times without any troubles. Trying to use the base branch I regularly got heap corruption errors on load with the same saved state.

The saved state for reference:
https://gist.github.com/Rosefield/cd5ea49034936be9ec9f02532339c044

Tried again with 128 windows, as a rough benchmark it takes ~1s after the quit confirmation for all windows to be visibly closed, and on open it takes ~18s for all windows to be visually open (showing a terminal). On load includes a sizeable amount of other applications freezing (such as the timer app) for multiple seconds, but that is probably reasonable behavior for spawning ~384 processes.

…-thread-safe Conflicts: src/cascadia/Remoting/Monarch.cpp src/cascadia/UnitTests_Remoting/RemotingTests.cpp

zadjii-msft

This seems good to me! Some of the other stuff lhecker mentioned might be nice too, but this is definitely better than before. Thanks for helping clean up some of my shortcomings 😄

zadjii-msft · 2021-09-14T23:44:25Z

src/cascadia/UnitTests_Remoting/RemotingTests.cpp

-        void RequestHideNotificationIcon() { throw winrt::hresult_error{}; };
-        void RequestQuitAll() { throw winrt::hresult_error{}; };
-        void Quit() { throw winrt::hresult_error{}; };
+        void AssignID(uint64_t /*id*/) { throw winrt::hresult_error(winrt::hresult{ (int32_t)0x800706ba }); };


may want to actually use RPC_S_SERVER_UNAVAILABLE (or RPC_E_SERVER_UNAVAILABLE, unclear what the difference is)

Where are these constants defined? I'm not seeing them.

winerror.h, which I would have presumed was already included. Oh but that might be an NTSTATUS, not a HRESULT

#define RPC_S_SERVER_UNAVAILABLE 1722L

Yah, that looks like a different constant

zadjii-msft · 2021-09-16T14:56:52Z

src/cascadia/Remoting/Monarch.h


        std::vector<Remoting::WindowActivatedArgs> _mruPeasants;
+        std::recursive_mutex _mruPeasantsMutex{};


Yea, I', not terribly worried about the perf here, there's not really going to be that many calls coming through here per second, and they're pretty much all user-input driven. Plus, O(N) where N=number of windows? That's gonna be pretty small unless you're actively trying to spawn 100's of windows per second and close them quickly.

…sive_mutex. This is done by making _getPeasant not remove from _mruPeasants on failure, and _getMostRecentPeasantID handle its own cleanup.

github-actions · 2021-09-16T17:32:51Z

@check-spelling-bot Report

Unrecognized words, please review:

ulock

Previously acknowledged words that are now absent

sid SPACEBAR Unregister xIcon yIcon

To accept these unrecognized words as correct (and remove the previously acknowledged and now absent words), run the following commands

... in a clone of the git@github.com:Rosefield/terminal.git repository
on the bug/make-monarch-more-thread-safe branch:

update_files() {
perl -e '
my @expect_files=qw('".github/actions/spelling/expect/alphabet.txt
.github/actions/spelling/expect/expect.txt
.github/actions/spelling/expect/web.txt"');
@ARGV=@expect_files;
my @stale=qw('"$patch_remove"');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
next if /^(?:$re)(?:(?:\r|\n)*$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spelling/expect/4793541c90be0baee7129bc0c138dbd862bb19b5.txt";
use File::Path qw(make_path);
use File::Basename qw(dirname);
make_path (dirname($new_expect_file));
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"$patch_add"');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a)."-".$a cmp lc($b)."-".$b} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;
system("git", "add", $new_expect_file);
'
}

comment_json=$(mktemp)
curl -L -s -S \
  --header "Content-Type: application/json" \
  "https://api.github.com/repos/microsoft/terminal/issues/comments/921096069" > "$comment_json"
comment_body=$(mktemp)
jq -r .body < "$comment_json" > $comment_body
rm $comment_json

patch_remove=$(perl -ne 'next unless s{^</summary>(.*)</details>$}{$1}; print' < "$comment_body")
  

patch_add=$(perl -e '$/=undef;
$_=<>;
s{<details>.*}{}s;
s{^#.*}{};
s{\n##.*}{};
s{(?:^|\n)\s*\*}{}g;
s{\s+}{ }g;
print' < "$comment_body")
  
update_files
rm $comment_body
git add -u

✏️ Contributor please read this

By default the command suggestion will generate a file named based on your commit. That's generally ok as long as you add the file to your commit. Someone can reorganize it later.

⚠️ The command is written for posix shells. You can copy the contents of each perl command excluding the outer ' marks and dropping any '"/"' quotation mark pairs into a file and then run perl file.pl from the root of the repository to run the code. Alternatively, you can manually insert the items...

If the listed items are:

... misspelled, then please correct them instead of using the command.
... names, please add them to .github/actions/spelling/allow/names.txt.
... APIs, you can add them to a file in .github/actions/spelling/allow/.
... just things you're using, please add them to an appropriate file in .github/actions/spelling/expect/.
... tokens you only need in one place and shouldn't generally be used, you can add an item in an appropriate file in .github/actions/spelling/patterns/.

See the README.md in each directory for more information.

🔬 You can test your commits without appending to a PR by creating a new branch with that extra change and pushing it to your fork. The check-spelling action will run in response to your push -- it doesn't require an open pull request. By using such a branch, you can limit the number of typos your peers see you make. 😉

🗜️ If you see a bunch of garbage

If it relates to a ...

well-formed pattern

See if there's a pattern that would match it.

If not, try writing one and adding it to a patterns/{file}.txt.

Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

Note that patterns can't match multiline strings.

binary-ish string

Please add a file path to the excludes.txt file instead of just accepting the garbage.

File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

DHowett · 2021-09-16T18:02:04Z

Curious ... does this do anything to help the issue where you can press win+backtick twice really fast and get two windows that think they're named _quake?

Rosefield · 2021-09-16T18:29:30Z

I just tested, no it does not. In fact it might make it more common. I think fixing that would require separate synchronization in the monarch.

Basically the following happens

the monarch receives 2 requests simultaneously for "is there a window named _quake", which the monarch happily responds "no"
Then after that two new shells are spawned with commandline args to be named _quake
Each of those shells then proposes their command line asking the monarch "is there a peasant named _quake", but there still isn't yet so the monarch says each shell should create a window
Finally those peasants name themself _quake and add themselves to the monarch
Which then result in adding two peasants with the name _quake.

There are locks each time _peasants is used in 1/3/4, which does serialize the requests within each item, but once one peasant hits (4) if there is another peasant at (3) it will be waiting on the lock before it adds itself. Basically there needs to be some sort of pre-registration in (3) of "I am going to be a _quake window, don't let anyone else try" before the peasant is actually added in (4).

…deadlock from being introduced.

…at take unique locks.

lhecker · 2021-09-16T19:56:30Z

src/cascadia/Remoting/Monarch.cpp

+            // I wish I had std::erase_if (C++20) but I don't yet
+            // partition the array into [not dead | dead ]
+            auto partition = std::stable_partition(_mruPeasants.begin(), _mruPeasants.end(), [&](const auto& p) {
+                const auto id = p.PeasantID();
+                const auto it = std::find(peasantsToErase.cbegin(), peasantsToErase.cend(), id);
+                // keep the element if it was not found in the list to erase.
+                return it == peasantsToErase.cend();
+            });


Why don't you use std::remove_if (erase-remove idiom)?

const auto it = std::remove_if(_mruPeasants.begin(), _mruPeasants.end(), [&](const auto& p) { const auto id = p.PeasantID(); const auto it = std::find(peasantsToErase.cbegin(), peasantsToErase.cend(), id); return it != peasantsToErase.cend(); }); _mruPeasants.erase(it, _mruPeasants.end());

Since you repeatedly do a std::find over a vector it would be preferable to use a std::unordered_set for peasantsToErase. Afterwards you can write:

_mruPeasants.erase(std::remove_if(_mruPeasants.begin(), _mruPeasants.end(), [&](const auto& p) { return peasantsToErase.count(p.PeasantID()); }), _mruPeasants.end());

That is definitely clearer. For some reason I couldn't find remove_if (maybe I was looking on std::vector directly).

I still don't think there is value in using unordered_set, especially with an expected small number of values to erase, but that is mostly me being stubborn. Really the problem is that the C++ std library is deficient and there is no convenient .contains on a vector.

You could also view it in a different light: peasantsToErase is almost always empty during runtime, so the performance benefit of using a vector normally doesn't apply. Neither container class allocates memory in such a case.
On the other hand if - due to some random misfortune - we do end up having a lot of peasants to erase, the unordered set will perform asymptotically better. It's also more readable in my opinion.

Although I honestly wouldn't block this PR over such a detail.

To be more pedantic, unordered_set.contains has worst case runtime of linear still :)

The hash function we use through the STL (FNV1a) makes that quite unlikely however.
std::unordered_set is about ~2x slower than std::vector for such worst cases IIRC (with small data sets).

I'm fairly stoked on adding robin_hood::unordered_map soon, as it uses a plain vector as the underlying container.

switched this and the other cases that used vectors to unordered_sets, and made the _clearOldMruEntries just take a set as parameter.

lhecker · 2021-09-16T19:58:01Z

src/cascadia/Remoting/Monarch.cpp

+        {
+            std::shared_lock lock{ _peasantsMutex };
+            names.reserve(_peasants.size());
+        }


BTW This makes me certain that we should create a thread safe proxy class for the Monarch in the future.
(Basically wrap the public methods of the class and make them thread-safe. The actual MonarchImpl class can then continue pretending threads don't exist.)

There is something to say about the simplicity of just throwing a std::unique_lock lock {_mutex} at the top of everything. There is a minor concern about event handlers that call back up to the monarch but that is a solvable problem too. 🤷

Yeah if you consider how we got event handlers, it's clear that recursive mutexes are the best solution here. You already have me entirely convinced about that. 😄
In either case I suppose that's something for a different time.

…ke stable_partition

zadjii-msft

Yea I'm still cool with this. I agree, this would probably be easier with one lock for all of the monarch internals, but I'm not going to stand in the way of correctness.

ghost · 2021-09-21T21:20:33Z

Hello @zadjii-msft!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

ghost · 2021-10-20T17:04:01Z

🎉Windows Terminal Preview v1.12.2922.0 has been released which incorporates this pull request.:tada:

Handy links:

Attempt to make the monarch more thread safe.

4eedfcc

spelling and formatting.

825dfc3

lhecker reviewed Sep 9, 2021

View reviewed changes

Rosefield added 5 commits September 9, 2021 19:43

Use _forEachPeasant correctly, and make the dead peasant use the erro…

91f1845

…r code we expect a dead peasant to have.

Some code review

e386d6f

Actually compiling the code is a good idea.

f765728

Change a fail_fast to an assert

5f632f4

More code review changes: be less trict about memory ordering

ee463aa

Rosefield mentioned this pull request Sep 13, 2021

Mysterious hang in Monarch::_getMostRecentPeasantID #11215

Closed

zadjii-msft self-assigned this Sep 13, 2021

Merge remote-tracking branch 'origin/main' into bug/make-monarch-more…

3aeaeed

…-thread-safe Conflicts: src/cascadia/Remoting/Monarch.cpp src/cascadia/UnitTests_Remoting/RemotingTests.cpp

zadjii-msft approved these changes Sep 16, 2021

View reviewed changes

Refactor _getMostRecentPeasantID so that we dont need to have a recur…

2f0db9e

…sive_mutex. This is done by making _getPeasant not remove from _mruPeasants on failure, and _getMostRecentPeasantID handle its own cleanup.

make the spellchecker happy

e1c2f53

Rosefield added 2 commits September 16, 2021 14:55

Explicitly unlock _mruPeasants when we are done with it to prevent a …

80646b3

…deadlock from being introduced.

Be more careful to ensure no deadlocks can occur. Annotate methods th…

be2cdbe

…at take unique locks.

lhecker reviewed Sep 16, 2021

View reviewed changes

Rosefield added 2 commits September 16, 2021 16:17

use an actually appropriate std method instead of something exotic li…

d95629b

…ke stable_partition

Use unordered_sets instead of vectors for erasing.

8fd2b75

lhecker approved these changes Sep 16, 2021

View reviewed changes

zadjii-msft approved these changes Sep 21, 2021

View reviewed changes

zadjii-msft added Area-Remoting Communication layer between windows. Often for windowing behavior, quake mode, etc. AutoMerge Marked for automatic merge by the bot when requirements are met labels Sep 21, 2021

ghost merged commit 591a671 into microsoft:main Sep 21, 2021

Rosefield deleted the bug/make-monarch-more-thread-safe branch September 21, 2021 21:29

zadjii-msft mentioned this pull request Sep 27, 2021

Mysterious crash in Monarch::_clearOldMruEntries #10028

Closed

This pull request was closed.

		std::unordered_map<uint64_t, winrt::Microsoft::Terminal::Remoting::IPeasant> _peasants;
		std::shared_mutex _peasantsMutex{};


		std::vector<Remoting::WindowActivatedArgs> _mruPeasants;
		std::recursive_mutex _mruPeasantsMutex{};

Attempt to make the monarch more thread safe. #11189

Attempt to make the monarch more thread safe. #11189

Conversation

Rosefield commented Sep 9, 2021 • edited Loading

Summary of the Pull Request

References

PR Checklist

Detailed Description of the Pull Request / Additional comments

Validation Steps Performed

github-actions bot commented Sep 9, 2021 • edited Loading

@check-spelling-bot Report

Unrecognized words, please review:

Rosefield commented Sep 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rosefield Sep 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rosefield Sep 9, 2021 • edited Loading

Choose a reason for hiding this comment

Rosefield commented Sep 10, 2021 • edited Loading

zadjii-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 16, 2021 • edited Loading

@check-spelling-bot Report

Unrecognized words, please review:

DHowett commented Sep 16, 2021

Rosefield commented Sep 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zadjii-msft left a comment

Choose a reason for hiding this comment

ghost commented Sep 21, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

ghost commented Oct 20, 2021

Rosefield commented Sep 9, 2021 •

edited

Loading

github-actions bot commented Sep 9, 2021 •

edited

Loading

Rosefield Sep 16, 2021 •

edited

Loading

Rosefield Sep 9, 2021 •

edited

Loading

Rosefield commented Sep 10, 2021 •

edited

Loading

github-actions bot commented Sep 16, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.