Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce usage of Json::Value throughout Terminal.Settings.Model #11184

Merged
30 commits merged into from
Sep 22, 2021

Conversation

lhecker
Copy link
Member

@lhecker lhecker commented Sep 9, 2021

This commit reduces the code surface that interacts with raw JSON data,
reducing code complexity and improving maintainability.
Files that needed to be changed drastically were additionally
cleaned up to remove any code cruft that has accrued over time.

In order to facility this the following changes were made:

  • Move JSON handling from CascadiaSettings into SettingsLoader
    This allows us to use STL containers for data model instances.
    For instance profiles are now added to a hashmap for O(1) lookup.
  • JSON parsing within SettingsLoader doesn't differentiate between user,
    inbox and fragment JSON data, reducing code complexity and size.
    It also centralizes common concerns, like profile deduplication and
    ensuring that all profiles are assigned a GUID.
  • Direct JSON modification, like the insertion of dynamic profiles into
    settings.json were removed. This vastly reduces code complexity,
    but unfortunately removes support for comments in JSON on first start.
  • ColorSchemes cannot be layered. As such its LayerJson API was replaced
    with FromJson, allowing us to remove JSON-based color scheme validation.
  • Profiles used to test their wish to layer using ShouldBeLayered, which
    was replaced with a GUID-based hashmap lookup on previously parsed profiles.

Further changes were made as improvements upon the previous changes:

  • Compact the JSON files embedded binary, saving 28kB
  • Prevent double-initialization of the color table in ColorScheme
  • Making til::color getters constexpr, allow better optimizations

The result is a reduction of:

  • 48kB binary size for the Settings.Model.dll
  • 5-10% startup duration
  • 26% code for the CascadiaSettings class
  • 1% overall code in this project

Furthermore this results in the following breaking changes:

  • The long deprecated "globals" settings object will not be detected and no
    warning will be created during load.
  • The initial creation of a new settings.json will not produce helpful comments.

Both cases are caused by the removal of manual JSON handling and the
move to representing the settings file with model objects instead

PR Checklist

Validation Steps Performed

  • Out-of-box-experience is identical to before ✔️
    (Except for the settings.json file lacking comments.)
  • Existing user settings load correctly ✔️
  • New WSL instances are added to user settings ✔️
  • New fragments are added to user settings ✔️
  • All profiles are assigned GUIDs ✔️

@@ -74,7 +74,7 @@
x:Uid="Globals_DefaultTerminal"
x:Load="false">
<ComboBox x:Name="DefaultTerminal"
ItemsSource="{x:Bind State.Settings.DefaultTerminals, Mode=OneWay}"
ItemsSource="{x:Bind State.Settings.DefaultTerminals}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.. why the change here? I guess we never reload this in practice...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I started testing my changes, the SUI was crashing on load and with @carlos-zamora I was able to determine that I made a mistake with my DefaultTerminals() implementation. After fixing my issue only this change remained. I believe this change to be correct as DefaultTerminals is unable to post updates (it's a pull-based API and cannot "push" updates).
But I do want to minimize the PR complexity and would thus be entirely happy to revert any change we deem unnecessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine? Only concern is if we're still updating the ItemsSource when we reload the page. But that's more a concern in Launch.cpp::OnNavigatedTo(). Mind double checking that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep it refreshes _State.

@lhecker lhecker force-pushed the dev/lhecker/settings-json-cleanup branch from 7412dfc to 1efe485 Compare September 9, 2021 18:04
Copy link
Member

@DHowett DHowett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

48/55 files reviewed! Would you mind sharing the output from running the LocalTests harness?

src/cascadia/TerminalSettingsModel/userDefaults.json Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/userDefaults.json Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/Profile.cpp Outdated Show resolved Hide resolved
static com_ptr<Profile> CloneInheritanceGraph(com_ptr<Profile> oldProfile, com_ptr<Profile> newProfile, std::unordered_map<void*, com_ptr<Profile>>& visited);
static com_ptr<Profile> CopySettings(com_ptr<Profile> source);
static void InsertParentHelper(com_ptr<Profile> child, com_ptr<Profile> parent, std::optional<size_t> index = std::nullopt);
static void CopyInheritanceGraph(std::unordered_map<const Profile*, winrt::com_ptr<Profile>>& visited, const std::vector<winrt::com_ptr<Profile>>& source, std::vector<winrt::com_ptr<Profile>>& target);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reduction of visited from many to one suggests that we'll have an inheritance issue -- you may need to explain why there are no longer multiple parents or interlinked branches in the tree.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a comment and renamed the two functions to make the intent more clear.
But they're still a basic algorithm to clone a directed acyclic graph now.

What I did here is to split up the previous function, which did "clone this node and its sub-graph", into:

  • "clone this node and its sub-graph" and
  • "clone these child-nodes and their sub-graphs"

The former uses the latter to clone the children of a node, whereas the latter uses the former to clone each child-node in a loop. The recursion thus now hops back and forth between two functions. This simplifies the implementation a lot.

settings->_ApplyDefaultsFromUserSettings();
settings->LayerJson(settings->_userSettings);
settings->_ValidateSettings();
"$help" : "https://aka.ms/terminal-documentation",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CURIOUS! What does this do?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I removed the helpful comments from userDefaults.json, I thought a documentation link might be useful for some. It's the first field in the settings.json.

src/cascadia/TerminalApp/AppLogic.cpp Show resolved Hide resolved
src/cascadia/TerminalApp/AppLogic.cpp Outdated Show resolved Hide resolved
VERIFY_ARE_EQUAL(L"NeitherShouldThisOne", settings->_allProfiles.GetAt(4).Name());
}

void DeserializationTests::TestExplodingNameOnlyProfiles()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we validate that this invariant is still true?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered this test in particular a duplicate of TestLayeringNameOnlyProfiles which is extremely similar. So I only fixed one of the two tests. Did I maybe miss something?

@zadjii-msft zadjii-msft self-assigned this Sep 9, 2021
Copy link
Member

@carlos-zamora carlos-zamora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok! I've taken a look at everything except...

  • the tests
  • Profile: copying the inheritance graph

A few memorable notes:

  • Take a look at Generate GUID's when it's requested (Profile) #7421 and Add support for profiles.defaults in defaults.json #5276. You might be able to close them out.
  • userDefaults.json still needs those comments. I didn't check if we're still populating the special variables in there (or I've forgotten how you do it). But we have to make sure we don't break that.
  • SettingsLoader would be extra nice if we moved away from knowing which layer we're in. I envision a world where we can easily load a chain of files. That's basically what you're already doing with fragments. So might as well change the wording a bit to check that off the list.

src/cascadia/TerminalSettingsModel/userDefaults.json Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/DefaultProfileUtils.h Outdated Show resolved Hide resolved
src/cascadia/TerminalApp/AppLogic.cpp Outdated Show resolved Hide resolved
{
pScheme->LayerJson(schemeJson);
throw SettingsException(SettingsLoadErrors::AllProfilesHidden);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a user only wants their dynamic/fragment profiles? Won't that trigger this exception?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sure! But this code is equivalent to the current code.

src/cascadia/TerminalSettingsModel/Profile.cpp Outdated Show resolved Hide resolved
@ghost ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Sep 10, 2021
Copy link
Member

@carlos-zamora carlos-zamora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Reviewed everything except 4 of the tests. But I'll get to those when the PR is out of draft.

README.md Show resolved Hide resolved
{
clone = CopySettings();
CopyInheritanceGraph(visited, _parents, clone->_parents);
clone->_FinalizeInheritance();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we also need to add the clone to visited here? That way if we encounter it again, clone will be populated above?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or just always emplace in CopyInheritanceGraph.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha! This is some evil C++ abuse! 😄
I'm using the operator[] a few lines above to get a reference a node in visited.
If the node doesn't exist the operator[] will default-initialize it (which is an empty com_ptr). In this if-condition I'm checking for emptiness and then assign clone with a clone. Since clone is a reference into the hashmap it writes it into the hashmap directly.

src/cascadia/TerminalSettingsModel/Profile.cpp Outdated Show resolved Hide resolved
@ghost ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Sep 10, 2021
@DHowett
Copy link
Member

DHowett commented Sep 14, 2021

You gotta write up a PR body!

@ghost ghost added Area-CodeHealth Issues related to code cleanliness, linting, rules, warnings, errors, static analysis, etc. Area-Settings Issues related to settings and customizability, for console or terminal Issue-Task It's a feature request, but it doesn't really need a major design. Product-Terminal The new Windows Terminal. labels Sep 14, 2021
@lhecker lhecker marked this pull request as ready for review September 14, 2021 20:33
azureCloudShellProfile.ConnectionType(AzureConnection::ConnectionType());
profiles.emplace_back(azureCloudShellProfile);
auto azureCloudShellProfile{ CreateDefaultProfile(AzureGeneratorNamespace, L"Azure Cloud Shell") };
azureCloudShellProfile->Commandline(L"Azure");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't. It's just ballast.

co_await winrt::resume_background();
auto v{ co_await task };
finalVal.emplace(co_await task);
latch.count_down();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for making this use latch ;P

if (fullFile.isMember(JsonKey(SchemesKey)))
// profiles.defaults
{
settings.baseLayerProfile = Profile::FromJson(defaultsObject);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works even if defaults has been deleted (and is therefore a json null), correct?

Copy link
Member Author

@lhecker lhecker Sep 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Most tests don't specify a defaults object for instance.

src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp Outdated Show resolved Hide resolved
@ghost ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Sep 14, 2021
@DHowett
Copy link
Member

DHowett commented Sep 17, 2021

@msftbot make sure @zadjii-msft signs off

@ghost ghost added the AutoMerge Marked for automatic merge by the bot when requirements are met label Sep 17, 2021
@ghost
Copy link

ghost commented Sep 17, 2021

Hello @DHowett!

Because you've given me some instructions on how to help merge this pull request, I'll be modifying my merge approach. Here's how I understand your requirements for merging this pull request:

  • I'll only merge this pull request if it's approved by @zadjii-msft

If this doesn't seem right to you, you can tell me to cancel these instructions and use the auto-merge policy that has been configured for this repository. Try telling me "forget everything I just told you".

@DHowett DHowett removed the AutoMerge Marked for automatic merge by the bot when requirements are met label Sep 17, 2021
Copy link
Member

@DHowett DHowett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels robust and effective, and I appreciate that I can now almost keep all the logic in my head at once. I've bothered Leonard numerous times during the evenings to tell him that I found "yet another" issue, and... I think I'm out of issues. I can't complain about the code style or the comments any more than I have, and I think this is ready to go.

@carlos-zamora, can you re-review?

I want 3 people to sign off on this as it's a big one. @zadjii-msft is my designated third. 😄

Copy link
Member

@carlos-zamora carlos-zamora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest also updating the PR body to mention any "breaking" changes we're ok with. That way it's a lot easier to track than having to look through this PR again. Some that come to mind are:

  • removal of "globals" warning
  • removal of comments on initial settings.json load

All the other comments I have don't have anything to do with the logic of this code. Happy to see this merge soon!

Comment on lines +58 to +59
ParsedSettings inboxSettings;
ParsedSettings userSettings;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(bump. Still want some kind of resolution on this one.)

src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/CascadiaSettings.cpp Outdated Show resolved Hide resolved
src/cascadia/TerminalSettingsModel/ColorScheme.h Outdated Show resolved Hide resolved
@lhecker
Copy link
Member Author

lhecker commented Sep 20, 2021

@carlos-zamora Due to the large number of comments and GitHub collapsing them (y u do dis?) let me comment here regarding inboxSettings/parentSettings & userSettings/childSettings:

Would it be possible to rename it to something like this? That way, if we ever want to add a chain of layers, we can reuse this?

I'd like to refrain from this for now. I feel like that a name that's as specific as possible helps newcomers to understand the code more easily. It also allows us to develop a mental model that solves exactly just the problem of only solving the problem of "merging builtin with user-given settings", instead of creating a more general solution. In my personal opinion it should be easy for any of us to rename the member variables and functions to something else without involving sweeping changes. After all the inbox/user terminology is concentrated entirely in SettingsLoader (and the CascadiaSettings constructor) and thus doesn't require us to touch public API code.

Copy link
Member

@zadjii-msft zadjii-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

46/67. Finished CascadiaSettingsSerialization, I suspect most of the remainder is tests. Boy GH does not like reviewing this diff thou, so big

Comment on lines 239 to 252
<data name="LegacyGlobalsProperty" xml:space="preserve">
<value>The "globals" property is deprecated - your settings might need updating. </value>
<comment>{Locked="\"globals\""} </comment>
</data>
<data name="LegacyGlobalsPropertyHrefUrl" xml:space="preserve">
<value>https://go.microsoft.com/fwlink/?linkid=2128258</value>
<comment>{Locked}This is a FWLink, so it will be localized with the fwlink tool</comment>
</data>
<data name="LegacyGlobalsPropertyHrefLabel" xml:space="preserve">
<value>For more info, see this web page.</value>
</data>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump for an answer to this one @DHowett

Comment on lines +58 to +59
ParsedSettings inboxSettings;
ParsedSettings userSettings;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meh, if there's a chain, then there's gonna be more than two anyways, so parent/child doesn't really matter, yea? it's just gonna be a vector of layers

if (!isDefaultSettings)
// FYI: The static_cast ensures we don't move the profile into
// `profilesByGuid`, even though we still need it later for `profiles`.
if (settings.profilesByGuid.emplace(profile->Guid(), static_cast<const winrt::com_ptr<Profile>&>(profile)).second)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If i'm getting this right, say we have two profiles in the settings.json:

{ "name": "foo" }, 
{ "name": "foo", "background": "#123456" },

we'll ignore the second one entirely, because it's a duplicate (and later use duplicateProfile to raise a warning?)

Don't we currently just layer the second one on top of the first?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can restore this old behavior if we'd like. I just found it somewhat inconsistent with the duplicateProfile warning and assumed that the likelihood of duplicate profiles in a single file is low.
I'll add a note to the PR body.

Should I restore the old behavior?

bool SettingsLoader::DisableDeletedProfiles()
{
const auto& state = winrt::get_self<ApplicationState>(ApplicationState::SharedInstance());
auto generatedProfileIds = state->GeneratedProfiles();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can GeneratedProfiles be null (like the first time the file is created, for ex)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No it returns a STL class actually (which is why I use get_self here).

_CatchRethrowSerializationExceptionWithLocationInfo(resultPtr->_userSettingsString);
// If inserted is true, then this is a generated profile that doesn't exist in the user's settings.
// While emplace() has already created an appropriate entry in .profilesByGuid, we still need to
// add it to .profiles (which is basically a sorted list of .profilesByGuid's values).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AH okay we CreateChild here because there wasn't an entry in the user's settings for this default profile. The child now acts like it belongs in the user settings, so when we write the file back out, we'll actually serialize that child to settings.json. Okay, that now makes some sense. Had to read this and the fragments version a few times to figure out why we were making a child here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment. 👍

Copy link
Member

@zadjii-msft zadjii-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm only blocking for:

  • The Opacity thing
  • The resources thing

y'all can feel free to dismiss me if that's sorted out after 5pm Wisconsin time.

I feel sad at the general loss of the json patching - I once had a mind to just do json patching for all settings writing, but the SUI is good enough at this point that it doesn't really matter anymore. I'll miss the comments, but whatever.

src/cascadia/TerminalSettingsModel/Profile.cpp Outdated Show resolved Hide resolved
osver.dwOSVersionInfoSize = sizeof(osver);
osver.dwBuildNumber = 21359;
osver.dwBuildNumber = 22000;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea this is probably a good call

@@ -1,673 +0,0 @@
// Copyright (c) Microsoft Corporation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these guys didn't really get resurrected anywhere, did they?

Copy link
Member Author

@lhecker lhecker Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestGenGuidsForProfiles was moved into ProfileTests.cpp.
I've removed the others because the SettingsLoader makes no distinction between the source of builtin profiles: "inbox" JSON data and generated profiles are treated the exact same way. SettingsLoader::GenerateProfiles calls SettingsLoader::_executeGenerator which basically only adds generated profiles straight to said inboxSettings.profiles vector.
Considering the extensive coverage of inbox JSON related tests in SerializationTests.cpp I've thus decided to not port these tests to the new API and saved myself some work.

Would you like me to restore the tests?

@ghost ghost added Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something and removed Needs-Author-Feedback The original author of the issue/PR needs to come back and respond to something labels Sep 21, 2021
@lhecker
Copy link
Member Author

lhecker commented Sep 22, 2021

I feel sad at the general loss of the json patching [...]

Yeah me too. It might be worthwhile to re-add such support in the future actually! Considering that we had no JSON comment support for our SUI convinced me that it's not a significant loss if we remove the remaining support for it (basically: splicing generated profiles into the settings file).
Unfortunately our current JSON comment support depends on JsonCPP which I wouldn't call a library I enjoy. Alternatives like the excellent nlohmann/json library don't support the preservation of comments the way we'd like to at all unfortunately.

@github-actions

This comment has been minimized.

Copy link
Member

@zadjii-msft zadjii-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RIP literally every pr

@zadjii-msft zadjii-msft removed their assignment Sep 22, 2021
@zadjii-msft zadjii-msft added the AutoMerge Marked for automatic merge by the bot when requirements are met label Sep 22, 2021
@ghost
Copy link

ghost commented Sep 22, 2021

Hello @zadjii-msft!

Because this pull request has the AutoMerge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@ghost ghost merged commit 168d28b into main Sep 22, 2021
@ghost ghost deleted the dev/lhecker/settings-json-cleanup branch September 22, 2021 16:27
@lhecker lhecker mentioned this pull request Oct 1, 2021
3 tasks
ghost pushed a commit that referenced this pull request Oct 5, 2021
This commit fixes various failing TestHostApp unit tests.
Most of these broke as part of 168d28b (#11184).

## PR Checklist
* [x] Closes #11339
* [x] I work here
* [x] Tests added/passed
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-CodeHealth Issues related to code cleanliness, linting, rules, warnings, errors, static analysis, etc. Area-Settings Issues related to settings and customizability, for console or terminal AutoMerge Marked for automatic merge by the bot when requirements are met Issue-Task It's a feature request, but it doesn't really need a major design. Product-Terminal The new Windows Terminal.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generate GUID's when it's requested (Profile) Add support for profiles.defaults in defaults.json
5 participants