Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding Humanizer to source-build #1767

Closed
crummel opened this issue Sep 24, 2020 · 25 comments
Closed

Consider adding Humanizer to source-build #1767

crummel opened this issue Sep 24, 2020 · 25 comments
Assignees
Labels
area-additional-repos Adding additional contributing repos area-patch-removal Removing patches for contributing repos from source-build area-product-experience Improvements in the end-user's product experience

Comments

@crummel
Copy link
Contributor

crummel commented Sep 24, 2020

Roslyn uses Humanizer to pluralize and convert to/from CamelCase for human-friendly automatic variable names and debug output. This was removed in #1766 to remove the prebuilt. We should either work with Roslyn to eliminate this dependency or add the Humanizer repo to source-build.

@crummel crummel added area-product-experience Improvements in the end-user's product experience area-additional-repos Adding additional contributing repos labels Sep 24, 2020
@dseefeld dseefeld added triaged area-patch-removal Removing patches for contributing repos from source-build labels Oct 8, 2020
dseefeld added a commit to dseefeld/source-build that referenced this issue Oct 23, 2020
Note: Removed aspnetcore version number patch because it is set to
a stable 5.0 version.
Note: Removed roslyn Humanizer removal patch because there is now
more usage of Humanizer in the code base.  May need to redo this
patch or add Humanizer.  (See dotnet#1767)
@dagood
Copy link
Member

dagood commented Oct 30, 2020

For 5.0 GA, #1845 removes the patch that had removed the Humanizer prebuilt. Humanizer got a lot more usage in GA and it wasn't reasonable to try to reapply it all while trying to get GA through. We'll either need to recreate the patch or add the repo. /cc @dseefeld

The repo looks like a normal-enough csproj... I think we should try that. I'm wondering if some of the new usages might be more impactful than the old ones, and would make more of a feature gap if we tried to patch them out manually.

@dagood
Copy link
Member

dagood commented Oct 30, 2020

@omajid @tmds I think adding this repo might be a reasonable thing for one of you to pick up if you can. (Shouldn't involve as much delving into binlogs and such.)

dseefeld added a commit to dseefeld/source-build that referenced this issue Oct 30, 2020
Note: Removed aspnetcore version number patch because it is set to
a stable 5.0 version.
Note: Removed roslyn Humanizer removal patch because there is now
more usage of Humanizer in the code base.  May need to redo this
patch or add Humanizer.  (See dotnet#1767)
@omajid
Copy link
Member

omajid commented Nov 2, 2020

Humanizer is also used by roslyn-analyzers

@omajid omajid self-assigned this Nov 2, 2020
@omajid
Copy link
Member

omajid commented Nov 2, 2020

The Microsoft SDK doesn't include Humanzier, does it?

$ tar tf dotnet-sdk-5.0.100-rc.2.20479.15-linux-x64.tar.gz | grep -i human
<empty>

Is this shipped with the SDK?

@dagood
Copy link
Member

dagood commented Nov 2, 2020

Huh... maybe none of the projects (as in csproj) that build using Humanizer actually ship in the .NET SDK? Considering we had to actually patch .cs files that contain using Humanizer; that's the only thing that makes sense to me right now.

If we can disable unused projects, that'd be ideal. If for some reason that doesn't work yet we do know the projects aren't used, we could put Humanizer in SBRP.

@crummel, do you know more about this, since you implemented the first patch?
@dseefeld, any insight on how this has potentially changed in GA?

The last possibility I can think of is Humanizer being used at build-time but not in the SDK, somehow. I don't have any evidence pointing to it. If we disable the projects or put Humanizer as a ref package, it seems more likely that we'll see a build break rather than something subtle (a string breaking somewhere in the SDK), so I don't think this is a big concern.

@crummel
Copy link
Contributor Author

crummel commented Nov 2, 2020

I'm pretty sure Microsoft.CodeAnalysis.CSharp.Workspaces should at least be shipping, the other code style projects maybe not. I'm taking a look now.

@omajid
Copy link
Member

omajid commented Nov 2, 2020

I tried building roslyn after removing the using Humanizer references and get:

/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/CodeStyle/CSharp/Analyzers/Microsoft.CodeAnalysis.CSharp.CodeStyle.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/CodeStyle/CSharp/Analyzers/Microsoft.CodeAnalysis.CSharp.CodeStyle.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(236,80): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
/home/omajid/devel/dotnet/roslyn/src/Workspaces/SharedUtilitiesAndExtensions/Compiler/CSharp/Extensions/SemanticModelExtensions.cs(262,46): error CS1061: 'string' does not contain a definition for 'Pluralize' and no accessible extension method 'Pluralize' accepting a first argument of type 'string' could be found (are you missing a using directive or an assembly reference?) [/home/omajid/devel/dotnet/roslyn/src/Workspaces/CSharp/Portable/Microsoft.CodeAnalysis.CSharp.Workspaces.csproj]
0 Warning(s)

The RC2 SDK includes Microsoft.CodeAnalysis.CSharp.CodeStyle.dll, one of the projects that are erroring out:

$ tar tf dotnet-sdk-5.0.100-rc.2.20479.15-linux-x64.tar.gz | grep -i Microsoft.CodeAnalysis.CSharp.CodeStyle.dll
./sdk/5.0.100-rc.2.20479.15/Sdks/Microsoft.NET.Sdk/codestyle/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll

@dagood
Copy link
Member

dagood commented Nov 2, 2020

It looks like Roslyn removed the references from a lot of projects except Microsoft.CodeAnalysis.CSharp.Workspaces: dotnet/roslyn#47297, and there are some (unfulfilled?) plans to add a Humanizer reference to dotnet/sdk. Our dotnet/roslyn commit seems to be in a weird spot: after Humanizer refs were added, but in the wrong places.

Microsoft.CodeAnalysis.CSharp.Workspaces should at least be shipping, the other code style projects maybe not.

I'm seeing kind of the opposite in terms of the DLLs being in the SDK. Nothing for find . -iname '*workspaces*' in the SDK, but Microsoft.CodeAnalysis.CSharp.CodeStyle.dll is where @omajid found it.

Which would be a good thing for us. Maybe Microsoft.CodeAnalysis.CSharp.Workspaces is VS-specific and we don't care about all this?

@dagood
Copy link
Member

dagood commented Nov 2, 2020

@333fred, can you help with this question for 5.0 roslyn source-buildability? Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK, and if not and we stop building it in source-build, will the Humanizer prebuilt dependency disappear?

@333fred
Copy link
Member

333fred commented Nov 2, 2020

@333fred, can you help with this question for 5.0 roslyn source-buildability? Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK, and if not and we stop building it in source-build, will the Humanizer prebuilt dependency disappear?

Yes, it is. In fact, dotnet/roslyn#48999 just added the CodeStyle layer because that is shipped in source-build, and that has an explicit dependency on the workspaces layer.

@dagood
Copy link
Member

dagood commented Nov 3, 2020

@333fred Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK? 5.0 GA, 5.0.1? It seems odd that we haven't been able to find it anywhere. What can we compare source-build against to make sure we have this right?

@tmds
Copy link
Member

tmds commented Nov 3, 2020

@333fred functionally roslyn doesn't need Humanizer, right? Would we want to consider removing it from roslyn all together for .NET 6?

@333fred
Copy link
Member

333fred commented Nov 3, 2020

Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK?

I'm not sure; the question was whether the Workspaces layer is in source-build, and that should be true.

functionally roslyn doesn't need Humanizer, right?

The compiler does not, but many of our analyzers do. For example, @dagood you mentioned the other day that you were trying to get the roslyn-analyzers repo into source-build, and that has a dependency on humanizer as well: https://github.com/dotnet/roslyn-analyzers/blob/79c2638f54b9e7ef0a3954cc838ec4c13ea63c1f/src/NetAnalyzers/Core/Microsoft.CodeQuality.Analyzers/Microsoft.CodeQuality.Analyzers.csproj#L15, which it uses for CA1714. It's possible that the IDE could move the humanizer dependency out of the Workspaces layer (I couldn't find a reference to the extension methods actually in the workspaces layer, mostly in the features layer), but I'm not sure whether that would actually get rid of the dependency given that roslyn-analyzers ship in-box in .NET 5 and we'd like to be able to source-build them as well.

@dagood
Copy link
Member

dagood commented Nov 3, 2020

Interesting, when/where do you expect Humanizer.dll to show up in the .NET SDK?

I'm not sure; the question was whether the Workspaces layer is in source-build, and that should be true.

That's not what I meant by my question--when I say the SDK here I mean the .NET SDK:

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK

(Even if something's built in source-build, that doesn't necessarily mean it's shipped in a .NET SDK, source-built or not. For example, we build dotnet/arcade.)

@jaredpar you're also listed as a Roslyn champ, maybe you can help figure this out?

@jaredpar
Copy link
Member

jaredpar commented Nov 3, 2020

@dagood what question do we need resolved here?

@dagood
Copy link
Member

dagood commented Nov 3, 2020

Two questions:

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK?

When/where do you expect Humanizer.dll to show up in the .NET SDK?

@333fred
Copy link
Member

333fred commented Nov 3, 2020

@dagood before that though, I do think it's important to address this question:

It's possible that the IDE could move the humanizer dependency out of the Workspaces layer (I couldn't find a reference to the extension methods actually in the workspaces layer, mostly in the features layer), but I'm not sure whether that would actually get rid of the dependency given that roslyn-analyzers ship in-box in .NET 5 and we'd like to be able to source-build them as well.

I can't exclude workspaces from source-build, particularly as we just added codestyle to source-build. And I don't know whether moving the Humanizer dependency from the workspaces layer to a higher layer would help, because other analyzers depend on humanizer and since they ship in-box with .NET 5, I expect they're in the SDK. Looking at the SDK, I see the analyzers shipped in-box are Microsoft.CodeAnalysis.(CSharp|VisualBasic).NetAnalyzers. Cracking them open in ILSpy, I see that they also reference Workspaces.

Is Microsoft.CodeAnalysis.CSharp.Workspaces shipped in the SDK?

So, Microsoft.CodeAnalysis.CodeStyle.dll is shipped in the SDK: 5.0.100-rc.2.20479.15/Sdks/Microsoft.NET.Sdk/codestyle/cs/Microsoft.CodeAnalysis.CodeStyle.dll. Somehow, the Workspaces dll is not currently shipped in the SDK, which is very odd to me as both CodeStyle and the in-box NetAnalyzers have an explicit dependency on Workspaces.

When/where do you expect Humanizer.dll to show up in the .NET SDK?

Well, I'm surprised Workspaces isn't showing up in the SDK. And I'm therefore surprised that the dependencies of Workspaces aren't showing up in the SDK. I think moving the Humanizer dlls to the features layer would be possible: I think all the functionality that depends on it is actually in the features layer, not the workspaces layer.

@omajid
Copy link
Member

omajid commented Nov 3, 2020

I have a question that's mostly unrelated to the issue at hand. I see terms like "workspaces" and "codestyle", and I dont know what they mean. Is there some document I can read up on that clarifies what these components are and which of these are shipped (or should be shipped) as part of the .NET SDK?

@333fred
Copy link
Member

333fred commented Nov 3, 2020

I see terms like "workspaces" and "codestyle", and I dont know what they mean.

These are roslyn feature layers. There's no real document that I know of, but here's the basics:

For every layer, there is a non-language-specific core, and 2 language specific implementations.

Layer 1: The actual compiler dlls. These are responsible for actually compiling code, and exposing APIs for analyzers to use.
Layer 1.5: csc and vbc. These are command-line executables that run the compiler. They have no upwards dependencies.
Layer 2: Workspaces. This layer provides abstractions that make it easier for analyzers to manage project state.
Layer 3: Features. This layer provides IDE features that are not tied to a specific editor. For example, fetching a list of completion items.
Layer 4: EditorFeatures. This layer provides IDE features that are tied to a specific editor. For example, the VS navbar services.
Layer 5: I know there's more IDE layers here, but I'm not actually sure what comes next 😅

The CodeStyle project is a set of analyzers that have a dependency on Layer 2 here, and they ship in the SDK. The .NET 5 in-box analyzers also have a dependency on layer 2.

@dagood
Copy link
Member

dagood commented Nov 3, 2020

Thanks, a lot of answers. 😄 I don't have any objections to adding Humanizer to source-build and it seemed like a reasonable dependency--just takes some work since it's not on the usual infra. (Maybe since it's now in .NET Foundation we will be able to work with the maintainers in the future to add arcade + arcade-powered source-build? 🤞) Removing it is interesting, but it sounds like we're beyond being able to do that for 5.0 without losing features vs. the Microsoft-built SDK. (Which is my focus.)

My main concern here is how weird and confusing it is, how some DLLs aren't showing up where we expect. I wonder if this rises to the level of something to investigate with the Microsoft-built .NET SDK build as well? For now I guess we can assume that as long as the source-built .NET SDK has similar files, we'll at least have parity.

On that note... here's a link to the current 5.0 SDK we're targeting for source-build 5.0 GA #1845:
https://dotnetcli.blob.core.windows.net/dotnet/Sdk/5.0.100-rtm.20526.5/dotnet-sdk-5.0.100-linux-x64.tar.gz

dseefeld added a commit that referenced this issue Nov 3, 2020
* Update to 5.0.100-rtm.20521.5, BAR Build ID 68494

* Update previously source-built to N-1

* Reconcile patches

Note: Removed aspnetcore version number patch because it is set to
a stable 5.0 version.
Note: Removed roslyn Humanizer removal patch because there is now
more usage of Humanizer in the code base.  May need to redo this
patch or add Humanizer.  (See #1767)

* Update to 5.0.100-rtm.20522.4, BAR Build ID 68583

* Update commits in Version.Details.xml and reconcile patches

* Remove temporary RC1 fix in aspnetcore

* Install built runtime for usage by aspnetcore

* Update roslyn to build CodeStyle packages

* Update / add installer patches

* Fix broken patch

* Update prevSourceBuilt to GA version

* Pin System.Security.Cryptography.Cng version in runtime build

* Update path to sdk installation

* Update smoke test nuget.config entries

* Update online prebuilt baseline

* Update package name for aspnetcore reference

* Add runtime feed to smoke-testNuGet.Config

* Update runtime config entry

* Update to 20201026.5, BAR Build ID 68880

* Set UseStableVersions flag

* Update online baseline

* Update offline prebuilt baseline

* Include patch from #1850

Include patch to remove import of restore/harvestPackages.targets
in runtime libraries build to fix issue with portable build

* Re-add portable build only prebuilts
@omajid
Copy link
Member

omajid commented Nov 16, 2020

Circling back since roslyn-analyzers was recently added: it looks like only one part of roslyn-analzyers needs Humanizer. And that part of roslyn-analzyers is not used by source-build.

@dagood
Copy link
Member

dagood commented Nov 16, 2020

We still need it for Roslyn though, right? @omajid are you still planning to work on this?

@omajid
Copy link
Member

omajid commented Nov 16, 2020

We still need it for Roslyn though, right?

I am still a little confused as to why it's not shipped as part of the Microsoft SDK ...

@omajid are you still planning to work on this?

If we (still) need it in source-build, sure, I can look into this.

@dagood
Copy link
Member

dagood commented Nov 17, 2020

When/where do you expect Humanizer.dll to show up in the .NET SDK?

Well, I'm surprised Workspaces isn't showing up in the SDK. And I'm therefore surprised that the dependencies of Workspaces aren't showing up in the SDK.

@333fred have you looked into this?

I am still a little confused as to why it's not shipped as part of the Microsoft SDK ...

We could throw it in SBRP if it's only compiled against and never executed, I guess. Since it's not shipping in the SDK that seems possible.

We weren't able to patch out usages in the 5.0.0 GA PR (#1767 (comment)) so it doesn't seem worthwhile to try that approach again.

omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 17, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 17, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 18, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 18, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 19, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 19, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 19, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
omajid added a commit to omajid/dotnet-source-build that referenced this issue Nov 19, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: dotnet#1767
dagood pushed a commit that referenced this issue Nov 19, 2020
This is https://github.com/Humanizr/Humanizer (different from
https://github.com/Humanizer/Humanizer !)

Humanizer shows up as a dependency of ASP.NET Core and roslyn-analyzers.
This tries to fix that. Supposedly it also ships with the SDK.

Fixes: #1767
@omajid
Copy link
Member

omajid commented Nov 20, 2020

Fixed with #1895

@omajid omajid closed this as completed Nov 20, 2020
MichaelSimons pushed a commit to dotnet/source-build-externals that referenced this issue Feb 8, 2022
* Update to 5.0.100-rtm.20521.5, BAR Build ID 68494

* Update previously source-built to N-1

* Reconcile patches

Note: Removed aspnetcore version number patch because it is set to
a stable 5.0 version.
Note: Removed roslyn Humanizer removal patch because there is now
more usage of Humanizer in the code base.  May need to redo this
patch or add Humanizer.  (See dotnet/source-build#1767)

* Update to 5.0.100-rtm.20522.4, BAR Build ID 68583

* Update commits in Version.Details.xml and reconcile patches

* Remove temporary RC1 fix in aspnetcore

* Install built runtime for usage by aspnetcore

* Update roslyn to build CodeStyle packages

* Update / add installer patches

* Fix broken patch

* Update prevSourceBuilt to GA version

* Pin System.Security.Cryptography.Cng version in runtime build

* Update path to sdk installation

* Update smoke test nuget.config entries

* Update online prebuilt baseline

* Update package name for aspnetcore reference

* Add runtime feed to smoke-testNuGet.Config

* Update runtime config entry

* Update to 20201026.5, BAR Build ID 68880

* Set UseStableVersions flag

* Update online baseline

* Update offline prebuilt baseline

* Include patch from dotnet/source-build#1850

Include patch to remove import of restore/harvestPackages.targets
in runtime libraries build to fix issue with portable build

* Re-add portable build only prebuilts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-additional-repos Adding additional contributing repos area-patch-removal Removing patches for contributing repos from source-build area-product-experience Improvements in the end-user's product experience
Projects
None yet
Development

No branches or pull requests

7 participants