Support package renames

[loic-sharma]

Background

Spec: Support package renames

Tasks

Minimum Viable Product

These are the highest priority work items. Once they are complete, we will have an internal MVP that our partner team can use to give feedback:

• NuGetGallery#7900 - Add new schema
• NuGetGallery#7898 - Apply popularity transfers in search
• NuGetEngineering#3087 - Validate MVP with internal team
• NuGetGallery#7944 - Add feature flag to undo popularity transfers

A/B test

Next we can A/B test the popularity transfers on PROD to ensure the new search rankings does not regress customer scenarios:

• NuGetGallery#7903 - Expose popularity transfers in search
• NuGetGallery#7905 - Monitor the popularity transfer freshness
• NuGetGallery#7966 - Monitor V3 jobs' feature flag staleness
• NuGetGallery#7977 - Unblock deployments to existing search instances
• NuGetGallery#7906 - A/B test popularity transfers

General Release

Once we feel confident that the feature addresses the needs of our internal partner, we can work on releasing this feature to all customers.

Add Gallery UI:

• NuGetGallery#7907 - Show replacement packages on "Display Package" page
• [ ] NuGetGallery#7908 - Add package renames form to "Manage Package" page
• [ ] NuGetGallery#7910 - Add message for pending popularity transfers on "Manage Package" page
  • This depends on NuGetGallery#7903 - Expose popularity transfers in search
• [ ] NuGetGallery#7911 - Reduce popularity transfer propagation times
• NuGetGallery#7943 - Add documentation

Testing:

• NuGetGallery#7897 - Add "categories" to NDCG data sets
• [ ] NuGetGallery#7909 - Add Gallery API for end-to-end testing
• [ ] NuGetGallery#7904 - Add end-to-end test
  • This depends on NuGetGallery#7911 - Reduce popularity transfer propagation times

Future Refinements

Once the feature has been released to all customers, we can consider the following future refinements. These are low priority:

• NuGetGallery#7901 - Consider letting new packages match on old packages' ID
• NuGetGallery#7902 - Refine the transfer popularity algorithm
• Engineering#3089 - Remove manual download overrides

Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

• [Package Renames] Add "categories" to NDCG data sets #7897
• [Package Renames] Post-MVP: Consider letting new packages match on old packages' ID  #7901

[Nirmal4G]

Also introduce a concept of Package family by namespaces.

for eg.: All NuGet.* packages by NuGet fall under NuGet family but 3rd party NuGet.* packages can listed under the same home but under 3rd party packages!!

[loic-sharma]

Thanks for the suggestion @Nirmal4G! NuGet.org has package ID prefix reservation that I think may already address this concern. For example, the NuGet. package ID prefix is reserved to the nuget account. As a result, NuGet.* packages owned by nuget account (like NuGet.Packaging) have a blue checkmark. NuGet.* packages not owned by nuget (like NuGet.Build.Packaging) do not have this blue checkmark. I hope that help!

[Nirmal4G]

ID prefix verifies who owns the prefix and not answering the question Are there any alternatives or different functionalities based on NuGet by 3rd parties available?

My solution answers to that. Basically 3rd party packages visibility. There are great solutions available by 3rd parties but they are not searchable much.

[loic-sharma]

I see. Maybe additional search filters on the owner and verified status could help. Thanks for the feedback!

[Nirmal4G]

I tried using NuGet search syntax. I can't exclude NuGet packages by NuGet owner.

Do we have -owner:nuget syntax for exclusion?
Update: Seems this https://nuget.org/packages?q=nuget+-author:nuget+-owner:nuget is not working.

[loic-sharma]

We don't have any exclusion syntax today, no.

I'm curious, what kind of packages would you like to find? Would you like to find packages that "extend" functionality provided by NuGet.* packages? Would a "reverse dependency" feature help? See: https://github.com/NuGet/Home/wiki/Reverse-Dependencies-(Dependent-Packages)

[Nirmal4G]

Not just for NuGet but for any other package that has an extended 3rd party solution. NuGetizer-3000 was hard find. If Miguel didn't tweet that, I wouldn't have found it, just by browsing through github/nuget.

Some packages like MSBuild tools and SDKs don't have dependencies, since they are self contained packages. Those are hard to find. Though the new PackageType filter in the search help but not for regular packages and those which don't specify the packageType.

[clairernovotny]

Does this feature address the diamond dependency issue?

A core problem is that renaming a package will cause conflicts in a graph containing packages that reference the new name and ones that reference the old name. Assuming that the namespaces in assemblies in the package can't be changed, this will cause lots of conflicts that are unsolvable by the end user.

[loic-sharma]

After talking to internal partners we have decided to abandon the "package renames" feature. We believe it would be better to improve the package deprecation feature instead.

We will keep the popularity transfer feature and will allow customers to apply using a process similar to our Package Id prefix reservation process. This work is tracked by #7943.

/cc @chgill-MSFT