Not all pushed packages are available

[sico85]

Hi Loïc,

I'm using BaGet for a month and I've pushed and used many packages without encounter any problem, but today I've pushed the 9th package and it is not listed in Visual Studio or in the server web. I've removed a package, and the last package I pushed appeared just for a while, after it has disapeared again.

In the server side I can see this warning:

warn: Microsoft.EntityFrameworkCore.Query[20500]
         The LINQ expression 'GroupBy([p].Id, [p])' could not be translated and will be evaluated locally.

System configuration
LXC over Ubuntu 18.04
BaGet commit ed0e62d
Hosting environment: Development (how to change it?)

Thank you for your help and the good job.

[loic-sharma]

That's weird. Does this only happen to a single package? If so, would you mind sharing this package? Also, are you using SQLite? If so, would you mind sharing the database file? How're you pushing packages?

Feel free to send this information to my email address sharma.loic@gmail.com for privacy.

In the server side I can see this warning:

warn: Microsoft.EntityFrameworkCore.Query[20500]
         The LINQ expression 'GroupBy([p].Id, [p])' could not be translated and will be evaluated locally.

This appears to be unrelated to your issue.

Hosting environment: Development (how to change it?)

Are you running BaGet on Docker? If so, BaGet only supports "Development" mode at the moment (see this issue). Otherwise, see ASP.NET's documentation.

[tomzo]

I think I managed to reproduce this in e2e tests. I will publish it soon.
Running following commands fast enough:

# push private package baget-two v1.0.0
dotnet nuget push baget-two.1.0.0.nupkg --source http://baget:9090/v3/index.json --api-key NUGET-SERVER-API-KEY
# nuget install latest package version (1.0.0)
nuget install baget-two -DisableParallelProcessing -NoCache -Source http://baget:9090/v3/index.json
# push private package baget-two v2.1.0
dotnet nuget push baget-two.2.1.0.nupkg --source http://baget:9090/v3/index.json --api-key NUGET-SERVER-API-KEY
# nuget install now should install latest package version (2.1.0)
nuget install baget-two -DisableParallelProcessing -NoCache -Source http://baget:9090/v3/index.json

But actually last command returns

     Feeds used:
       http://baget:9090/v3/index.json
     
     Installing package 'baget-two' to '/ide/work/e2e/test_update-package/nuget'.
       CACHE http://baget:9090/v3/registration/baget-two/index.json
     Package "baget-two.1.0.0" is already installed.

And then after a few seconds it would pass, installing v2.1.0. So it must be a race condition or caching issue.

Once I am done with docker and pipeline setup, I'll have a look to fix this.

Edit

Above is actually failing because client is caching registration endpoint.
Last command can be replaced by following and then test passes:

nuget locals http-cache -clear
nuget install baget-two -DisableParallelProcessing -NoCache -DirectDownload -Source http://baget:9090/v3/index.json

[solidsloth]

Don't we want the skip and take to occur after the grouping? I think what's currently happening is the total list of ungrouped packages are being limited/paged, and then the limited list is being grouped. For example, if you had a single package with 20+ versions, the default query would skip 0 and take 20, which would only be grabbing the first 20 versions of the package, and the group that single package.

I think

var packages = await search
    .Where(p => p.Listed)
    .OrderByDescending(p => p.Downloads)
    .Skip(skip)
    .Take(take)
    .GroupBy(p => p.Id) // Move this
    .ToListAsync();

should be:

var packages = await search
    .Where(p => p.Listed)
    .OrderByDescending(p => p.Downloads)
    .GroupBy(p => p.Id) // To here
    .Skip(skip)
    .Take(take)
    .ToListAsync();

Now we're getting packages grouped by package id, and then paging on those results.

[tomzo]

The issue I found is with just 2 versions of same package and it is not using search query, so I guess paging would not be the problem here.
Still, what you are proposing makes sense.

[solidsloth]

I was having this issue myself and this resolved the issue for me.

The problem is that now the local evaluation problem is multiplied, because this causes the Take and Skip to be evaluated locally as well, and now that all of the packages in the database are being retrieved in order to be grouped properly this would have substantial performance issues when the database grows large. For the life of me I can't figure out what is causing the local evaluation.

[sico85]

Sorry for the late response.
I cannot share the database or the package because they belong to the private developments of my company.

Anyway I've seen that you guys have a better idea on the problem.

For me, the bug affects to 2 of my packages that have many versions each. At first, I thinked that the problem was on the name on one of the packages that includes a number, but I have also another package not being displayed that doesn't contain a number.

Thank you guys for all your efforts.

[loic-sharma]

This should be fixed by ef59709#diff-86483935118f023a4828d37181ae9416. Feel free to open a new issue if anything else comes up!