[BUG] Detected loop in FindAll({0})

[kamilwarowny]

Version
.Net 8.0
LiteDB.dll 5.0.21.0

Describe the bug
About 16000 records in file. File size 33MB

Code to Reproduce

using (var db = new LiteDatabase(dbFilePAth))
{                   
    var col = db.GetCollection<UserHash>(CollectionName);

    var userHashes = col.Query().ToList();

    db.Checkpoint();

    db.Rebuild();

    return userHashes;
}

line db.Rebuild(); throws error below:

at LiteDB.Engine.IndexService.FindAll(CollectionIndex index, Int32 order)+MoveNext()
at LiteDB.LinqExtensions.<>c__DisplayClass2_02.<<DistinctBy>g___|0>d.MoveNext() at LiteDB.Engine.LiteEngine.<>c__DisplayClass5_0.<EnsureIndex>b__0(TransactionService transaction) at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn)
at LiteDB.Engine.LiteEngine.EnsureIndex(String collection, String name, BsonExpression expression, Boolean unique)
at LiteDB.Engine.LiteEngine.RebuildContent(IFileReader reader)
at LiteDB.Engine.RebuildService.Rebuild(RebuildOptions options)
at LiteDB.Engine.LiteEngine.Rebuild(RebuildOptions options)
at LiteDB.LiteDatabase.Rebuild(RebuildOptions options)

[holychaos]

I had this problem too. I tried to move the 'db' file out of the original path, run the program once to generate a new 'db' file, and finally overwrite the moved 'db' file back, and everything worked.

[marss72]

Just BTW, not regarding your issue, but it might be possible to improve the provided code by maybe better performance and clarity.

If it's really necessary to use List, ignore this one. But you can improve memory efficiency and lazily load all the entries. I am not an expert on LiteDB so this might cause issues since you are rebuilding it right after.

Also, you can modify col.Query().ToList(); to use FindAll() instead of the Query(). Try it yourself, you might get a small performance benefit or not, but I haven't measured it.

[k0ka]

I have the same problem. It happens if any table on rebuild has more than 2550 records. Here is why:

1. RedbuildService creates a new database https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Services/RebuildService.cs#L55
2. It starts rebuilding it using the main IFileReader https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Engine/Rebuild.cs#L52
3. When it creates IndexService it passes _disk.MAX_ITEM_COUNT from the newly created database. Which means it is always minimum value of 2550. https://github.com/mbdavid/LiteDB/blob/c4db4ab3b1e594439921d594f403775c82f9e8c7/LiteDB/Engine/Engine/Rebuild.cs#L63

I'm not sure how to properly fix it. The easiest way would be to just pass the correct MAX_ITEM_COUNT from the RebuildService where it still knows this value.

[zalza13]

This property was added in version 5.0.019 and I believe it is causing that calculation to be incorrect.

    /// <summary>
    /// This method calculates the maximum number of items (documents or IndexNodes) that this database can have.
    /// The result is used to prevent infinite loops in case of problems with pointers
    /// Each page support max of 255 items. Use 10 pages offset (avoid empty disk)
    /// </summary>
    public uint MAX_ITEMS_COUNT => (uint)(((_dataLength + _logLength) / PAGE_SIZE) + 10) * byte.MaxValue;

In my case I have a collection with 5000+ documents and my application is collapsing.

Possibles workaround:

1. Go back to the version 5.0.18
2. Comment the lines ENSURE(counter++ < _maxItemsCount, "Detected loop in GetNodeList({0})" (I guess no a good idea)
3. Wait for a fix

@mbdavid I know you are not involved in the project anymore and that's great, but could you at least give us a hint on how to solve it or how we can at least apply a temporary solution? --> 663f749

Thank you so much!! :D