occ files:scan's memory consumption grows with amount of files scanned

[danimo]

Expected behavior

The memory consumption of the scanner is constant.

Actual behavior

The memory consumption of the scanner grows with the amount of files, requiring to increase the memory limit again and again for users with a large amount of data.

Details

ownCloud: 8.1.3
Total amount of file in user's share: 45308
Memory required: > 256 MB, < 512 MB

[karlitschek]

@icewind1991

[oparoz]

It's by design #10922.
All the information is collected first, before being sent to the database, in order to speed up the process.
So the solution seems to be to give large amounts of memory (in the GB range) to the cli.

Note: Scanned folders are unavailable while the command runs

[oparoz]

I see one possible enhancement: Collect structure data first and launch non-recursive scans from the bottom of the tree. Each folder update will be committed separately.

[PVince81]

This could be due to stat caches.

When scanning a local storage, PHP will store stuff in its stat cache. Not sure if that one is capped.

Regarding SMB, we added a cap to the stat cache recently: #21648

If this is still happening in 8.2.2 or 9.0 RC1, please reopen.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.