Some spiders create millions of files in a directory

[jpmckinney]

This creates performance issues on some filesystems. The approach taken by caching libraries is to generate and fill in arbitrary directory trees; if we take the same approach, there is existing code/libraries to reuse. Otherwise, we can consider other ways to segment files, which might be specific to those spiders that have a lot of files.

[jpmckinney]

cc @jakubkrafka if you have some suggestion.

[yolile]

Note that this won't be an issue for some spiders after #944 anymore.

[jpmckinney]

For clarity, this issue is still relevant to some Digiwhist sources (line-delimited JSON).

[jpmckinney]

ProZorro actually hit the limit of files per directory (perhaps max_dir_size_kb), causing the (misleading) error message: OSError: [Errno 28] No space left on device

[yolile]

From a conversation with @jpmckinney:

Creating arbitrary (but predictable) directories is the best approach.

Ruby on Rails’ FileStore does:

hash = Zlib.adler32(fname)
hash, dir_1 = hash.divmod(0x1000)
dir_2 = hash.modulo(0x1000)

And then formats the numbers dir_1 and dir_2 as "%03X"

File.join(cache_path, DIR_FORMATTER % dir_1, DIR_FORMATTER % dir_2, fname)

We could follow the same approach and also use adler32, as other existing hashing libraries for Python (such as hashlib) only return byte or hex strings and not integers.

We should implement this as part of the FilesStore extension and for all the existing spiders, to be consistent with the output.

[jpmckinney]

DatabaseStore's yield_items_from_directory probably needs to use os.walk instead of os.scandir.