Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* for postgres queries, there is no need to make an extra DB query, after processing the results of the first query. * in Mongo map-reduce, we could get to a situation where the query returned less than the limit, even though there are still relevant entries. * in Postgres, this is not happening, since the map_common_prefixes, iterates until it finds enough records (or until it reaches the end). * Performing the extra DB query in postgres, can cause a very long query that eventually returns 0 entries. One scenario is where we have a directory that is the last one returned in the previous query, and it has a large number of objects under it (.e.g /folder/file0 .. /folder/file9999999). when returning for the extra query, the marker that is used is 'folder/', so map_common_prefixes starts iterating from 'folder/', looking for all objects that do not start with 'folder/', but ends with a '/' (the delimiters). for postgres there is no smart way to do it, so it just going over all objects under 'folder/' and filetering them out. this can take over several minutes to complete. Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
- Loading branch information