Skip to content

Commit

Permalink
fix for list_objects
Browse files Browse the repository at this point in the history
* for postgres queries, there is no need to make an extra DB query,
  after processing the results of the first query.
* in Mongo map-reduce, we could get to a situation where the query
  returned less than the limit, even though there are still relevant
entries.
* in Postgres, this is not happening, since the map_common_prefixes,
  iterates until it finds enough records (or until it reaches the end).
* Performing the extra DB query in postgres, can cause a very long query
  that eventually returns 0 entries. One scenario is where we have a
directory that is the last one returned in the previous query, and it
has a large number of objects under it (.e.g /folder/file0 ..
/folder/file9999999). when returning for the extra query, the marker
that is used is 'folder/', so map_common_prefixes starts iterating from
'folder/', looking for all objects that do not start with 'folder/', but
ends with a '/' (the delimiters). for postgres there is no smart way to
do it, so it just going over all objects under 'folder/' and filetering
them out. this can take over several minutes to complete.

Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
  • Loading branch information
dannyzaken committed May 30, 2024
1 parent 426e7c7 commit 219699d
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion src/server/object_services/object_server.js
Original file line number Diff line number Diff line change
Expand Up @@ -1211,7 +1211,9 @@ function _list_add_results(state, results) {
// this case avoids another last query when we got less results and no common prefixes
// with common prefixes we cannot avoid the last query because the results might be
// less than the requested limit although there are more results to fetch
if (!has_common_prefixes && count >= state.user_limit) {
//
// for postgres we should not do another query, since the list command returns the required limit
if (config.DB_TYPE === 'postgres' || (!has_common_prefixes && count >= state.user_limit)) {
state.done = true;
}
}
Expand Down

0 comments on commit 219699d

Please sign in to comment.