Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

Merged
merged 16 commits into from
Mar 1, 2021
Merged

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

merged 16 commits into from
Mar 1, 2021

Commits on Nov 22, 2020

  1. block: precalculate hashes if enabled and use them during compaction 

    Added the possibility to ignore certain directories in
objstore.{Download,DownloadDir}. Do not download files which have the
same hash as in remote object storage. Wire up `--hash-func` so that
writers could specify what hash function to use when uploading. There is
no performance impact if no hash function has been explicitly specified.
Clean up the removal of files logic in Thanos Compact to ensure we do
not remove something that exists on disk already.

Tested manually + new tests cover all of this more or less.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Nov 22, 2020
    Configuration menu
    Copy the full SHA
    8bae707 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2020

  1. Merge remote-tracking branch 'origin/master' into add_hash_method 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    4ca451f View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2020

  1. block: expose GatherFileStats and use it 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    259c70b View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into add_hash_method 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    a938561 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2020

  1. Revert "block: expose GatherFileStats and use it" 

    This reverts commit 259c70b.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    e8023b8 View commit details
    Browse the repository at this point in the history

  2. block: do not calc hash for dirs, add locks 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    6e8ebc8 View commit details
    Browse the repository at this point in the history

  3. docs/tools: update 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    68df43e View commit details
    Browse the repository at this point in the history

  4. shipper: pass s.hashFunc 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    455ea4c View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2021

  1. Merge remote-tracking branch 'origin/master' into add_hash_method 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    0bd177a View commit details
    Browse the repository at this point in the history

  2. Fix according to Bartek's comments 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    43dd78b View commit details
    Browse the repository at this point in the history

  3. compact: clean up comment 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    e8fac48 View commit details
    Browse the repository at this point in the history

  4. block: close with log on error 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    fab4562 View commit details
    Browse the repository at this point in the history

  5. *: remove unused FNs 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    883550f View commit details
    Browse the repository at this point in the history

  6. compact: add e2e test for new hash functionality 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    cad16e4 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2021

  1. Fix according to Bartek's comments 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Feb 26, 2021
    Configuration menu
    Copy the full SHA
    fd00457 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2021

  1. Merge remote-tracking branch 'origin/main' into add_hash_method 

    Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
    @GiedriusS
    GiedriusS committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    2175598 View commit details
    Browse the repository at this point in the history