Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compact: Update filtered blocks list before second downsample pass #7492

Merged
merged 4 commits into from
Jul 13, 2024

Commits on Jun 27, 2024

  1. compact: Update filtered blocks list before second downsample pass

    If the second downsampling pass is given the same filteredMetas
    list as the first pass, it will create duplicates of blocks
    created in the first pass.
    
    It will also not be able to do further downsampling e.g 5m->1h
    using blocks created in the first pass, as it will not be aware
    of them.
    
    The metadata was already being synced before the second pass,
    but not updated into the filteredMetas list.
    
    Signed-off-by: Thomas Hartland <thomas.hartland@diamond.ac.uk>
    tghartland committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    3c37062 View commit details
    Browse the repository at this point in the history
  2. Update changelog

    Signed-off-by: Thomas Hartland <thomas.hartland@diamond.ac.uk>
    tghartland committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    de859e1 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. e2e/compact: Fix number of blocks cleaned assertion

    The value was increased in 2ed48f7 to fix the test,
    with the reasoning that the hardcoded value must
    have been taken from a run of the CI that didn't
    reach the max value due to CI worker lag.
    
    More likely the real reason is that commit 68bef3f
    the day before had caused blocks to be duplicated
    during downsampling.
    
    The duplicate block is immediately marked for deletion,
    causing an extra +1 in the number of blocks cleaned.
    
    Subtracting one from the value again now that the
    block duplication issue is fixed.
    
    Signed-off-by: Thomas Hartland <thomas.hartland@diamond.ac.uk>
    tghartland committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    4fae4fc View commit details
    Browse the repository at this point in the history
  2. e2e/compact: Revert change to downsample count assertion

    Combined with the previous commit this effectively reverts
    all of 2ed48f7, in which two assertions were changed to
    (unknowingly) account for a bug which had just been
    introduced in the downsampling code, causing duplicate blocks.
    
    This assertion change I am less sure on the reasoning for,
    but after running through the e2e tests several times locally,
    it is consistent that the only downsampling happens in the
    "compact-working" step, and so all other steps would report 0
    for their total downsamples metric.
    
    Signed-off-by: Thomas Hartland <thomas.hartland@diamond.ac.uk>
    tghartland committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    bdcf63b View commit details
    Browse the repository at this point in the history