Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport sort and finalize fixes #1859

Merged
merged 7 commits into from
Oct 1, 2024
Merged

Commits on Sep 30, 2024

  1. Fixes for sort and finalize (#1763)

    Fixes: #1737 #1736 #1735
    
    <details>
      <summary>
       Checklist for code changes...
      </summary>
    
    - [ ] Have you updated the relevant docstrings, documentation and
    copyright notice?
    - [ ] Is this contribution tested against [all ArcticDB's
    features](../docs/mkdocs/docs/technical/contributing.md)?
    - [ ] Do all exceptions introduced raise appropriate [error
    messages](https://docs.arcticdb.io/error_messages/)?
     - [ ] Are API changes highlighted in the PR description?
    - [ ] Is the PR labelled as enhancement or bug so it appears in
    autogenerated release notes?
    </details>
    
    <!--
    Thanks for contributing a Pull Request to ArcticDB! Please ensure you
    have taken a look at:
    - ArcticDB's Code of Conduct:
    https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
    - ArcticDB's Contribution Licensing:
    https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
    -->
    
    ---------
    
    Co-authored-by: Vasil Pashov <vasil.pashov@man.com>
    vasil-pashov and Vasil Pashov committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    946cbf9 View commit details
    Browse the repository at this point in the history
  2. Fix symbol list bug on compaction (#1324)

    - Only adds a symbol list entry on append if there is no previous
      version.
    - Fixes `finalize_staged_data` to write to the symbol list.
    - Adds a python test for the new behavior.
    IvoDD authored and Vasil Pashov committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    c430234 View commit details
    Browse the repository at this point in the history
  3. More sort and finalize fixes (#1799)

    <!--Example: Fixes #1234. See also #3456.-->
    Fixes #1738
    Fixes #1781
    Fixes #1466
    Fixes #1795
    Fixes #1797
    Fixes #1807
    Fixes #1828
    
    A notable change is that staged writes no longer validate the index is
    sorted. The validation is done at the moment
    compact_incompletes/finalize_staged_data/sort_and_finalize_staged_data
    is called. This is because sort_and_finalize_staged_data does not
    require the segments to be sorted, but the call for adding a staged
    segment is the same. We should add a separate call for that.
    
    Note also that all incomplete keys for a symbol are deleted if any of
    the finalize calls fail. The other option is to leave the segments. In
    that case the user will have the responsibility of calling
    `delete_staged_data`.
    
    <details>
      <summary>
       Checklist for code changes...
      </summary>
    
    - [ ] Have you updated the relevant docstrings, documentation and
    copyright notice?
    - [ ] Is this contribution tested against [all ArcticDB's
    features](../docs/mkdocs/docs/technical/contributing.md)?
    - [ ] Do all exceptions introduced raise appropriate [error
    messages](https://docs.arcticdb.io/error_messages/)?
     - [ ] Are API changes highlighted in the PR description?
    - [ ] Is the PR labelled as enhancement or bug so it appears in
    autogenerated release notes?
    </details>
    
    <!--
    Thanks for contributing a Pull Request to ArcticDB! Please ensure you
    have taken a look at:
    - ArcticDB's Code of Conduct:
    https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
    - ArcticDB's Contribution Licensing:
    https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
    -->
    
    ---------
    
    Co-authored-by: Vasil Pashov <vasil.pashov@man.com>
    vasil-pashov and Vasil Pashov committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0fe71d6 View commit details
    Browse the repository at this point in the history
  4. Fix row slicing with sort_and_finalize throw on column slicing when t…

    …he column group is larger than the segment column size (#1838)
    
    #### Reference Issues/PRs
    <!--Example: Fixes #1234. See also #3456.-->
    
    #### What does this implement or fix?
    Closes #1851 
    Closes #1848 
    #### Any other comments?
    
    #### Checklist
    
    <details>
      <summary>
       Checklist for code changes...
      </summary>
     
    - [ ] Have you updated the relevant docstrings, documentation and
    copyright notice?
    - [ ] Is this contribution tested against [all ArcticDB's
    features](../docs/mkdocs/docs/technical/contributing.md)?
    - [ ] Do all exceptions introduced raise appropriate [error
    messages](https://docs.arcticdb.io/error_messages/)?
     - [ ] Are API changes highlighted in the PR description?
    - [ ] Is the PR labelled as enhancement or bug so it appears in
    autogenerated release notes?
    </details>
    
    <!--
    Thanks for contributing a Pull Request to ArcticDB! Please ensure you
    have taken a look at:
    - ArcticDB's Code of Conduct:
    https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
    - ArcticDB's Contribution Licensing:
    https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
    -->
    
    ---------
    
    Co-authored-by: Vasil Pashov <vasil.pashov@man.com>
    vasil-pashov and Vasil Pashov committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    624b432 View commit details
    Browse the repository at this point in the history
  5. Remove read_dataframe_merged. It was removed with #1698.

    Vasil Pashov committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    8eadb29 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Add drop staged data argument

    Add drop_staged_data_on_failure flag for sort and finalize
    
    Add unit tests to make sure it behaves as expected
    
    Add drop_staged_data_on_failure flag for compact incompletes
    
    Add tests for drop_staged_data_on_failure for compact_incompletes
    
    Add documentation for finalize_staged_data and sort_and_finalize_staged_data
    
    Stylistic change on checking whether to delete incomplete keys
    
    Don't throw in destructor
    
    Remove unused headers
    
    Rename drop_staged_data_on_failure to delete_staged_data_on_failure
    
    Address review comments
    
    Fix conda build
    
    Removes spaces from `environment_unix.yml` which new versions of mamba
    are not processing correctly.
    
    Fix row slicing with sort_and_finalize throw on column slicing when the column group is larger than the segment column size (#1838)
    
    <!--Example: Fixes #1234. See also #3456.-->
    
    Closes #1851
    Closes #1848
    
    <details>
      <summary>
       Checklist for code changes...
      </summary>
    
    - [ ] Have you updated the relevant docstrings, documentation and
    copyright notice?
    - [ ] Is this contribution tested against [all ArcticDB's
    features](../docs/mkdocs/docs/technical/contributing.md)?
    - [ ] Do all exceptions introduced raise appropriate [error
    messages](https://docs.arcticdb.io/error_messages/)?
     - [ ] Are API changes highlighted in the PR description?
    - [ ] Is the PR labelled as enhancement or bug so it appears in
    autogenerated release notes?
    </details>
    
    <!--
    Thanks for contributing a Pull Request to ArcticDB! Please ensure you
    have taken a look at:
    - ArcticDB's Code of Conduct:
    https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
    - ArcticDB's Contribution Licensing:
    https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
    -->
    
    ---------
    
    Co-authored-by: Vasil Pashov <vasil.pashov@man.com>
    
    Fix incomplete segments raii when exception is thrown before read incompletes to pipeline
    
    Fix docstrings
    Vasil Pashov committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    79dd396 View commit details
    Browse the repository at this point in the history
  2. Fix compilation errors

    Vasil Pashov committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9adfa72 View commit details
    Browse the repository at this point in the history