Bugfix 1683: Add validate_index arg to staged data finalization in both V1 and V2 APIs #1694
Conversation
| prune_previous_versions: bool, default=False | ||
| Removes previous (non-snapshotted) versions from the database. | ||
| metadata : Any, default=None | ||
| Optional metadata to persist along with the symbol. | ||
| validate_index: bool, default=True |
There was a problem hiding this comment.
It would be good to describe why anyone would want to set this to False. Is it just for a performance boost?
There was a problem hiding this comment.
No, it is for if you want to allow unordered data. It is following the same pattern as the docstrings for write etc, but the phrasing has to be a bit more complex because you're creating the new version from incompletes.
| lib.write(sym, df_1, staged=True) | ||
| if validate_index is None: | ||
| # Test default behaviour when arg isn't provided | ||
| with pytest.raises(SortingException): |
There was a problem hiding this comment.
Also need tests for cases where the validation should pass
There was a problem hiding this comment.
Agree, I'll add it to test_parralel.py though since the v2 API just passes through anyway
| // This is correct because incomplete segments aren't column sliced | ||
| sorting::check<ErrorCode::E_UNSORTED_DATA>( | ||
| inserted, | ||
| "Cannot finalize staged data as incomplete segments overlap one another"); |
There was a problem hiding this comment.
Should be possible to make the error message describe the overlap?
There was a problem hiding this comment.
Added, and also made all index timestamps human readable
Rebased |
…th V1 and V2 APIs
…nd improve error messages for overlapping segments
alexowens90
force-pushed
the
bugfix/1637/support-validate-index-arg-with-staged-data
branch
from
4b2cdc3
to
f1b35ed
Compare
alexowens90
deleted the
bugfix/1637/support-validate-index-arg-with-staged-data
branch
Reference Issues/PRs
Fixes #1683
What does this implement or fix?
V2 API
finalize_staged_datacontinues to validate that index values are sorted and non-overlapping by default, but now checking can be disabled by passingvalidate_index=Falsein line with other writing methods.V1 API
compact_incompletenow allows unsorted timeseries index data by default, and checking can be enabled by passingvalidate_index=Truein line with other writing methods.Also means that when writing incomplete segments, the
validate_indexargument is used to check sortedness or not, previously it was assumed that incompletes had to beASCENDING.