Add support for converting pyarrow DataFrame to numpy #98
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduced a new function `convert_to_numpy` to reverse convert pyarrow DataFrame to numpy dtypes. Added a unit test to ensure the conversions are accurate and consistent with expected numpy dtypes. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Included convert_to_numpy in __all__ for module exports and updated relevant imports in tests. This change ensures convert_to_numpy is accessible and properly tested across the library. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Previously, strings were not explicitly handled in the reverse converter, which could lead to inconsistencies. This commit ensures that columns with string dtype are converted to object type, improving the dtype consistency and compatibility. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Ensure the conversion from pandas to numpy via pyarrow is idempotent by asserting the result is unchanged on a second conversion. Additionally, check that the original dataframe remains unaltered during the process. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #98 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 7 8 +1
Lines 59 77 +18
=========================================
+ Hits 59 77 +18 ☔ View full report in Codecov by Sentry. |
Changed the paths in the Codecov workflow configuration to ensure it only runs when relevant files, including the codecov.yml, are modified. This improves the efficiency of our CI/CD pipeline by reducing unnecessary runs. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Created tests for converting data back from PyArrow to Numpy. These tests ensure the correct data types after conversion for timestamps, half floats, durations, and strings. They validate that PyArrow types do not persist in the resulting Numpy dataframes. Signed-off-by: DanielAvdar <66269169+DanielAvdar@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduced a new function
convert_to_numpyto reverse convert pyarrow DataFrame to numpy dtypes. Added a unit test to ensure the conversions are accurate and consistent with expected numpy dtypes.