fix(python): Create delta compatible schema during writing#10165

Merged

stinodego merged 27 commits intopola-rs:mainfrom ion-elgreco:feature/fix_schema_for_delta_write

Aug 10, 2023

+267-53

Commits on Aug 10, 2023

Added method to reconstruct schema with delta compatible timestamps

ion-elgreco
authored and
stinodego
committed
Added uint to int logic

ion-elgreco
authored and
stinodego
committed
Disable TZ for now, will have to check with delta-rs team why TZ gets casted into no TZ

ion-elgreco
authored and
stinodego
committed
Fix most formatting

ion-elgreco
authored and
stinodego
committed
Use inbuilt pyarrow method to check dtype and use mapping of ints

ion-elgreco
authored and
stinodego
committed
Fix bug some dtypes didn't return the reconstructed field, but only the last field

ion-elgreco
authored and
stinodego
committed
Formatting

ion-elgreco
authored and
stinodego
committed
Added one test case with multiple uints and datetimes in unnested and nested format

ion-elgreco
authored and
stinodego
committed
Fix to pass all tests

ion-elgreco
authored and
stinodego
committed
formatting

ion-elgreco
authored and
stinodego
committed
Use lazy import

ion-elgreco
and
stinodego
committed
Formatting fix

ion-elgreco
authored and
stinodego
committed
add doc string, with explanation of the casting, and link to the delta protocol

ion-elgreco
authored and
stinodego
committed
Include largetypes to types schema fixing and cast data to delta compatible schema before write, removes the workaround

ion-elgreco
authored and
stinodego
committed
Formatting

ion-elgreco
authored and
stinodego
committed
Added parametrized tests

ion-elgreco
authored and
stinodego
committed
Cast to UTC to prevent delta-rs write issue

ion-elgreco
authored and
stinodego
committed
Remove TZ info from timestamp types

ion-elgreco
authored and
stinodego
committed
Reduce specific ifelse for timestamp

ion-elgreco
authored and
stinodego
committed
Improve order of ifelse statements for readability

ion-elgreco
authored and
stinodego
committed
Move list into function directly

ion-elgreco
authored and
stinodego
committed
Add helper function to created nested data type

ion-elgreco
authored and
stinodego
committed
Added if statement to check if custom schema has been passed

ion-elgreco
authored and
stinodego
committed
Formatting

ion-elgreco
authored and
stinodego
committed
Refactor
stinodego
committed
Refactor 2
stinodego
committed
Formatting
stinodego
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(python): Create delta compatible schema during writing#10165

Commits on Aug 10, 2023

Added method to reconstruct schema with delta compatible timestamps

Added uint to int logic

Disable TZ for now, will have to check with delta-rs team why TZ gets casted into no TZ

Fix most formatting

Use inbuilt pyarrow method to check dtype and use mapping of ints

Fix bug some dtypes didn't return the reconstructed field, but only the last field

Formatting

Added one test case with multiple uints and datetimes in unnested and nested format

Fix to pass all tests

formatting

Use lazy import

Formatting fix

add doc string, with explanation of the casting, and link to the delta protocol

Include largetypes to types schema fixing and cast data to delta compatible schema before write, removes the workaround

Formatting

Added parametrized tests

Cast to UTC to prevent delta-rs write issue

Remove TZ info from timestamp types

Reduce specific ifelse for timestamp

Improve order of ifelse statements for readability

Move list into function directly

Add helper function to created nested data type

Added if statement to check if custom schema has been passed

Formatting

Refactor

Refactor 2

Formatting