add empty chunk handling, importing python crate dependencies, tx filtering by address, support for new datatypes in python crate

[davidthegardens]

Motivation

1. Closes pip installation does not require arrow #137, also pip did not install polars and some other libraries, this was also fixed.
2. Added filtering transactions by --address, which acts as an OR over filtering by --to-address and --from-address.
3. Adding the argument --write-empty to prevent from writing empty dataframes to disk
4. Added support for the new datatypes in the python crate

Solution

1. added the packages and versions into the build file
2. created a closure that copies the conditions of to and from filtering, adding the from-address into env.execution, and creating a global vector to log the tx hashes that have been moved (in order to prevent duplicates that arise from async)
3. added support for the flag, and added a couple lines that checks the dataframes shape. If shape is 0 and --write-empty is false, then it will continue rather than write to disk.
4. allowed the python crate to take in the new datatypes (like transaction), I can't take much of the credit, this was mostly already supported.

PR Checklist

• Added Tests
• Added Documentation
• Breaking changes

[sslivkoff]

hi. finally have some time to go through PR's

some of this PR is ready to merge. other parts need a couple iterations. left some comments (if you would like some parts to be merged faster you can split into multiple PRs)

[sslivkoff]

what is the reason for using TRANSACTIONS_SET here?

[davidthegardens]

In cases where you prompt 2 or more addresses, anytime both the sender and receiver of the transaction are in --addresses, you will get duplicated transactions in the df. Originally, I used TRANSACTIONS_SET to prevent duplicates.

I've since made things more efficient and avoided global variables by making the list of addresses accessible.
This is how it works now:

from_bool = addresses.contains(from_address)
to_bool = addresses.contains(to_address)
if from_bool and to_bool:
    return from_address==current_address
else:
    return from_address==current_address or to_address==current_address

I'm unsure if the way I'm getting the list of addresses is the best way, but I couldn't figure out how else to do it. Currently, it gets the addresses by parsing 'cli_command' from ExecutionEnv.

[sslivkoff]

nice, looking very clean now

additionally I think the request.addresses() will give the relevant info without need for get_addresses(). no need to parse cli arg data, request.addresses() will give you the list of addresses relevant to the current chunk (which might be different from all of the addresses, if youre chunking by address)

[davidthegardens]

Thanks! I can't seem to find .addresses(), in request. There is .address though, which only has one address :/
Since cryo filters transactions by each address separately, I'd need the complete list of addresses in addition to the address for the current chunk to prevent duplicates.

[sslivkoff]

looking nice I think it's almost ready

these various python changes will be very useful for an upcoming python accessibility push I've been working on

[sslivkoff]

nice, looking very clean now

additionally I think the request.addresses() will give the relevant info without need for get_addresses(). no need to parse cli arg data, request.addresses() will give you the list of addresses relevant to the current chunk (which might be different from all of the addresses, if youre chunking by address)

[sslivkoff]

heyo. everything is looking great on the pr. the last thing is get_addresses(). does request.addresses() have everything you need to avoid the get_addresses() function?

[davidthegardens]

heyo. everything is looking great on the pr. the last thing is get_addresses(). does request.addresses() have everything you need to avoid the get_addresses() function?

Hey, thank you! So request.addresses() doesn't seem to exist and request.address doesn't have all the information needed unfortunately.