Starknet updates and fixes #57
Conversation
Wizard1209
commented
Apr 1, 2024
•
Wizard1209
commented
- add raw ingester(to jsonl.gz)
- add parquet writer from raw
- fix data loss in case 0 chunks were filled
- backport to starknet feature for worker to start indexing from last block that have data
- refactor _to_sync_gen usage and typing
| @@ -166,8 +166,8 @@ def flush(): | |||
| self._report() | |||
| last_report = current_time | |||
|
|
|||
| if self._writer.buffered_bytes() > 0 and last_block == write_range[1]: | |||
| flush() | |||
| # NOTE: In case no chunks were made we still need to save data we received | |||
There was a problem hiding this comment.
What is the reason behind abandoning the check above?
Although I don't remember why it exists, I know, that we don't want to write half sized chunk unless it is the last one in the specified range.
There was a problem hiding this comment.
Since bytesize counts as sum of chunks for all tables, if no chuncks were created this check leads to data loss for smaller data volumes
|
@eldargab |
