SPE-15 migrate dex_aggregator.trades to dunesql engine
#4320
Conversation
jeff-dude
added
WIP
jeff-dude
changed the title
dex_aggregator.trades to dunesql enginedex_aggregator.trades to dunesql engine
|
#4285 (zeroex op spells) is ready for review. |
|
#4273 (zeroex polygon spells) is ready for review. |
|
#4410 (zeroex arbitrum, avaxc, fantom spells) is ready for review. |
|
Added some missing |
aalan3
force-pushed
the
migrate-dex-agg
branch
from
e5ed223 to
2e1458c
Compare
|
I removed the partioning, the table was overpartitioned with block_month now it's slightly under partitioned but I think it's fine. |
|
@jeff-dude I removed the deduplication, let me know if we should add it back. |
| block_time, | ||
| block_number, | ||
| tx_hash, | ||
| evt_index, | ||
| CAST(NULL as array<bigint>) as trace_address, | ||
| ARRAY[-1] as trace_address, |
There was a problem hiding this comment.
We should use a null array here
There was a problem hiding this comment.
why is that? we are using this across all models it seems, not sure why though. One thing to keep in mind is that this is used in the unique key and nulls can cause issues there sometimes.
There was a problem hiding this comment.
I guess it was null before though so shouldn't really be a problem. @jeff-dude @couralex6 do you have any context on why we are using ARRAY[-1] vs null array?
There was a problem hiding this comment.
I will merge this and do a sweeping PR (since they are all over the place already) of ARRAY[-1] replacements if we decide that it's needed.
There was a problem hiding this comment.
yeah, the column is a unique key, so null doesn't find a match on merge joins, therefore inserting rows again. null arrays must have worked differently on spark then. i vaguely recall a small difference during migration (may have been different table/column). you can give it a test on one of the base-level project models that use the hardcoded -1 to see what happens
todo: