add back changes for solana token 2022

[jeff-dude]

fyi @andrewhong5297
we tried to merge this PR and let it rebuild in prod + it's downstream dependencies. in the current prod setup, this fails on spill to disk error.

steps taken prior to revert:

• run on current setup, spill to disk failure
• reduce thread count, hoping less models running will help, same error
• work with @belen-pruvost on the infra GH actions to increase cluster size, leveraging memory-intensive machines but same amount of workers, still failed on spill
• increase disk size from 500 --> 1000GB (belen helped here, not part of GH action at this time)
• change dbt cloud job to run solana transfers alone, then let the rest run in normal modified step
  • new step for transfers in simple select
  • exclude spell in modified step
• then we hit dbt cloud errors
  • manually canceled a scheduled run that kicked off while cluster was rebuilding
  • instead of canceling like normal, one step within keeps spinning forever and can't cancel it
  • job itself marked as "succeeded" while this step still runs
  • future manual runs of the job fail on "manifest not available from previous run" which points to run above that is stuck

i have reverted the original PR and modified to dbt cloud job to try and get around these bugs. only incremental step is enabled, the setting for comparing to previous run is turned off to get around error. the cluster has been reverted back to its normal sizing.

assuming incremental runs as normal, we can reconfigure the dbt cloud job back to normal. then we would need to take steps again like above to try and get the solana spells to run in prod without failing.

fyi @aalan3

[jeff-dude]

due to the dbt cloud bugs, we haven't tried running with increased disk size yet. the next time we try this, we should consider:

• memory focused machines
• more workers
• increased disk size

[andrewhong5297]

thanks for pushing on this! I appreciate it 🙏

[aalan3]

I think we need to split these models into intermediary models in order for this to work, otherwise the next time we do a full refresh the bigger cluster might not work. I also suspect that running these with partition by creates memory issues.

For metadata:

• Try to avoid using partition by this is really heavy usually
• Split between the different token types - spl_token and token22

For transfers

• Try to avoid partition by (it's done here when getting fees)
• Split out fees into its own model
• Split spl_token and token22

[andrewhong5297]

I think we need to split these models into intermediary models in order for this to work, otherwise the next time we do a full refresh the bigger cluster might not work. I also suspect that running these with partition by creates memory issues.

For metadata:

• Try to avoid using partition by this is really heavy usually

• Split between the different token types - spl_token and token22

For transfers

• Try to avoid partition by (it's done here when getting fees)

• Split out fees into its own model

• Split spl_token and token22

tokens_solana.fungible runs fairly fast, I believe its transfers thats the problem.

We can do tokens_solana.fungible and the fee updates model in one PR, then the transfers model in another PR.

Does that work?

[aalan3]

We can do tokens_solana.fungible and the fee updates model in one PR, then the transfers model in another PR.

Yes that works

[andrewhong5297]

this PR should only be merged after this one #6113

[aalan3]

@andrewhong5297 could these be inner joins?

[andrewhong5297]

Yes changed

[kasiafi]

We only care about the latest fee for a transfer (per predicate WHERE latest_fee = 1).
Could we join to a more narrow window of fees? Depending on how ofter there are fee updates, it could be for example one day:
... AND tr.call_block_time >= f.fee_time AND tr.call_block_time - interval '1 day' < f.fee_time
That would limit the cost of TopNRanking operator downstream.

[andrewhong5297]

Unfortunately not, because the latest fee might be set even a year or two ago.