Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tornado Cash: Fix sources, lowercase and improve materialization strategy #1460

Merged
merged 12 commits into from
Sep 1, 2022
+77 −117
Merged

Tornado Cash: Fix sources, lowercase and improve materialization strategy #1460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
bed9de8
Fix sources, lowercase and improve materialization strat
soispoke Aug 30, 2022
f707c2e
ALTER TABLE, not VIEW
soispoke Aug 30, 2022
6fbecf5
Revert "ALTER TABLE, not VIEW"
soispoke Aug 30, 2022
1e9f8ae
changed avalanche to avalanche_c in blockchain name for consistency a…
hildobby Aug 30, 2022
326dbcf
Lowercase for blockchains and versiosn on schema.yml
soispoke Aug 31, 2022
bccaa5d
avalanche to avalanche_c in schema and macro files
soispoke Aug 31, 2022
c2f497b
Merge branch 'main' into T-15760-fix-tornado-cash-materialization-str…
soispoke Aug 31, 2022
861dba2
transactions_006 to transactions
soispoke Sep 1, 2022
c449a7e
Merge branch 'main' into T-15760-fix-tornado-cash-materialization-str…
soispoke Sep 1, 2022
5d81c41
Update alter_table_properties.sql
soispoke Sep 1, 2022
edc1780
fix transactions_006
soispoke Sep 1, 2022
66f0bf3
Replace LEFT BY INNER JOINS on base transactions tables
soispoke Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions macros/alter_table_properties.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -579,16 +579,16 @@ ALTER TABLE airdrop_optimism.addresses SET TBLPROPERTIES ('dune.public'='true',

{% set tornado_cash_deposits %}
ALTER TABLE tornado_cash.deposits SET TBLPROPERTIES ('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum", "bnb", "avalanche", "gnosis", "optimism", "arbitrum"]',
'dune.data_explorer.blockchains'='["ethereum", "bnb", "avalanche_c", "gnosis", "optimism", "arbitrum"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='tornado_cash',
'dune.data_explorer.contributors'='["hildobby", "dot2dotseurat"]');
{% endset %}

{% set tornado_cash_withdrawals %}
ALTER TABLE tornado_cash.withdrawals SET TBLPROPERTIES ('dune.public'='true',
'dune.data_explorer.blockchains'='["ethereum", "bnb", "avalanche", "gnosis", "optimism", "arbitrum"]',
ALTER VIEW tornado_cash.withdrawals SET TBLPROPERTIES ('dune.public'='true',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soispoke looks like this one didn't get updated back to 'table'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was fixed here https://github.com/duneanalytics/spellbook/pull/1475/files#diff-89d8392c4db5b996c368bef69b027f7ccd442b9135924630007065e3a1d5569cR590

'dune.data_explorer.blockchains'='["ethereum", "bnb", "avalanche_c", "gnosis", "optimism", "arbitrum"]',
'dune.data_explorer.category'='abstraction',
'dune.data_explorer.abstraction.type'='project',
'dune.data_explorer.abstraction.name'='tornado_cash',
Expand Down
3 changes: 0 additions & 3 deletions models/ens/ens_ethereum_sources.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,6 @@ sources:
error_after: { count: 24, period: hour }
loaded_at_field: evt_block_time
- name: ENSRegistry_evt_NewOwner
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
loaded_at_field: evt_block_time
- name: ENSRegistryWithFallback_evt_NewOwner
freshness:
Expand Down
3 changes: 0 additions & 3 deletions models/sudoswap/ethereum/sudoswap_ethereum_sources.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,6 @@ sources:
error_after: { count: 24, period: hour }
loaded_at_field: call_block_time
- name: LSSVMPair_general_call_swapTokenForAnyNFTs
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
loaded_at_field: call_block_time
- name: LSSVMPair_general_call_swapTokenForSpecificNFTs
freshness:
Expand Down
70 changes: 35 additions & 35 deletions models/tornado_cash/tornado_cash_deposits.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2' AS currency_contract
, 'ETH' AS currency_symbol
, 'Ethereum' AS blockchain
, 'Classic' AS tornado_version
, 'ethereum' AS blockchain
, 'classic' AS tornado_version
, et.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x12d66f87a04a9e220743712ce6d9bb1b5616b8fc' THEN 0.1
Expand All @@ -29,19 +29,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_ethereum','eth_evt_Deposit') }} tc
LEFT JOIN {{ source('ethereum','transactions_0006') }} et
INNER JOIN {{ source('ethereum','transactions') }} et
ON et.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND et.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_ethereum','eth_evt_Deposit') }})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't comment on the actual line but line 32 source('ethereum','transactions_0006') should be source('ethereum','transactions').

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe that source was used specifically for performance reasons, but agreed to revert back if we're comfortable with new performance enhancement changes (could then also remove from source file)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That issue has been resolved now (since last week). We shouldn't refer to these ever though because they can (and will) change without warning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soispoke can you update this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeff-dude Done

{% endif %}
{% if is_incremental() %}
AND et.block_time >= (select max(block_time) from {{ this }})
AND et.block_time >= date_trunc("day", now() - interval '1 week')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the LEFT JOIN above be an INNER JOIN instead? And then you could drop the where filter below.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this common check to review doc, but many existing queries from v1 will have left joins to look out for

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are cases where you'd want to do a left join as I discovered here. But generally it seems like it should be inner.

{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_ethereum','eth_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand Down Expand Up @@ -94,8 +94,8 @@ FROM
WHEN tc.contract_address='0x610b717796ad172b316836ac95a2ffad065ceab4' THEN 'WBTC'
WHEN tc.contract_address='0xbb93e510bbcd0b7beb5a853875f9ec60275cf498' THEN 'WBTC'
END AS currency_symbol
, 'Ethereum' AS blockchain
, 'Classic' AS tornado_version
, 'ethereum' AS blockchain
, 'classic' AS tornado_version
, et.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0xd4b88df4d29f5cedd6857912842cff3b20c8cfa3' THEN 100
Expand Down Expand Up @@ -126,19 +126,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_ethereum','erc20_evt_Deposit') }} tc
LEFT JOIN {{ source('ethereum','transactions_0006') }} et
INNER JOIN {{ source('ethereum','transactions') }} et
ON et.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND et.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_ethereum','erc20_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND et.block_time >= (select max(block_time) from {{ this }})
AND et.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_ethereum','erc20_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand All @@ -147,8 +147,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0xb8c77482e45f1f44de1745f52c74426c631bdd52' AS currency_contract
, 'BNB' AS currency_symbol
, 'BNB' AS blockchain
, 'Classic' AS tornado_version
, 'bnb' AS blockchain
, 'classic' AS tornado_version
, bt.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x84443cfd09a48af6ef360c6976c5392ac5023a1f' THEN 0.1
Expand All @@ -161,19 +161,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_bnb','TornadoCashBNB_evt_Deposit') }} tc
LEFT JOIN {{ source('bnb','transactions') }} bt
INNER JOIN {{ source('bnb','transactions') }} bt
ON bt.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND bt.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_bnb','TornadoCashBNB_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND bt.block_time >= (select max(block_time) from {{ this }})
AND bt.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_bnb','TornadoCashBNB_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand All @@ -182,8 +182,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0x6b175474e89094c44da98b954eedeac495271d0f' AS currency_contract
, 'xDAI' AS currency_symbol
, 'Gnosis' AS blockchain
, 'Classic' AS tornado_version
, 'gnosis' AS blockchain
, 'classic' AS tornado_version
, gt.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x1e34a77868e19a6647b1f2f47b51ed72dede95dd' THEN 100
Expand All @@ -196,19 +196,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_gnosis','eth_evt_Deposit') }} tc
LEFT JOIN {{ source('gnosis','transactions') }} gt
INNER JOIN {{ source('gnosis','transactions') }} gt
ON gt.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND gt.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_gnosis','eth_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND gt.block_time >= (select max(block_time) from {{ this }})
AND gt.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_gnosis','eth_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand All @@ -217,8 +217,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2' AS currency_contract
, 'ETH' AS currency_symbol
, 'Optimism' AS blockchain
, 'Classic' AS tornado_version
, 'optimism' AS blockchain
, 'classic' AS tornado_version
, ot.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x84443cfd09a48af6ef360c6976c5392ac5023a1f' THEN 0.1
Expand All @@ -231,19 +231,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_optimism','ETHTornado_evt_Deposit') }} tc
LEFT JOIN {{ source('optimism','transactions') }} ot
INNER JOIN {{ source('optimism','transactions') }} ot
ON ot.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND ot.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_optimism','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND ot.block_time >= (select max(block_time) from {{ this }})
AND ot.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_optimism','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand All @@ -252,8 +252,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0x85f138bfEE4ef8e540890CFb48F620571d67Eda3' AS currency_contract
, 'AVAX' AS currency_symbol
, 'Avalanche' AS blockchain
, 'Classic' AS tornado_version
, 'avalanche_c' AS blockchain
, 'classic' AS tornado_version
, at.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x330bdfade01ee9bf63c209ee33102dd334618e0a' THEN 10
Expand All @@ -265,19 +265,19 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_avalanche_c','ETHTornado_evt_Deposit') }} tc
LEFT JOIN {{ source('avalanche_c','transactions') }} at
INNER JOIN {{ source('avalanche_c','transactions') }} at
ON at.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND at.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_avalanche_c','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND at.block_time >= (select max(block_time) from {{ this }})
AND at.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_avalanche_c','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}

UNION
Expand All @@ -286,8 +286,8 @@ FROM
SELECT tc.evt_block_time AS block_time
, '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2' AS currency_contract
, 'ETH' AS currency_symbol
, 'Arbitrum' AS blockchain
, 'Classic' AS tornado_version
, 'arbitrum' AS blockchain
, 'classic' AS tornado_version
, at.from AS depositor
, tc.contract_address AS contract_address
, CASE WHEN tc.contract_address='0x84443cfd09a48af6ef360c6976c5392ac5023a1f' THEN 0.1
Expand All @@ -300,18 +300,18 @@ FROM
, tc.evt_index
, TRY_CAST(date_trunc('DAY', tc.evt_block_time) AS date) AS block_date
FROM {{ source('tornado_cash_arbitrum','ETHTornado_evt_Deposit') }} tc
LEFT JOIN {{ source('arbitrum','transactions') }} at
INNER JOIN {{ source('arbitrum','transactions') }} at
ON at.hash=tc.evt_tx_hash
{% if not is_incremental() %}
AND at.block_time >= (select min(evt_block_time) from {{ source('tornado_cash_arbitrum','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
AND at.block_time >= (select max(block_time) from {{ this }})
AND at.block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
{% if not is_incremental() %}
WHERE tc.evt_block_time >= (select min(evt_block_time) from {{ source('tornado_cash_arbitrum','ETHTornado_evt_Deposit') }})
{% endif %}
{% if is_incremental() %}
WHERE tc.evt_block_time >= (select max(block_time) from {{ this }})
WHERE tc.evt_block_time >= date_trunc("day", now() - interval '1 week')
{% endif %}
)
8 changes: 4 additions & 4 deletions models/tornado_cash/tornado_cash_schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ version: 2
models:
- name: tornado_cash_deposits
meta:
blockchain: ethereum, bnb, avalanche, gnosis, optimism, arbitrum #, polygon
blockchain: ethereum, bnb, avalanche_c, gnosis, optimism, arbitrum #, polygon
project: tornado_cash
contibutors: [hildobby, dot2dotseurat]
config:
tags: ['tornado_cash', 'deposits', 'ethereum', 'bnb', 'avalanche', 'gnosis', 'optimism', 'arbitrum']
tags: ['tornado_cash', 'deposits', 'ethereum', 'bnb', 'avalanche_c', 'gnosis', 'optimism', 'arbitrum']
description: "Tornado Cash Deposits"
columns:
- &block_time
Expand All @@ -25,13 +25,13 @@ models:
name: blockchain
tests:
- accepted_values:
values: ['Ethereum', 'BNB', 'Avalanche', 'Gnosis', 'Optimism', 'Arbitrum', 'Polygon']
values: ['ethereum', 'bnb', 'avalanche_c', 'gnosis', 'optimism', 'arbitrum', 'polygon']
description: "Blockchain"
- &tornado_version
name: tornado_version
tests:
- accepted_values:
values: ['Classic', 'Nova']
values: ['classic', 'nova']
description: "Version of Tornado Cash used for this transaction"
- &depositor
name: depositor
Expand Down
30 changes: 0 additions & 30 deletions models/tornado_cash/tornado_cash_sources.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ sources:
- name: tornado_cash_bnb
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: TornadoCashBNB_evt_Deposit
loaded_at_field: evt_block_time
Expand All @@ -16,18 +15,6 @@ sources:
- name: commitment
- name: leafIndex
- name: timestamp
- name: ETHTornado_evt_Withdrawal
loaded_at_field: evt_block_time
columns:
- name: contract_address
- name: evt_tx_hash
- name: evt_index
- name: evt_block_time
- name: evt_block_number
- name: fee
- name: nullifierHash
- name: relayer
- name: to
- name: TornadoCashBNB_evt_Withdrawal
loaded_at_field: evt_block_time
columns:
Expand All @@ -44,7 +31,6 @@ sources:
- name: tornado_cash_arbitrum
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: ETHTornado_evt_Deposit
loaded_at_field: evt_block_time
Expand Down Expand Up @@ -73,7 +59,6 @@ sources:
- name: tornado_cash_optimism
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: ETHTornado_evt_Deposit
loaded_at_field: evt_block_time
Expand All @@ -86,18 +71,6 @@ sources:
- name: commitment
- name: leafIndex
- name: timestamp
- name: erc20_evt_Withdrawal
loaded_at_field: evt_block_time
columns:
- name: contract_address
- name: evt_tx_hash
- name: evt_index
- name: evt_block_time
- name: evt_block_number
- name: fee
- name: nullifierHash
- name: relayer
- name: to
- name: ETHTornado_evt_Withdrawal
loaded_at_field: evt_block_time
columns:
Expand All @@ -114,7 +87,6 @@ sources:
- name: tornado_cash_avalanche_c
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: ETHTornado_evt_Deposit
loaded_at_field: evt_block_time
Expand Down Expand Up @@ -143,7 +115,6 @@ sources:
- name: tornado_cash_ethereum
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: erc20_evt_Deposit
loaded_at_field: evt_block_time
Expand Down Expand Up @@ -191,7 +162,6 @@ sources:
- name: tornado_cash_gnosis
freshness:
warn_after: { count: 12, period: hour }
error_after: { count: 24, period: hour }
tables:
- name: eth_evt_Deposit
loaded_at_field: evt_block_time
Expand Down
Loading