Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EIP-1559 fields (cli, airflow, dataflow) #96

Merged
merged 16 commits into from
Dec 19, 2022
Merged

Conversation

charlielewisme
Copy link
Collaborator

@charlielewisme charlielewisme commented Nov 14, 2022

What?

Add EIP-1559 fields to CLI, Airflow and Dataflow
Following up on and consolidating @cffls PRs #64 #65 #66

Why?

EIP-1559

How?

Testing?

  • Merge from main to add CLI tests
  • Add fields to pass existing CLI tests
  • Add CLI tests specific for post EIP-1559 blocks
  • In dev composer run export & load dags 2022-01-17 (pre EIP-1559)
  • In dev composer run export & load dags 2022-01-19 (post EIP-1559)
  • In dev composer run load_all_partitions (mix of pre and post EIP-1559)
  • Check updated BigQuery dev tables, diff against prod, see example
    • blocks, transactions identical but with extra fields (except for "miner" field which is known to have changed)
    • contracts, logs, token_transfers identical
    • tokens near identical but total_supply changes (as expected)
    • traces differ, seems to be a separate node data quality issue and not connected to Gulshan's re-export (slack thread)
  • Enable dev streamers and test dataflow into BigQuery dev tables

Notes from previous PR #64:

This commit is similar to the EIP-1559 pull request for ethereum-etl: blockchain-etl/ethereum-etl#256
The changes has been tested with export_blocks_and_transactions command.

I created separate pull requests for airflow and dataflow changes, so the deployment will be easier. Dependent PRs:

These pull requests will address this issue: #56

Fields added:

  • base_fee_per_gas (block) - base fee per gas in protocol, which can move up or down each block according to a formula which is a function of gas used in parent block and gas target (block gas limit divided by elasticity multiplier) of parent block.
  • max_fee_per_gas (tx) - total fee which covers both the priority fee and the block's network fee per gas
  • max_priority_fee_per_gas (tx) - maximum fee per gas tx senders are willing to give to miners to incentivize them to include their transaction
  • transaction_type (tx) - an envelope for future transaction types
  • effective_gas_price (receipt) - a replacement for gasUsed field

Note: There is one uncertain change about the postgres table. The file cli/polygonetl/streaming/postgres_tables.py is in the repo and I added new fields to it, but I can't find any table schema definition like this one for Ethereum. I am wondering if postgres_tables.py is even used. Please point me to the schema definition files if they exist, and I will change them accordingly.


Notes from previous PR #65

This change adds new fields introduced by EIP-1559 to airflow. This pull request depends on #64.

Reference changes for Ethereum:


Notes from previous PR #66

This change adds new fields introduced by EIP-1559 to dataflow module.
It has a dependency on #64.

Reference change for Ethereum:


cffls and others added 6 commits March 24, 2022 13:38
This commit is similar to the EIP-1559 pull request for ethereum-etl: blockchain-etl/ethereum-etl#256

Fields added:

 - base_fee_per_gas (block) - base fee per gas in protocol, which can move up or down each block according to a formula which is a function of gas used in parent block and gas target (block gas limit divided by elasticity multiplier) of parent block.
 - max_fee_per_gas (tx) - total fee which covers both the priority fee and the block's network fee per gas
 - max_priority_fee_per_gas (tx) - maximum fee per gas tx senders are willing to give to miners to incentivize them to include their transaction
 - transaction_type (tx) - an envelope for future transaction types
 - effective_gas_price (receipt) - a replacement for gasUsed field
@charlielewisme charlielewisme marked this pull request as ready for review November 16, 2022 12:57
@charlielewisme charlielewisme changed the title Eip 1559 etl Add fields introduced by EIP-1559 (2) Nov 16, 2022
This was referenced Nov 16, 2022
@charlielewisme charlielewisme changed the title Add fields introduced by EIP-1559 (2) EIP-1559 (cli, airflow, dataflow) Nov 16, 2022
@charlielewisme charlielewisme changed the title EIP-1559 (cli, airflow, dataflow) Add EIP-1559 fields (cli, airflow, dataflow) Nov 16, 2022
@charlielewisme charlielewisme force-pushed the EIP-1559_etl branch 2 times, most recently from 67a5aaf to f29b616 Compare December 19, 2022 12:05
Based on nansen-ai/evmchain-etl/pull/55

Add ds postfix for load and enrich tasks
Add expiration for temp raw tables
Remove unused enrichment sql
Support load_all_partitions for export files missing EIP-1559 fields
@charlielewisme charlielewisme merged commit fabc1aa into main Dec 19, 2022
@charlielewisme charlielewisme deleted the EIP-1559_etl branch December 19, 2022 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants