diff --git a/README.md b/README.md index 6a5c095..88103ad 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,9 @@ Klaytn ETL lets you convert Klaytn blockchain data into convenient formats like JSONs, CSVs and relational databases. This is a fork of [Ethereum ETL](https://github.com/blockchain-etl/ethereum-etl). -***Notice: Klaytn ETL is still on the beta version. However, CLIs are all functional.*** - -***Documents about command reference will be soon updated.*** +[Full documentation available here](http://klaytn-etl.readthedocs.io/). -***Please check issues to figure out current works.*** +***Notice: Klaytn ETL is still on the beta version. However, CLIs are all functional.*** ## Quickstart Install Klaytn ETL: diff --git a/docs/commands.md b/docs/commands.md new file mode 100644 index 0000000..07053e7 --- /dev/null +++ b/docs/commands.md @@ -0,0 +1,240 @@ +# Commands + +All the commands accept `-h` parameter for help, e.g.: + +```bash +> klaytnetl export_blocks_and_transactions -h + +Usage: klaytnetl export_blocks_and_transactions [OPTIONS] + +Options: + -s, --start-block INTEGER Start block [default: 0] + -e, --end-block INTEGER End block [required] + -b, --batch-size INTEGER The number of blocks to export at a time. + [default: 100] + -p, --provider-uri TEXT The URI of the web3 provider e.g. + file://$HOME/var/kend/data/klay.ipc or + https://cypress.fandom.finance/archive + [default: + https://cypress.fandom.finance/archive] + -w, --max-workers INTEGER The maximum number of workers. [default: 5] + --blocks-output TEXT The output file for blocks. If not provided + blocks will not be exported. Use "-" for stdout + --transactions-output TEXT The output file for transactions. If not + provided transactions will not be exported. Use + "-" for stdout + --network TEXT Input either baobab or cypress to obtain public + provider.If not provided, the option will be + disabled. + -h, --help Show this message and exit. +``` + +For the `--output` parameters the supported types are csv and json. The format type is inferred from the output file name. + +#### export_blocks_and_transactions + +```bash +> klaytnetl export_blocks_and_transactions --start-block 0 --end-block 500000 \ +--provider-uri https://cypress.fandom.finance/archive \ +--blocks-output blocks.csv --transactions-output transactions.csv +``` + +Omit `--blocks-output` or `--transactions-output` options if you want to export only transactions/blocks. + +You can tune `--batch-size`, `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Blocks and transactions schema](schema.md#blockscsv). + +#### export_token_transfers + +```bash +> klaytnetl export_token_transfers --start-block 0 --end-block 500000 \ +--provider-uri https://cypress.fandom.finance/archive --batch-size 100 --output token_transfers.csv +``` + +Include `--tokens --tokens ` to filter only certain tokens, e.g. + +```bash +> klaytnetl export_token_transfers --start-block 42397700 --end-block 42397800 \ +--provider-uri https://cypress.fandom.finance/archive --output token_transfers.csv \ +--tokens 0xcee8faf64bb97a73bb51e115aa89c17ffa8dd167 --tokens 0x34d21b1e550d73cee41151c77f3c73359527a396 +``` + +You can tune `--batch-size`, `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Token transfers schema](schema.md#token_transferscsv). + +#### export_receipts_and_logs + +First extract transactions from [export_blocks_and_transactions](#export_blocks_and_transactions) + +Then export receipts and logs from transactions.csv file: + +```bash +> klaytnetl export_receipts_and_logs --transactions transactions.csv \ +--provider-uri https://cypress.fandom.finance/archive --receipts-output receipts.csv --logs-output logs.csv +``` + +Omit `--receipts-output` or `--logs-output` options if you want to export only logs/receipts. + +You can tune `--batch-size`, `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Receipts and logs schema](schema.md#receiptscsv). + +#### extract_token_transfers + +First export receipt logs with [export_receipts_and_logs](#export_receipts_and_logs). + +Then extract transfers from the logs.csv file: + +```bash +> klaytnetl extract_token_transfers --logs logs.csv --output token_transfers.csv +``` + +You can tune `--batch-size`, `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Token transfers schema](schema.md#token_transferscsv). + +#### export_contracts + +First extract receipts from [export_receipts_and_logs](#export_receipts_and_logs) + +Then export contracts: + +```bash +> klaytnetl export_contracts --receipts receipts.csv \ +--provider-uri https://cypress.fandom.finance/archive --output contracts.csv +``` + +You can tune `--batch-size`, `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Contracts schema](schema.md#contractscsv). + +#### export_tokens + +First extract token addresses from `contracts.json` +(Exported with [export_contracts](#export_contracts)): + +```bash +> klaytnetl filter_items -i contracts.json -p "item['is_erc20'] or item['is_erc721'] or item['is_erc1155']" | \ +klaytnetl extract_field -f address -o token_addresses.txt +``` + +Then export ERC20 / ERC721 tokens: + +```bash +> klaytnetl export_tokens --token-addresses token_addresses.txt \ +--provider-uri https://cypress.fandom.finance/archive --output tokens.csv +``` + +You can tune `--max-workers` for performance. + +You can select either `baobab` or `cypress` in `--network`. + +[Tokens schema](schema.md#tokenscsv). + +#### export_traces + +Also called internal transactions. +Since this is rerunning a block, this will take a long time based on the transactions that block contains. +Make sure your node is an archive node with at least 8GB of memory, or else you will face timeout errors. +See [this issue](https://github.com/blockchain-etl/ethereum-etl/issues/137) + +```bash +> klaytnetl export_traces --start-block 0 --end-block 500000 \ +--provider-uri https://cypress.fandom.finance/archive --batch-size 100 --output traces.csv +``` + +By adding `--enrich` flag, you can enrich output files with additional fields like `block-timestamp`. + +You can tune `--batch-size`, `--max-workers` for performance. + +You can set `--timeout` appropriately. + +You can set `--file-format` to either `csv` or `json` and manipulate by `--file-maxlines` and `--compress` + +You can export to cloud storage by adding `--s3-bucket` flag. + +You can select either `baobab` or `cypress` in `--network`. + +[Traces schema](schema.md#tracescsv). + +#### export_block_group + +Exports block groups - blocks, transactions, receipts, logs, token transfer - from Klaytn node. + +```bash +> klaytnetl export_block_group --start-block 0 --end-block 500000 \ +--provider-uri https://cypress.fandom.finance/archive --batch-size 100 \ +--blocks-output blocks.csv --transactions-output transactions.csv \ +--receipts-output receipts.csv --logs-output logs.csv --token-transfers-output token_transfer.csv +``` + +Omit `--blocks-output`/`--transactions-output`/`--receipts-output`/`--logs-output`/`--token-transfers-output` options +if you want to export only transactions/blocks/receipts/logs/token transfers. + +By adding `--enrich` flag, you can enrich output files with additional fields like `block-timestamp`. + +You can tune `--batch-size`, `--max-workers` for performance. + +You can set `--timeout` appropriately. + +You can set `--file-format` to either `csv` or `json` and manipulate by `--file-maxlines` and `--compress` + +You can export to cloud storage by adding `--s3-bucket` or `--gcs-bucket` flag. + +You can select either `baobab` or `cypress` in `--network`. + + +#### export_trace_group + +Exports trace groups - traces, contracts, tokens - from Klaytn node. +Since this is rerunning a block, this will take a long time based on the transactions that block contains. +Make sure your node is an archive node with at least 8GB of memory, or else you will face timeout errors. + +```bash +> klaytnetl export_trace_group --start-block 0 --end-block 500000 \ +--provider-uri https://cypress.fandom.finance/archive --batch-size 100 \ +--traces-output traces.csv --tokens-output tokens.csv --contracts-output contracts.csv +``` + +Omit `--traces-output`/`--tokens-output`/`--contracts-output` options +if you want to export only traces/tokens/contracts. + +By adding `--enrich` flag, you can enrich output files with additional fields like `block-timestamp`. + +You can tune `--batch-size`, `--max-workers` for performance. + +You can set `--timeout` appropriately. + +You can set `--file-format` to either `csv` or `json` and manipulate by `--file-maxlines` and `--compress` + +You can export to cloud storage by adding `--s3-bucket` or `--gcs-bucket` flag. + +Use `--detailed-trace-log` and `--log-percentage-step` to get trace count with wanted steps. + +You can select either `baobab` or `cypress` in `--network`. + +#### get_block_range_for_date + +```bash +> klaytnetl get_block_range_for_date --provider-uri=https://cypress.fandom.finance/archive --date 2020-01-01 +16369455,16455852 +``` + +#### get_keccak_hash + +```bash +> klaytnetl get_keccak_hash -i "transfer(address,uint256)" +0xa9059cbb2ab09eb219583f4a59a5d0623ade346d962bcd4e46b11da047c9049b +``` diff --git a/docs/exporting-the-blockchain.md b/docs/exporting-the-blockchain.md new file mode 100644 index 0000000..f97073b --- /dev/null +++ b/docs/exporting-the-blockchain.md @@ -0,0 +1,30 @@ +## Exporting the Blockchain + +1. Install python 3.7.2+: [https://www.python.org/downloads/](https://www.python.org/downloads/) + +1. Launch an endpoint node (https://docs.klaytn.foundation/getting-started/quick-start/launch-an-en) or use pre-existing endpoint (https://docs.klaytn.foundation/dapp/json-rpc/public-en) + +1. Install Klaytn ETL: `> pip3 install klaytn-etl-cli` + +1. Export all: + +```bash +> klaytnetl export_all --help +> klaytnetl export_all -s 0 -e 5999999 -b 100000 -p https://cypress.fandom.finance/archive -o output +``` + +In case `klaytnetl` command is not available in PATH, use `python3 -m klaytnetl` instead. + +The result will be in the `output` subdirectory, partitioned in Hive style: +```bash +output/blocks/start_block=00000000/end_block=00099999/blocks_00000000_00099999.csv +output/blocks/start_block=00100000/end_block=00199999/blocks_00100000_00199999.csv +... +output/transactions/start_block=00000000/end_block=00099999/transactions_00000000_00099999.csv +... +output/token_transfers/start_block=00000000/end_block=00099999/token_transfers_00000000_00099999.csv +... +``` + +Should work on Linux, Mac, Windows. +Since `debug_traceBlockByNumber` takes a long time, please cautious when running anything related to trace. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..0e38b09 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,18 @@ +# Overview + +Klaytn ETL lets you convert Klaytn blockchain data into convenient formats like JSONs, CSVs and relational databases. +This is a fork of [Ethereum ETL](https://github.com/blockchain-etl/ethereum-etl). + +## Features + +Easily export: + +* Blocks +* Transactions +* ERC20 / ERC721 / ERC 1155 tokens +* Token transfers +* Receipts +* Logs +* Contracts +* Traces (Internal transactions) + diff --git a/docs/limitations.md b/docs/limitations.md new file mode 100644 index 0000000..13dcccc --- /dev/null +++ b/docs/limitations.md @@ -0,0 +1,14 @@ +# Limitation + +- In case the contract is a proxy, which forwards all calls to a delegate, interface detection doesn’t work, +which means `is_erc20`/`is_erc721`/`is_erc1155` will always be false for proxy contracts and they will be missing in the `tokens` +table. +- The metadata methods (`symbol`, `name`, `decimals`, `total_supply`) for ERC20 are optional, so around 10% of the +contracts are missing this data. +- `token_transfers.value`, `tokens.decimals` and `tokens.total_supply` have type `STRING` in BigQuery tables, +because numeric types there can't handle 32-byte integers. You should use +`cast(value as FLOAT64)` (possible loss of precision) or +`safe_cast(value as NUMERIC)` (possible overflow) to convert to numbers. +- The contracts that don't implement `decimals()` function but have the +[fallback function](https://solidity.readthedocs.io/en/v0.4.21/contracts.html#fallback-function) that returns a `boolean` +will have `0` or `1` in the `decimals` column in the CSVs. \ No newline at end of file diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..ebd8366 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,35 @@ +# Quickstart +Install Klaytn ETL: + +```bash +pip3 install klaytn-etl-cli +``` + +Export blocks and transactions + +```bash +> klaytnetl export_blocks_and_transactions --start-block 0 --end-block 5000 \ +--blocks-output blocks.json --transactions-output transactions.json +``` + +Export ERC20 and ERC721 transfers + +```bash +> klaytnetl export_token_transfers --start-block 0 --end-block 5000 \ +--output token_transfers.json +``` + +Export traces + +```bash +> klaytnetl export_traces --start-block 0 --end-block 5000 \ +--output traces.json +``` + +Find other commands [here](commands.md). + +For the latest version, check out the repo and call +```bash +> pip3 install -e . +> python3 klaytnetl.py +``` diff --git a/docs/schema.md b/docs/schema.md new file mode 100644 index 0000000..6c07360 --- /dev/null +++ b/docs/schema.md @@ -0,0 +1,219 @@ +# Schema + +## blocks.csv + +Column | Type | +--------------------|------------| +number | bigint | +hash | hex_string | +parent_hash | hex_string | +logs_bloom | hex_string | +transactions_root | hex_string | +state_root | hex_string | +receipts_root | hex_string | +size | bigint | +extra_data | hex_string | +gas_used | numeric | +block_timestamp | timestamp | +block_unix_timestamp| numeric | +transaction_count | bigint | +block_score | numeric | +total_block_score | numeric | +governance_data | hex_string | +vote_data | hex_string | +committee | struct | +proposer | hex_string | +reward_address | adresss | +base_fee_per_gas | numeric | + +--- + +## transactions.csv + +Column | Type | +------------------------|--------------| +hash | hex_string | +nonce | bigint | +block_hash | hex_string | +block_number | bigint | +transaction_index | bigint | +from_address | address | +to_address | address | +value | numeric | +gas | bigint | +gas_price | bigint | +input | hex_string | +fee_payer | hex_string | +fee_payer_signatures | struct | +fee_ratio | bigint | +sender_tx_hash | hex_string | +signatures | struct | +tx_type | string | +tx_type_int | bigint | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +receipt_gas_used | numeric | +receipt_contract_address| adresss | +receipt_status | bigint | +max_priority_fee_per_gas| numeric | +max_fee_per_gas | numeric | +access_list | struct | + +--- + +## token_transfers.csv + +Column | Type | +----------------------------|------------| +token_address | address | +from_address | address | +to_address | address | +value | string | +block_hash | hex_string | +block_number | bigint | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +transaction_hash | hex_string | +transaction_receipt_status | bigint | +log_index | bigint | + +--- + +## receipts.csv + +Column | Type | +-----------------------------|------------| +transaction_hash | hex_string | +transaction_index | bigint | +block_hash | hex_string | +block_number | bigint | +gas | numeric | +gas_price | numeric | +gas_used | numeric | +effective_gas_price | numeric | +contract_address | address | +logs_bloom | hex_string | +nonce | bigint | +fee_payer | hex_string | +fee_payer_signatures | struct | +fee_ratio | bigint | +code_format | string | +human_readable | boolean | +tx_error | string | +key | hex_string | +input_data | hex_string | +from_address | address | +to_address | address | +type_name | string | +type_int | bigint | +sender_tx_hash | hex_string | +signatures | struct | +status | bigint | +value | string | +input_json | struct | +access_list | struct | +chain_id | bigint | +max_priority_fee_per_gas | numeric | +max_fee_per_gas | numeric | +block_timestamp | timestamp | +block_unix_timestamp | numeric | + +--- + +## logs.csv + +Column | Type | +-----------------------------|------------| +block_number | bigint | +block_hash | hex_string | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +transaction_hash | hex_string | +transaction_index | bigint | +transaction_receipt_status | bigint | +log_index | bigint | +address | address | +data | string | +topics | string | + +--- + +## contracts.csv + +Column | Type | +-----------------------------|------------| +address | address | +bytecode | hex_string | +function_sighashes | string | +is_erc20 | boolean | +is_erc721 | boolean | +is_erc1155 | boolean | +block_number | bigint | +block_hash | hex_string | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +transaction_hash | hex_string | +transaction_index | bigint | +transaction_receipt_status | bigint | +trace_index | bigint | +trace_status | bigint | +creator_address | address | + +--- + +## tokens.csv + +Column | Type | +-----------------------------|------------| +address | address | +symbol | string | +name | string | +decimals | bigint | +total_supply | string | +function_sighashes | string | +is_erc20 | boolean | +is_erc721 | boolean | +is_erc1155 | boolean | +block_number | bigint | +block_hash | hex_string | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +transaction_hash | hex_string | +transaction_index | bigint | +transaction_receipt_status | bigint | +trace_index | bigint | +trace_status | bigint | +creator_address | address | + +--- + +## traces.csv + +Column | Type | +-----------------------------|------------| +block_number | bigint | +block_hash | hex_string | +block_timestamp | timestamp | +block_unix_timestamp | numeric | +transaction_hash | hex_string | +transaction_index | bigint | +transaction_receipt_status | bigint | +from_address | address | +to_address | address | +value | numeric | +input | hex_string | +output | hex_string | +trace_type | string | +call_type | string | +gas | numeric | +gas_used | numeric | +subtraces | bigint | +trace_address | string | +error | string | +status | bigint | +trace_index | string | + +You can find column descriptions in [https://github.com/klaytn/klaytn-etl](https://github.com/klaytnetl/klaytn-etl/schema) + +Note: for the `address` type all hex characters are lower-cased. +`boolean` type can have 2 values: `True` or `False`. diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..4a669c4 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,12 @@ +site_name: Klaytn ETL +nav: + - Overview: index.md + - Quickstart: quickstart.md + - Exporting the Blockchain: exporting-the-blockchain.md + - References: + - Commands: commands.md + - Schema: schema.md + - Limitations: limitations.md +theme: readthedocs +repo_url: https://github.com/klaytn/klaytn-etl +edit_uri: edit/main/docs \ No newline at end of file diff --git a/schemas/blocks.py b/schemas/blocks.py new file mode 100644 index 0000000..d048b2b --- /dev/null +++ b/schemas/blocks.py @@ -0,0 +1,128 @@ +BLOCKS_SCHEMA = [ + { + "name": "number", + "type": "INT64", + "mode": "NULLABLE", + "description": "The block number", + }, + { + "name": "hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "parent_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the parent block", + }, + { + "name": "logs_bloom", + "type": "STRING", + "mode": "NULLABLE", + "description": "The bloom filter for the logs of the block", + }, + { + "name": "transactions_root", + "type": "STRING", + "mode": "NULLABLE", + "description": "The root of the transaction trie of the block", + }, + { + "name": "state_root", + "type": "STRING", + "mode": "NULLABLE", + "description": "The root of the final state trie of the block", + }, + { + "name": "receipts_root", + "type": "STRING", + "mode": "NULLABLE", + "description": "The root of the receipts trie of the block", + }, + { + "name": "size", + "type": "INT64", + "mode": "NULLABLE", + "description": "The size of this block in bytes", + }, + { + "name": "extra_data", + "type": "STRING", + "mode": "NULLABLE", + "description": "The extra data field of this block", + }, + { + "name": "gas_used", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "The total used gas by all transactions in this block", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_count", + "type": "INT64", + "mode": "NULLABLE", + "description": "The number of transactions in the block", + }, + { + "name": "block_score", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Former difficulty. Always 1 in the BFT consensus engine", + }, + { + "name": "total_block_score", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Integer of the total blockScore of the chain until this block", + }, + { + "name": "governance_data", + "type": "STRING", + "mode": "NULLABLE", + "description": "RLP encoded governance configuration", + }, + { + "name": "vote_data", + "type": "STRING", + "mode": "NULLABLE", + "description": "RLP encoded governance vote of the proposer", + }, + { + "name": "committee", + "type": "STRING", + "mode": "REPEATED", + "description": "Array of addresses of committee members of this block. The committee is a subset of validators participated in the consensus protocol for this block", + }, + { + "name": "proposer", + "type": "STRING", + "mode": "NULLABLE", + "description": "The address of the block proposer", + }, + { + "name": "reward_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "The address of the beneficiary to whom the block rewards were givem", + }, + { + "name": "base_fee_per_gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "The base fee per gas. This value is returned only when EthTxTypeCompatibleBlock is activated for that block number", + }, +] diff --git a/schemas/contracts.py b/schemas/contracts.py new file mode 100644 index 0000000..72e9536 --- /dev/null +++ b/schemas/contracts.py @@ -0,0 +1,98 @@ +CONTRACTS_SCHEMA = [ + { + "name": "address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the contract", + }, + { + "name": "bytecode", + "type": "STRING", + "mode": "NULLABLE", + "description": "Bytecode of the contract", + }, + { + "name": "function_sighashes", + "type": "STRING", + "mode": "REPEATED", + "description": "4-byte function signature hashes", + }, + { + "name": "is_erc20", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC20 contract", + }, + { + "name": "is_erc721", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC721 contract", + }, + { + "name": "is_erc1155", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC1155 contract", + }, + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "transaction_receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "trace_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Index of the trace", + }, + { + "name": "trace_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure, due to any operation that can cause the call itself or any top-level call to revert)", + }, + { + "name": "creator_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Token creator address", + }, +] diff --git a/schemas/logs.py b/schemas/logs.py new file mode 100644 index 0000000..10be4b3 --- /dev/null +++ b/schemas/logs.py @@ -0,0 +1,68 @@ +LOGS_SCHEMA = [ + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "log_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the log index position in the block", + }, + { + "name": "address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address from which this log originated", + }, + { + "name": "data", + "type": "STRING", + "mode": "NULLABLE", + "description": "Contains one or more 32 Bytes non-indexed arguments of the log", + }, + { + "name": "topics", + "type": "STRING", + "mode": "REPEATED", + "description": "Indexed log arguments (0 to 4 32-byte hex strings). (In solidity: The first topic is the hash of the signature of the event (e.g. Deposit(address,bytes32,uint256)), except you declared the event with the anonymous specifier.)", + }, +] diff --git a/schemas/receipts.py b/schemas/receipts.py new file mode 100644 index 0000000..81434c4 --- /dev/null +++ b/schemas/receipts.py @@ -0,0 +1,235 @@ +RECEIPTS_SCHEMA = [ + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas provided by the sender", + }, + { + "name": "gas_price", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas price provided by the sender in Peb", + }, + { + "name": "gas_used", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "The total used gas by all transactions in this block", + }, + { + "name": "effective_gas_price", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "The actual value per gas deducted from the senders account", + }, + { + "name": "contract_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "The contract address created, if the transaction was a contract creation, otherwise null", + }, + { + "name": "logs_bloom", + "type": "STRING", + "mode": "NULLABLE", + "description": "The bloom filter for the logs", + }, + { + "name": "nonce", + "type": "INT64", + "mode": "NULLABLE", + "description": "The number of transactions made by the sender prior to this one", + }, + { + "name": "fee_payer", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) Address of the fee payer", + }, + { + "name": "fee_payer_signatures", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "V", "type": "STRING"}, + {"name": "R", "type": "STRING"}, + {"name": "S", "type": "STRING"}, + ], + "description": "(optional) An array of fee payer’s signature objects. A signature object contains three fields (V, R, and S). V contains ECDSA recovery id. R contains ECDSA signature r while S contains ECDSA signature s", + }, + { + "name": "fee_ratio", + "type": "INT64", + "mode": "NULLABLE", + "description": "(optional) Fee ratio of the fee payer. If it is 30, 30% of the fee will be paid by the fee payer. 70% will be paid by the sender", + }, + { + "name": "code_format", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) The code format of smart contract code", + }, + { + "name": "human_readable", + "type": "BOOL", + "mode": "NULLABLE", + "description": "(optional) true if the address is humanReadable, false if the address is not humanReadable", + }, + { + "name": "tx_error", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) detailed error code if status is equal to zero", + }, + { + "name": "key", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) key of the newly created account", + }, + { + "name": "input_data", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) The data sent along with the transaction", + }, + { + "name": "from_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the sender", + }, + { + "name": "to_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the receiver. Null when it is a contract creation transaction", + }, + { + "name": "type_name", + "type": "STRING", + "mode": "NULLABLE", + "description": "A string representing the type of the transaction", + }, + { + "name": "type_int", + "type": "INT64", + "mode": "NULLABLE", + "description": "An integer representing the type of the transaction", + }, + { + "name": "sender_tx_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of a transaction that is signed only by the sender", + }, + { + "name": "signatures", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "V", "type": "STRING"}, + {"name": "R", "type": "STRING"}, + {"name": "S", "type": "STRING"}, + ], + "description": "An array of signature objects. A signature object contains three fields (V, R, and S). V contains ECDSA recovery id. R contains ECDSA signature r while S contains ECDSA signature s", + }, + { + "name": "status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure)", + }, + { + "name": "value", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) Integer of the value sent with this transaction", + }, + { + "name": "input_json", + "type": "STRUCT", + "mode": "NULLABLE", + "fields": [ + {"name": "blockCount", "type": "INT64"}, + {"name": "blockHash", "type": "STRING"}, + {"name": "blockNumber", "type": "INT64"}, + {"name": "id", "type": "STRING"}, + {"name": "parentHash", "type": "STRING"}, + {"name": "receiptsRoot", "type": "STRING"}, + {"name": "stateRoot", "type": "STRING"}, + {"name": "transactionsRoot", "type": "STRING"}, + {"name": "txCount", "type": "INT64"}, + ], + "description": "(optional) The data sent along with the transaction", + }, + { + "name": "access_list", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "address", "type": "STRING"}, + { + "name": "storage_keys", + "type": "STRUCT", + "fields": [{"name": "hash", "type": "STRING"}], + }, + ], + "description": "An array of accessList", + }, + { + "name": "chain_id", + "type": "INT64", + "mode": "NULLABLE", + "description": "id of the chain", + }, + { + "name": "max_priority_fee_per_gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "A maximum amount to pay for the transaction to execute", + }, + { + "name": "max_fee_per_gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas tip cap for dynamic fee transaction in peb", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, +] diff --git a/schemas/token_transfers.py b/schemas/token_transfers.py new file mode 100644 index 0000000..79324b4 --- /dev/null +++ b/schemas/token_transfers.py @@ -0,0 +1,74 @@ +TOKEN_TRANSFERS_SCHEMA = [ + { + "name": "token_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Token address", + }, + { + "name": "from_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the sender", + }, + { + "name": "to_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the receiver", + }, + { + "name": "value", + "type": "STRING", + "mode": "NULLABLE", + "description": "Amount of tokens transferred (ERC20) / id of the token transferred (ERC721). Use safe_cast for casting to NUMERIC or FLOAT64", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "transaction_receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "log_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the log index position in the block", + }, +] diff --git a/schemas/tokens.py b/schemas/tokens.py new file mode 100644 index 0000000..03dac9d --- /dev/null +++ b/schemas/tokens.py @@ -0,0 +1,116 @@ +TOKENS_SCHEMA = [ + { + "name": "address", + "type": "STRING", + "mode": "NULLABLE", + "description": "The address of the token", + }, + { + "name": "symbol", + "type": "STRING", + "mode": "NULLABLE", + "description": "The symbol of the token", + }, + { + "name": "name", + "type": "STRING", + "mode": "NULLABLE", + "description": "The name of the token", + }, + { + "name": "decimals", + "type": "INT64", + "mode": "NULLABLE", + "description": "The number of decimals the token uses. Use safe_cast for casting to NUMERIC or FLOAT64", + }, + { + "name": "total_supply", + "type": "STRING", # Due to cases where total_supply is set up really high + "mode": "NULLABLE", + "description": "The total token supply. Use safe_cast for casting to NUMERIC or FLOAT64", + }, + { + "name": "function_sighashes", + "type": "STRING", + "mode": "REPEATED", + "description": "4-byte function signature hashes", + }, + { + "name": "is_erc20", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC20 contract", + }, + { + "name": "is_erc721", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC721 contract", + }, + { + "name": "is_erc1155", + "type": "BOOLEAN", + "mode": "NULLABLE", + "description": "Whether this contract is an ERC1155 contract", + }, + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "transaction_receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "trace_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Trace index of the trace", + }, + { + "name": "trace_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure, due to any operation that can cause the call itself or any top-level call to revert)", + }, + { + "name": "creator_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Token creator address", + }, +] diff --git a/schemas/traces.py b/schemas/traces.py new file mode 100644 index 0000000..fe8cdcc --- /dev/null +++ b/schemas/traces.py @@ -0,0 +1,128 @@ +TRACES_SCHEMA = [ + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "transaction_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Hash of the transactions", + }, + { + "name": "transaction_receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "from_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the sender, null when trace_type is reward", + }, + { + "name": "to_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the receiver if trace_type is call, address of new contract or null if trace_type is create (or create 2), beneficiary address if trace_type is suicide, miner address if trace_type is reward", + }, + { + "name": "value", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Value transferred in Peb", + }, + { + "name": "input", + "type": "STRING", + "mode": "NULLABLE", + "description": "The data sent along with the message call", + }, + { + "name": "output", + "type": "STRING", + "mode": "NULLABLE", + "description": "The output of the message call, bytecode of contract when trace_type is create", + }, + { + "name": "trace_type", + "type": "STRING", + "mode": "NULLABLE", + "description": "One of call, create, create2, suicide, reward", + }, + { + "name": "call_type", + "type": "STRING", + "mode": "NULLABLE", + "description": "One of call, callcode, delegatecall, staticcall", + }, + { + "name": "gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas provided with the message call", + }, + { + "name": "gas_used", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas used by the message call", + }, + { + "name": "subtraces", + "type": "INT64", + "mode": "NULLABLE", + "description": "Number of subtraces", + }, + { + "name": "trace_address", + "type": "STRING", + "mode": "REPEATED", + "description": "Comma separated list of trace address in call tree", + }, + { + "name": "error", + "type": "STRING", + "mode": "NULLABLE", + "description": "Error message", + }, + { + "name": "status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure, due to any operation that can cause the call itself or any top-level call to revert)", + }, + { + "name": "trace_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Trace index of the trace", + }, +] diff --git a/schemas/transactions.py b/schemas/transactions.py new file mode 100644 index 0000000..3630199 --- /dev/null +++ b/schemas/transactions.py @@ -0,0 +1,176 @@ +TRANSACTIONS_SCHEMA = [ + { + "name": "hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the transaction", + }, + { + "name": "nonce", + "type": "INT64", + "mode": "NULLABLE", + "description": "The number of transactions made by the sender prior to this one", + }, + { + "name": "block_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of the block", + }, + { + "name": "block_number", + "type": "INT64", + "mode": "NULLABLE", + "description": "Block number corresponding", + }, + { + "name": "transaction_index", + "type": "INT64", + "mode": "NULLABLE", + "description": "Integer of the transactions index position in the block", + }, + { + "name": "from_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the sender", + }, + { + "name": "to_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "Address of the receiver. null when its a contract creation transaction", + }, + { + "name": "value", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Value transferred in Peb", + }, + { + "name": "gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas provided by the sender", + }, + { + "name": "gas_price", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas price provided by the sender in Peb", + }, + { + "name": "input", + "type": "STRING", + "mode": "NULLABLE", + "description": "The data sent along with the transaction", + }, + { + "name": "fee_payer", + "type": "STRING", + "mode": "NULLABLE", + "description": "(optional) Address of the fee payer", + }, + { + "name": "fee_payer_signatures", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "V", "type": "STRING"}, + {"name": "R", "type": "STRING"}, + {"name": "S", "type": "STRING"}, + ], + "description": "(optional) An array of fee payer’s signature objects. A signature object contains three fields (V, R, and S). V contains ECDSA recovery id. R contains ECDSA signature r while S contains ECDSA signature s", + }, + { + "name": "fee_ratio", + "type": "INT64", + "mode": "NULLABLE", + "description": "(optional) Fee ratio of the fee payer. If it is 30, 30% of the fee will be paid by the fee payer. 70% will be paid by the sender", + }, + { + "name": "sender_tx_hash", + "type": "STRING", + "mode": "NULLABLE", + "description": "Hash of a transaction that is signed only by the sender", + }, + { + "name": "signatures", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "V", "type": "STRING"}, + {"name": "R", "type": "STRING"}, + {"name": "S", "type": "STRING"}, + ], + "description": "An array of signature objects. A signature object contains three fields (V, R, and S). V contains ECDSA recovery id. R contains ECDSA signature r while S contains ECDSA signature s", + }, + { + "name": "tx_type", + "type": "STRING", + "mode": "NULLABLE", + "description": "A string representing the type of the transaction", + }, + { + "name": "tx_type_int", + "type": "INT64", + "mode": "NULLABLE", + "description": "An integer representing the type of the transaction", + }, + { + "name": "block_timestamp", + "type": "TIMESTAMP", + "mode": "NULLABLE", + "description": "The UTC timestamp for when the block was collated", + }, + { + "name": "block_unix_timestamp", + "type": "FLOAT64", + "mode": "NULLABLE", + "description": "The unix timestamp for when the block was collated", + }, + { + "name": "receipt_gas_used", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "The amount of gas used by this specific transaction alone", + }, + { + "name": "receipt_contract_address", + "type": "STRING", + "mode": "NULLABLE", + "description": "The contract address created, if the transaction was a contract creation, otherwise null", + }, + { + "name": "receipt_status", + "type": "INT64", + "mode": "NULLABLE", + "description": "Either 1 (success) or 0 (failure) (post Byzantium)", + }, + { + "name": "max_priority_fee_per_gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "A maximum amount to pay for the transaction to execute", + }, + { + "name": "max_fee_per_gas", + "type": "DECIMAL", + "mode": "NULLABLE", + "description": "Gas tip cap for dynamic fee transaction in peb", + }, + { + "name": "access_list", + "type": "STRUCT", + "mode": "REPEATED", + "fields": [ + {"name": "address", "type": "STRING"}, + { + "name": "storage_keys", + "type": "STRUCT", + "fields": [{"name": "hash", "type": "STRING"}], + }, + ], + "description": "An array of accessList", + }, +]