Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize postgresql queries for API Server scanner #1878

Merged
merged 7 commits into from
Mar 27, 2025

Conversation

OBorce
Copy link
Contributor

@OBorce OBorce commented Feb 5, 2025

  • added a cache table for the latest delegation states so that we can efficiently retrieve all the latest data for each delegation of a pool
  • added delegations balance in the pool endpoints

@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch from 6e25173 to 1b1d9de Compare February 5, 2025 14:31
@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch 2 times, most recently from 2df3f28 to 36c2f3e Compare February 19, 2025 10:36
@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch 2 times, most recently from 5f7cbf7 to 39a0c55 Compare March 12, 2025 02:29
@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch from 39a0c55 to 9babfb2 Compare March 13, 2025 07:59
async fn get_address_balances(
&self,
address: &str,
) -> Result<Vec<(CoinOrTokenId, Amount, u8)>, ApiServerStorageError> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The u8 is cryptic. In such a case it's better to at least leave a comment, e.g.

Result<Vec<(CoinOrTokenId, Amount, /*number of decimals*/ u8)>

(though rustfmt may not like such comments sometimes)

But since this tuple appears in public interfaces, it's better to make it a struct with named fields and use the struct everywhere.

P.S. it's also a bit strange that a vec is returned instead of a map with CoinOrTokenId as the key.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with DecimalAmount

Comment on lines 2095 to 2098
self.tx
.execute(
"INSERT INTO ml.coin_or_token_decimals (coin_or_token_id, block_height, number_of_decimals) VALUES ($1, $2, $3) ON CONFLICT (coin_or_token_id) DO NOTHING;",
&[&coin_or_token_id.encode(), &height, &(issuance.number_of_decimals as i16)],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"ON CONFLICT (coin_or_token_id) DO NOTHING" IMO looks strange.

In the context of this particular function probably UPDATE would make more sense? I mean, what does set_fungible_token_issuance mean in the context of the storage - the token somehow got "re-issued", so probably we should just update the data.

In general, storage methods should make sense on their own. And same methods from the postgres and in-memory storage should probably behave in the same way.
And e.g. the in-memory storage makes no particular assumptions on when a token re-issuance can occur - it just adds a new issuance object for the provided token_id/block_height.
So I guess postgres should do the same, assuming that the same token can be issued multiple times at different heights. I.e. coin_or_token_decimals should have block_height as a part of the key and queries that use coin_or_token_decimals should take this into account.
But then the next question arises - why isn't token decimals value a part of ml.fungible_token? And the coin decimals never change, so there isn't much sense in putting them into the db. So coin_or_token_decimals seems redundant.

P.S. another weird thing about coin_or_token_decimals is that it's partially exposed from the storage interface via del_coin_or_token_decimals_above_height.
Where the in-memory storage implementation just does nothing.
IMO it's another sign that it should go.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the on conflict as a token can be issued only once, if there are multiple then it should be an error.

separated the set_fungible_token_issuance into 2 set_fungible_token_issuance and set_fungible_token_data, first for the initial issuance that can happen only once per token_id and the later for any modification to the token data from mint/change token authority ...

The new table is just a cache for faster lookups when updating address balances.

Comment on lines 2427 to 2439
.await
.map_err(|e| ApiServerStorageError::LowLevelStorageError(e.to_string()))?;

let coin_or_token_id = CoinOrTokenId::TokenId(token_id);
self.tx
.execute(
"INSERT INTO ml.coin_or_token_decimals (coin_or_token_id, block_height, number_of_decimals) VALUES ($1, $2, $3);",
&[&coin_or_token_id.encode(), &height, &0_i16],
)
.await
.map_err(|e| ApiServerStorageError::LowLevelStorageError(e.to_string()))?;

Ok(())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment.

And btw probably the above "INSERT INTO ml.nft_issuance" should also have "ON CONFLICT ... UPDATE" so that it behaves just like its in-memory counterpart.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't have an on conflict because it cannot happen same as the insert of the nft above it.
Added a panic in the in-memory counterpart.

@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch 2 times, most recently from ec854aa to 4186a49 Compare March 24, 2025 16:20
@OBorce OBorce force-pushed the refactor/optimize-api-server-db-queries branch from 4186a49 to 2b01a6b Compare March 24, 2025 19:07
OBorce and others added 2 commits March 26, 2025 23:07
…ken-authority

Add address token authority endpoint to API Server
@OBorce OBorce merged commit dbaee41 into master Mar 27, 2025
18 checks passed
@OBorce OBorce deleted the refactor/optimize-api-server-db-queries branch March 27, 2025 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants