Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Get slot number by provided date-time endpoint #387

Merged
merged 118 commits into from
Apr 10, 2024
Merged
Show file tree
Hide file tree
Changes from 114 commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
9366d71
fix CardanoStakeAddress error handling
Mr-Leshiy Mar 29, 2024
bdeee7a
refactor, add sync_state_get endpoint
Mr-Leshiy Mar 29, 2024
a083cb3
refactor types
Mr-Leshiy Mar 29, 2024
8adcfdc
refactor
Mr-Leshiy Mar 29, 2024
bdc15ca
add block_hash validation
Mr-Leshiy Mar 29, 2024
313efa2
Merge branch 'main' into feat/utxo-stake-test
Mr-Leshiy Mar 29, 2024
602e476
wip
Mr-Leshiy Mar 29, 2024
171f5f8
Merge branch 'main' into feat/utxo-stake-test
Mr-Leshiy Mar 29, 2024
07a5e00
wip
Mr-Leshiy Mar 29, 2024
6f75eb4
wip
Mr-Leshiy Mar 29, 2024
76da85b
wip
Mr-Leshiy Mar 29, 2024
9ed1781
add check_network fn
Mr-Leshiy Mar 29, 2024
199f217
fix
Mr-Leshiy Mar 29, 2024
6ad503f
fix schematisis test
Mr-Leshiy Mar 29, 2024
4c6b92c
try
Mr-Leshiy Mar 29, 2024
f465002
wip
Mr-Leshiy Mar 29, 2024
a1f5677
try
Mr-Leshiy Mar 30, 2024
bed8663
try
Mr-Leshiy Mar 30, 2024
03793e0
try
Mr-Leshiy Mar 30, 2024
6b0a021
try
Mr-Leshiy Mar 30, 2024
ffb51ca
wip
Mr-Leshiy Mar 30, 2024
6a5130c
try
Mr-Leshiy Mar 30, 2024
a9376b9
try
Mr-Leshiy Mar 30, 2024
91c56e4
fix
Mr-Leshiy Mar 30, 2024
e223d21
update Network
Mr-Leshiy Mar 30, 2024
421c83b
add test_utxo test
Mr-Leshiy Mar 30, 2024
61802c6
try
Mr-Leshiy Mar 30, 2024
66cf0b3
fix
Mr-Leshiy Mar 30, 2024
8bf2ee9
try
Mr-Leshiy Mar 30, 2024
b6c2887
fix
Mr-Leshiy Mar 30, 2024
2f0db56
wip
Mr-Leshiy Mar 30, 2024
a813114
fix
Mr-Leshiy Mar 30, 2024
f90d7f3
fix docket-compose.yml file
Mr-Leshiy Mar 30, 2024
85c650a
try
Mr-Leshiy Mar 31, 2024
afb3fec
try
Mr-Leshiy Mar 31, 2024
991cf13
fix
Mr-Leshiy Mar 31, 2024
0eed553
try
Mr-Leshiy Mar 31, 2024
e64cef6
try
Mr-Leshiy Mar 31, 2024
c80b7d5
try
Mr-Leshiy Mar 31, 2024
7b98df8
Merge branch 'main' into feat/utxo-stake-test
Mr-Leshiy Mar 31, 2024
25a2298
try
Mr-Leshiy Mar 31, 2024
d01fe41
wip
Mr-Leshiy Mar 31, 2024
4a494ca
fix
Mr-Leshiy Mar 31, 2024
7476c3e
wip
Mr-Leshiy Mar 31, 2024
4d42d1f
try
Mr-Leshiy Mar 31, 2024
1bf8132
try
Mr-Leshiy Mar 31, 2024
4981e20
wip
Mr-Leshiy Mar 31, 2024
09cffdd
try
Mr-Leshiy Mar 31, 2024
178df34
try
Mr-Leshiy Mar 31, 2024
9368b13
revert
Mr-Leshiy Mar 31, 2024
46fa7a4
wip
Mr-Leshiy Apr 1, 2024
bbde2c7
Merge branch 'feat/utxo-stake-test' into feat/ada_stake_test
Mr-Leshiy Apr 1, 2024
81a3202
wip
Mr-Leshiy Apr 1, 2024
9fc334e
wip
Mr-Leshiy Apr 1, 2024
0351671
fix
Mr-Leshiy Apr 1, 2024
cda6204
fix
Mr-Leshiy Apr 1, 2024
03b560d
Merge branch 'feat/utxo-stake-test' into feat/ada_stake_test
Mr-Leshiy Apr 1, 2024
884aa6d
fix
Mr-Leshiy Apr 1, 2024
5cb9adb
remove mithril_snapshot loader
Mr-Leshiy Apr 1, 2024
7ba24dd
wip
Mr-Leshiy Apr 1, 2024
5350f9a
wip
Mr-Leshiy Apr 1, 2024
f5c88bb
wip
Mr-Leshiy Apr 1, 2024
c9df0c7
wip
Mr-Leshiy Apr 2, 2024
005f14c
wip
Mr-Leshiy Apr 2, 2024
da4049a
wip
Mr-Leshiy Apr 2, 2024
53b4e3d
wip
Mr-Leshiy Apr 2, 2024
6d2f5a8
Merge branch 'main' into feat/sync_test
Mr-Leshiy Apr 2, 2024
359f42b
add stake addr bech32 encode utils function
Mr-Leshiy Apr 3, 2024
840844f
wip
Mr-Leshiy Apr 3, 2024
4473898
wip
Mr-Leshiy Apr 3, 2024
e61ee23
update indexing of the utxo data
Mr-Leshiy Apr 3, 2024
617f8e0
fix spelling
Mr-Leshiy Apr 3, 2024
fbe584e
wip
Mr-Leshiy Apr 4, 2024
4569945
wip
Mr-Leshiy Apr 4, 2024
f33e52a
Merge branch 'main' into feat/sync_test
Mr-Leshiy Apr 4, 2024
556fe69
finish utxo test
Mr-Leshiy Apr 4, 2024
fc38366
fix deny
Mr-Leshiy Apr 4, 2024
4c93bf7
fix check
Mr-Leshiy Apr 4, 2024
a050be7
fix
Mr-Leshiy Apr 4, 2024
b260868
fix
Mr-Leshiy Apr 4, 2024
23fe829
update earthly builder versions
Mr-Leshiy Apr 4, 2024
ffeb3f3
wip
Mr-Leshiy Apr 4, 2024
3acbc27
Merge branch 'main' into feat/sync_test
Mr-Leshiy Apr 5, 2024
f64e9f0
ignore test_utxo.py in CI
Mr-Leshiy Apr 5, 2024
6cb52e1
dont ignore tests
Mr-Leshiy Apr 5, 2024
ba4e117
add date_time_to_slot_number_get endpoint
Mr-Leshiy Apr 5, 2024
6b450b5
add sql queries
Mr-Leshiy Apr 5, 2024
98c6d59
fix
Mr-Leshiy Apr 5, 2024
a966b82
update slot info, fix follower indexing block time issue
Mr-Leshiy Apr 5, 2024
e0cbd89
add previous slot info field
Mr-Leshiy Apr 5, 2024
ad4bd27
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 8, 2024
adc143a
fix
Mr-Leshiy Apr 8, 2024
d825f2c
refactor
Mr-Leshiy Apr 8, 2024
78ef30c
fix sync_state_get
Mr-Leshiy Apr 8, 2024
fd7c12d
wip
Mr-Leshiy Apr 8, 2024
4b9f5e6
fix check
Mr-Leshiy Apr 8, 2024
69e0184
try
Mr-Leshiy Apr 8, 2024
b327f44
fix
Mr-Leshiy Apr 8, 2024
d232211
finish slot_info test, fix queries
Mr-Leshiy Apr 8, 2024
51d5718
fix
Mr-Leshiy Apr 8, 2024
d3a2197
cleanup
Mr-Leshiy Apr 8, 2024
1b2528b
wip
Mr-Leshiy Apr 8, 2024
45305ca
wip
Mr-Leshiy Apr 8, 2024
b67ff65
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 8, 2024
dac3cdc
wip
Mr-Leshiy Apr 8, 2024
136f691
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 8, 2024
94fb538
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 9, 2024
fb306f7
increase max_response time
Mr-Leshiy Apr 9, 2024
1ffb25f
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 9, 2024
747b182
update slot_info examples
Mr-Leshiy Apr 9, 2024
d63158d
cleanup get slot info db functions
Mr-Leshiy Apr 9, 2024
e1764d3
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 9, 2024
2b5b005
wip
Mr-Leshiy Apr 9, 2024
9a8e72c
make templated sql query
Mr-Leshiy Apr 10, 2024
1c09566
refactor
Mr-Leshiy Apr 10, 2024
0d3894e
refactor
Mr-Leshiy Apr 10, 2024
96236f8
disable fuzzer-api test
Mr-Leshiy Apr 10, 2024
40cbb89
Merge branch 'main' into feat/get_slot_number
Mr-Leshiy Apr 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@ FROM debian:stable-slim

# check-markdown markdown check using catalyst-ci.
check-markdown:
DO github.com/input-output-hk/catalyst-ci/earthly/mdlint:v2.11.0+CHECK
DO github.com/input-output-hk/catalyst-ci/earthly/mdlint:v2.11.1+CHECK

# markdown-check-fix markdown check and fix using catalyst-ci.
markdown-check-fix:
LOCALLY

DO github.com/input-output-hk/catalyst-ci/earthly/mdlint:v2.11.0+MDLINT_LOCALLY --src=$(echo ${PWD}) --fix=--fix
DO github.com/input-output-hk/catalyst-ci/earthly/mdlint:v2.11.1+MDLINT_LOCALLY --src=$(echo ${PWD}) --fix=--fix

# check-spelling Check spelling in this repo inside a container.
check-spelling:
DO github.com/input-output-hk/catalyst-ci/earthly/cspell:v2.11.0+CHECK
DO github.com/input-output-hk/catalyst-ci/earthly/cspell:v2.11.1+CHECK

repo-docs:
# Create artifacts of extra files we embed inside the documentation when its built.
Expand Down
40 changes: 39 additions & 1 deletion catalyst-gateway/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion catalyst-gateway/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,11 @@ local-ip-address = "0.5.7"
gethostname = "0.4.3"
hex = "0.4.3"
async-recursion = "1.0.5"
pallas = { git = "https://github.com/input-output-hk/catalyst-pallas.git", branch = "fix/immutable-secondary", version = "0.24.0" }
handlebars = "5.1.2"
anyhow = "1.0.71"
cddl = "0.9.2"
ciborium = "0.2"
pallas = { git = "https://github.com/input-output-hk/catalyst-pallas.git", branch = "fix/immutable-secondary", version = "0.24.0" }
cardano-chain-follower= { git = "https://github.com/input-output-hk/hermes.git", version="0.0.1" }

[workspace.lints.rust]
Expand Down
2 changes: 1 addition & 1 deletion catalyst-gateway/Earthfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ VERSION --try --global-cache 0.7

# Set up our target toolchains, and copy our files.
builder:
DO github.com/input-output-hk/catalyst-ci/earthly/rust:v2.11.0+SETUP
DO github.com/input-output-hk/catalyst-ci/earthly/rust:v2.11.1+SETUP

COPY --dir .cargo .config Cargo.* clippy.toml deny.toml rustfmt.toml bin crates .
COPY --dir ./event-db/queries ./event-db/queries
Expand Down
5 changes: 3 additions & 2 deletions catalyst-gateway/bin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -66,5 +66,6 @@ async-recursion = { workspace = true }
pallas = { workspace = true }
cardano-chain-follower= { workspace = true }
anyhow = { workspace = true }
cddl = "0.9.2"
ciborium = "0.2"
handlebars = { workspace = true }
cddl = { workspace = true }
ciborium = { workspace = true }
149 changes: 109 additions & 40 deletions catalyst-gateway/bin/src/event_db/follower.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
//! Follower Queries

use cardano_chain_follower::Network;
use handlebars::Handlebars;

use crate::event_db::{Error, EventDB};

/// Block time
pub type BlockTime = chrono::DateTime<chrono::offset::Utc>;
pub type DateTime = chrono::DateTime<chrono::offset::Utc>;
/// Slot
pub type SlotNumber = i64;
/// Epoch
Expand All @@ -15,99 +16,167 @@ pub type BlockHash = String;
/// Unique follower id
pub type MachineId = String;

/// `slot_no` column name
const SLOT_NO_COLUMN: &str = "slot_no";
/// `block_hash` column name
const BLOCK_HASH_COLUMN: &str = "block_hash";
/// `block_time` column name
const BLOCK_TIME_COLUMN: &str = "block_time";
/// `ended` column name
const ENDED_COLUMN: &str = "ended";

/// `insert_slot_index.sql`
const INSERT_SLOT_INDEX_SQL: &str =
include_str!("../../../event-db/queries/follower/insert_slot_index.sql");
/// `select_slot_info_by_datetime.sql.tmpl`
const SLOT_INFO_SQL_TMPL: &str =
include_str!("../../../event-db/queries/follower/select_slot_info_by_datetime.sql.tmpl");
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved

/// Query type
pub(crate) enum SlotInfoQueryType {
/// Previous slot info query type
Previous,
/// Current slot info query type
Current,
/// Next slot info query type
Next,
}

/// Slot info query template fields
#[derive(serde::Serialize)]
struct SlotInfoQueryTmplFields {
/// `sign` field from the sql template
sign: &'static str,
/// `ordering` field from the sql template
ordering: Option<&'static str>,
}

impl SlotInfoQueryType {
/// Get SQL query
fn get_sql_query(&self) -> Result<String, Error> {
let mut reg = Handlebars::new();
// disable default `html_escape` function
// which transforms `<`, `>` symbols to `&lt`, `&gt`
reg.register_escape_fn(|s| s.into());
let query = match self {
SlotInfoQueryType::Previous => {
Mr-Leshiy marked this conversation as resolved.
Show resolved Hide resolved
reg.render_template(SLOT_INFO_SQL_TMPL, &SlotInfoQueryTmplFields {
sign: "<",
ordering: Some("DESC"),
})
},
SlotInfoQueryType::Current => {
reg.render_template(SLOT_INFO_SQL_TMPL, &SlotInfoQueryTmplFields {
sign: "=",
ordering: None,
})
},
SlotInfoQueryType::Next => {
reg.render_template(SLOT_INFO_SQL_TMPL, &SlotInfoQueryTmplFields {
sign: ">",
ordering: None,
})
},
};
query.map_err(|e| Error::Unknown(e.to_string()))
}
}

impl EventDB {
/// Index follower block stream
pub(crate) async fn index_follower_data(
&self, slot_no: SlotNumber, network: Network, epoch_no: EpochNumber, block_time: BlockTime,
&self, slot_no: SlotNumber, network: Network, epoch_no: EpochNumber, block_time: DateTime,
block_hash: BlockHash,
) -> Result<(), Error> {
let conn = self.pool.get().await?;

let network = match network {
Network::Mainnet => "mainnet".to_string(),
Network::Preview => "preview".to_string(),
Network::Preprod => "preprod".to_string(),
Network::Testnet => "testnet".to_string(),
};

let _rows = conn
.query(
include_str!("../../../event-db/queries/follower/insert_slot_index.sql"),
&[
&slot_no,
&network,
&epoch_no,
&block_time,
&hex::decode(block_hash).map_err(|e| Error::DecodeHex(e.to_string()))?,
],
)
.query(INSERT_SLOT_INDEX_SQL, &[
&slot_no,
&network.to_string(),
&epoch_no,
&block_time,
&hex::decode(block_hash).map_err(|e| Error::DecodeHex(e.to_string()))?,
])
.await?;

Ok(())
}

/// Get slot info for the provided date-time and network and query type
pub(crate) async fn get_slot_info(
&self, date_time: DateTime, network: Network, query_type: SlotInfoQueryType,
) -> Result<(SlotNumber, BlockHash, DateTime), Error> {
let conn = self.pool.get().await?;

let rows = conn
.query(&query_type.get_sql_query()?, &[
&network.to_string(),
&date_time,
])
.await?;

let Some(row) = rows.first() else {
return Err(Error::NotFound);
};

let slot_number: SlotNumber = row.try_get(SLOT_NO_COLUMN)?;
let block_hash = hex::encode(row.try_get::<_, Vec<u8>>(BLOCK_HASH_COLUMN)?);
let block_time = row.try_get(BLOCK_TIME_COLUMN)?;
Ok((slot_number, block_hash, block_time))
}

/// Check when last update occurred.
/// Start follower from where previous follower left off.
pub(crate) async fn last_updated_metadata(
&self, network: Network,
) -> Result<(SlotNumber, BlockHash, BlockTime), Error> {
) -> Result<(SlotNumber, BlockHash, DateTime), Error> {
let conn = self.pool.get().await?;

let network = match network {
Network::Mainnet => "mainnet".to_string(),
Network::Preview => "preview".to_string(),
Network::Preprod => "preprod".to_string(),
Network::Testnet => "testnet".to_string(),
};

let rows = conn
.query(
include_str!("../../../event-db/queries/follower/select_update_state.sql"),
&[&network],
&[&network.to_string()],
)
.await?;

let Some(row) = rows.first() else {
return Err(Error::NotFound);
};

let slot_no = row.try_get("slot_no")?;
let block_hash = hex::encode(row.try_get::<_, Vec<u8>>("block_hash")?);
let last_updated = row.try_get("ended")?;
let slot_no = row.try_get(SLOT_NO_COLUMN)?;
let block_hash = hex::encode(row.try_get::<_, Vec<u8>>(BLOCK_HASH_COLUMN)?);
let last_updated = row.try_get(ENDED_COLUMN)?;

Ok((slot_no, block_hash, last_updated))
}

/// Mark point in time where the last follower finished indexing in order for future
/// followers to pick up from this point
pub(crate) async fn refresh_last_updated(
&self, last_updated: BlockTime, slot_no: SlotNumber, block_hash: BlockHash,
&self, last_updated: DateTime, slot_no: SlotNumber, block_hash: BlockHash,
network: Network, machine_id: &MachineId,
) -> Result<(), Error> {
let conn = self.pool.get().await?;

// Rollback or update
let update = true;

let (id, network) = match network {
Network::Mainnet => (1, "mainnet".to_string()),
Network::Preview => (2, "preview".to_string()),
Network::Preprod => (3, "preprod".to_string()),
Network::Testnet => (4, "testnet".to_string()),
};
let network_id: u64 = network.into();

// An insert only happens once when there is no update metadata available
// All future additions are just updates on ended, slot_no and block_hash
let _rows = conn
.query(
include_str!("../../../event-db/queries/follower/insert_update_state.sql"),
&[
&i64::from(id),
&i64::try_from(network_id)
.map_err(|_| Error::Unknown("Network id out of range".to_string()))?,
&last_updated,
&last_updated,
&machine_id,
&slot_no,
&network,
&network.to_string(),
&hex::decode(block_hash).map_err(|e| Error::DecodeHex(e.to_string()))?,
&update,
],
Expand Down
18 changes: 2 additions & 16 deletions catalyst-gateway/bin/src/event_db/utxo.rs
Original file line number Diff line number Diff line change
Expand Up @@ -82,17 +82,10 @@ impl EventDB {
) -> Result<(), Error> {
let conn = self.pool.get().await?;

let network = match network {
Network::Mainnet => "mainnet".to_string(),
Network::Preview => "preview".to_string(),
Network::Preprod => "preprod".to_string(),
Network::Testnet => "testnet".to_string(),
};

let _rows = conn
.query(
include_str!("../../../event-db/queries/utxo/insert_txn_index.sql"),
&[&tx_id, &slot_no, &network],
&[&tx_id, &slot_no, &network.to_string()],
)
.await?;

Expand All @@ -105,17 +98,10 @@ impl EventDB {
) -> Result<(StakeAmount, SlotNumber), Error> {
let conn = self.pool.get().await?;

let network = match network {
Network::Mainnet => "mainnet".to_string(),
Network::Preview => "preview".to_string(),
Network::Preprod => "preprod".to_string(),
Network::Testnet => "testnet".to_string(),
};

let row = conn
.query_one(
include_str!("../../../event-db/queries/utxo/select_total_utxo_amount.sql"),
&[&stake_credential, &network, &slot_num],
&[&stake_credential, &network.to_string(), &slot_num],
)
.await?;

Expand Down
Loading
Loading