Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(zksync_cli): Health checkpoint improvements #3193

Merged
merged 70 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
8dd58dc
feat: add dummy node version heathcheck
manuelmauro Oct 29, 2024
98167a1
feat: add version information to healthcheck
manuelmauro Oct 30, 2024
d08ecae
refactor: simplify static health check
manuelmauro Oct 30, 2024
a058388
feat: add last migration to system_dal
manuelmauro Oct 30, 2024
a5f571b
feat: add database healthcheck
manuelmauro Oct 31, 2024
a5db49a
feat: add more information to database heathcheck
manuelmauro Oct 31, 2024
b57027c
style: format code
manuelmauro Oct 31, 2024
5d23bd6
chore: prepare sqlx queries
manuelmauro Oct 31, 2024
959542b
fix: remove outdated query file
manuelmauro Oct 31, 2024
28ecc96
feat: improve bytes encoding in healthcheck
manuelmauro Oct 31, 2024
3d7e82c
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Oct 31, 2024
de3a5e1
feat: add dummy health check tasks for state keeper and eth sender
manuelmauro Oct 31, 2024
582754b
fix: do not unwrap
manuelmauro Oct 31, 2024
1b8e50e
feat: retrieve failed L1 transactions and next operator nonce
manuelmauro Oct 31, 2024
68210aa
feat: add information on last saved/mined batches to healthcheck
manuelmauro Nov 4, 2024
2fc3da9
feat: get last miniblock number from DB
manuelmauro Nov 4, 2024
098e54f
feat: add protocol version information to healthcheck
manuelmauro Nov 4, 2024
4cbb890
feat: add last processed L1 batch to health check
manuelmauro Nov 4, 2024
f60e3a9
refactor: use SELECT MAX instead of ORDER BY
manuelmauro Nov 4, 2024
927cddb
refactor: rename LastBatchIndex to BatchNumbers
manuelmauro Nov 4, 2024
80db2ce
fix: revert code committed by mistake
manuelmauro Nov 6, 2024
2c9ca8d
feat: add config parameters for healthcheck polling intervals
manuelmauro Nov 6, 2024
98f8d72
fix: fix house keeper config from env test
manuelmauro Nov 6, 2024
988b957
fix: fix house keeper config parameters naming in unit test
manuelmauro Nov 6, 2024
293484d
fix: use u64 for failed_l1_txns
manuelmauro Nov 7, 2024
3ad3fd2
fix: return u64 in get_number_of_failed_transactions
manuelmauro Nov 7, 2024
b31233b
feat: use connection_tagged for better code instumentation
manuelmauro Nov 7, 2024
5396e5a
feat: add reactive health check to state keeper
manuelmauro Nov 7, 2024
f51785c
fix: update state keeper health at the right moment
manuelmauro Nov 7, 2024
bb5acfc
refactor: use ORDER BY to query last database migration
manuelmauro Nov 7, 2024
fd35543
feat: add reactive health check to eth sender
manuelmauro Nov 8, 2024
416ea50
style: clippy
manuelmauro Nov 8, 2024
d0ac511
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 8, 2024
8d4c176
style: move field before reserved ones
manuelmauro Nov 13, 2024
eb2fb68
refactor: rename PostgresMetricsLayer to PostgresLayer
manuelmauro Nov 13, 2024
80986e4
refactor: rename postgres_metrics_layer to postgres_layer
manuelmauro Nov 13, 2024
9bffd99
refactor: rename module postgres_layer to postgres
manuelmauro Nov 13, 2024
5e235b1
refactor: move database health check task to postgres layer
manuelmauro Nov 13, 2024
884b864
feat: implement Serialize and Deserialize directly on AggregatedActio…
manuelmauro Nov 13, 2024
c701399
refactor: make DatabaseHealthTask fields private
manuelmauro Nov 13, 2024
53cca2f
refactor: remove redundant health status updates
manuelmauro Nov 13, 2024
fe339c2
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 13, 2024
9fe98d2
feat: make health mod private
manuelmauro Nov 13, 2024
36ad7c5
refactor: remove StateKeeperTask constructor
manuelmauro Nov 13, 2024
1585d5d
refactor: use getter for health updater
manuelmauro Nov 13, 2024
f9e2ebf
refactor: clippy
manuelmauro Nov 14, 2024
26aa824
feat: add git information to RustcMetadata
manuelmauro Nov 14, 2024
f06a415
refactor: rename rustc module to binary
manuelmauro Nov 14, 2024
bdc50f4
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 14, 2024
39f3864
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 15, 2024
195aee4
refactor: remove redundant health status update
manuelmauro Nov 15, 2024
4e14447
refactor: remove redundant health check update
manuelmauro Nov 15, 2024
2acadca
fix: remove unused dependencies
manuelmauro Nov 15, 2024
d5d78fc
revert: revert formatting changes
manuelmauro Nov 15, 2024
f1d618c
Merge branch 'main' into manuel-add-more-components-to-healthcheck
Deniallugo Nov 15, 2024
ec9ae20
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 15, 2024
416c187
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 18, 2024
9964ee7
feat: use Duration for migrations' execution_time
manuelmauro Nov 18, 2024
19275b6
refactor: use same interval for Postgres metrics exporter and healthc…
manuelmauro Nov 18, 2024
8911cc2
refactor: do not split use and mod declarations
manuelmauro Nov 18, 2024
6bc90e2
refactor: use Option type instead of "unknown"
manuelmauro Nov 18, 2024
d2c983c
feat: update state keeper health from cursor right away
manuelmauro Nov 18, 2024
25d8b26
refactor: merge tx status into tx details
manuelmauro Nov 18, 2024
d7d62ec
refactor: clippy
manuelmauro Nov 18, 2024
a994cc6
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 18, 2024
93b3153
Merge branch 'main' into manuel-add-more-components-to-healthcheck
manuelmauro Nov 19, 2024
16d1358
feat: integrate binary metadata into AppHealth
manuelmauro Nov 19, 2024
b6bdda4
feat: split git metrics from rust metrics
manuelmauro Nov 21, 2024
785fe5b
style: format code
manuelmauro Nov 21, 2024
e30ebac
refactor: nit BinMetadata creation
manuelmauro Nov 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ members = [
# Test infrastructure
"core/tests/loadnext",
"core/tests/vm-benchmark",
"core/lib/bin_metadata",
]
resolver = "2"

Expand Down Expand Up @@ -275,6 +276,7 @@ zksync_health_check = { version = "0.1.0", path = "core/lib/health_check" }
zksync_l1_contract_interface = { version = "0.1.0", path = "core/lib/l1_contract_interface" }
zksync_mempool = { version = "0.1.0", path = "core/lib/mempool" }
zksync_merkle_tree = { version = "0.1.0", path = "core/lib/merkle_tree" }
zksync_bin_metadata = { version = "0.1.0", path = "core/lib/bin_metadata" }
zksync_mini_merkle_tree = { version = "0.1.0", path = "core/lib/mini_merkle_tree" }
zksync_object_store = { version = "0.1.0", path = "core/lib/object_store" }
zksync_protobuf_config = { version = "0.1.0", path = "core/lib/protobuf_config" }
Expand Down
4 changes: 2 additions & 2 deletions core/bin/external_node/src/metrics/framework.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use zksync_node_framework::{
implementations::resources::pools::{MasterPool, PoolResource},
FromContext, IntoContext, StopReceiver, Task, TaskId, WiringError, WiringLayer,
};
use zksync_shared_metrics::rustc::RUST_METRICS;
use zksync_shared_metrics::BIN_METRICS;
use zksync_types::{L1ChainId, L2ChainId, SLChainId};

use super::EN_METRICS;
Expand Down Expand Up @@ -39,7 +39,7 @@ impl WiringLayer for ExternalNodeMetricsLayer {
}

async fn wire(self, input: Self::Input) -> Result<Self::Output, WiringError> {
RUST_METRICS.initialize();
BIN_METRICS.initialize();
EN_METRICS.observe_config(
self.l1_chain_id,
self.sl_chain_id,
Expand Down
8 changes: 4 additions & 4 deletions core/bin/external_node/src/node_builder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ use zksync_node_framework::{
NodeStorageInitializerLayer,
},
pools_layer::PoolsLayerBuilder,
postgres_metrics::PostgresMetricsLayer,
postgres::PostgresLayer,
prometheus_exporter::PrometheusExporterLayer,
pruning::PruningLayer,
query_eth_client::QueryEthClientLayer,
Expand Down Expand Up @@ -125,8 +125,8 @@ impl ExternalNodeBuilder {
Ok(self)
}

fn add_postgres_metrics_layer(mut self) -> anyhow::Result<Self> {
self.node.add_layer(PostgresMetricsLayer);
fn add_postgres_layer(mut self) -> anyhow::Result<Self> {
self.node.add_layer(PostgresLayer);
Ok(self)
}

Expand Down Expand Up @@ -582,7 +582,7 @@ impl ExternalNodeBuilder {
// so until we have a dedicated component for "auxiliary" tasks,
// it's responsible for things like metrics.
self = self
.add_postgres_metrics_layer()?
.add_postgres_layer()?
.add_external_node_metrics_layer()?;
// We assign the storage initialization to the core, as it's considered to be
// the "main" component.
Expand Down
10 changes: 4 additions & 6 deletions core/bin/zksync_server/src/node_builder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ use zksync_node_framework::{
object_store::ObjectStoreLayer,
pk_signing_eth_client::PKSigningEthClientLayer,
pools_layer::PoolsLayerBuilder,
postgres_metrics::PostgresMetricsLayer,
postgres::PostgresLayer,
prometheus_exporter::PrometheusExporterLayer,
proof_data_handler::ProofDataHandlerLayer,
query_eth_client::QueryEthClientLayer,
Expand Down Expand Up @@ -138,8 +138,8 @@ impl MainNodeBuilder {
Ok(self)
}

fn add_postgres_metrics_layer(mut self) -> anyhow::Result<Self> {
self.node.add_layer(PostgresMetricsLayer);
fn add_postgres_layer(mut self) -> anyhow::Result<Self> {
self.node.add_layer(PostgresLayer);
Ok(self)
}

Expand Down Expand Up @@ -760,9 +760,7 @@ impl MainNodeBuilder {
self = self.add_eth_tx_manager_layer()?;
}
Component::Housekeeper => {
self = self
.add_house_keeper_layer()?
.add_postgres_metrics_layer()?;
self = self.add_house_keeper_layer()?.add_postgres_layer()?;
}
Component::ProofDataHandler => {
self = self.add_proof_data_handler_layer()?;
Expand Down
18 changes: 18 additions & 0 deletions core/lib/bin_metadata/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[package]
name = "zksync_bin_metadata"
version.workspace = true
edition.workspace = true
authors.workspace = true
homepage.workspace = true
repository.workspace = true
license.workspace = true
keywords.workspace = true
categories.workspace = true

[dependencies]
serde.workspace = true
vise.workspace = true
tracing.workspace = true

[build-dependencies]
rustc_version.workspace = true
72 changes: 72 additions & 0 deletions core/lib/bin_metadata/build.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
//! Build script for the external node binary.

use std::{
env, fs,
io::{self, Write},
path::Path,
process::Command,
};

use rustc_version::{Channel, LlvmVersion};

fn print_binary_meta(out: &mut impl Write) -> io::Result<()> {
let rustc_meta = rustc_version::version_meta().expect("Failed obtaining rustc metadata");

writeln!(
out,
"pub const BIN_METADATA: BinMetadata = BinMetadata {{ \
rustc_version: {semver:?}, \
rustc_commit_hash: {commit_hash:?}, \
rustc_commit_date: {commit_date:?}, \
rustc_channel: {channel:?}, \
host: {host:?}, \
llvm: {llvm:?}, \
git_branch: {git_branch:?}, \
git_revision: {git_revision:?} \
}};",
semver = rustc_meta.semver.to_string(),
commit_hash = rustc_meta.commit_hash,
commit_date = rustc_meta.commit_date,
channel = match rustc_meta.channel {
Channel::Dev => "dev",
Channel::Beta => "beta",
Channel::Nightly => "nightly",
Channel::Stable => "stable",
},
host = rustc_meta.host,
llvm = rustc_meta.llvm_version.as_ref().map(LlvmVersion::to_string),
git_branch = git_branch(),
git_revision = git_revision()
)
}

/// Outputs the current git branch as a string literal.
pub fn git_branch() -> Option<String> {
run_cmd_opt("git", &["rev-parse", "--abbrev-ref", "HEAD"])
}

/// Outputs the current git commit hash as a string literal.
pub fn git_revision() -> Option<String> {
run_cmd_opt("git", &["rev-parse", "--short", "HEAD"])
}

fn run_cmd_opt(cmd: &str, args: &[&str]) -> Option<String> {
let output = Command::new(cmd).args(args).output().ok()?;
if output.status.success() {
String::from_utf8(output.stdout)
.ok()
.map(|s| s.trim().to_string())
} else {
None
}
}

fn main() {
let out_dir = env::var("OUT_DIR").expect("`OUT_DIR` env var not set for build script");
let metadata_module_path = Path::new(&out_dir).join("metadata_values.rs");
let metadata_module =
fs::File::create(metadata_module_path).expect("cannot create metadata module");
let mut metadata_module = io::BufWriter::new(metadata_module);

print_binary_meta(&mut metadata_module).expect("failed printing binary metadata");
}
36 changes: 36 additions & 0 deletions core/lib/bin_metadata/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
use serde::Serialize;
use vise::{EncodeLabelSet, Info, Metrics};

use self::values::BIN_METADATA;

pub mod values {
use super::BinMetadata;
include!(concat!(env!("OUT_DIR"), "/metadata_values.rs"));
}

/// Metadata of the compiled binary.
#[derive(Debug, EncodeLabelSet, Serialize)]
pub struct BinMetadata {
pub rustc_version: &'static str,
pub rustc_commit_hash: Option<&'static str>,
pub rustc_commit_date: Option<&'static str>,
pub rustc_channel: &'static str,
pub host: &'static str,
pub llvm: Option<&'static str>,
pub git_branch: Option<&'static str>,
pub git_revision: Option<&'static str>,
}

#[derive(Debug, Metrics)]
#[metrics(prefix = "rust")]
Deniallugo marked this conversation as resolved.
Show resolved Hide resolved
pub struct BinMetrics {
/// General information about the compiled binary.
info: Info<BinMetadata>,
}

impl BinMetrics {
pub fn initialize(&self) {
tracing::info!("Metadata for this binary: {BIN_METADATA:?}");
self.info.set(BIN_METADATA).ok();
}
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion core/lib/dal/src/eth_sender_dal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -669,7 +669,7 @@ impl EthSenderDal<'_, '_> {
Ok(())
}

pub async fn get_number_of_failed_transactions(&mut self) -> anyhow::Result<i64> {
pub async fn get_number_of_failed_transactions(&mut self) -> anyhow::Result<u64> {
sqlx::query!(
r#"
SELECT
Expand All @@ -683,6 +683,7 @@ impl EthSenderDal<'_, '_> {
.fetch_one(self.storage.conn())
.await?
.count
.map(|c| c as u64)
.context("count field is missing")
}

Expand Down
Loading
Loading