Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(db): fix total amount of unique blobs aggregation #666

Merged
merged 4 commits into from
Dec 12, 2024
Merged

fix(db): fix total amount of unique blobs aggregation #666

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e218060
fix(db): handle null values when updating blob's first block number
PJColombo Dec 5, 2024
db58186
fix(db): correct total unique blobs stats calculation
PJColombo Dec 5, 2024
55bd1b5
chore: add changesets
PJColombo Dec 6, 2024
23fe45b
test: update snapshot
PJColombo Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/breezy-cups-double.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@blobscan/db": patch
---

Corrected an issue in upsertMany operations where the firstBlockNumber column was not updated if it contained a null value
5 changes: 5 additions & 0 deletions .changeset/sixty-flies-own.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@blobscan/db": patch
---

Enhanced the calculation of unique blob daily aggregations to account for the uniqueness of blobs relative to previously existing blobs, rather than only within the current aggregation scope.
26 changes: 13 additions & 13 deletions clis/stats-aggregation-cli/test/commands/__snapshots__/daily.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 6,
"totalBlocks": 1,
"totalTransactions": 2,
"totalUniqueBlobs": 2,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -232,7 +232,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 6,
"totalBlocks": 1,
"totalTransactions": 2,
"totalUniqueBlobs": 2,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -258,7 +258,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 2,
"totalBlocks": 1,
"totalTransactions": 2,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 2,
"totalUniqueSenders": 2,
},
Expand All @@ -284,7 +284,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 3,
"totalBlocks": 1,
"totalTransactions": 3,
"totalUniqueBlobs": 3,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 3,
"totalUniqueSenders": 3,
},
Expand All @@ -310,7 +310,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 5,
"totalBlocks": 1,
"totalTransactions": 5,
"totalUniqueBlobs": 4,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 3,
"totalUniqueSenders": 5,
},
Expand All @@ -336,7 +336,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 1,
"totalBlocks": 1,
"totalTransactions": 1,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand Down Expand Up @@ -388,7 +388,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 1,
"totalBlocks": 1,
"totalTransactions": 1,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -414,7 +414,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 1,
"totalBlocks": 1,
"totalTransactions": 1,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -440,7 +440,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 3,
"totalBlocks": 1,
"totalTransactions": 1,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -466,7 +466,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 4,
"totalBlocks": 1,
"totalTransactions": 2,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 2,
},
Expand All @@ -492,7 +492,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 3,
"totalBlocks": 1,
"totalTransactions": 1,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 1,
"totalUniqueSenders": 1,
},
Expand All @@ -518,7 +518,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 2,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 2,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 2,
"totalUniqueSenders": 2,
},
Expand Down Expand Up @@ -570,7 +570,7 @@ exports[`Daily command > when aggregating daily stats > should aggregate all dat
"totalBlobs": 5,
"totalBlocks": 3,
"totalTransactions": 3,
"totalUniqueBlobs": 3,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 2,
"totalUniqueSenders": 3,
},
Expand Down
12 changes: 6 additions & 6 deletions clis/stats-aggregation-cli/test/commands/__snapshots__/overall.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 2,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand All @@ -73,7 +73,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 6,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 0,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand All @@ -98,7 +98,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 6,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 0,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand Down Expand Up @@ -153,7 +153,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 3,
"totalBlocks": 3,
"totalTransactions": 3,
"totalUniqueBlobs": 1,
"totalUniqueBlobs": 0,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand All @@ -178,7 +178,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 6,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 0,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand All @@ -203,7 +203,7 @@ exports[`Overall command > when incrementing overall stats > should increment th
"totalBlobs": 6,
"totalBlocks": 2,
"totalTransactions": 2,
"totalUniqueBlobs": 0,
"totalUniqueBlobs": 1,
"totalUniqueReceivers": 0,
"totalUniqueSenders": 0,
},
Expand Down
26 changes: 13 additions & 13 deletions packages/api/test/__snapshots__/stats.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
5,
],
"totalUniqueBlobs": [
3,
1,
],
}
`;
Expand All @@ -32,8 +32,8 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
5,
],
"totalUniqueBlobs": [
0,
1,
3,
],
}
`;
Expand All @@ -56,9 +56,9 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
5,
],
"totalUniqueBlobs": [
4,
1,
3,
0,
1,
],
}
`;
Expand All @@ -84,10 +84,10 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
5,
],
"totalUniqueBlobs": [
2,
4,
0,
1,
0,
1,
3,
],
}
`;
Expand Down Expand Up @@ -117,10 +117,10 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
],
"totalUniqueBlobs": [
3,
2,
4,
0,
1,
0,
1,
3,
],
}
`;
Expand Down Expand Up @@ -154,10 +154,10 @@ exports[`Stats router > getBlobDailyStats > when getting stats for a specific ti
"totalUniqueBlobs": [
3,
3,
2,
4,
0,
1,
0,
1,
3,
],
}
`;
Expand Down
4 changes: 2 additions & 2 deletions packages/api/test/stats.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ describe("Stats router", async () => {
"totalBlobs": 29,
"totalBlocks": 8,
"totalTransactions": 16,
"totalUniqueBlobs": 9,
"totalUniqueBlobs": 8,
"totalUniqueReceivers": 4,
"totalUniqueSenders": 7,
}
Expand All @@ -84,7 +84,7 @@ describe("Stats router", async () => {
{
"totalBlobSize": "422116",
"totalBlobs": 29,
"totalUniqueBlobs": 9,
"totalUniqueBlobs": 8,
}
`);
});
Expand Down
5 changes: 4 additions & 1 deletion packages/db/prisma/extensions/base.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,10 @@ export const baseExtension = Prisma.defineExtension((prisma) =>
commitment = EXCLUDED.commitment,
proof = EXCLUDED.proof,
size = EXCLUDED.size,
first_block_number = LEAST(b.first_block_number, EXCLUDED.first_block_number),
first_block_number = CASE
WHEN b.first_block_number IS NULL THEN EXCLUDED.first_block_number
ELSE LEAST(b.first_block_number, EXCLUDED.first_block_number)
END,
updated_at = NOW()
`;
},
Expand Down
6 changes: 5 additions & 1 deletion packages/db/prisma/sql/aggregateBlobDailyStats.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@ SELECT
tx.category,
tx.rollup,
COUNT(bl_tx.blob_hash)::INT AS total_blobs,
COUNT(DISTINCT b.versioned_hash)::INT AS total_unique_blobs,
COUNT(
DISTINCT CASE
WHEN b.first_block_number = bl_tx.block_number THEN bl_tx.blob_hash
END
) AS total_unique_blobs,
SUM(b.size) AS total_blob_size
FROM blob b
JOIN blobs_on_transactions bl_tx ON bl_tx.blob_hash = b.versioned_hash
Expand Down
2 changes: 1 addition & 1 deletion packages/db/prisma/sql/aggregateBlobOverallStats.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ SELECT
tx.rollup,
COALESCE(COUNT(bl_txs.blob_hash)::INT, 0) AS total_blobs,
COALESCE(
COUNT(DISTINCT CASE WHEN bl.first_block_number BETWEEN $1 AND $2 THEN bl.versioned_hash END)::INT,
COUNT(DISTINCT CASE WHEN bl.first_block_number = bl_txs.block_number THEN bl.versioned_hash END)::INT,
0
) AS total_unique_blobs,
COALESCE(SUM(bl.size), 0) AS total_blob_size,
Expand Down
18 changes: 14 additions & 4 deletions packages/db/test/base-extension.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,13 +249,23 @@ describe("Base Extension", () => {
expect(updatedBlob?.firstBlockNumber).toBe(lowerBlockNumber);
});

it("should not update a blob's first block number when new one is lower", async () => {
const higherBlockNumber = 1006;
it("should not update a blob's first block number when new one is higher", async () => {
const versionedHash =
"0x000000000000000000000000000000000000000000000000000000000000b2";

const blob = await prisma.blob.findUniqueOrThrow({
where: {
versionedHash,
},
});

await prisma.blob.upsertMany([
{
...oldBlob,
firstBlockNumber: higherBlockNumber,
commitment: blob.commitment,
proof: blob.proof,
size: blob.size,
versionedHash: blob.versionedHash,
firstBlockNumber: (blob?.firstBlockNumber ?? 0) + 1,
},
]);

Expand Down
13 changes: 5 additions & 8 deletions packages/db/test/stats-extension.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,14 +383,11 @@ describe("Stats Extension", () => {
BigInt(0)
);
const totalUniqueBlobs = new Set(
txBlobs
.filter(({ firstBlockNumber }) =>
blockNumberRange && firstBlockNumber
? firstBlockNumber >= blockNumberRange.from &&
firstBlockNumber <= blockNumberRange.to
: true
)
.map(({ versionedHash }) => versionedHash)
transactions.flatMap((tx) =>
tx.blobs
.filter((b) => b.firstBlockNumber === tx.blockNumber)
.map((b) => b.versionedHash)
)
).size;
const totalBlocks = new Set(transactions.map((tx) => tx.blockNumber)).size;
const totalTransactions = transactions.length;
Expand Down
Loading
Loading