Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Write makeSnapshot telemetry to slog #6188

Merged
merged 13 commits into from
Sep 15, 2022

Conversation

gibson042
Copy link
Member

closes: #6164

Description

Endow SnapStore instances with a now function so save can return duration and size telemetry all the way up to VatKeeper saveSnapshot where it is written into the slog.

Security Considerations

Filesystem access is generally more powerful than a timer, so this should be acceptable.

Documentation Considerations

n/a

Testing Considerations

I'm not actually testing slog contents, is that something we do?

@gibson042 gibson042 requested a review from mhofman September 13, 2022 09:44
@gibson042 gibson042 force-pushed the gibson-6164-snapshot-slog-telemetry branch 2 times, most recently from d514ce5 to 2718bdb Compare September 13, 2022 16:34
Copy link
Member

@mhofman mhofman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly stylistic nits, but I'd like to measure the hashing time since it's sequential, and we should stay consistent regarding time formatting in the slog.

packages/swing-store/src/snapStore.js Outdated Show resolved Hide resolved
packages/swing-store/src/snapStore.js Outdated Show resolved Hide resolved
packages/swing-store/src/snapStore.js Outdated Show resolved Hide resolved
packages/swing-store/src/snapStore.js Outdated Show resolved Hide resolved
packages/swing-store/src/swingStore.js Outdated Show resolved Hide resolved
packages/SwingSet/src/kernel/state/vatKeeper.js Outdated Show resolved Hide resolved
@gibson042
Copy link
Member Author

@mhofman I've taken all your suggestions. PTAL.

@@ -231,12 +237,13 @@ export function makeSnapStore(
if (fileStat) {
return freeze(info);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we stated the file, should we include the compressedByteCount with a 0 or undefined compressSeconds ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point... I think we do want to emit fields consistently, including data types. I've opted to read compressedByteCount from stat info, use 0 for compressSeconds, and introduce a boolean newFile to support exclusion of those values for accurate statistics.

packages/swing-store/src/swingStore.js Outdated Show resolved Hide resolved
async function atomicWrite(destFilePath, writeContents) {
// Atomicity requires remaining on the same filesystem,
// so we perform all operations in the destination directory.
const dir = resolve(destFilePath, '..');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use dirname?

Suggested change
const dir = resolve(destFilePath, '..');
const dir = path.dirname(destFilePath);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve is an endowment to makeSnapStore, and I didn't want to require another one or risk conflict between it and the built-in path module (however unlikely that may be). But I've restored root-relative behavior to avoid that, and this should be refactored in the performance-improving followup anyway.

@gibson042 gibson042 force-pushed the gibson-6164-snapshot-slog-telemetry branch from 48e7fd5 to fd1b27d Compare September 15, 2022 17:58
@gibson042 gibson042 requested a review from mhofman September 15, 2022 17:59
@gibson042
Copy link
Member Author

gibson042 commented Sep 15, 2022

@mhofman Back to you. And should I start work on the performance improvements, or did you plan to take that on?

Copy link
Member

@mhofman mhofman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gibson042 gibson042 added the automerge:rebase Automatically rebase updates, then merge label Sep 15, 2022
@turadg turadg force-pushed the gibson-6164-snapshot-slog-telemetry branch from 3188a60 to 45bc40d Compare September 15, 2022 19:53
@mergify mergify bot merged commit dd2e2b9 into master Sep 15, 2022
@mergify mergify bot deleted the gibson-6164-snapshot-slog-telemetry branch September 15, 2022 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge:rebase Automatically rebase updates, then merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Telemetry for Heap Snapshotting
2 participants