[Execution Node] Evaluate compressing the checkpoint file or some types of data in checkpoint file to reduce file size

[fxamacker]

Problem

The checkpoint file size is growing fast.

PR #1944 started looking into compressing the checkpoint file but that work was postponed to tackle higher priority items.

Updates #1744

The Proposed Solution

Continue evaluating ways to compress the entire checkpoint and/or some types of data in the checkpoint. Also use concurrency to reduce the performance hit from using compression.

OPTIONAL: Maybe also revisit the possibility of splitting the checkpoint file into multiple files and having a metadata file with filenames, checksums, and dates.

[fxamacker]

Maybe also evaluate transparent folder or file system compression instead of reinventing the wheel.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.