The remoting client currently buffers all fetched blobs in memory before storing to LMDB.

[jsirois]

That's here:

pants/src/rust/engine/fs/store/src/remote.rs

Lines 383 to 402 in d8fba9a

	let read_result_closure = async {
	let mut buf = BytesMut::with_capacity(digest.size_bytes);
	while let Some(response) = stream.next().await {
	// Record the observed time to receive the first response for this read.
	if let Some(start) = start_opt.take() {
	if let Some(workunit_store_handle) = workunit_store::get_workunit_store_handle() {
	let timing: Result<u64, _> =
	Instant::now().duration_since(start).as_micros().try_into();
	if let Ok(obs) = timing {
	workunit_store_handle
	.store
	.record_observation(ObservationMetric::RemoteStoreTimeToFirstByteMicros, obs);
	}
	}
	}
	buf.extend_from_slice(&(response?).data);
	}
	Ok(buf.freeze())
	};

For things like ~GB pytorch wheel zips; that could be problematic.

[stuhood]

One option would be to store into a temporary file, and then copy into LMDB with:

pants/src/rust/engine/fs/store/src/local.rs

Lines 317 to 331 in abd3bfe

	///
	/// Store data in two passes, without buffering it entirely into memory. Prefer
	/// `Self::store_bytes` for small values which fit comfortably in memory.
	///
	pub async fn store<F, R>(
	&self,
	entry_type: EntryType,
	initial_lease: bool,
	data_is_immutable: bool,
	data_provider: F,
	) -> Result<Digest, String>
	where
	R: Read + Debug,
	F: Fn() -> Result<R, io::Error> + Send + 'static,
	{

But that would use more passes over the data than we strictly need. We do need to re-compute and validate the Digest of the data (because we don't immediately trust the data that we get over the wire), but the Digest could be computed while writing the data to the temporary file, rather than via reading it in two passes as fn store will do.

[stuhood]

One other much more fundamental idea would be to actually begin to "size split" our local::Store, and to store files larger than a threshold as files on disk. LMDB is great for small files and directory entries, but it's advantages diminish for files which are too large to buffer in RAM.

That would have advantages for this codepath, but it would also have advantages for #17282, because if we chose our heuristics well, we could symlink directly from a large-file store, rather than first copying a large file out of the store and into a real file. cc @thejcannon

[stuhood]

One other much more fundamental idea would be to actually begin to "size split" our local::Store, and to store files larger than a threshold as files on disk. LMDB is great for small files and directory entries, but it's advantages diminish for files which are too large to buffer in RAM.

That would have advantages for this codepath, but it would also have advantages for #17282, because if we chose our heuristics well, we could symlink directly from a large-file store, rather than first copying a large file out of the store and into a real file. cc @thejcannon

Opened #18048 for this idea.

[huonw]

But that would use more passes over the data than we strictly need. We do need to re-compute and validate the Digest of the data (because we don't immediately trust the data that we get over the wire), but the Digest could be computed while writing the data to the temporary file, rather than via reading it in two passes as fn store will do.

I've opened #18231 for this, since #18054 solves the basic "avoid buffering into memory" issue, without optimising the hashing.