Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: stream JLAP repodata writes #891

Merged
merged 5 commits into from
Oct 7, 2024
Merged

refactor: stream JLAP repodata writes #891

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fd647d4
refactor: stream JLAP repodata reads and writes
AaronOpfer Oct 4, 2024
0d7b4ad
put back fs_err
AaronOpfer Oct 4, 2024
19ce3cb
switch to memmap; apply PR comments
AaronOpfer Oct 5, 2024
c4a5d9a
revert memmap changes and add std::mem::drop
AaronOpfer Oct 6, 2024
cdc1239
Update crates/rattler_repodata_gateway/Cargo.toml
baszalmstra Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions crates/rattler_repodata_gateway/src/fetch/jlap/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,6 @@ use reqwest_middleware::ClientWithMiddleware;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use serde_with::serde_as;
use std::io::Write;
use std::iter::Iterator;
use std::path::Path;
use std::str;
Expand Down Expand Up @@ -537,13 +536,13 @@ fn apply_jlap_patches(
reporter.on_jlap_decode_start(index);
}

tracing::info!("parsing cached repodata.json as JSON");
// Read the contents of the current repodata to a string
let repo_data_contents = fs::read_to_string(repo_data_path).map_err(JLAPError::FileSystem)?;
let repo_data_file = fs::File::open(repo_data_path).map_err(JLAPError::FileSystem)?;
let repo_data_reader = std::io::BufReader::with_capacity(64 * 1024, repo_data_file);

// Parse the JSON so we can manipulate it
tracing::info!("parsing cached repodata.json as JSON");
let mut repo_data =
serde_json::from_str::<Value>(&repo_data_contents).map_err(JLAPError::JSONParse)?;
let mut repo_data: Value =
serde_json::from_reader(repo_data_reader).map_err(JLAPError::JSONParse)?;
AaronOpfer marked this conversation as resolved.
Show resolved Hide resolved

if let Some((reporter, index)) = report {
reporter.on_jlap_decode_completed(index);
Expand All @@ -569,9 +568,6 @@ fn apply_jlap_patches(
reporter.on_jlap_encode_start(index);
}

// Convert the json to bytes, but we don't really care about formatting.
let updated_json = serde_json::to_string(&repo_data).map_err(JLAPError::JSONParse)?;

// Write the content to disk and immediately compute the hash of the file contents.
tracing::info!("writing patched repodata to disk");
let mut hashing_writer = NamedTempFile::new_in(
Expand All @@ -581,9 +577,12 @@ fn apply_jlap_patches(
)
.map_err(JLAPError::FileSystem)
.map(rattler_digest::HashingWriter::<_, Blake2b256>::new)?;
hashing_writer
.write_all(&updated_json.into_bytes())
.map_err(JLAPError::FileSystem)?;
serde_json::to_writer(
std::io::BufWriter::with_capacity(64 * 1024, &mut hashing_writer),
AaronOpfer marked this conversation as resolved.
Show resolved Hide resolved
&repo_data,
)
.map_err(JLAPError::JSONParse)?;

let (file, hash) = hashing_writer.finalize();
file.persist(repo_data_path)
.map_err(|e| JLAPError::FileSystem(e.error))?;
Expand Down