Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement inputs and outputs hash based task skipping #933

Merged
merged 17 commits into from
Mar 15, 2024

Conversation

wolfv
Copy link
Member

@wolfv wolfv commented Mar 8, 2024

TODO:

  • Glob needs to accept folder

@ruben-arts ruben-arts changed the title implement sources / generates hash based task skipping [wip] [skip ci] feature: implement inputs and outputs hash based task skipping Mar 8, 2024
@ruben-arts ruben-arts changed the title feature: implement inputs and outputs hash based task skipping feat: implement inputs and outputs hash based task skipping Mar 8, 2024
@wolfv wolfv marked this pull request as ready for review March 8, 2024 22:37
@wolfv wolfv force-pushed the task-skip branch 2 times, most recently from 92bf1f5 to 46ae4d5 Compare March 10, 2024 10:31
src/consts.rs Outdated Show resolved Hide resolved
src/task/executable_task.rs Outdated Show resolved Hide resolved
src/task/executable_task.rs Outdated Show resolved Hide resolved
@wolfv
Copy link
Member Author

wolfv commented Mar 10, 2024

One thing I noted is that "globbing" folders doesn't work (e.g. build/). I think build/** will work just fine but we should also fix the mentioned case before merging.

@ruben-arts
Copy link
Contributor

What doesn't work for me in the cpp-sdl example is:

pixi run start
rm .build -rf
pixi run start

@wolfv
Copy link
Member Author

wolfv commented Mar 11, 2024

I fixed that issue - we were taking into account any .gitignore files as well as ignoring hidden folders.

@ruben-arts ruben-arts linked an issue Mar 12, 2024 that may be closed by this pull request
@wolfv
Copy link
Member Author

wolfv commented Mar 13, 2024

We now store the task cache under .pixi/task-cache/<env-name>-<task>.json.
It's just a "forward" mapping of these two variables, so any - in the env name pose no problem.

Things we should consider:

  • changing the name of the task or environment will invalidate the cache - might be annoying when downloading large files?
  • We could think about a fast-cache mode where we only check the existence of files

Maybe we should spend a few minutes thinking about these items, but IMO it's good to merge now.

@ruben-arts
Copy link
Contributor

ruben-arts commented Mar 14, 2024

What doesn't work for me in the cpp-sdl example is:

pixi run start
rm .build -rf
pixi run start

This doesn't work yet. I tried adding outputs = [".build"] to the configure task but that didn't help.

It will error on not having the .build/bin/sdl_example in the start task but and the cache will be hit for the configure and build

Edit: This works now but only with .build/ instead of .build which I would like to both be working as it would have taken me some time to debug this.

@ruben-arts ruben-arts requested a review from baszalmstra March 14, 2024 13:19
src/task/executable_task.rs Outdated Show resolved Hide resolved
src/task/executable_task.rs Outdated Show resolved Hide resolved
src/task/executable_task.rs Outdated Show resolved Hide resolved
};

let files =
FileHashes::from_files(&task.working_directory().unwrap(), inputs.iter()).await?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will unwrap when a user accidentally add as file instead of a dir in the cwd

};

let files =
FileHashes::from_files(&task.working_directory().unwrap(), outputs.iter()).await?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here aswell

@ruben-arts ruben-arts enabled auto-merge (squash) March 15, 2024 15:26
@wolfv wolfv disabled auto-merge March 15, 2024 16:52
@wolfv wolfv merged commit 9fb8cae into prefix-dev:main Mar 15, 2024
18 checks passed
@wolfv wolfv deleted the task-skip branch March 15, 2024 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add generates and sources directives to tasks for caching
3 participants