fix: prevent multiple downloads of same file #2477
Conversation
|
It makes sense to me, but I don't know if I am really the best authority. My Rust knowledge is just enough to be dangerous. |
|
@ry could you take a look at this PR? Tests are failing but I believe they are not replicating "real life" scenario - I don't think it's possible to change Also I had other idea: update |
bartlomieju
force-pushed
the
test-prevent_multiple_downloads
branch
from
4889a5c to
d14adf2
Compare
|
@ry I think I managed to find solution using initial approach that involves marking which URLs had already been downloaded. It's much simpler compared to saving whole response. I changed two tests to behave like a real world scenario and added a test for multiple downloads (verifying that headers file modified timestamp is unchanged). Let me know what you think. But as you said earlier this is definitely to be rewritten from ground up. |
bartlomieju
changed the title
| let headers_file_metadata_2 = headers_file_2.metadata().unwrap(); | ||
| let headers_file_modified_2 = headers_file_metadata_2.modified().unwrap(); | ||
|
|
||
| assert_eq!(headers_file_modified, headers_file_modified_2); |
There was a problem hiding this comment.
Very nice! I've been trying to come up with a test for this failure for a long time. I have verified this test does indeed fail when applied to master.
Thank you for solving this bug!
There was a problem hiding this comment.
Very nice! I've been trying to come up with a test for this failure for a long time. I have verified this test does indeed fail when applied to master.
Thank you for solving this bug!
Great, thanks! 🎉
This PR is proof-of-concept to prevent multiple downloads of same file as described in #2442.
For example given in #2442, here's downloads before this patch:
And after patch:
It builds on idea of
State.compiled- once file is downloaded it's placed inHashSetto prevent another download of same file. Code is hacky to prevent necessity of huge changes inDenoDir- I don't intend this to be merged but rather be another POV for refactor of DenoDir.CC @kitsonk
Fixes #2442