Crates that are fetched but not added to the index because of a crash will be lost

[jyn514]

Secondary problem: apparently crates that are fetched but not added to the index will be lost and not build. Relevant code: https://github.com/rust-lang/docs.rs/blob/master/src/docbuilder/queue.rs#L13

Originally posted by @jyn514 in #626 (comment)

crates-index-diff just released a change in 5.1.0 that will let us peek without consuming the changes 🎉 🎉 . This will give us much better durability in the event of a crash, I just need to implement using the new API.

[jyn514]

Mentoring instructions:

• In get_new_crates(), change the line from using fetch_changes() to using peek_changes().
• After adding all new crates to the queue, call index.last_seen_reference().set_target(changes), as per the instructions in the changelog. See also git2::Reference.
• Optionally, change add_crate_to_queue to add multiple crates at once, so the operation will be atomic and we won't add crates that we've already built if we crash halfway through adding crates.

[harrisonbrock]

If this is still open, I would like to work on it.

[jyn514]

@harrisonbrock go for it! If you have questions, feel free to reach out here or on Discord :)

[jyn514]

There was a new API added recently that will make this a little easier: https://docs.rs/crates-index-diff/6.2.0/crates_index_diff/struct.Index.html#method.set_last_seen_reference

Also note that we currently use crates-index-diff 4.0, so it will need to be upgraded to at least 5 (preferably 6) to use this new API.

[harrisonbrock]

Thanks. I have been a little sick and have not had a chance to start yet.

[jyn514]

Get well soon! Your health is more important than code :)

[Byron]

I hope you are doing better, @harrisonbrock!
In case you need additional support related to how to use crates-index-diff, I am happy to help (I have authored crates-index-diff).

[jyn514]

#746 only logged the failure, it will still not be added to the index.

[jyn514]

Ok, this is a slightly different issue than originally - we record the failure internally, it won't be lost, we just don't display it anywhere but in the logs. So it seems from the outside as if the build was never there, but we can still rerun it with UPDATE queue SET attempts = 0;

[jyn514]

Closing in favor of #903, the original bug this was for has been fixed.