add "priority" to the build queue, and decouple builds from reading new crates #344
Conversation
| let mut doc_builder = DocBuilder::new(opts); | ||
|
|
||
| /// Represents the current state of the builder thread. | ||
| enum BuilderState { |
There was a problem hiding this comment.
When i initially wrote this enum, i had thought that the logic would be more complicated than it wound up being. This can probably be reduced to an Option<usize>, but i left this state enum in to make the usage code easier to read.
There was a problem hiding this comment.
Update: I tried to reduce this to a usize and then to an Option<usize>, and proceeded to introduce bugs into the implementation. I think i'll keep this state enum as-is, since it provides a more readable implementation, IMO. (It'll take up the same space on the stack as an Option<usize> regardless.)
| debug!("Queue is empty, going back to sleep"); | ||
| continue; | ||
| } | ||
| if let Err(e) = doc_builder.load_cache() { |
There was a problem hiding this comment.
This load/save dance is here in case the on-disk cache has been updated since the builder thread has last saved it. Since it's represented in-memory as a BTreeSet, reloading everything in doesn't create any duplicates in the cache when it's saved into disk later.
QuietMisdreavus
force-pushed
the
build-queue-priority
branch
from
8a83cc3 to
5ab05c9
Compare
QuietMisdreavus
force-pushed
the
build-queue-priority
branch
from
5ab05c9 to
94cf10f
Compare
This PR implements the first couple steps of #301, paving the way for manually batching builds while still allowing for new crate releases to be posted in a timely manner.
The largest change is the first commit, which breaks up the queue thread in the daemon into two: one which loads new crates from crates.io once a minute, and one which checks the build queue once a minute to start builds. This turned out to be a little more complicated than i initially imagined, because of the way the
DocBuildercaches what has been built already. Juggling the caches made it a little more awkward than it otherwise would have been, but i think it turned out alright.This alone allows docs.rs to avoid the confusing scenario where a long-running build job jams up the queue and makes it appear like the builder died (whereas now the queue will start accumulating entries and the builder will merely appear stuck, which is more reflective of reality). cc #335
The later commits are a logically-distinct change which is the core of #301 - the notion of a "priority" in the build queue, and a change in how builds are sorted. This allows us to queue up a batch of rebuilds without blocking new crate releases from being built.
Finally, for ease of maintenance, i added a
cratesfyi queue addcommand, which allows administrators to add a crate to the build queue without having to manually enter a row into the database for it. (This means we can easily wire up a script for it, a la the currentbuild-from-psql-select.) By default, crates entered through this command get priority 5 (which means they'll be sorted below the priority 0 that new crates get), though there is an available--priorityflag you can use to override that. Since the column (and the types used in the code) are signed, it's technically possible to add queue entries above new crates, so some care is necessary. However, this is a much easier scheme than the current "lock the world so we can run builds outside the main log" system.