Start working on docsrs-2.0

[onur]

I created a new branch (docsrs-2.0) and this is first patch against it. It's basically wiping out everything and making a fresh start. Also created a tag for 2.0 related work.

[QuietMisdreavus]

Is creating a branch with nothing in it necessary? I thought it was possible to make the changes fairly incrementally:

• Land Add database migration  #305
• Convert project to 2018 edition and idioms
• Land design discussion: build queue overhaul #301
• Break up the web server and daemon processes (by adding a daemon-only command if it doesn't already exist)
• Convert our build container from LXC to Docker (or create a new container and start using that)
• Convert the cratesfyi doc command into the new docsrs-build-agent command we talked about
• Change the build queue thread to manage containers via k8s/EKS (the docsrs-master command we talked about)
• Change the web server from Iron to some other web framework

Most of these changes are independent of each other, and most of them don't need to care about how the rest of the application is structured, meaning we can swap out components as we go. Deleting literally everything and starting from scratch without at least keeping the templates around feels like it's discarding too much.

[onur]

I strongly believe starting from scratch and porting components bit by bit is really necessary. Almost every aspect of our current system is not compatible with the new system I mentioned. And I really don't think its really worth the effort to patch current system to make it work with new one. There is so many changes I'd like to make and I think it will be much easier if we start from scratch.

For example I've been working on very core part of docs.rs for 2.0; building documentation. It is basically build_doc function right? Problem is there is even so many issues in this module. There is so many code repeats between build_doc and get_package. try!s all over the place. Metadata created and destroyed a couple of times in this function and in the other places etc. We are currently getting all crate data when we are adding a crate into database (db/add_package module). We need to dump this idea and get crate details when we are building it for new system (I'll explain this later). Making a change like this in current code base requires so many changes all over the place to make it work.

We are also not dumping master to anywhere. I think web module is the only reusable part of current docs.rs and 2.0. Not because its perfect but its just working, and we can easily get it from master when we need it. Yes we will lose commit history when we do this but I think its not really a big deal.

[QuietMisdreavus]

In that case, we might as well include "convert our use of the Cargo library to the Cargo binary and cargo_metadata crate" in the 2.0 wishlist, while we're rewriting everything.

Since we're going to be starting over anyway, i wonder if it would be better to do this on a new repository, so we don't have to coordinate over the tighter PR/CI requirements of this one. (On the other hand, it seems like CI didn't run here, so maybe it's moot.) I feel like having the work on a side branch that's totally different from the master one may create some confusion if we try to bring more people on for it. We can swap out the repo when we want to release it.

[dekellum]

2¢: You could also start with a different repo, and later copy all of that new repo's commits into this repo, on another branch, or another directory. Moving PR/issue history around may be more difficult though, not sure if that it's relevant.

3¢: You could also just keep 2.0 in its own git-tracked directory in this repo, then use git filter-branch or related to move the 2.0 code into root position later.

[QuietMisdreavus]

I've talked about it with Onur for the last few days, as they work on the initial steps in the rewrite. I think i've come around to using this new branch instead of a new repo or a branch on someone's fork:

• Having it in the main repo helps legitimize the effort and helps coordinate work (if we make sure that PRs to the docsrs-2.0 branch get visibly tagged/labeled, and they go to the correct branch)
• Making it here helps keep the contribution history in the easiest way, since we can rename the branches into place when we want to switch (as opposed to making one commit that copies everything over, or using some complicated git commands to splice in the other repo's history on top of this one)
• We don't have a high enough volume of contributions right now for the 2.0 work to get lost in existing PR noise (or vice-versa)

So i'm going to merge this. Let's start the work toward a brighter docs.rs!