Add tool to import from git index into the database

[arlosi]

Adds a new admin tool to import data from the git index into the database:

• versions table: checksum, links
• dependencies table: explicit_name

Split out from #5066
r? @Turbo87

[Turbo87]

👍

[Turbo87]

fetching git repo
HEAD is at d0f936875bafc75ffe094d5d5b65f4f99ba33a66
found 90367 crates
continue? yes
Error: failed to update crate: Crate { name: "ak", vers: "0.1.1", deps: [Dependency { name: "async-trait", req: "^0.1.13", features: [], optional: false, default_features: true, target: None, kind: Some(Normal), package: None }, Dependency { name: "codegen", req: "^0.1.0", features: [], optional: false, default_features: true, target: None, kind: Some(Normal), package: Some("ak-codegen") }, Dependency { name: "futures", req: "^0.3.0-alpha.19", features: ["async-await"], optional: false, default_features: true, target: None, kind: Some(Normal), package: Some("futures-preview") }, Dependency { name: "pin-utils", req: "^0.1.0-alpha.4", features: [], optional: false, default_features: true, target: None, kind: Some(Normal), package: None }, Dependency { name: "rt", req: "^0.1.0", features: [], optional: false, default_features: true, target: None, kind: Some(Normal), package: Some("ak-rt") }, Dependency { name: "tokio", req: "^0.2.0-alpha.6", features: [], optional: false, default_features: false, target: None, kind: Some(Normal), package: None }], cksum: "05f4361c1f56f791eec4ad40e20a7798ca4d2e39ca3c412f0c58d05edea3d7be", features: {}, features2: None, yanked: Some(false), links: None, v: None }

Caused by:
    NotFound

I had to restart the import after a few crates due to connection troubles. It looks like the import script is not a fan of that currently 😅

[arlosi]

Do we need to add re-try support for connection issues? Did it eventually finish? How long did it take?

[Turbo87]

How long did it take?

With --delay 10 it estimated between 20 and 40 minutes

Do we need to add re-try support for connection issues?

It was the connection from my side to heroku that was failing, so I don't think we have an option there to reestablish the connection.

Did it eventually finish?

Unfortunately not. When I ran it the second time it immediately errored out (see above). I'm not sure where exactly the NotFound came from because it was not showing a stacktrace and there was not much else in terms of context on the error. My assumption is that updating the crates in the first run caused the crate or dependency to no longer be found in the second run and erroring out then.