Improve the OSV.dev database situation #3850
Description
Tracking issue for database improvements.
In short, the Bug Datastore entity is unwieldy and affecting our ability to make performance improvements. We'd also like to move away from our dependence on Datastore.
We want to store full vulnerabilities in GCS, with smaller query indexes in Datastore to improve things.
Additionally, we want to separate vulnerability ingestion from enrichment (e.g. computing affected versions and commits), to reduce the delay between vulns being published upstream and them being available in OSV.dev.
| Task | Test | Prod | |
|---|---|---|---|
| Create & start populating new Datastore entities and GCS objects | #3708 | #4011 | ☑️ |
| Ensure consistency between GCS and Datastore entities | #3821 #3957 | #4011 | ☑️ |
| Migrate API to new entities | #3776 | #4058 | ☑️ |
| Migrate website to new entities | #4469 | #4469 | |
| Change importer to create entities before sending tasks to worker | #4344 | #4344 | ☑️ |
| Change exporter to read/write from new entities | #4197 | #4197 | ☑️ |
| Change alias/upstream cron to read from new entities | #4395 | #4395 | ☑️ |
| Split OSS-Fuzz-specific handling away from main importer/worker | N/A | #4041 |