determine status of /api/v1/index/retrieve endpoint before Rekor 1.0 release

[bobcallaway]

Description

The current implementation of searching the log index is still "proof-of-concept" quality:

• Indices are inserted into the index in a best-effort fashion (as there is no transaction support for atomically committing to both the index and the transaction log currently)
• If there are errors generating index keys for an entry, those errors are logged but are not exposed to clients.
• There is no background job presently that attempts to ensure that all possible indices for a logged entry are in the index

Proposal

I believe the right long term design likely involves emitting an event from Rekor to an external pub/sub system and building an index based on that event, versus having a tight coupling within the Rekor runtime.

I would propose that we either:

• mark the endpoint as experimental and look to re-implement this capability in a future release (ideally quickly post 1.0); the experimental flag would mean that there are no security or consistency guarantees from this API endpoint, and that we reserve the right to change/remove this endpoint as deemed appropriate in the future.
• remove the /api/v1/index/retrieve endpoint completely before Rekor 1.0

@asraa @priyawadhwa @dlorenc @SantiagoTorres @lukehinds @var-sdk @haydentherapper

[priyawadhwa]

I don't have a strong opinion, am good with either option. Slightly leaning towards removing it, as it might make it easier to implement the long term design later on and doesn't come with any sort of time pressure to do it quickly post 1.0

[haydentherapper]

I'm hesitant to entirely remove search before GA because there is client reliance on the search capabilities. If we want to at least separate search into its own repo, we could remove it from the repo, operate it as its own "experimental" lightweight service, and best-effort push entries from Rekor to the service (effectively having the same guarantees as now, but with the ability to iterate). Search could be served on the same API endpoint to not break clients.

Long term, I am in agreement that search should be removed from Rekor and maintained separately. Sigstore should maintain its own search cluster with a minimal set of indices in my opinion, and we should encourage the community to operate more complex search capabilities in their own instances. I'd also like to make search indices verifiable.

Implementation details to chat about later: I think it can be implemented either with pub/sub to entirely decouple it from Rekor, or having log monitors also maintain search indices by requesting all entries between two checkpoints that have been verified for consistency.

[dlorenc]

having log monitors also maintain search indices by requesting all entries between two checkpoints that have been verified for consistency.

I like having log monitors back the searches, they can provide more guarantees than pub/sub would be able to. But I also like the idea of having pub/sub support for other use cases!

[priyawadhwa]

Shall we go with marking it experimental for now then?

[bobcallaway]

it seems like we have consensus to mark as experimental.