Returning index_state during index report creation

[crozzy]

Problem

Clair purposely does not have a mechanism for re-indexing, we just surface the index_state. This means that integrations must have their own re-submission logic by judging the index_state. In Quay's case the process looks something like:

• There is a table that holds indexing statuses (manifest, error/success, index_state)
• When the security worker starts it grabs the /index_state and looks up all manifests that don't have that state
• Quay submits those to Clair and updates the DB row with the new index_state (gleaned from the original call to /index_state)

This leads to the possibility of a race condition if the index_state changes after the initial state request and before the worker finishes. Under this condition, Quay would save the wrong state with the manifest, this is somewhat mitigated against as the next worker run will recognize the manifest as a candidate for re-indexing and save the new state.

e.g

• Quay worker gets the /index_state (a)
• It identifies manifests that don't have this index_state (!a)
• It starts indexing these manifests
• A new version of Clair is rolled out
• Different indexing requests hit different indexer pods with distinct Clair versions (and for the sake of argument different index_states (a and b)
• The states are saved back to the Quay DB identifying all the indexed manifests as having the original state (a) despite some of them having been indexed with the new state (b)

Possible solution

To avoid the "rollout-cornercase", we could return the index_state with the index_report creation, either as a header or as a part of the index_report.

Open Questions

• Should it be a part of report creation and report retrieval?
• If that's a yes, then I think it would need to be a part of the index_report so it is persisted. That would also give one the ability to see from a Clair context how consistent manifests are with the current state (although you might have to parse JSON depending on how it'd be implemented).

[hdonnay]

I don't think it should be part of the index report. Internally, an index's "freshness" is a function of the set of indexers one is using. I think that's the important invariant, instead of the indexer state.

I think reporting the indexer state via an HTTP header makes a lot of sense, there is/was a scheme to try to make it work via Etag, which I'd need to page-in again.

As far as the consistency issue goes, I think it's much easier to handle at the HTTP layer at the load balancer. A cut over and cache expiration scheme would paper over a lot of this, I think.

[paulaldridge]

Just to make sure I understand correctly, is the set of indexers/scanners (including their versions) not represented by the index state? Is there another way you could discover an index reports freshness?

How I'm understanding it is that there currently is no way of seeing if an index report is up to date, as the state it was indexed at isn't stored. Whereas if you stored the index state in the index report you could refer to that.

[paulaldridge]

One slightly worse scenario of the rollout cornercase I can see is if:

• A new version of Clair is in the middle of rolling out, with some clair instances updated and some yet to be
• worker gets the /index_state of the new version (b)
• It identifies manifests that don't have this index_state (!b)
• It starts indexing these manifests
• Different indexing requests hit different indexer pods with distinct Clair versions (and for the sake of argument different index_states (a and b))
• The states are saved back to the DB identifying all the indexed manifests as having the new state (b) despite some of them having been indexed with the old state (a)