[Feature/multi_tenancy] Add ifSeqNo and ifPrimaryTerm for update concurrency

[dbwiddis]

Description

Tracks SeqNo and PrimaryTerm used for strong concurrency for updates.

• Adds to update requests
• Implements tracking of this version in DynamoDB

Still TODO (@arjunkumargiri will do in another PR):

• DDB: Initialize SeqNo to 0 for new documents, but increment on overwrite
• DDB: Increment on successful update
• DDB: Return -2 on delete

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[arjunkumargiri]

In case of delete, why does seq no be updated? can we safely delete document as long as seq no matches?

[dbwiddis]

@arjunkumargiri This is somewhat of an artifact of how "soft deletes" work in OpenSearch.

When you call the Delete API, it only marks the document as deleted in the metadata but it doesn't actually remove it until the segment is merged. It's actually still available in search results unless you use the refresh parameter.

So it still tracks the seq_no as a change to the document for replication/tracking purposes until the document actually goes away on a merge, at which point I assume that tracking disappears.

[dhrubo-os]

How can we distinguish between 'If the document doesn't exist' and 'If the document never existed'?

[dbwiddis]

@dhrubo-os @arjunkumargiri

How can we distinguish between 'If the document doesn't exist' and 'If the document never existed'?

We can only do so on OpenSearch for some time period following the deletion.

However, this TODO is in the context of other data stores, and I'm not sure we can, or that we want to.

My intent in writing these comments was to document OpenSearch behavior, specifically:

1. The seq_no is incremented when a document is deleted
2. The seq_no is maintained in the segment metadata for at least as long as it takes for all merge and indexing and replication operations to safely complete, as well as remaining part of the translog.
3. At some point, depending on usage of the cluster, the translog is eventually trimmed, and the reference to the document's seq_no is deleted.

From a practical standpoint, the only difference would be seen if someone created a new document with the same document ID as a previously deleted document, within the time period that the old seq_no remains. In that case the value would be a number higher than 0.

I'd assume that a reasonable implementation for a deletion would be to increment and return the seq_no, but then not track it any more, so the next creation would start again at 0. THat would match current behavior except in the rare edge case of deleting and recreating the same doc-id.

[dhrubo-os]

May be not a part of this PR, but we need to address this use case. @dbwiddis could you please track this action item in your end? Thanks.

[dbwiddis]

May be not a part of this PR, but we need to address this use case. @dbwiddis could you please track this action item in your end? Thanks.

I'm not clear what the action item is? The code currently throws an exception and returns an error to the user; wouldn't it be exactly the same error? Are you referring to per-tenant keys or something else?

[dhrubo-os]

May be not a part of this PR, but we need to address this use case. @dbwiddis could you please track this action item in your end? Thanks.

I'm not clear what the action item is? The code currently throws an exception and returns an error to the user; wouldn't it be exactly the same error? Are you referring to per-tenant keys or something else?

So currently Optype is Index by default. But for config key creation, we want OpType.CREATE specifically. From current sdkClient interface we can't achieve this.

[dbwiddis]

we want OpType.CREATE specifically. From current sdkClient interface we can't achieve this.

Got it... I'm not sure we want to use "OpType" generaly. "Upsert" is the standard language here. We probably want to implement conditional writes on the DynamoDB version.

Let's align on a name for it... will track separately.

[dbwiddis]

May be not a part of this PR, but we need to address this use case. @dbwiddis could you please track this action item in your end? Thanks.

So currently Optype is Index by default. But for config key creation, we want OpType.CREATE specifically. From current sdkClient interface we can't achieve this.

@dhrubo-os @arjunkumargiri #2640