CLDSRV-497 Fix BackbeatClient.putMetadata with versionID #5525

nicolas2bert · 2024-01-26T14:35:19Z

Issue: When Cloudserver BackbeatClient.putMetadata() option fields are sent to Metadata through the query string, they are converted to strings. As a result, Metadata interprets the value undefined in the versionId field as an empty string ('').

Background: Previously, the 'crrExistingObject' script used this bug/behavior as a workaround to generate an internal version ID to replicate null version (= objects created before versioning was enabled). However, this approach has led to inconsistencies, occasionally resulting in the creation of multiple null internal versions.

Resolution: To address this issue, the 'crrExistingObject' workaround will be deprecated. Instead, Backbeat will be enhanced to support the replication of null versions directly, thereby ensuring more reliable and consistent behavior in handling versioning.

bert-e · 2024-01-26T14:35:23Z

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2024-01-26T14:40:51Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

francoisferrand · 2024-01-26T15:59:02Z

lib/routes/routeBackbeat.js

+        // To prevent this, the versionId field is only included in options when it is defined.
+        if (versionId !== undefined) {
+            options.versionId = versionId;
+        }


can you confirm this will have no impact when backend is mongo?

Yes, I confirm

tests/functional/raw-node/test/routes/routeBackbeatForReplication.js

tests/functional/raw-node/utils/makeRequest.js

nicolas2bert · 2024-02-15T09:08:31Z

/create_integration_branches

bert-e · 2024-02-15T09:08:49Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.6/bugfix/CLDSRV-497/putmetadata with contents from w/7.70/bugfix/CLDSRV-497/putmetadata
and development/8.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.6/bugfix/CLDSRV-497/putmetadata origin/development/8.6
 $ git merge origin/w/7.70/bugfix/CLDSRV-497/putmetadata
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.6/bugfix/CLDSRV-497/putmetadata

The following options are set: create_integration_branches

bert-e · 2024-02-15T10:42:22Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.7/bugfix/CLDSRV-497/putmetadata with contents from w/8.6/bugfix/CLDSRV-497/putmetadata
and development/8.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.7/bugfix/CLDSRV-497/putmetadata origin/development/8.7
 $ git merge origin/w/8.6/bugfix/CLDSRV-497/putmetadata
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.7/bugfix/CLDSRV-497/putmetadata

The following options are set: create_integration_branches

bert-e · 2024-02-15T14:08:39Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-497/putmetadata into
development/7.10
w/7.70/bugfix/CLDSRV-497/putmetadata will be merged into development/7.70
w/8.6/bugfix/CLDSRV-497/putmetadata will be merged into development/8.6
w/8.7/bugfix/CLDSRV-497/putmetadata will be merged into development/8.7
w/8.8/bugfix/CLDSRV-497/putmetadata will be merged into development/8.8

The following branches will NOT be impacted:

development/7.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-02-15T14:08:40Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

benzekrimaha · 2024-02-15T14:54:11Z

tests/functional/raw-node/package.json

@@ -11,7 +11,7 @@
    "test-aws": "AWS_ON_AIR=true mocha -t 40000 test/",
    "test-gcp": "mocha -t 40000 test/GCP/",
    "test-routes": "mocha -t 40000 test/routes/",
-    "test": "mocha -t 40000 test/",
+    "test": "mocha -t 40000 test/routes/routeBackbeatForReplication.js",


shouldn't this test be renamed as it's targetting now specific tests (unless this was just for testing purpose)

Good catch, I force-pushed it unwillingly :) https://github.com/scality/cloudserver/compare/44a1c1c4b492ec1a7c16617a3ef91da7430ec290..6a37bf33d836379203a0049ebae8e411040f59fc

anurag4DSB

LGTM, but I want to know about Maha's comment as well, are we not planning to run the remaining tests?

anurag4DSB · 2024-02-15T15:08:48Z

tests/functional/raw-node/test/routes/routeBackbeatForReplication.js

+        'default', { signatureVersion: 'v4' });
+    const s3 = bucketUtil.s3;
+
+    const bucketSource = 'backbeatbucket-replication-source';


Should we use unique bucket names? Does it matter?

Same for destination

Unique bucket names might hide cleaning issues when done without a clear purpose. Why would we need it?

Issue: When Cloudserver BackbeatClient.putMetadata() option fields are sent to Metadata through the query string, they are converted to strings. As a result, Metadata interprets the value undefined in the versionId field as an empty string (''). Background: Previously, the 'crrExistingObject' script used this bug/behavior as a workaround to generate an internal version ID to replicate null version (= objects created before versioning was enabled). However, this approach has led to inconsistencies, occasionally resulting in the creation of multiple null internal versions. Resolution: To address this issue, the 'crrExistingObject' workaround will be deprecated. Instead, Backbeat will be enhanced to support the replication of null versions directly, thereby ensuring more reliable and consistent behavior in handling versioning.

bert-e · 2024-02-15T16:31:52Z

History mismatch

Merge commit #6a37bf33d836379203a0049ebae8e411040f59fc on the integration branch
w/7.70/bugfix/CLDSRV-497/putmetadata is merging a branch which is neither the current
branch bugfix/CLDSRV-497/putmetadata nor the development branch
development/7.70.

It is likely due to a rebase of the branch bugfix/CLDSRV-497/putmetadata and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

nicolas2bert · 2024-02-15T16:33:42Z

@bert-e reset

bert-e · 2024-02-15T16:33:52Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2024-02-15T16:34:08Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.6/bugfix/CLDSRV-497/putmetadata with contents from w/7.70/bugfix/CLDSRV-497/putmetadata
and development/8.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.6/bugfix/CLDSRV-497/putmetadata origin/development/8.6
 $ git merge origin/w/7.70/bugfix/CLDSRV-497/putmetadata
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.6/bugfix/CLDSRV-497/putmetadata

The following options are set: create_integration_branches

bert-e · 2024-02-15T16:58:07Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.7/bugfix/CLDSRV-497/putmetadata with contents from w/8.6/bugfix/CLDSRV-497/putmetadata
and development/8.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.7/bugfix/CLDSRV-497/putmetadata origin/development/8.7
 $ git merge origin/w/8.6/bugfix/CLDSRV-497/putmetadata
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.7/bugfix/CLDSRV-497/putmetadata

The following options are set: create_integration_branches

bert-e · 2024-02-15T17:58:31Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-497/putmetadata into
development/7.10
w/7.70/bugfix/CLDSRV-497/putmetadata will be merged into development/7.70
w/8.6/bugfix/CLDSRV-497/putmetadata will be merged into development/8.6
w/8.7/bugfix/CLDSRV-497/putmetadata will be merged into development/8.7
w/8.8/bugfix/CLDSRV-497/putmetadata will be merged into development/8.8

The following branches will NOT be impacted:

development/7.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-02-15T17:58:32Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

nicolas2bert · 2024-02-15T18:43:12Z

@bert-e approve

bert-e · 2024-02-15T18:43:34Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/7.10
✔️ development/7.70
✔️ development/8.6
✔️ development/8.7
✔️ development/8.8

The following branches will NOT be impacted:

development/7.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2024-02-15T19:00:50Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/7.10
✔️ development/7.70
✔️ development/8.6
✔️ development/8.7
✔️ development/8.8

The following branches have NOT changed:

development/7.4

Please check the status of the associated issue CLDSRV-497.

Goodbye nicolas2bert.

scality deleted a comment from bert-e Jan 26, 2024

francoisferrand reviewed Jan 26, 2024

View reviewed changes

jonathan-gramain approved these changes Feb 14, 2024

View reviewed changes

tests/functional/raw-node/test/routes/routeBackbeatForReplication.js Outdated Show resolved Hide resolved

tests/functional/raw-node/test/routes/routeBackbeatForReplication.js Outdated Show resolved Hide resolved

tests/functional/raw-node/utils/makeRequest.js Show resolved Hide resolved

nicolas2bert force-pushed the bugfix/CLDSRV-497/putmetadata branch 2 times, most recently from 44a1c1c to 6a37bf3 Compare February 15, 2024 08:58

benzekrimaha reviewed Feb 15, 2024

View reviewed changes

benzekrimaha approved these changes Feb 15, 2024

View reviewed changes

anurag4DSB reviewed Feb 15, 2024

View reviewed changes

nicolas2bert force-pushed the bugfix/CLDSRV-497/putmetadata branch from 6a37bf3 to c52a3a6 Compare February 15, 2024 16:31

bert-e merged commit a6b283f into development/7.10 Feb 15, 2024
10 checks passed

bert-e deleted the bugfix/CLDSRV-497/putmetadata branch February 15, 2024 19:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDSRV-497 Fix BackbeatClient.putMetadata with versionID #5525

CLDSRV-497 Fix BackbeatClient.putMetadata with versionID #5525

nicolas2bert commented Jan 26, 2024

bert-e commented Jan 26, 2024

bert-e commented Jan 26, 2024

francoisferrand Jan 26, 2024

nicolas2bert Feb 1, 2024 •

edited

Loading

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

benzekrimaha Feb 15, 2024

nicolas2bert Feb 15, 2024

anurag4DSB left a comment

anurag4DSB Feb 15, 2024 •

edited

Loading

anurag4DSB Feb 15, 2024

nicolas2bert Feb 15, 2024

bert-e commented Feb 15, 2024

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

bert-e commented Feb 15, 2024

CLDSRV-497 Fix BackbeatClient.putMetadata with versionID #5525

CLDSRV-497 Fix BackbeatClient.putMetadata with versionID #5525

Conversation

nicolas2bert commented Jan 26, 2024

bert-e commented Jan 26, 2024

Hello nicolas2bert,

bert-e commented Jan 26, 2024

Request integration branches

francoisferrand Jan 26, 2024

Choose a reason for hiding this comment

nicolas2bert Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

Conflict

bert-e commented Feb 15, 2024

Conflict

bert-e commented Feb 15, 2024

Integration data created

bert-e commented Feb 15, 2024

Waiting for approval

benzekrimaha Feb 15, 2024

Choose a reason for hiding this comment

nicolas2bert Feb 15, 2024

Choose a reason for hiding this comment

anurag4DSB left a comment

Choose a reason for hiding this comment

anurag4DSB Feb 15, 2024 • edited Loading

Choose a reason for hiding this comment

anurag4DSB Feb 15, 2024

Choose a reason for hiding this comment

nicolas2bert Feb 15, 2024

Choose a reason for hiding this comment

bert-e commented Feb 15, 2024

History mismatch

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

Reset complete

bert-e commented Feb 15, 2024

Conflict

bert-e commented Feb 15, 2024

Conflict

bert-e commented Feb 15, 2024

Integration data created

bert-e commented Feb 15, 2024

Waiting for approval

nicolas2bert commented Feb 15, 2024

bert-e commented Feb 15, 2024

In the queue

bert-e commented Feb 15, 2024

nicolas2bert Feb 1, 2024 •

edited

Loading

anurag4DSB Feb 15, 2024 •

edited

Loading