[BUG] repository-azure plugin hangs in OpenSearch >= 1.2.0

[uncycler]

Describe the bug
Since 1.2.0, the repository-azure plugin stop working correctly. The PUT command to create the new repository hanging forever and the thread pool queue is filling up with 120 generic tasks and the master node is eating all the cpu resources it got:

"CNLPL4MfQ1aOeA1io2LXKw:44940" : {
    "node" : "CNLPL4MfQ1aOeA1io2LXKw",
    "id" : 44940,
    "type" : "transport",
    "action" : "cluster:admin/snapshot/get",
    "start_time_in_millis" : 1639552118629,
    "running_time_in_nanos" : 205051397113,
    "cancellable" : false,
    "parent_task_id" : "uY6TEyVlSQCxiJxkMJq6Sg:10583",
    "headers" : { }
  },

Nothing is logged. Is there anyway to enable debug logging on plugins?

Also, if you look at transactions/sec metrics in the azure storage account, there is thousands of them:

To Reproduce
Steps to reproduce the behavior:

1. Add Azure Storage Account info (name and sas token) in keystore
   azure.client.default.account
   azure.client.default.sas_token

2. Create the snapshot repository.

PUT _snapshot/azure
{
  "type": "azure",
  "settings": {
    "client": "default",
    "container": "opensearch"
    "base_path": "subfolder"
  }
}

This should hangs forever.
3. See the thread pool or running tasks

GET /_cat/thread_pool
GET _tasks

Expected behavior

{
  "acknowledged" : true
}

Plugins

• repository-azure

Host/Environment (please complete the following information):

• opensearch 1.2.1 docker image running in Kubernetes

[juntezhang]

This bug is serious and prevents from upgrading to OpenSearch 1.2.1 to fix the log4j2 REC issue. I also created a thread on the forums: https://discuss.opendistrocommunity.dev/t/snapshot-stuck-in-in-progress-in-all-but-shows-as-succeeded-in-status/8014/2

[reta]

@juntezhang looking into it

[reta]

@juntezhang any exceptions on the server side (logs/ folder) you could share?

[juntezhang]

@reta there are no exceptions logged by OpenSearch. It just hangs.

[reta]

Could you please capture the thread dumps for the Opensearch server process? I think the issue is figured out

[reta]

@PaulLesur @juntezhang so the issue is closely related to FasterXML/jackson-databind#3322 and in the nutshell, Azure Blob APIs V12 heavily relies on the fact that empty XML elements / attributes are going to be nullified.

However, sadly, it highly depends on XMLInputReader instance being picked up at runtime: the Woodstox does that, whereas the default one from JDK com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl does not. It leads to infinite loop within listBlobsByHierarchy or listBlobs - the page navigation only understands null as termination condition.

Working on the fix now.

[dblock]

Let's talk about a release for this in opensearch-project/opensearch-build#1365? Will a release of just the plugin with version 1.2.2.1 work?

[reta]

Let's talk about a release for this in opensearch-project/opensearch-build#1365? Will a release of just the plugin with version 1.2.2.1 work?

@dblock I think technically it will work, but from the code perspective, it will go to 1.2 branch, could we track the release of the plugin to particular commit? (wondering how we could match binary and source artifacts since it is the same repository).

[dblock]

Let's talk about a release for this in opensearch-project/opensearch-build#1365? Will a release of just the plugin with version 1.2.2.1 work?

@dblock I think technically it will work, but from the code perspective, it will go to 1.2 branch, could we track the release of the plugin to particular commit? (wondering how we could match binary and source artifacts since it is the same repository).

We'll increment the version and make a tag like we always do. 1.2 is just the line for all the 1.2.x releases.

[dblock]

Scratch my idea of incrementing a version to 1.2.2.1, there are waaay too many assumptions in OpenSearch about a 3-digit version to fix in a patch release. I tried for 30 minutes and got dozens of changes ;)

Can I have everybody's opinion on this thread, @reta @uncycler @juntezhang - which one do you think we should do:

1. Build another 1.2.2, pluck repository-azure-1.2.2.zip out of it, sign it, and replace only the zip in our existing 1.2.2 distribution. We cannot overwrite published maven artifacts I believe.
2. Make a complete 1.2.3 release of OpenSearch with all the plugins.

[peternied]

I vote for option 2 follows our semver principles. 4 hours ago when we didn't yet release OpenSearch 1.2.2 to maven we would have had other options.

[peterzhuamazon]

Since we also release on maven I vote for 2 as we cannot override now:
https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/plugin/repository-azure/1.2.2/

[reta]

I vote for option 2 follows our semver principles. 4 hours ago when we didn't yet release OpenSearch 1.2.2 to maven we would have had other options.

+1 for 2nd option (sorry @dblock), just to clarify, the release of 1.2.2 without this bugfix was by intention

[stockholmux]

I both love and hate a 1.2.3. Having a shadow release is madness and against semver. On the other hand, I dislike releasing so often, wolf crying comes to mind.

[reta]

I both love and hate a 1.2.3. Having a shadow release is madness and against semver. On the other hand, I dislike releasing so often, wolf crying comes to mind.

Sorry about that, the timing with log4j is not good

[juntezhang]

I think option 2 is probably cleanest and most transparent, but I can image you would like to combine it with more features or bug fixes and not do 3 releases within 1 week. I think those who want to have this fix can fork it and install the plugin from file. Or switch to snapshots to FS instead. Anyway, thanks for the quick fix and looking forward to the release (at some point).

…

On Thu, Dec 16, 2021, 9:59 PM Kyle J. Davis ***@***.***> wrote: I both love and hate a 1.2.3. Having a shadow release is madness and against semver. On the other hand, I dislike releasing so often, wolf crying comes to mind. — Reply to this email directly, view it on GitHub <#1734 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKDT2UTHOZCWJQNNVSDUGTURJHMJANCNFSM5KDR6RHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[reta]

@nknize @dblock do you think guys it is worth extracting the repository plugins off the main repo? (it looks doable in general)

[dblock]

I think option 2 is probably cleanest and most transparent, but I can image you would like to combine it with more features or bug fixes and not do 3 releases within 1 week. I think those who want to have this fix can fork it and install the plugin from file. Or switch to snapshots to FS instead. Anyway, thanks for the quick fix and looking forward to the release (at some point).
…

I would be ok with this, but we are telling everyone to upgrade to 1.2.2 because of log4j, and there’s nothing to upgrade to for users of this plug-in

[dblock]

@nknize @dblock do you think guys it is worth extracting the repository plugins off the main repo? (it looks doable in general)

I was going to suggest that. There’s no reason for these plugins to be tied to OpenSearch IMO. Appreciate if you could open an issue either way.

[reta]

Done, #1754

[dblock]

@reta @uncycler @juntezhang care to confirm that @reta's fix works in https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.2/102/linux/x64/builds/opensearch/core-plugins/repository-azure-1.2.2.zip, this does not have a version increment, we're going to do this and go to 1.2.3.

If you need an OpenSearch-min, https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.2/102/linux/x64/builds/opensearch/dist/opensearch-min-1.2.2-linux-x64.tar.gz

[uncycler]

@reta @uncycler @juntezhang care to confirm that @reta's fix works in https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.2/102/linux/x64/builds/opensearch/core-plugins/repository-azure-1.2.2.zip, this does not have a version increment, we're going to do this and go to 1.2.3.

I confirm that the updated plugin is working as expected.

[dblock]

Full x64 distribution build, https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/126/linux/x64/dist/opensearch/opensearch-1.2.3-linux-x64.tar.gz

Anyone to try this on Azure?

Other manifests/components:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/126/linux/arm64/builds/opensearch/manifest.yml
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/126/linux/arm64/dist/opensearch/manifest.yml
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/126/linux/x64/builds/opensearch/manifest.yml
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/126/linux/x64/dist/opensearch/manifest.yml

[jcgraybill]

Confirmed that 1.2.3 produces the correct behavior for me:

./bin/opensearch-plugin install repository-azure
./bin/opensearch-keystore add azure.client.default.account
./bin/opensearch-keystore add azure.client.default.key
./bin/opensearch -d -p opensearch.pid
/usr/bin/curl http://localhost:9200/songs/_doc -X POST -H 'Content-Type: application/json' -d '{"title": "Inside Out", "artist": "Eve 6"}'
/usr/bin/curl http://localhost:9200/songs/_doc -X POST -H 'Content-Type: application/json' -d '{"title": "Semi-Charmed Life", "artist": "Third Eye Blind"}'
/usr/bin/curl http://localhost:9200/_snapshot/testbackup -X PUT -H 'Content-Type: application/json' -d '{"type": "azure", "settings": {"container": "testbackup"}}'
{"acknowledged":true}
/usr/bin/curl http://localhost:9200/_snapshot/testbackup/1 -X PUT  
{"accepted":true}

Transactions / sec is back to an expected range: