fail to pass CI integration test secret_store_spec

[kaisecheng]

Build scan:
integration tests - part 2 #3886
integration tests (persistent queues) - part 2 #3886

Repro line:
https://github.com/elastic/logstash/blob/master/qa/integration/specs/secret_store_spec.rb#L58

Reproduces locally?:
no

Applicable branches:
master, 7.x, 7.15

Failure history:

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:'2021-09-26T22:00:00.000Z',mode:absolute,to:'2021-10-06T21:59:59.999Z'))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b646ed00-7efc-11e8-bf69-63c8ef516157,key:process.exit-code,negate:!f,params:(query:1,type:phrase),type:phrase,value:'1'),query:(match:(process.exit-code:(query:1,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:b646ed00-7efc-11e8-bf69-63c8ef516157,key:build.job-name,negate:!f,params:(query:elastic%2Blogstash%2B7.15%2Bmultijob-integration-pq-2,type:phrase),type:phrase,value:elastic%2Blogstash%2B7.15%2Bmultijob-integration-pq-2),query:(match:(build.job-name:(query:elastic%2Blogstash%2B7.15%2Bmultijob-integration-pq-2,type:phrase))))),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:logstash),sort:!(process.time-start,desc))

The build failed after merging #13270

Failure excerpt:

10:41:29     Failures:
10:41:29 
10:41:29       1) Test that Logstash expands secret store variables from config
10:41:29          Failure/Error: Unable to find matching line from backtrace
10:41:29          
10:41:29          Errno::ENOENT:
10:41:29            No such file or directory - /tmp/studtmp-737eb858a87a456ac1f2e0f2796d8f5e77a1429e814e2248e1cd46dc4bcf/mypath/logstash_secretstore_test.output
10:41:29 
10:41:29     Finished in 17 minutes 36 seconds (files took 5.4 seconds to load)
10:41:29     33 examples, 1 failure
10:41:29 
10:41:29     Failed examples:
10:41:29 
10:41:29     rspec ./specs/secret_store_spec.rb:58 # Test that Logstash expands secret store variables from config

[yaauie]

Looking further up at the console output for the process being tested, we see that it fails to start up because the mutate filter is not installed. Since it fails to start up and run, the assertions we make against a successful run fail.

     [ERROR] 2021-10-06 15:29:24.776 [Converge PipelineAction::Create<mypipeline>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:mypipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any filter plugin named 'mutate'. Are you sure this is correct? Trying to load the mutate filter plugin resulted in this error: Unable to load the requested plugin named mutate of type filter. The plugin is not installed.",

This is preceded by a test that attempts to validate our ability to uninstall the mutate filter:

08:16:48     CLI > logstash-plugin remove
...
08:16:48       without internet connection (linux seccomp wrapper)
...
08:17:27         when no other plugins depends on this plugin
...
08:18:30           successfully remove the plugin

Which is:

        it "successfully remove the plugin" do
          execute = @logstash_plugin.remove("logstash-filter-mutate")


          expect(execute.exit_code).to eq(0)
          expect(execute.stderr_and_stdout).to match(/Successfully removed logstash-filter-mutate/)


          presence_check = @logstash_plugin.list("logstash-filter-mutate")
          expect(presence_check.exit_code).to eq(1)
          expect(presence_check.stderr_and_stdout).to match(/ERROR: No plugins found/)


          @logstash_plugin.install("logstash-filter-mutate")
        end

-- qa/integration/specs/cli/remove_spec.rb@1299f336:77-88

While this spec certainly makes an attempt to reinstall the mutate filter, it makes no assertions that it has done so successfully.

My best guess is that installing our mutate filter doesn't work when offline after #13270 because we now attempt to update the shared dependency, and fail to do so because we are offline.