Avoid retrieving a worker which is being removed by the idle worker c…

…lean-up process (#3725)

* prevent returning stopping workers

* clear only the specific worker

* Add ibc-go v8 to integration tests (#3694)

* Add simapp v8 to CI jobs

* Add simapp v8 to nix flake

* Handle CLI breaking changes of Cosmos SDK v0.50 in test bootstrap

* Handle genesis config 'voting_perdiod' and CLI 'query txs' changes for Cosmos SDK v0.50.1

* Use 'MsgSubmitProposal' instead of deprecated 'UpgradeProposal' to initiate a chain upgrade

* Update 'tx upgrade-chain' CLI template

* Fix 'tx chain-upgrade' tests

* Update chain upgrade for compatibility between different ibc-go versions

* Improve assertion for client upgrade tests

* Update ibc-proto-rs to v0.39.0

* Add changelog entry

* Fix and improve guide section for client upgrade

* Wait before querying client state for client upgrade tests

* Apply suggestions from code review

Co-authored-by: Romain Ruetschi <romain@informal.systems>
Signed-off-by: Luca Joss <43531661+ljoss17@users.noreply.github.com>

* Rename method 'ibc_version()' to 'version_specs'

* Extract the verification of legacy version in a method

* Implement 'FromStr' instead of 'TryFrom<String>' for 'ProposalStatus'

* Fix cargo-doc warning

* Add changelog entry for CLI

* Change the '--gov-account' flag as optional but will fail if trying to upgrade a chain with ibc-go v8+

* Update guide

* Move and update changelog entry for CLI

* Return a 'Specs' struct in 'version_specs()' method

* Fix clippy errors

---------

Signed-off-by: Luca Joss <43531661+ljoss17@users.noreply.github.com>
Co-authored-by: Romain Ruetschi <romain@informal.systems>

* Update Gaia used in tests from v12 to v13 & v14 (#3700)

* Update Gaia v12 to v13 and v14 in tests

* Fix Celestia CI job

* Replace Gaia v12 with v13 and v14 in multi-chain tests (#3701)

* Add changelog entry

---------

Signed-off-by: Luca Joss <43531661+ljoss17@users.noreply.github.com>
Co-authored-by: beer-1 <147697694+beer-1@users.noreply.github.com>
Co-authored-by: Romain Ruetschi <romain@informal.systems>

.changelog/unreleased/bug-fixes/ibc-relayer/3703-avoid-returning-stopped-worker.md

@@ -0,0 +1,3 @@
+- Avoid retrieving a worker which is being removed by the idle worker clean-up
+  process.
+  process ([\#3703](https://github.com/informalsystems/hermes/issues/3703))

crates/relayer/src/worker/map.rs

@@ -125,7 +125,17 @@ impl WorkerMap {
         config: &Config,
     ) -> &WorkerHandle {
         if self.workers.contains_key(&object) {
-            &self.workers[&object]
+            if self.workers[&object].shutdown_stopped_tasks() {
+                self.remove_stopped(
+                    self.workers[&object].id(),
+                    self.workers[&object].object().clone(),
+                );
+
+                let worker = self.spawn_worker(src, dst, &object, config);
+                self.workers.entry(object).or_insert(worker)
+            } else {
+                &self.workers[&object]
+            }
         } else {
             let worker = self.spawn_worker(src, dst, &object, config);
             self.workers.entry(object).or_insert(worker)