Webhook not received from agent with OOB and a credential/proof attachment.

[nodlesh]

This issue affects Out Of Band Connection scenarios with credential or proof attachments. The scenario can look like this,
(taken from the OOB 0434 tests in AATH)

      Given we have "2" agents
         | name | role   |
         | Acme | issuer |
         | Bob  | holder |
      Given "Acme" is ready to issue a "indy" credential
      When "Acme" creates a credential offer
      And "Acme" sends a connectionless out of band invitation to "Bob" with "credential-offer"
      And "Bob" receives the invitation
      And "Bob" requests the credential
      And "Acme" issues the credential
      And "Bob" acknowledges the credential issue
      Then "Bob" has the credential issued

The failure here is on the And "Bob" requests the credential
After the receiver/holder agent does the receive-invitation with the credential attachement. The test passes a thread_id to the receiver/holder controller and the controller tries to lookup the credential_exchange_id with the thread_id in a webhook message. This has been working like this for a long time. This webhook message containing the appropriate thread_id and cred_ex_id doesn’t seem to be there anymore.

Stack Trace

2024-04-17 15:55:36,774 aries_cloudagent.core.oob_processor DEBUG Retrieving OOB record using pthid 28cd086a-91f8-4536-8349-31ebe3a15667 for message type https://didcomm.org/issue-credential/1.0/offer-credential
2024-04-17 15:55:36,781 aries_cloudagent.core.oob_processor DEBUG Found out of band record for inbound message with type https://didcomm.org/issue-credential/1.0/offer-credential: 0a6cebcd-19a3-40db-93b5-3e3cf7378b2a
2024-04-17 15:55:36,781 aries_cloudagent.messaging.models.base ERROR ServiceDecorator message validation error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
    schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
    return self._do_load(
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': ['Invalid input type.']}
2024-04-17 15:55:36,783 aries_cloudagent.core.conductor ERROR Exception in message handler:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
    schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
    return self._do_load(
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': ['Invalid input type.']}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 210, in handle_message
    await handler(context, responder)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_offer_handler.py", line 46, in handle
    oob_record = await oob_processor.find_oob_record_for_inbound_message(context)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/oob_processor.py", line 262, in find_oob_record_for_inbound_message
    ServiceDecorator.deserialize(oob_record.their_service),
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/models/base.py", line 200, in deserialize
    raise BaseModelError(f"{cls.__name__} schema validation failed") from err
aries_cloudagent.messaging.models.base.BaseModelError: ServiceDecorator schema validation failed
2024-04-17 15:55:36,784 aries_cloudagent.core.conductor ERROR DON'T shutdown on BaseModelError ServiceDecorator schema validation failed
2024-04-17 15:55:36,784 aries_cloudagent.core.dispatcher ERROR Handler error: Dispatcher.handle_message
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
    schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
    return self._do_load(
  File "/usr/local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'_schema': ['Invalid input type.']}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 210, in handle_message
    await handler(context, responder)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/protocols/issue_credential/v1_0/handlers/credential_offer_handler.py", line 46, in handle
    oob_record = await oob_processor.find_oob_record_for_inbound_message(context)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/oob_processor.py", line 262, in find_oob_record_for_inbound_message
    ServiceDecorator.deserialize(oob_record.their_service),
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/models/base.py", line 200, in deserialize
    raise BaseModelError(f"{cls.__name__} schema validation failed") from err
aries_cloudagent.messaging.models.base.BaseModelError: ServiceDecorator schema validation failed
Exception: Timeout waiting for web callback to retrieve the thread id based on the exchange id
Traceback (most recent call last):
  File "/aries-backchannels/python/agent_backchannel.py", line 275, in _post_command_backchannel
    (resp_status, resp_text) = await self.make_agent_POST_request(command)
  File "/aries-backchannels/acapy/acapy_backchannel.py", line 794, in make_agent_POST_request
    cred_ex_id = await self.swap_thread_id_for_exchange_id(
  File "/aries-backchannels/acapy/acapy_backchannel.py", line 514, in swap_thread_id_for_exchange_id
    raise TimeoutError(
TimeoutError: Timeout waiting for web callback to retrieve the thread id based on the exchange id

This test scenario can be seen failing here,
https://allure.vonx.io/api/allure-docker-service/projects/acapy/reports/latest/index.html?redirect=false#behaviors/c6209898fdef7b8cf04993b6a123eb05

[dbluhm]

Fixed by #2910