Generic: All chain service instances failed

[yaslama]

LOG {"level": "TRACE", "line": "put; add idle connection for ("https", mempool.emzy.de)"}
LOG {"level": "DEBUG", "line": "pooling idle connection for ("https", mempool.emzy.de)"}
LOG {"level": "ERROR", "line": "Call to chain service https://mempool.emzy.de/api failed: error decoding response body: expected value at line 1 column 1"}
LOG {"level": "TRACE", "line": "client tx closed"}
LOG {"level": "TRACE", "line": "pool closed, canceling idle interval"}
LOG {"level": "TRACE", "line": "State::close_read()"}
LOG {"level": "TRACE", "line": "State::close_write()"}
LOG {"level": "TRACE", "line": "flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }"}
LOG {"level": "TRACE", "line": "shut down IO complete"}
LOG Generic: All chain service instances failed

[ok300]

We log the raw response on debug for all calls we make in ChainService:

https://github.com/breez/breez-sdk/blob/529340447a1ff377850a43d6c192eb293114b69f/libs/sdk-core/src/input_parser.rs#L261-L269

I think if the user would set this logger to DEBUG

https://github.com/breez/breez-sdk/blob/529340447a1ff377850a43d6c192eb293114b69f/libs/sdk-core/src/breez_services.rs#L1702

the logs would include the raw response body.

[roeierez]

It seems that we are not checking the status code before attempting to deserialize the json. mempool space returns 404 for example for an not found transaction:
07c9d3fbffc20f96ea7c93ef3bcdf346c8a8456c25850ea76be62b24a7cf690c

We should only attempt to deserialize the json if we got 200 response.
any other status code should return an error immediately.
The error details in such cases are included as raw text in the response body so logging this plus the original request on error is a good idea to understand what happened here.

[roeierez]

Digging a bit more it seems that there are several existing channels that their funding transactions are not found onchain:

f135ed194147cd5f29f429b1561bc49a38755b222b3f2996a974c6ca5615a096
8e03c585a995666d117d37037fc36fcd50f335206f7cbd4ce1d351f64dc00fb5
f128623e3a1057ac91d50cb4c50ba8791ba95130b7e191dc19e7b5002d9e3d50

While we attempting to find their corresponding closing tx we are using this API:
https://mempool.space/api/tx/{txid}/outspends

Which fails from the reason I mentioned above.
So in addition to being more robust in interpreting chain service result, we should be tollerant while looking for the closing tx on chain (which happens during sync) and not fail the sync if we fail to fetch the closing tx.