soroban-rpc: Audit and enforce timeouts everywhere we need them

[paulbellamy]

What problem does your feature solve?

At the moment our timeouts (if we have them) are set pretty ad-hoc.

What would you like to see?

We should go through the code, and figure out where we have existing timeouts, and where we should add new timeouts. Then do it. We should also consider adding configuration options for any timeouts that make sense.

Examples:

• http req/resp
• different parts of the startup
• when filling the db
• db queries
• core txsub sending
• (any other external service calls?)

What alternatives are there?

[sreuland]

@tsachiherman , it looks like some areas may be checked off based on prior work you've merged like rpc timeouts with DurationLimiter in #804 and panic handling in #856

[stellarsaur]

After taking a look, we currently have timeouts for:

• RPC requests (see soroban-rpc: limit the execution duration of the jrpc requests #804 )
• Streaming ledgers on initialization
• Retrying ingestion (every 1s for 5s)
• DB operations

Adding timeouts for history archive HTTP connections were considered, but decided not to be needed (see stellar/go#1829 for context).

Therefore, everything relevant is covered, so I'm going to close this issue. Feel free to reopen if I missed something.