Allow data pushing specs to be updated remotely

[bbenligiray]

Currently, when we need to fetch a new kind of signed data from an API provider, we just call their signed HTTP gateway with the new parameters. In the current state of the signed API pusher, a redeployment would be needed for each new dAPI. This is by design, as it prevents the availability of the signed data from being interrupted by us. However, @metobom thinks this will require far too many redeployments, annoying the user.

A middle ground could be to have two groups of pushed data

1. An established group of signed data that is pushed by a pusher with a static config
2. A newer group of signed data that is pushed by a pusher with remotely updated config that will graduate to group 1 in batches to avoid frequent redeployment

This could be implemented by the signed API having an endpoint that signals to the pusher the additional Beacons that it wants to push data for. In the signed API GET interface, for each Beacon, it would be specified (for example with a flag) that the signed data is pushed based on a static config (safe to use) or remotely controlled config (not as safe to use) for transparency.

Note that this can also be implemented with a scheme where the pusher config can be specified to be remote (as in https://api3workspace.slack.com/archives/C05S589E7B4/p1695730031236799) and having the API provider deploy two separate pushers but this approach has some problems

• Handling two deployments will be significantly more difficult for the API provider
• It assumes the remote config is provided by the pusher operator so it is allowed to control anything (like rate limiting parameters)
• It doesn't let the signed API user differentiate between signed data that only the API provider can deny access to vs. signed data the config provider can also deny access to (though this may have to be that way, as an API provider can self-host a remote config/a remote config doesn't necessarily indicate that it's not safe)

[Siegrift]

In the signed API GET interface, for each Beacon, it would be specified (for example with a flag) that the signed data is pushed based on a static config (safe to use) or remotely controlled config (not as safe to use) for transparency.

The signed API should not be trusted so this should ideally come directly from the pusher and be verifiable (e.g. by signature).

and having the API provider deploy two separate pushers but this approach has some problems

I dislike this for all the reasons you mention.

I think there are two discussion points:

1. Do we want to allow pusher configuration to be updated remotely (to avoid redeployments)?
   • After going through the deployment flow I think this might not be as big of a problem, considering the complexity introduced by this feature.
2. Who controls the remote configuration (pusher or signed API)?
   • It should be pusher otherwise it's much harder to reason about the security. Also the data provider owns their data and should be in complete charge what is sent to us.

Alternative ideas

1. Let the pusher configuration be remote, but make it operable only by pusher operator. This way, when we need them to send additional data, they don't have to go through redeployment, but just update the configuration. We still need to bother them to do so, but it's simpler then redeployment. Followup to this would be that we can update their configuration and send it to them, but they still need to approve so. This would require help from frontend which would display the diff for them that they could approve.
2. Try to reduce the number of redeployments. Let's say there is an endpoint with large number of assets. Say we push all of them (or at least the most common ones for dAPIs) even if we are not needing them immediately. Then we need redeployment only if there is a new data provider endpoint. This has a disadvantage that we need to configure many more beacons, signed API receives data it does not need (I guess data providers would not like this) and pusher needs to submit larger payloads (not an issue in practice).

[bbenligiray]

The signed API should not be trusted so this should ideally come directly from the pusher and be verifiable (e.g. by signature).

Good point

Do we want to allow pusher configuration to be updated remotely (to avoid redeployments)?

I think we'll add new data feeds very frequently (mostly because each chain integration means adding a dAPI for the native currency and we'll be doing a lot of chain integrations) and even if the effort of redeployment is manageable, redeployments are very error-prone so we'd be taking a reliability hit by requiring more frequent redeployments. That being said, requiring redeployments for each added data feed is definitely not out of the question.

Who controls the remote configuration (pusher or signed API)?

Maybe each signed API controls the remote configuration of extra data that will be signed to itself? That's a bit complex though.

Let the pusher configuration be remote, but make it operable only by pusher operator. This way, when we need them to send additional data, they don't have to go through redeployment, but just update the configuration. We still need to bother them to do so, but it's simpler then redeployment. Followup to this would be that we can update their configuration and send it to them, but they still need to approve so. This would require help from frontend which would display the diff for them that they could approve.

That's already the plan, but the "approval" is them redeploying. I can't imagine a configuration update method that will be simpler than this+the diff frontend.

Try to reduce the number of redeployments

There are way too many potential feeds for this to be feasible, especially if you consider feeds such as https://github.com/api3dao/tasks/issues/42 It's also pretty common for us to ask API providers to add assets specifically for us so they will not exist at deployment time.

[Siegrift]

That's already the plan, but the "approval" is them redeploying. I can't imagine a configuration update method that will be simpler than #41 (comment) diff frontend.

I think updating just the file can be much simpler. You could do it with a simple POST request or something without going through AWS where you can screw up way more. That said, I don't think the improvement is marginal (and users would actually now have two "redeployment" ways).

There are way too many potential feeds for this to be feasible, especially if you consider feeds such as https://github.com/api3dao/tasks/issues/42 It's also pretty common for us to ask API providers to add assets specifically for us so they will not exist at deployment time.

I see.

I'd revisit this issue in a couple of weeks/months if there is a demand for this.

[bbenligiray]

Okay. An interim (or even permanent tbh) solution is to depend on the HTTP gateway for this, as in we would run a bot at our end that fetches this kind of data and pushes it to the signed API, and every once in a while the API provider would redeploy and the bot would be wound down

[bbenligiray]

I'm dropping the v0.2 milestone, @metobom see the discussion above