Make feedback more flexible and light

[jklaise]

Currently the send-feedback API takes in a batch of samples (request+response) but only one float for the reward which means that design choices must be made when distributing this reward between all samples in the batch (e.g. #336). One potential fix would be to change reward to be of type DefaultData so that each sample in a batch gets its own reward.

Additionally, we need to re-evaluate sending both request and response as part of the feedback as this could be wasteful with large payloads.

[janvdvegt]

I agree with both statements, aggregating up front seems like throwing away information unnecessarily and shutting off options for more elaborate choices. Aggregating them later if needed seems better.

With regards to the request and response, would there be a possibility to send a unique identifier instead of the full request if needed? That way we could still have a full audit to link the reward to an earlier request but not have to send the full input multiple times. In other places I also believe that using an identifier instead of the full payload is useful.

[ukclivecox]

Related to new spec in https://github.com/SeldonIO/mlgraph

[seldondev]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
/lifecycle stale

[ukclivecox]

Iter8 integration allows for more flexible usage now. We will work further on this for v2.