http ratelimit: option to reduce budget on stream done

[mathetake]

Commit Message: ratelimit: option to excute action on stream done

Additional Description:
This adds a new option apply_on_stream_done to the rate limit
policy corresponding to each descriptor. This basically allows to configure
descriptors to be executed in a response content-aware way and do not
enforce the rate limit (in other words "fire-and-forget"). Since addend
can be currently controlled via metadata per descriptor,
another filter can be used to set the value to reflect their intent there,
for example, by using Lua or Ext Proc filters.

This use case arises from the LLM API services which usually return
the usage statistics in the response body. More specifically,
they have "streaming" APIs whose response is a line-by-line event
stream where the very last line of the response line contains the
usage statistics. The lazy nature of this action is perfectly fine
as in these use cases, the rate limit happens like "you are forbidden
from the next time".

Besides the LLM specific, I've also encountered the use case from the
data center resource allocation case where the operators want to
"block the computation from the next time since you used this much
resources in this request".

Ref: envoyproxy/gateway#4756

Risk Level: low
Testing: done
Docs Changes: done
Release Notes: TODO
Platform Specific Features: n/a

[repokitteh-read-only]

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #37548 was opened by mathetake.

see: more, trace.

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @wbpcode
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #37548 was opened by mathetake.

see: more, trace.

[mathetake]

i guess the impl can be a bit large, so I might do that in separate PRs - anyways will think about it after the API gets approved

[wbpcode]

wow, we have a similar requirement internally and I finally figured out a similar way. It is super surprised and happy to see this.

[mathetake]

cool glad to hear that you came to the similar idea!

[mathetake]

@wbpcode thank you for the valuable feedback offline! I think I will go ahead and try implementing the idea - i don't think the change won't be that huge

[wbpcode]

Thanks for this contribution. This extend the usage of the rate limit filter largely. Some comments are added.

[wbpcode]

Thanks for the update. It's much better now. Some more comments are added. And please keep the route and route specific config in the filter at the begin of the requet. Or the route refreshment may results in that the encoding phase has different configuration with the decoding phase.

[mathetake]

meanwhile i am working on the integration tests now ... some cases are failing

[wbpcode]

LGTM. Thanks for the contribution.

[wbpcode]

cc @mattklein123 cc @tyxia for any additional comments.