[Experimental] DeliverySpec Timeout

[slinkydeveloper]

Description
Sink services are usually very heterogeneous, each one with different response time characteristics, depending on their functionality, the way they're implemented, the guarantees the user wants. Because of that, timeouts of a request, when dispatching an event, may vary. DeliverySpec doesn't give the ability to the user to specify the timeout of the single request, so we usually just default to something like 10 seconds. This is not enough, and doesn't cover a lot of use cases where this parameter needs to be tuned.

This experimental feature proposed to add a new field to the DeliverySpec to define such timeout for each dispatched request.

Exit Criteria
DeliverySpec allows to configure the timeout of the single request.

Experimental flag name: delivery-timeout

Experimental feature stages plan

Below the proposed plan for the feature stages (this list implicitly includes the requirements defined in the process)

• Alpha: Included in 0.24
  • Add in the API code the DeliverySpec.Timeout field Delivery timeout #5149
  • Implement in kncloudevents module the handling of timeouts per-request RetryConfig.RequestTimeout to configure the timeout of a single request #5503
  • Implement the glue code between the api field and kncloudevents module Glue Delivery.Timeout to RetryConfig.RequestTimeout #5507
  • e2e test Glue Delivery.Timeout to RetryConfig.RequestTimeout #5507
  • User documentation Documentation for DeliverySpec.Timeout docs#3793
• Beta graduation as soon as 1 release after the inception.
• Beta:
  • User documentation stabilization and improvements
  • Add conformance tests Test DeliverySpec.Timeout for Trigger #5565
• Stable graduation as 2 releases after the beta graduation
• Stable:
  • Add the requirement to support DeliverySpec.Timeout to the knative/specs repo: Add timeout to the delivery spec specs#15 (comment)

Affected WG

• Event Delivery WG

Prior discussion

• Support configuration of Dispatcher CircuitBreaker knative-extensions/eventing-kafka-broker#757

[slinkydeveloper]

/assign

[matzew]

I think it is very reasonable to have some timeout notion configurable

/cc @lionelvillard

[slinkydeveloper]

Spec PR: knative/specs#15
API PR: #5149

After these get accepted, I'll proceed with implementing it in broker and channel.

[n3wscott]

I believe this should be pushed to after we become v1.0 ready. It changes the spec of so many resources.

[slinkydeveloper]

As explained here knative/specs#15 (comment), I think this is an important feature and we should do it before v1, and it's not a drastic change. It's trivial to implement for most components out there (at least the ones i know).

[n3wscott]

And as I said here: it has long lasting effects: knative/specs#15 (comment)

If this value changes in the core, then every CRD needs to be updated or they can not round trip with the timeout.

[slinkydeveloper]

This is not a problem, we can deal with it in the weeks prior to the next release, we've plenty of time for it.

We're also going to remove v1beta1 APIs #5201, so it's even less a problem.

[n3wscott]

This is a very very real problem if v1 continues to be a sliding scale for anyone who has developed any component that exists outside the knative repos. We need to put a stop to the feature creep until we can cut a v1/GA version of Eventing.

[slinkydeveloper]

Refactored this issue as experimental feature, per proposal https://docs.google.com/document/d/1AoH0FyLeuHIg5snrlCKzELQv-NEA6bbjTU2Qv3zlW5k/edit?usp=sharing

[Shashankft9]

Hey, I have recently been looking at the option of configuring timeouts based on the workload to which the cloudevent is going. I have a knative service which takes a bit of time to do some processing to give back the response (basically file conversion use case). Currently I am just directly using the knative service URL to do the work - where we can set the maxTimeout period (in serving), but I am evaluating how it will fit in the eventing part.
I believe without a configurable timeout option in channels, triggers, sources etc, the request will never actually happen successfully, so a feature like this would be really helpful!

[slinkydeveloper]

Inclusion of this feature is approved! https://groups.google.com/g/knative-dev/c/7h1p0eBmUuo/m/lTzaSULKAQAJ

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

/remove-lifecycle stale

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

/assign

[pierDipi]

Feature promoted to beta in 1.7, so enabled by default.

[pfilaretov42]

Hi, I'm trying to set delivery timeout for kafka source and it seems it does not work.

Here is the kafka source with the delivery timeout set to 30s:

apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: source-knative-demo-kafka
  namespace: default
spec:
  consumerGroup: knative-group
  bootstrapServers:
    - my-cluster-kafka-bootstrap.kafka:9092
  topics:
    - knative-demo-kafka-source-topic
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: knative-demo
    uri: /events/long
  delivery:
    backoffDelay: PT3S
    backoffPolicy: linear
    retry: 2
    timeout: PT30S

And here are logs from kafka-source-dispatcher pod:

{"@timestamp":"2024-11-06T07:38:12.324Z","@version":"1","message":"Received a failure status code that is not 2xx. failed to send event to subscriber ...", ..., "statusCode":504}
{"@timestamp":"2024-11-06T07:38:30.328Z","@version":"1","message":"Received a failure status code that is not 2xx. failed to send event to subscriber ...",..., "statusCode":504}
{"@timestamp":"2024-11-06T07:38:51.332Z","@version":"1","message":"Received a failure status code that is not 2xx. failed to send event to subscriber ...",..., "statusCode":504}
{"@timestamp":"2024-11-06T07:38:51.332Z","@version":"1","message":"Failed to send event to subscriber ...", ...}
{"@timestamp":"2024-11-06T07:38:51.333Z","@version":"1","message":"Failed to send event to dead letter sink ...", ...}

As you can see the time between "Received a failure status code" messages is less than 30s.

Could you advice on how can I set the timeout for kafka source, please?