Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: timeout configuration not work #4789

Open
BangDaeng opened this issue Jun 11, 2024 · 1 comment
Open

bug: timeout configuration not work #4789

BangDaeng opened this issue Jun 11, 2024 · 1 comment
Labels
bug Something isn't working feedback-wanted Request for feedback

Comments

@BangDaeng
Copy link

BangDaeng commented Jun 11, 2024

Describe the bug

The timeout setting of api_server and runner is not working in bentoml.
i'm using bentoml 1.0.20.post11 version
The default configuration is as follows

version: 1
api_server:
  workers: ~ # cpu_count() will be used when null
  timeout: 60
  backlog: 2048
  # the maximum number of connections that will be made to any given runner server at once
  max_runner_connections: 16
  metrics:
    enabled: true
    namespace: bentoml_api_server
    duration:
      # https://github.com/prometheus/client_python/blob/f17a8361ad3ed5bc47f193ac03b00911120a8d81/prometheus_client/metrics.py#L544
      buckets: [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0]
      min: ~
      max: ~
      factor: ~
  logging:
    access:
      enabled: true
      request_content_length: true
      request_content_type: true
      response_content_length: true
      response_content_type: true
      format:
        trace_id: 032x
        span_id: 016x
  ssl:
    enabled: false
    certfile: ~
    keyfile: ~
    keyfile_password: ~
    ca_certs: ~
    version: 17 # ssl.PROTOCOL_TLS_SERVER
    cert_reqs: 0 # ssl.CERT_NONE
    ciphers: TLSv1 # default ciphers
  http:
    host: 0.0.0.0
    port: 3000
    cors:
      enabled: false
      access_control_allow_origins: ~
      access_control_allow_credentials: ~
      access_control_allow_methods: ~
      access_control_allow_headers: ~
      access_control_allow_origin_regex: ~
      access_control_max_age: ~
      access_control_expose_headers: ~
    response:
      trace_id: false
  grpc:
    host: 0.0.0.0
    port: 3000
    max_concurrent_streams: ~
    maximum_concurrent_rpcs: ~
    max_message_length: -1
    reflection:
      enabled: false
    channelz:
      enabled: false
    metrics:
      host: 0.0.0.0
      port: 3001
  runner_probe: # configure whether the API server's health check endpoints (readyz, livez, healthz) also check the runners
    enabled: true
    timeout: 1
    period: 10
runners:
  resources: ~
  workers_per_resource: 1
  timeout: 300
  batching:
    enabled: true
    max_batch_size: 100
    max_latency_ms: 10000
  logging:
    access:
      enabled: true
      request_content_length: true
      request_content_type: true
      response_content_length: true
      response_content_type: true
  metrics:
    enabled: true
    namespace: bentoml_runner
tracing:
  exporter_type: ~
  sample_rate: ~
  excluded_urls: ~
  timeout: ~
  max_tag_value_length: ~
  zipkin:
    endpoint: ~
    local_node_ipv4: ~
    local_node_ipv6: ~
    local_node_port: ~
  jaeger:
    protocol: thrift
    collector_endpoint: ~
    thrift:
      agent_host_name: ~
      agent_port: ~
      udp_split_oversized_batches: ~
    grpc:
      insecure: ~
  otlp:
    protocol: ~
    endpoint: ~
    compression: ~
    http:
      certificate_file: ~
      headers: ~
    grpc:
      headers: ~
      insecure: ~
monitoring:
  enabled: true
  type: default
  options:
    log_config_file: ~
    log_path: monitoring

I modified the timeout of the api_server and the timeout of the runners, but the settings are not applied
When I write service.py as below and enter kentoml serve and test it, it doesn't work normally.

I tried changing the default configuration
I wrote configuration.yaml and tried changing it.
I also checked that it changed properly by printing the config setting.

but not work, please help me

@svc.api(
    input=JSON.from_sample(samples),
    output=NumpyNdarray(),
    route=BUILD_NAME,
)
async def predict(sentences):
    time.sleep(350)
    output = await scappy_runner.predict.async_run(sentences)
    return output

Please Help Me

To reproduce

No response

Expected behavior

No response

Environment

bentoml: 1.0.20.post11
python: 3.10.12

@BangDaeng BangDaeng added the bug Something isn't working label Jun 11, 2024
@frostming
Copy link
Contributor

Can you try on the latest version of BentoML?

@frostming frostming added the feedback-wanted Request for feedback label Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feedback-wanted Request for feedback
Projects
None yet
Development

No branches or pull requests

2 participants