Question: worker processes rebooting causing 502 error with worker class=gevent

[rujutashinde]

Hello

I have a flask app which I run using gunicorn. This app runs on a container on Openshift. This app does get/put/post using external apis
Initially I had used worker_class = sync and workers=4 and gunicorn version 19.8.1. This works fine when i dont have multiple requests calling this app at the same time.
But recently the usage of this app has increased and when we send a lot of requests we were seeing worker process Booting 502 error.

[2022-07-22 14:13:51 +0000] [1] [WARNING] Worker with pid 29 was terminated due to signal 9
[2022-07-22 14:13:51 +0000] [148] [INFO] Booting worker with pid: 148

From your FAQ I see that this can happen especially when running on containers due to memory issues.
https://docs.gunicorn.org/en/stable/faq.html#why-are-workers-silently-killed

So i increased the memory to this

        resources:
          limits:
            cpu: 750m
            memory: 2Gi
          requests:
            cpu: 300m
            memory: 256Mi

The performance improved where i am now seeing less 502 errors than before, but they still occur
I also changed the worker_class = gevent, upgraded to gunicorn 20.1.0 and changed some of the configuration for gunicorn. I still land up seeing one or two 502 errors where my posts fail and app exits, but I also see in the logs that the worker processes are being booted randomly. This is what i have for the configuration:

 [2022-07-21 13:48:52 -0400] [7388] [DEBUG] Current configuration:
  config: ./rp_preproc/gunicorn_config.py
  wsgi_app: None
  bind: ['0.0.0.0:8000']
  backlog: 2048
  workers: 4
  worker_class: gevent
  threads: 1
  worker_connections: 1000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 3600
  graceful_timeout: 60
  keepalive: 7200
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: True
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  print_config: False
  preload_app: False
  sendfile: None
  reuse_port: False
  chdir: /home/rushinde/rpv5_py39/rp_preproc
  daemon: False
  raw_env: []
  pidfile: None
  worker_tmp_dir: /dev/shm
  user: 105754
  group: 105754
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  forwarded_allow_ips: ['127.0.0.1']
  accesslog: /dev/stdout
  disable_redirect_access_to_syslog: False
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  errorlog: -
  loglevel: DEBUG
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  dogstatsd_tags: 
  statsd_prefix: 
  proc_name: None
  default_proc_name: rp_preproc.app:app
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x7f8f6c31daf0>
  on_reload: <function OnReload.on_reload at 0x7f8f6c31dc10>
  when_ready: <function WhenReady.when_ready at 0x7f8f6c31dd30>
  pre_fork: <function Prefork.pre_fork at 0x7f8f6c31de50>
  post_fork: <function Postfork.post_fork at 0x7f8f6c31df70>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f8f6c3300d0>
  worker_int: <function WorkerInt.worker_int at 0x7f8f6c3301f0>
  worker_abort: <function WorkerAbort.worker_abort at 0x7f8f6c330310>
  pre_exec: <function PreExec.pre_exec at 0x7f8f6c330430>
  pre_request: <function PreRequest.pre_request at 0x7f8f6c330550>
  post_request: <function PostRequest.post_request at 0x7f8f6c3305e0>
  child_exit: <function ChildExit.child_exit at 0x7f8f6c330700>
  worker_exit: <function WorkerExit.worker_exit at 0x7f8f6c330820>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f8f6c330940>
  on_exit: <function OnExit.on_exit at 0x7f8f6c330a60>
  proxy_protocol: False
  proxy_allow_ips: ['127.0.0.1']
  keyfile: None
  certfile: None
  ssl_version: 2
  cert_reqs: 0
  ca_certs: None
  suppress_ragged_eofs: True
  do_handshake_on_connect: False
  ciphers: None
  raw_paste_global_conf: []
  strip_header_spaces: False
[2022-07-21 13:48:52 -0400] [7388] [INFO] Starting gunicorn 20.1.0
[2022-07-21 13:48:52 -0400] [7388] [DEBUG] Arbiter booted
[2022-07-21 13:48:52 -0400] [7388] [INFO] Listening at: http://0.0.0.0:8000 (7388)
[2022-07-21 13:48:52 -0400] [7388] [INFO] Using worker: gevent
[2022-07-21 13:48:52 -0400] [7407] [INFO] Booting worker with pid: 7407
[2022-07-21 13:48:52 -0400] [7408] [INFO] Booting worker with pid: 7408
[2022-07-21 13:48:52 -0400] [7409] [INFO] Booting worker with pid: 7409
[2022-07-21 13:48:52 -0400] [7410] [INFO] Booting worker with pid: 7410
[2022-07-21 13:48:52 -0400] [7388] [DEBUG] 4 workers

My questions are: :

1. for my flask app, do you think the above configuration is suitable?
2. Are there some specific parameters that I can modify to fine tune gunicorn for my use case?
3. The worker_class of gevent , is it appropriate?
4. I looked up design doc, and it seems i should be using gevent or some asynchronous type and not sync since the code uses other apis -> https://docs.gunicorn.org/en/stable/design.html#choosing-a-worker-type

I would really appreciate some feedback on this
Thanks
Rujuta