Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat(batching): refactor; troubleshoot back pressure #630

Merged
merged 6 commits into from
May 11, 2020

Conversation

bojiang
Copy link
Member

@bojiang bojiang commented May 6, 2020

(Thanks for sending a pull request! Please make sure to read the contribution guidelines, then fill out the blanks below.)

What changes were proposed in this pull request?

Does this close any currently open issues?

How was this patch tested?

@pep8speaks
Copy link

pep8speaks commented May 6, 2020

Hello @hrmthw, Thanks for updating this PR.

There are currently no PEP 8 issues detected in this PR. Cheers! 🍻

Comment last updated at 2020-05-11 05:56:49 UTC

@bojiang bojiang force-pushed the batching branch 2 times, most recently from 279597e to 4b528b0 Compare May 6, 2020 06:07
@parano parano self-assigned this May 6, 2020
@parano parano self-requested a review May 6, 2020 06:20

def log_outbound_time(self, info):
if info[0] < 5: # skip all small batch
self.o_stat = FixedBucket(self.N_OUTBOUND_SAMPLE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add inline comments here to explain these variables?

def __init__(self, max_wait_time, max_size, shared_sema: callable = None):
def __init__(
self,
max_expected_time: int,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename max_latency_in_ms?

@@ -141,6 +141,7 @@ def __init__(
self.setup_routes_from_pb(self.bento_service_metadata_pb)
if psutil.POSIX:
import resource

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update the docstring in MarshalService class to reflect the changes in this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage is almost the same as before.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we changed the semantic of mb_max_latency?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously it was closer to max_wait_time?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@parano just updated the docstring

import time


class FixedBucket:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider deque with max length for this?

from collections import deque
deque(maxlen=100)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not used anymore?



class Optimizer:
N_OUTBOUND_SAMPLE = 500
N_OUTBOUND_WAIT_SAMPLE = 20
N_OUTBOUND_SAMPLE = 50
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add inline comments about these variables

@bojiang bojiang force-pushed the batching branch 3 times, most recently from 1e4e514 to 54dcd0e Compare May 9, 2020 14:58
@codecov-io
Copy link

codecov-io commented May 9, 2020

Codecov Report

Merging #630 into master will decrease coverage by 0.25%.
The diff coverage is 19.10%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #630      +/-   ##
==========================================
- Coverage   54.40%   54.14%   -0.26%     
==========================================
  Files         102      103       +1     
  Lines        7827     7890      +63     
==========================================
+ Hits         4258     4272      +14     
- Misses       3569     3618      +49     
Impacted Files Coverage Δ
bentoml/marshal/dispatcher.py 21.37% <16.79%> (-1.21%) ⬇️
bentoml/marshal/marshal.py 28.05% <20.00%> (-0.31%) ⬇️
bentoml/utils/alg.py 27.02% <27.02%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5428fe5...0033606. Read the comment docs.

@bojiang bojiang force-pushed the batching branch 2 times, most recently from 8d65e36 to 2aac052 Compare May 11, 2020 05:54
@parano parano merged commit 04be66e into bentoml:master May 11, 2020
aarnphm pushed a commit to aarnphm/BentoML that referenced this pull request Jul 29, 2022
* Feat(batching): refactor; troubleshoot back pressure

* Chore: add dispatcher benchmark

* Fix(batching): freeze at beginning with small max_expected_time

* Style: docs for batching

* Chore(batching): handle exceptions out of parade dispatcher

* Fix: event loop lazy loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants