Support Fireworks batching (#8) #12052

ZixinYang · 2023-10-19T21:12:18Z

Description

Add _generate and _agenerate to support Fireworks batching.
Add stop words test cases
Opt out retry mechanism

Issue - Not applicable
Dependencies - None
Tag maintainer - @baskaryan

vercel · 2023-10-19T21:12:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Oct 26, 2023 6:37pm

eyurtsev · 2023-10-19T21:25:21Z

libs/langchain/langchain/chat_models/fireworks.py

@@ -55,7 +67,7 @@ def _convert_delta_to_message_chunk(
    elif role or default_class == ChatMessageChunk:
        return ChatMessageChunk(content=content, role=role)
    else:
-        return default_class(content=content)
+        return default_class(type="", content=content)


type should not be specified -- it's assigned by default depending on the message class

I got a linter error says argument type is missing. Let me update it later to see if it passes the CI here.

The type should not be provided -- an empty string is incorrect here as it'll break serialization of messages if these messages are every serialized directly using pydantic.

Could you include the linter message here? And perhaps use a type: ignore[ speciifc error] to mute it -- sounds like the linter may be incorrect

Have passed CI here without specifying value to type, so should be the linter I used was incorrect.

libs/langchain/langchain/chat_models/fireworks.py

eyurtsev · 2023-10-19T21:33:17Z

libs/langchain/langchain/llms/fireworks.py

@@ -179,6 +228,74 @@ async def _completion_with_retry(**kwargs: Any) -> Any:
    return await _completion_with_retry(**kwargs)


+def completion_with_retry_batching(


we handle retries via -- fireworks should not contain any logic for retries

https://api.python.langchain.com/en/latest/schema.runnable/langchain.schema.runnable.retry.RunnableRetry.html

fireworks.client has own error handling wrapper
e.g. in _create_retry_decorator

fireworks.client.error.RateLimitError, fireworks.client.error.ServiceUnavailableError,

Can we keep it to use our fireworks library and use the custom error handling?

RunnableRetry allows using custom exceptions;

Once Fireworks inherits from BaseLLM, one can do this:

llm = Fireworks(..).with_retry() # Will automatically retry any exception 3 times

retryable_exceptions = (fireworks.client.error.RateLimitError, fireworks.client.error.ServiceUnavailableError) llm = Fireworks(..).with_retry(retry_if_exception_type=retryable_exceptions) # Will automatically retry any exception 3 times

Would either of these options work?

Got it. But then users have to handle retries themselves. We handled retries from the beginning. Is there any way we can still handle retries on our side to avoid breaking users' workflow?

Following the comment in the slack, we keep retry mechanism but allow users to opt out with use_retry=False.

eyurtsev

Sorry didn't meant to accept the previous version...

tjaffri · 2023-10-25T22:54:29Z

libs/langchain/langchain/llms/fireworks.py

+        choices = []
+        for _prompts in sub_prompts:
+            response = await acompletion_with_retry_batching(
+                self, self.use_retry, prompt=_prompts, run_manager=run_manager, **params


I think you need to send the stop words in to acompletion_with_retry_batching?

stop is in params

See other comment, I don't believe this is correct.

libs/langchain/tests/integration_tests/chat_models/test_fireworks.py

libs/langchain/tests/integration_tests/llms/test_fireworks.py

libs/langchain/langchain/llms/fireworks.py

* Support Fireworks batching * Support ChatFireworks batching

libs/langchain/langchain/llms/fireworks.py

libs/langchain/tests/integration_tests/chat_models/test_fireworks.py

libs/langchain/tests/integration_tests/llms/test_fireworks.py

eyurtsev

@baskaryan this looks good to me.

@baskaryan

Description * Add _generate and _agenerate to support Fireworks batching. * Add stop words test cases * Opt out retry mechanism Issue - Not applicable Dependencies - None Tag maintainer - @baskaryan

dosubot bot added Ɑ: models Related to LLMs or chat model modules 🤖:improvement Medium size change to existing code to handle new use-cases labels Oct 19, 2023

eyurtsev self-requested a review October 19, 2023 21:27

eyurtsev self-assigned this Oct 19, 2023

eyurtsev approved these changes Oct 19, 2023

View reviewed changes

eyurtsev requested changes Oct 19, 2023

View reviewed changes

ZixinYang force-pushed the master branch from cf2686e to 6b96e83 Compare October 24, 2023 16:52

ZixinYang requested review from eyurtsev and hwchase17 October 24, 2023 18:21

tjaffri suggested changes Oct 25, 2023

View reviewed changes

tjaffri reviewed Oct 25, 2023

View reviewed changes

libs/langchain/langchain/llms/fireworks.py Show resolved Hide resolved

ZixinYang force-pushed the master branch from 440d857 to fe69c14 Compare October 25, 2023 23:26

ZixinYang commented Oct 25, 2023

View reviewed changes

libs/langchain/langchain/llms/fireworks.py Outdated Show resolved Hide resolved

ZixinYang force-pushed the master branch 2 times, most recently from a0f6adb to df2cb06 Compare October 26, 2023 04:34

ZixinYang added 6 commits October 26, 2023 17:55

Support Fireworks batching (#8)

c3ec6f3

* Support Fireworks batching * Support ChatFireworks batching

Support Fireworks batching

71f628b

Fix lint issue

88f3be1

Add serialisation props to Fireworks and ChatFireworks

72bdc5e

Add type annotation

eb90d57

Update retry mechanism with new error type

b80e462

ZixinYang force-pushed the master branch 2 times, most recently from d923d9c to b80e462 Compare October 26, 2023 18:04

Add invoke/batch test cases and pass stop to batch retry

c2f6ce4

ZixinYang requested a review from tjaffri October 26, 2023 18:20

tjaffri suggested changes Oct 26, 2023

View reviewed changes

libs/langchain/langchain/llms/fireworks.py Outdated Show resolved Hide resolved

libs/langchain/tests/integration_tests/chat_models/test_fireworks.py Outdated Show resolved Hide resolved

libs/langchain/tests/integration_tests/llms/test_fireworks.py Outdated Show resolved Hide resolved

Remove redundant test cases

2c1926a

ZixinYang requested a review from tjaffri October 26, 2023 18:38

eyurtsev approved these changes Oct 26, 2023

View reviewed changes

eyurtsev merged commit 6ce276e into langchain-ai:master Oct 26, 2023
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Fireworks batching (#8) #12052

Support Fireworks batching (#8) #12052

ZixinYang commented Oct 19, 2023 •

edited

Loading

vercel bot commented Oct 19, 2023 •

edited

Loading

eyurtsev Oct 19, 2023

ZixinYang Oct 19, 2023

eyurtsev Oct 24, 2023

ZixinYang Oct 25, 2023

eyurtsev Oct 19, 2023

ZixinYang Oct 23, 2023

eyurtsev Oct 24, 2023

ZixinYang Oct 24, 2023

ZixinYang Oct 25, 2023 •

edited

Loading

eyurtsev left a comment

tjaffri Oct 25, 2023

ZixinYang Oct 25, 2023

tjaffri Oct 25, 2023 •

edited

Loading

eyurtsev left a comment

		@@ -179,6 +228,74 @@ async def _completion_with_retry(**kwargs: Any) -> Any:
		return await _completion_with_retry(**kwargs)


		def completion_with_retry_batching(

Support Fireworks batching (#8) #12052

Support Fireworks batching (#8) #12052

Conversation

ZixinYang commented Oct 19, 2023 • edited Loading

vercel bot commented Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZixinYang Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

eyurtsev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjaffri Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

eyurtsev left a comment

Choose a reason for hiding this comment

ZixinYang commented Oct 19, 2023 •

edited

Loading

vercel bot commented Oct 19, 2023 •

edited

Loading

ZixinYang Oct 25, 2023 •

edited

Loading

tjaffri Oct 25, 2023 •

edited

Loading