Make Cohere provider AF3 compatible #51396
Conversation
boring-cyborg
bot
added
area:providers
provider:cohere
provider:pinecone
provider:weaviate
labels
eladkal
left a comment
eladkal
left a comment
There was a problem hiding this comment.
Can you please ammed commit message? Changes are on multiple providers.
Changes are specific to Cohere only. We have some integration system tests which are from other providers |
Then what is the motivation to include them here? If there is another reason for having the changes best to do it in a seperated PR |
We have interation system DAGs from pinecone and Weaviate which uses cohere operator in these tests are kind of dependent on these changes. |
|
Looking at failing tests |
…nto fix-cohere-provider
|
I will raise separate PR for Pinecone and Weaviate. |
|
@vatsrahul1001 #50867 tracks an effort to make the serialisation work for cohere embeddings, it is not due to pickling imo, it probably is because we do not have support for ser / deser pydantic types as |
* make cohere provider AF3 compatible
|
Yes. I agree with @amoghrajesh. If |
| return embeddings | ||
| if response.embeddings.float_ is None: | ||
| raise ValueError("Embeddings response is missing float_ field") | ||
| return response.embeddings.float_ |
There was a problem hiding this comment.
I would like to provide some of my findings about this change. As shown https://github.com/cohere-ai/cohere-python/blob/main/src/cohere/types/embed_by_type_response_embeddings.py. There could be a case that the embedding is stored in other fields. I also share some thoughts in the comment, and it will be better if the pydantic model can be handled by XComs serde.
There was a problem hiding this comment.
For now its ok, just added a comment so that people arent confused: #51517
5 participants
I noticed that when we try to create embeddings using Cohere, we encounter a serialization error in Airflow 3. Since pickling has been removed from XCom in AF3, this might be causing the error:
TypeError: cannot serialize object of type <class 'cohere.types.embed_by_type_response_embeddings.EmbedByTypeResponseEmbeddings'>.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.