Skip to content

Commit

Permalink
fix: uninstall bson for mongo connector (#3104)
Browse files Browse the repository at this point in the history
### Summary

Closes #3049. Reenables the MongoDB connector test, which was disabled
previously in #3047 due to incompatibility between the `pymongo` and the
`bson` package from `pip`, which is a dependency for the Astra
connector. Per the `pymongo` docs below, `pymongo` ships with its own
version of `bson` and installing `bson` from `pip` breaks `pymongo`.

- https://pymongo.readthedocs.io/en/stable/installation.html

### Testing

Ingest tests ran successfully for the [source
connector](https://github.com/Unstructured-IO/unstructured/actions/runs/9273154676/job/25512636315)
and the [destination
connector](https://github.com/Unstructured-IO/unstructured/actions/runs/9273154676/job/25512635546).
  • Loading branch information
MthwRobinson authored May 28, 2024
1 parent 6b400b4 commit 3158169
Show file tree
Hide file tree
Showing 4 changed files with 16 additions and 10 deletions.
7 changes: 7 additions & 0 deletions test_unstructured_ingest/dest/mongodb.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ function cleanup() {

trap cleanup EXIT

# NOTE(robinson) - per pymongo docs, pymongo ships with its own version of the bson library,
# which is incompatible with the bson installed from pypi. bson is installed as part of the
# astra dependencies.
# ref: https://pymongo.readthedocs.io/en/stable/installation.html
pip uninstall -y bson pymongo
make install-ingest-mongodb

python "$SCRIPT_DIR"/python/test-ingest-mongodb.py \
--uri "$MONGODB_URI" \
--database "$MONGODB_DATABASE_NAME" \
Expand Down
7 changes: 6 additions & 1 deletion test_unstructured_ingest/src/mongodb.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,12 @@ if [ -z "$MONGODB_URI" ] && [ -z "$MONGODB_DATABASE_NAME" ]; then
exit 8
fi

# trap cleanup EXIT
# NOTE(robinson) - per pymongo docs, pymongo ships with its own version of the bson library,
# which is incompatible with the bson installed from pypi. bson is installed as part of the
# astra dependencies.
# ref: https://pymongo.readthedocs.io/en/stable/installation.html
pip uninstall -y bson pymongo
make install-ingest-mongodb

PYTHONPATH=. ./unstructured/ingest/main.py \
mongodb \
Expand Down
5 changes: 1 addition & 4 deletions test_unstructured_ingest/test-ingest-dest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,7 @@ all_tests=(
'sqlite.sh'
'vectara.sh'
'weaviate.sh'
# NOTE(robinson) - mongo conflicts with astra because it ships with its
# own version of bson, and installing bson from pip causes mongo to fail
# ref: https://pymongo.readthedocs.io/en/stable/installation.html
# 'mongodb.sh'
'mongodb.sh'
)

full_python_matrix_tests=(
Expand Down
7 changes: 2 additions & 5 deletions test_unstructured_ingest/test-ingest-src.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ all_tests=(
'confluence-diff.sh'
'confluence-large.sh'
'airtable-diff.sh'
# NOTE(ryan): This test is disabled because it is triggering too many requests to the API
# # NOTE(ryan): This test is disabled because it is triggering too many requests to the API
# 'airtable-large.sh'
'local-single-file.sh'
'local-single-file-basic-chunking.sh'
Expand All @@ -62,10 +62,7 @@ all_tests=(
'local-embed-voyageai.sh'
'sftp.sh'
'opensearch.sh'
# NOTE(robinson) - mongo conflicts with astra because it ships with its
# own version of bson, and installing bson from pip causes mongo to fail
# ref: https://pymongo.readthedocs.io/en/stable/installation.html
# 'mongodb.sh'
'mongodb.sh'
)

full_python_matrix_tests=(
Expand Down

0 comments on commit 3158169

Please sign in to comment.