MongoDB: More ad hoc fixes for supporting real-world data #255

amotl · 2024-09-02T18:28:21Z

About

Add support for UUID types.
Improve reading timestamps in previous BSON formats like end_date: Long('1455141600000').
Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.
Add relevant end-to-end integration test.
Sanitize lists of varying objects.
~~Add --treatment option for applying special treatments to certain items on real-world data.~~
Improve Zyp Transformations (--transformation option) for applying special treatments to certain items on real-world data.
Use pagination on source collection, for creating batches towards CrateDB.
Unlock importing MongoDB Extended JSON files using file+bson://....

Documentation

https://cratedb-toolkit--255.org.readthedocs.build/io/mongodb/loader.html

Install

pip install --upgrade 'cratedb-toolkit[mongodb] @ git+https://github.com/crate/cratedb-toolkit.git@mongodb-more'

Backlog

Add relevant software tests.
Documentation. For example, parameter batch-size.

/cc @hlcianfagna, @hammerhead, @lservini, @zolbatar, @juanpardo, @widmogrod

amotl · 2024-09-02T19:40:02Z

cratedb_toolkit/io/mongodb/export.py

 if isinstance(value, dict):
 if len(value) == 1:
+ if "$binary" in value and value["$binary"]["subType"] in ["03", "04"]:
+ decoded = UUID(bytes=base64.b64decode(value["$binary"]["base64"]))
+ return extract_value(decoded, parent_type)
 for k, v in value.items():
 if k.startswith("$"):
 return extract_value(v, k.lstrip("$"))


Sure enough, decoding MongoDB Extended JSON should not be performed manually, but should use _json_convert instead, like the CDC subsystem is doing it already.

Through decode_bson, the machinery now uses BSON's _json_convert.

cratedb-toolkit/cratedb_toolkit/io/mongodb/copy.py

Lines 52 to 59 in bc9362b

# Converge multiple MongoDB documents into SQL parameters for `executemany` operation.

parameters: t.List[DocumentDict] = []

for document in data:

record = self.converter.convert(self.decode_bson(document))

oid: str = self.get_document_key(record)

parameters.append({"oid": oid, "record": record})

return SQLOperation(sql, parameters)

cratedb_toolkit/io/mongodb/extract.py

amotl · 2024-09-02T19:45:12Z

tests/io/mongodb/test_cli.py

+DOCUMENT_IN = {
+ "id": bson.Binary.from_uuid(UUID("d575540f-759d-4653-a4c4-4a9e410f1aa1")),
+ "value": {
+ "name": "foobar",
+ "active": True,
+ "created": dateutil.parser.parse("2020-06-19T15:03:53.727Z"),
+ "timestamp": bson.datetime_ms.DatetimeMS(1455141600000),
+ },
+}
+DOCUMENT_OUT = {
+ "__id": mock.ANY,
+ "id": "d575540f-759d-4653-a4c4-4a9e410f1aa1",
+ "value": {
+ "name": "foobar",
+ "active": True,
+ "created": 1592579033000,
+ "timestamp": 1455141600000,
+ },
+}
+DOCUMENT_DDL = """
+CREATE TABLE IF NOT EXISTS "testdrive"."demo" (
+ "__id" TEXT,
+ "id" TEXT,
+ "value" OBJECT(DYNAMIC) AS (
+ "name" TEXT,
+ "active" BOOLEAN,
+ "created" TIMESTAMP WITH TIME ZONE,
+ "timestamp" TIMESTAMP WITH TIME ZONE
+ )
+)""".lstrip()


Best view into the integration test added with bf78869 and 249cac1.

wierdvanderhaar

Lets a go.

amotl · 2024-09-05T08:15:09Z

cratedb_toolkit/io/mongodb/export.py

+ """
+ # Manual treatment.
+ # Some nested objects have been defined as strings, probably in previous schema versions.
+ if "users" in value:
+ for user_item in value["users"]:
+ if "user" in user_item and not isinstance(user_item["user"], dict):
+ user_item["user"] = {"id": user_item["user"]}
+ """
+
+ if "createdBy" in value and not isinstance(value["createdBy"], dict):
+ value["createdBy"] = {"id": value["createdBy"]}
+
+ # Prune invalid date representations.
+ for key in ["start_date", "end_date"]:
+ if key in value:
+ if not isinstance(value[key], dict):
+ del value[key]
+ elif "date" in value[key]:
+ if isinstance(value[key]["date"], str):
+ del value[key]


While others are configurable per --treatment=Path option using a corresponding YAML file now, those three special treatments have not been generalized yet.

Those rules now have been generalized further to be controlled using the treatment file.

convert_dict: - name: createdBy wrapper_name: id - name: user wrapper_name: id prune_invalid_date: - "start_date" - "end_date"

The --treatment option has been dissolved again. Special treatments have been integrated into Zyp.

Zyp Treatments: A slightly tailored transformation subsystem commons-codec#46

zyp-treatment-all.yaml is an example file representing a Zyp project that includes definitions of special treatments for a specific collection.

amotl · 2024-09-10T13:23:21Z

pyproject.toml

- "commons-codec[mongodb,zyp]>=0.0.14",
+ "commons-codec[mongodb,zyp] @ git+https://github.com/crate/commons-codec@zyp-treatments",


Of course, this needs another release of commons-codec beforehand.

commons-codec v0.0.15 has been released.

-- https://pypi.org/project/commons-codec/

... to use double leading underscores.

bson.errors.InvalidBSON: year 292278994 is out of range Consider Using CodecOptions(datetime_conversion=DATETIME_AUTO) or MongoClient(datetime_conversion='DATETIME_AUTO'). See: https://pymongo.readthedocs.io/en/stable/examples/datetimes.html#handling-out-of-range-datetimes

By default, assume `TEXT` as inner type.

... based on the first 10,000 documents.

This means relevant column definitions will not be included into the SQL DDL.

- Use `--transformation` option for applying special treatments. Certain fields should be stored as lists, some need to be ignored for now, others need to be treated manually, etc. - Use pagination on source collection, for creating batches towards CrateDB.

amotl force-pushed the mongodb-more branch from 0a6ba2f to 54937b1 Compare September 2, 2024 19:26

amotl commented Sep 2, 2024

View reviewed changes

amotl force-pushed the mongodb-more branch from 0ff5bb7 to 249cac1 Compare September 2, 2024 19:44

amotl commented Sep 2, 2024

View reviewed changes

amotl force-pushed the mongodb-more branch 15 times, most recently from 66399b9 to c899a30 Compare September 4, 2024 00:08

amotl requested a review from wierdvanderhaar September 4, 2024 17:21

wierdvanderhaar approved these changes Sep 4, 2024

View reviewed changes

amotl force-pushed the mongodb-more branch 6 times, most recently from 7c8e999 to bc9362b Compare September 5, 2024 08:13

amotl commented Sep 5, 2024

View reviewed changes

amotl force-pushed the mongodb-more branch 2 times, most recently from 6ec4aab to 5802e19 Compare September 6, 2024 08:24

amotl mentioned this pull request Sep 7, 2024

Zyp Treatments: A slightly tailored transformation subsystem crate/commons-codec#46

Merged

amotl force-pushed the mongodb-more branch 3 times, most recently from 60193ce to f9ddac5 Compare September 10, 2024 13:09

amotl commented Sep 10, 2024

View reviewed changes

amotl marked this pull request as ready for review September 10, 2024 13:23

amotl mentioned this pull request Sep 10, 2024

MongoDB: General backlog #260

Open

8 tasks

amotl added 12 commits September 10, 2024 17:41

MongoDB: Rename columns with leading underscores

00862af

... to use double leading underscores.

MongoDB: Add support for UUID types

8f3aeab

MongoDB: Add integration test

b15743d

MongoDB: Improve test case about different types of timestamp fields

9d14596

MongoDB: Fix processing empty arrays/lists

a4be365

By default, assume `TEXT` as inner type.

MongoDB: For ctk load table, use "partial" scan for inferring schema

dcfb8b7

... based on the first 10,000 documents.

MongoDB: Skip leaking UNKNOWN fields into SQL DDL

abf1e81

This means relevant column definitions will not be included into the SQL DDL.

MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) strategy

2ccc0b2

MongoDB: Sanitize lists of varying objects

befd579

MongoDB: Unlock importing MongoDB Extended JSON using file+bson://...

cf8213d

amotl force-pushed the mongodb-more branch from f9ddac5 to cf8213d Compare September 10, 2024 15:42

amotl merged commit 8fc1a22 into main Sep 10, 2024
31 checks passed

amotl deleted the mongodb-more branch September 10, 2024 17:05

amotl mentioned this pull request Sep 11, 2024

MongoDB: Improve support for reading JSON/BSON files #261

Merged

amotl mentioned this pull request Sep 18, 2024

MongoDB: Code refactoring and generalization crate/commons-codec#49

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MongoDB: More ad hoc fixes for supporting real-world data #255

MongoDB: More ad hoc fixes for supporting real-world data #255

amotl commented Sep 2, 2024 •

edited

Loading

amotl Sep 2, 2024

amotl Sep 5, 2024

amotl Sep 2, 2024

wierdvanderhaar left a comment

amotl Sep 5, 2024 •

edited

Loading

amotl Sep 6, 2024

amotl Sep 8, 2024 •

edited

Loading

amotl Sep 10, 2024

amotl Sep 10, 2024

	# Converge multiple MongoDB documents into SQL parameters for `executemany` operation.
	parameters: t.List[DocumentDict] = []
	for document in data:
	record = self.converter.convert(self.decode_bson(document))
	oid: str = self.get_document_key(record)
	parameters.append({"oid": oid, "record": record})

	return SQLOperation(sql, parameters)

		"commons-codec[mongodb,zyp]>=0.0.14",
		"commons-codec[mongodb,zyp] @ git+https://github.com/crate/commons-codec@zyp-treatments",

MongoDB: More ad hoc fixes for supporting real-world data #255

MongoDB: More ad hoc fixes for supporting real-world data #255

Conversation

amotl commented Sep 2, 2024 • edited Loading

About

Documentation

Install

Backlog

amotl Sep 2, 2024

Choose a reason for hiding this comment

amotl Sep 5, 2024

Choose a reason for hiding this comment

amotl Sep 2, 2024

Choose a reason for hiding this comment

wierdvanderhaar left a comment

Choose a reason for hiding this comment

amotl Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Sep 6, 2024

Choose a reason for hiding this comment

amotl Sep 8, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Sep 10, 2024

Choose a reason for hiding this comment

amotl Sep 10, 2024

Choose a reason for hiding this comment

amotl commented Sep 2, 2024 •

edited

Loading

amotl Sep 5, 2024 •

edited

Loading

amotl Sep 8, 2024 •

edited

Loading