Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MongoDB: More ad hoc fixes for supporting real-world data #255

Merged
merged 12 commits into from
Sep 10, 2024
Merged

MongoDB: More ad hoc fixes for supporting real-world data #255

merged 12 commits into from
Sep 10, 2024

Commits on Sep 10, 2024

  1. MongoDB: Rename columns with leading underscores 

    ... to use double leading underscores.
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    00862af View commit details
    Browse the repository at this point in the history

  2. MongoDB: Add support for UUID types

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8f3aeab View commit details
    Browse the repository at this point in the history

  3. MongoDB: Add integration test

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    b15743d View commit details
    Browse the repository at this point in the history

  4. MongoDB: Improve reading timestamps in previous BSON formats 

    bson.errors.InvalidBSON: year 292278994 is out of range

Consider Using CodecOptions(datetime_conversion=DATETIME_AUTO) or MongoClient(datetime_conversion='DATETIME_AUTO').

See: https://pymongo.readthedocs.io/en/stable/examples/datetimes.html#handling-out-of-range-datetimes
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8cdbd02 View commit details
    Browse the repository at this point in the history

  5. MongoDB: Improve test case about different types of timestamp fields

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    9d14596 View commit details
    Browse the repository at this point in the history

  6. MongoDB: Fix processing empty arrays/lists 

    By default, assume `TEXT` as inner type.
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    a4be365 View commit details
    Browse the repository at this point in the history

  7. MongoDB: For ctk load table, use "partial" scan for inferring schema 

    ... based on the first 10,000 documents.
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    dcfb8b7 View commit details
    Browse the repository at this point in the history

  8. MongoDB: Skip leaking UNKNOWN fields into SQL DDL 

    This means relevant column definitions will not be included into the
SQL DDL.
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    abf1e81 View commit details
    Browse the repository at this point in the history

  9. MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) strategy

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    2ccc0b2 View commit details
    Browse the repository at this point in the history

  10. MongoDB: Sanitize lists of varying objects

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    befd579 View commit details
    Browse the repository at this point in the history

  11. MongoDB: Accept Zyp Treatments, and ingress pagination / egress batching 

    - Use `--transformation` option for applying special treatments.
  Certain fields should be stored as lists, some need to be ignored for
  now, others need to be treated manually, etc.

- Use pagination on source collection, for creating batches towards
  CrateDB.
    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    4297f5d View commit details
    Browse the repository at this point in the history

  12. MongoDB: Unlock importing MongoDB Extended JSON using file+bson://...

    @amotl
    amotl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    cf8213d View commit details
    Browse the repository at this point in the history