Skip to content

Releases: crate/cratedb-toolkit

v0.0.29

13 Oct 19:52
Compare
Choose a tag to compare

What's Changed

  • MongoDB: Added Zyp transformations to the CDC subsystem,
    making it more symmetric to the full-load procedure.
  • Query Converter: Added very basic expression converter utility with
    CLI interface
  • DynamoDB: Added query expression converter for relocating object
    references, to support query migrations after the breaking change
    with the SQL DDL schema, by v0.0.27.

Full Changelog: v0.0.28...v0.0.29

v0.0.28

09 Oct 09:27
Compare
Choose a tag to compare

What's Changed

  • IO: Improved BulkProcessor when running per-record operations by
    also checking rowcount for handling INSERT OK, 0 rows responses
  • MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps
    by updating to commons-codec 0.0.21.
  • Testcontainers: Don't always pull the OCI image before starting.
    It is unfortunate in disconnected situations.

Details

  • Update poethepoet requirement from <0.29 to <0.30 by @dependabot in #283
  • Update validate-pyproject requirement from <0.20 to <0.21 by @dependabot in #284
  • IO: Improve BulkProcessor when running per-record operations by @amotl in #286
  • Testcontainers: Don't always pull the OCI image before starting. by @amotl in #287

Full Changelog: v0.0.27...v0.0.28

v0.0.27

30 Sep 22:24
Compare
Choose a tag to compare

What's Changed

  • DynamoDB: Change CrateDB data model to use (pk, data, aux) columns by @amotl in #281
    Attention: This is a breaking change.

Dependencies

Full Changelog: v0.0.26...v0.0.27

v0.0.26

25 Sep 23:50
Compare
Choose a tag to compare

What's Changed

  • MongoDB: Configure MongoDBCrateDBConverter for updating commons-codec by @amotl in #275
  • DynamoDB CDC: Make MODIFY operation also propagate deleted attributes by @amotl in #276

Dependabot

Full Changelog: v0.0.25...v0.0.26

v0.0.25

22 Sep 12:24
Compare
Choose a tag to compare

What's Changed

  • Table Loader: Improved conditional handling of "transformation" parameter
  • Table Loader: Improved status reporting and error logging in BulkProcessor
  • MongoDB: Improve error reporting
  • MongoDB Full: Polars' read_ndjson doesn't load MongoDB JSON data well, use fsspec and orjson instead
  • MongoDB Full: Improved initialization of transformation subsystem
  • MongoDB Adapter: Improved performance of when computing collection cardinality by using collection.estimated_document_count()
  • MongoDB Full: Optionally use limit parameter as number of total records
  • MongoDB Adapter: Evaluate _id filter field by upcasting to bson.ObjectId, to convey a filter that makes ctk load table process a single document, identified by its OID
  • MongoDB Dependencies: Update to commons-codec 0.0.17

Full Changelog: v0.0.24...v0.0.25

v0.0.24

19 Sep 14:18
Compare
Choose a tag to compare

What's Changed

  • MongoDB Full: Refactor transformation subsystem to commons-codec
  • MongoDB: Update to commons-codec v0.0.16

Details

  • AWS DMS: Add first working version by @amotl in #213
  • MongoDB Full: Refactor transformation subsystem to commons-codec by @amotl in #269

Full Changelog: v0.0.23...v0.0.24

v0.0.23

16 Sep 11:12
Compare
Choose a tag to compare

What's Changed

  • MongoDB: Unlock processing multiple collections, either from server database, or from filesystem directory
  • MongoDB: Unlock processing JSON files from HTTP resource, using https+bson://
  • MongoDB: Optionally filter server collection using MongoDB query expression
  • MongoDB: Improve error handling wrt. bulk operations vs. usability
  • DynamoDB CDC: Add ctk load table interface for processing CDC events
  • DynamoDB CDC: Accept a few more options for the Kinesis Stream:
    batch-size, create, create-shards, start, seqno, idle-sleep, buffer-time
  • DynamoDB Full: Improve error handling wrt. bulk operations vs. usability

Details

  • MongoDB: Improve support for reading JSON/BSON files by @amotl in #261
  • MongoDB: Improve error handling wrt. bulk operations vs. usability by @amotl in #262
  • DynamoDB: Add ctk load table interface for processing CDC events by @amotl in #247
  • Update datasets requirement from <3 to <4 by @dependabot in #264
  • Update aiobotocore requirement from <2.15 to <2.16 by @dependabot in #265
  • Update cattrs requirement from <24 to <25 by @dependabot in #267
  • DynamoDB Full: Improve error handling wrt. bulk operations vs. usability by @amotl in #263

Full Changelog: v0.0.22...v0.0.23

v0.0.22

10 Sep 17:20
Compare
Choose a tag to compare

What's Changed

  • MongoDB: Rename columns with leading underscores to use double leading underscores
  • MongoDB: Add support for UUID types
  • MongoDB: Improve reading timestamps in previous BSON formats
  • MongoDB: Fix processing empty arrays/lists. By default, assume TEXT as inner type.
  • MongoDB: For ctk load table, use "partial" scan for inferring the collection schema,
    based on the first 10,000 documents.
  • MongoDB: Skip leaking UNKNOWN fields into SQL DDL.
    This means relevant column definitions will not be included into the SQL DDL.
  • MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.
  • MongoDB: Sanitize lists of varying objects
  • MongoDB: Add treatment option for applying special treatments to certain items
    on real-world data
  • MongoDB: Use pagination on source collection, for creating batches towards CrateDB
  • MongoDB: Unlock importing MongoDB Extended JSON files using file+bson://...

Details

Full Changelog: v0.0.21...v0.0.22

v0.0.21

02 Sep 14:24
Compare
Choose a tag to compare

What's Changed

  • DynamoDB: Add special decoding for varied lists.
    Store them into a separate OBJECT(IGNORED) column in CrateDB.
  • DynamoDB: Add pagination support for full-load table loader

Details

  • DynamoDB: Add software test for decoding list of objects by @amotl in #246
  • DynamoDB: Improve software test about decoding a list with empty string by @amotl in #248
  • Update polars requirement from <1.6 to <1.7 by @dependabot in #250
  • Update poethepoet requirement from <0.28 to <0.29 by @dependabot in #251
  • Revert "CI: Codecov API is currently down" by @amotl in #253
  • DynamoDB: Add pagination support for full-load table loader by @amotl in #252

Full Changelog: v0.0.20...v0.0.21

v0.0.20

27 Aug 09:47
Compare
Choose a tag to compare

What's Changed

  • DMS/DynamoDB: Fix table name quoting within CDC processor handler by @amotl in #245

Full Changelog: v0.0.19...v0.0.20