v0.12.0

Attention: Please be aware that this version introduces a new check to make sure that replica identities of the tables are set up correctly. A misconfigured replica identity may prevent deletes or updates on tables, hence issues in other parts of the system.

Changelog

• c06c1da Added documentation on how to configure the CDC / replication user
• 32cdc23 Added a replica identity sanity check, to make sure that issues with updates and deletes are prevented
• c80e5a5 Fixed AWS SQS integration test
• a6c2508 Fixes CAGG name on registration for system catalog
• 4d4f3bb Fixes #132: Race condition due to byte-array being sliced not cloned
• e1dada4 Added configuration option for stats port
• 9dc7afc Fixed #117: if the statestorage folder cannot be created and doesn't exist yet, fail hard
• 57ab5e8 Upgraded to Go 1.21.x, exchanged lo./functional. for slices. functionality
• 80d01a7 Added REPLCIA IDENTITY FULL information
• be2ffdd Added information for decompression markers
• 6f28cf0 Bumped localstack docker image to 3.0.1
• 0c99aa9 Update main.go
• 957e7ec Update README.md
• 1ca8348 Bump github.com/aws/aws-sdk-go from 1.45.19 to 1.48.15
• c7faa32 Bump github.com/samber/lo from 1.38.1 to 1.39.0
• 10d63fb Bump golang.org/x/net from 0.18.0 to 0.19.0
• 0e3ae8c Bump github.com/antonmedv/expr from 1.15.3 to 1.15.5
• 4ed0872 Bump github.com/testcontainers/testcontainers-go/modules/redis
• 023aefe Bump github.com/testcontainers/testcontainers-go/modules/localstack
• 3cdbe33 Bump github.com/testcontainers/testcontainers-go from 0.25.0 to 0.26.0
• bf5fa71 Bump github.com/twpayne/go-geom from 1.5.2 to 1.5.3
• bb80ac6 Bump github.com/gookit/goutil from 0.6.12 to 0.6.14
• d622ab8 Bump github.com/jackc/pgx/v5 from 5.4.3 to 5.5.0
• 4a79b91 Bump github.com/IBM/sarama from 1.41.2 to 1.42.1
• 22c5d74 Bump github.com/docker/docker
• 10aa4a8 Bump golang.org/x/net from 0.15.0 to 0.18.0
• 11368ac Bump github.com/nats-io/nats.go from 1.30.2 to 1.31.0
• b7130a0 Bump github.com/go-errors/errors from 1.5.0 to 1.5.1

v0.11.0

Release v0.11.0 provides extended support for TimescaleDB 2.12+, which provides a special marking capability to filter insert statements which occur due to some internal decompressions (when inserting, updating, deleting into compressed chunks). To enable this behavior, set the timescaledb.enable_decompression_logrep_markers=on in postgresql.conf.

Furthermore, TimescaleDB 2.12+ supports for REPLICA IDENTITY FULL (USING INDEX isn't fully working yet) which provides the possibility to capture old values when deleting or updating rows in hypertables.

Special thanks to @flexwende and @vidosits 🙏

Changelog

• 5bf7e89 Merge pull request #76 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.45.19
• b51439d Merge pull request #85 from noctarius/unit-test-decompression
• 5f9913a Fixed broken unit test
• 6574927 Merge pull request #87 from noctarius/split-unit-pg-tests
• 1904dc5 Made decompression events work with pre-, and post-2.12
• c55911b Merge branch 'main' into split-unit-pg-tests
• 68e3695 Merge pull request #86 from noctarius/added-integration-test-http
• 1dc6bf2 Split unit tests and pg tests apart for a small speedup in builds
• 68c2dcd Enabled automatic integration test for the http sink
• 5ff11aa Merge pull request #67 from vidosits/feature/http-sink
• 75ff2dd Disabled test Test_Hypertable_Compression_Partial_Insert_Events on <PG14 and <TSDB2.12
• f009f5b Removed leftover logging
• 0b78967 Cloning the message content to prevent data race
• 05da944 Added unit test for implicit decompressions
• c626ac3 Merge pull request #84 from noctarius/unit-test-replica-identity
• 68f9da1 Fixes #70: Implemented unit test and check for replica identity
• e6ab3d9 Merge pull request #83 from noctarius/unit-test-logical-rep-message
• 46bf440 Added unit test for logical replication message
• a6f6cf5 Merge pull request #82 from noctarius/add-tsdb-version-to-workflow
• a7be322 Merge pull request #81 from noctarius/snapshot-restarting
• 556c578 Merge pull request #80 from noctarius/fix-79
• 0a6a72a Added a forced pre-2.12 build and a latest build to capture both 2.12+ versions with support for decompression markers and <2.12 builds with transaction collectors
• 633c673 Removed ineffective assignment for final offset value which isn't used as of now
• 60efe62 Bump github.com/aws/aws-sdk-go from 1.45.11 to 1.45.19
• fa1f4aa Merge pull request #77 from noctarius/dependabot/go_modules/github.com/testcontainers/testcontainers-go/modules/localstack-0.25.0
• df16b36 Fixes #79: potential race condition in cases where the dispatcher is stopped before fully started
• 43d8a7c Added initial snapshotting for vanilla PG tables
• 8e52139 Bump github.com/testcontainers/testcontainers-go/modules/localstack
• abe7fed Merge pull request #78 from noctarius/dependabot/go_modules/github.com/testcontainers/testcontainers-go/modules/redis-0.25.0
• 8de87a7 Bump github.com/testcontainers/testcontainers-go/modules/redis
• 08787c7 Merge pull request #60 from noctarius/dependabot/go_modules/github.com/testcontainers/testcontainers-go-0.24.1
• 946f788 Merge pull request #74 from noctarius/dependabot/go_modules/github.com/nats-io/nats.go-1.30.2
• d14490c Bump github.com/nats-io/nats.go from 1.30.0 to 1.30.2
• e86c659 Started reimplementing snapshotting with support for pg vanilla tables
• 79847be Extended offset to provide space for the current snapshot keyset in addition to an offset, as well as a bool which tells if the snapshotting was finished
• 776fed6 Fix: Add Content-Type: application/json to HTTP sink request headers
• d5c63f5 Use http server instance in http integration test so we can cleanly tear it down
• 0900af5 Bump github.com/testcontainers/testcontainers-go from 0.23.0 to 0.24.1
• 396ab1c Merge pull request #68 from noctarius/dependabot/go_modules/github.com/antonmedv/expr-1.15.3
• 28fb26f HTTP sink: Infer TLS presence from url prefix
• bdde6c4 HTTP sink: Emit schema along with the payload
• ec2d34c Bump github.com/antonmedv/expr from 1.15.2 to 1.15.3
• 4d03d03 Tests: Add integration test for http sink
• 6382614 Fix: Linting issues in sink/http/http.go
• 7e6542a Document HTTP Sink
• 5d931d8 Implement Http Sink
• 3481100 Merge pull request #62 from noctarius/dependabot/go_modules/github.com/nats-io/nats.go-1.30.0
• c724fe3 Merge pull request #66 from vidosits/fix/typos
• e6a67f5 Fix: Minor typos
• 4f5344a Bump github.com/nats-io/nats.go from 1.29.0 to 1.30.0

v0.10.1

Changelog

• 35b3eb5 Merge pull request #64 from noctarius/fix-issue-63
• a3f5f9c Fixed remaining booleans
• ff504cf Fixes #63 event property configuration is ignored

v0.10.0

Changelog

• 96e477d Merge pull request #48 from noctarius/dependabot/go_modules/github.com/nats-io/nats.go-1.29.0
• 6a4a951 Merge pull request #51 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.45.11
• 60ed29a Merge pull request #50 from noctarius/dependabot/go_modules/github.com/go-errors/errors-1.5.0
• 4b3e242 Merge pull request #52 from noctarius/support-for-vanilla-tables
• 64447bd Added truncate events for vanilla PG tables
• 47c68c7 Added documentation for vanilla PG tables
• 36e505a Added support to replicate vanilla PostgreSQL tables
• 0971187 Bump github.com/aws/aws-sdk-go from 1.45.9 to 1.45.11
• e685bbb Bump github.com/go-errors/errors from 1.4.2 to 1.5.0
• 886a43a Bump github.com/nats-io/nats.go from 1.28.0 to 1.29.0

v0.9.2

Changelog

• f3833ee Merge pull request #42 from noctarius/dependabot/go_modules/github.com/antonmedv/expr-1.15.2
• 505ca65 Merge branch 'main' into dependabot/go_modules/github.com/antonmedv/expr-1.15.2
• ee4f6f4 Bump github.com/antonmedv/expr from 1.14.3 to 1.15.2
• 899266f Merge pull request #45 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.45.9
• 8754762 Merge pull request #34 from noctarius/dependabot/go_modules/github.com/docker/docker-24.0.6incompatible
• 061a977 Merge pull request #35 from noctarius/dependabot/go_modules/golang.org/x/net-0.15.0
• 8c3da3f Merge pull request #43 from noctarius/dependabot/go_modules/github.com/IBM/sarama-1.41.2
• ba55475 Bump github.com/aws/aws-sdk-go from 1.44.331 to 1.45.9
• 9edff0c Bump github.com/IBM/sarama from 1.41.0 to 1.41.2
• c5491d1 Bump golang.org/x/net from 0.14.0 to 0.15.0
• bbba4c1 Bump github.com/docker/docker
• 3314506 Merge pull request #47 from noctarius/nullable-columns
• 072d3c0 Fixed nullable columns

v0.9.0

Changelog

• 82bf72b Disabled proc stats for anything other but Linux for now... Cross compiling needs to be fixed first.
• c9b4f9a Moved Darwin release build back into the default build
• acc7c4e Merge branch 'prepare-release-0.9.0'
• a37bad6 Disabled darwin(MacOS) ARM64 build for now. Can still be built locally.
• bfb3071 Merge branch 'prepare-release-0.9.0'
• fcb8dd2 Fixed build for Windows / FreeBSD / Darwin
• 793bc4d Added newly supported data types to the table
• 0f5e657 Merge pull request #25 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.331
• a12b51b Merge pull request #23 from noctarius/postgis-support
• abcb05c Removed ARM builds, as the 2GB limit may be a hard sell
• 25ec2a5 Bump github.com/aws/aws-sdk-go from 1.44.330 to 1.44.331
• 5fafbcf Unified codec implementation for all PostGIS types, as all of them use the same encoding
• d222381 Added support for PostGIS's geography data type
• 0916039 Added a generic text array codec that supports the delimiter value, known to pgcatalog, when parsing the text representation of an array
• fbee5d3 Added geometry and gemometry array data type support
• 1eea4b4 Merge pull request #24 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.330
• c5ecc4f Bump github.com/aws/aws-sdk-go from 1.44.329 to 1.44.330
• bd761cb Merge pull request #22 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.329
• f27cb99 Merge branch 'main' into dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.329
• 3983895 Merge pull request #21 from noctarius/dependabot/go_modules/github.com/IBM/sarama-1.41.0
• ef75011 Merge branch 'main' into dependabot/go_modules/github.com/IBM/sarama-1.41.0
• 438835f Merge pull request #19 from noctarius/dependabot/go_modules/github.com/antonmedv/expr-1.14.3
• 08ada52 Bump github.com/aws/aws-sdk-go from 1.44.324 to 1.44.329
• fdd7b72 Bump github.com/IBM/sarama from 1.40.1 to 1.41.0
• 1b8b99b Bump github.com/antonmedv/expr from 1.13.0 to 1.14.3
• 58ee59c Merge pull request #13 from noctarius/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.324
• f797d94 Bump github.com/aws/aws-sdk-go from 1.44.322 to 1.44.324
• 4f98244 Added dependabot configuration
• e7ed3d5 Merge pull request #11 from noctarius/upgrade-dependency-versions
• e7611b3 Bumped x/net dependency
• 02ff50c Bumped nats dependency
• 8b4bdcc Bumped pgx dependency
• e0a5637 Bumped slog dependency
• 1ae70fa Bumped testcontainers dependency
• 751ccbe Bumped aws-sdk dependency
• 6a29eda Bumped expr dependency
• 1ac56de Bumped sarama dependency
• 3d528d2 Added support for PostGIS data types

v0.4.0

Changelog

• 48ed86e Added support for enum types. Enums use schema type STRING.
• f77871c Fixed default schema assignment for repl_user and created extensions in schemas
• f0a3525 Fixed some logging issues when logging redirection to stderr was enabled
• 0a17e5c More api extraction to break and prevent import cycles in the future
• a1a2fcc SideChannel cleanup and clarification of SQL function names
• 22f0cfb Added missing license headers
• ff5f3ee Lowered contention overhead
• 36d9962 Fixed read lock which should be a write lock
• 8006ddd Added a concurrent map wrapper (generified the sync.Map) and used it for the planner cache
• 50936e5 Added decoder plan caching for streaming values
• 4d0267b Added unnamed fields to create a compile time check that providers and the factory methods match
• ef14086 Added external json encoding library to improve encoding speed and allocation rate when transforming envelopes to json documents
• bb81f97 Updated all references of internal.dispatcher.maxqueuesize to the new internal.dispatcher.initialqueuecapacity
• 592af63 Added additional logging around restart points and skipped messages
• c2cf9ad Added functions to check for a specific log level to be enabled. Those function can be used as guards around more complex log prepartion.
• ebbc4a1 Fixed exceeding queue size when snapshotting large tables
• cbcaf4c Cleaned up the context naming mess after renaming the replicationcontext package
• 354e62a Renamed package context to replicationcontext
• 1d7a9e9 Added internal.snapshotter.parallelism property to define the partition count for the snapshotter
• 2a128be Splitting package 'supporting' into separate, more matching packages: Part 3, functional/testing
• aef0a4c Splitting package 'supporting' into separate, more matching packages: Part 2, containers
• 380f033 Splitting package 'supporting' into separate, more matching packages: Part 1, erroring
• 0f4a159 Introduced basic dependency injection to simplify the service construction and overriding (in tests)
• 7b718d1 Removed 'impl' from struct names
• 2095493 Extracted SideChannel into it's own package, breaking the dependency on ReplicationContext
• fc4e34e Added missing license headers
• 40fc4ad Added license header check
• bd0f30b Adding dependency license check
• e3de968 Unified method parameter style
• d1317a3 Fixed formating error
• 2f69317 Added unit tests to check config properties and tags
• b5c5eac More cleanup and removing internal code for samber/lo imports
• 5adbcd5 A lot of cleanup and separation of concern / merging of packages

v0.3.1

Changelog

• d16f06d Added build constraints for the plugin system since it's only available on freebsd, linux, and macos
• 12a5cbc Added a basic plugin infrastructure to provide additional strategies for naming, sink, and state storage
• 40a6483 Renamed file to match package name
• 67b3103 Simplified sink registry initialization in alignment with the other two registries (namingstrategy, statestorage)
• ceba2e1 Fixed registry initialization (namingstrategy and statestorage) after package merging
• 9dbf998 Fixed formating error
• 478a3ae Merged statestorage packages
• 08e3738 Merged namingstrategy packages
• 4037761 Moved all testing related packaged to top level. They don't really belong in internal
• a7dd030 Merged multiple schema packages to simplify structure
• 2b5816a Fixed index field schema property not being set
• 3c83872 Merged multiple pg packages to simplify structure
• 5fbfb20 Fixed unit test compile error
• 8dbd436 Fixed import cycle
• f6df294 Implemented Format function on Column
• 4b29372 Implemented Format function on PgType
• dd418ca Constified the OIDs not officially supported by pgx
• f69abf8 Fixed case fallthrough in some oid tests
• b0917d1 Fixed quoted char in the README
• 7025b3f Added length property for fixed length characters and bit data types

v0.3.0

Changelog

• a49245d Added a table of all supported and tested data types, as well as their schema type mapping
• d4bc36c Enabled BPChar Array
• 28bc47c Renamed PG Name to Name
• db0599a Added xml data type support
• c830f79 Fixed ltree scanning
• ece0651 Added filtering of previously known state in the state storage to remove known but non-existing chunks - may be removed or compressed while offline
• adfba62 Moved normal PG tests back to hosted
• a1b7000 Try to run integration tests on hosted runners again
• 33caf55 Fixed unreachable function due to reorganization of integration tests
• 92c0d50 Split tests into unit, pg (standard tests) and integration tests
• 9f5d88d Revert "Removed self-hosted runner for testing"
• c5e9e26 Removed self-hosted runner for testing
• 3190220 Moving the creation of well defined SchemaBuilders to reusable functions
• 7f63fa7 Various optimizations, mostly preventing object allocation
• 332733c Improved wait strategy for TimescaleDB container startup
• 4b0ecb1 Added immediate fail test when errors happen when connecting to the TimescaleDB container
• 3cee042 More SchemaBuilder usage
• 2aed16b Initial bits use the new SchemaBuilder and removed the Schema type, going for Struct directly
• 8083820 Add IsPrimitive to Type
• 1c6db03 Little bit cleanup
• b119baf Fixed message for creating PostGIS extension
• f32ee4e Changed SchemaBuilder to immediately return the schema as Struct
• 3f0322b Implementation of the new SchemaBuilder
• beddf99 Renamed schemabuilder.go -> schema.go
• 7581a4a Added Bytes Schema
• dfd0889 Removed all old SchemaBuilder references
• 5ba1135 Renamed SchemaType to Type
• 4105d20 Renamed BaseSchemaType to SchemaType
• d3c64f7 Added namespace to PgType
• 61b739c Fixed lint issue
• 08012da Unified the type registrations between core and dynamic, but optimized, types
• 3272a68 Added support for ltree, fixed some other types or fixed their representation
• c7e3c03 Fixed numeric (array) type, added basic range types

v0.2.0

Changelog

• d2bff1a Added a pattern to .gitignore to exclude test config files
• 2dd9ff4 Merge pull request #9 from noctarius/pgtype-rework
• c32eef8 Added the now supported array types to the fast track types
• 3930d0f Enabled more array types
• 9c22a75 Disabled fail fast for build matrix errors
• e722d23 Disabled cache on setup-go
• 648a2be Removed old macos part of the test process
• c8be041 Remove go module caching
• 5db7881 Move pg integration tests to self-hosted runner
• 73e5462 Fixed potential nil pointer
• d6c75d4 Implemented the first set of array column data types
• bed5377 First steps to implement a simple Array type test
• 5d2d6da Updated tests with the new packagename
• 9dab01a Implemented schema type resolution against the new TypeManager and Type classes
• 4b16131 Moving all schema building related types and interfaces together into the new schemamodel package
• f6f79f6 Started SchemaBuilder implementation and split out the schema model from the schema manager
• 4a26a8f Made logging for catalog updates consistent with normal log message: no uppercase shouting
• e1a365e Added log message about loaded type to discovered system information
• 2352df8 Fixed message about minimum PG version
• e1fd96a Added internal.dispatcher.maxqueuesize property to change the size of the queue before backpressure kicks in
• 02a94d4 Added some wait time to the test to force the new backpressure on ringbuffers to kick in forcibly
• be62f74 Fixed ringbuffer to not drop entries but wait and retry to ingest it, producing some type of backpressure
• f100986 Fixed shutdown bug where the task queue wouldn't be closed, causing the shutdown to hang
• 53ecb2c Fixed lint issues
• 22466d7 Incomplete support for ltree and geometry
• 9548e97 Exchanged the internal queue with a channel based ringbuffer implementation
• 240cd1c Added support to start the tool with the Go CPU profiler
• 5d2648f Substantial rework for pg and schema type resolution in preparation for struct and array type support
• d381115 Added tests for data types that are currently supported, and prepared missing ones
• 4cc241e Increased processed LSN by 1 byte before sending (according to https://www.postgresql.org/docs/current/protocol-replication.html#PROTOCOL-REPLICATION-STANDBY-STATUS-UPDATE that's how it's supposed to work
• 7a1054d Added test to make sure PG13 / PG14 empty transactions (only COMMIT / BEGIN) are handled correctly
• 3e7ec11 Added constant for PG15 version number
• c6d6ea2 Added pg and timescaledb version retrieval to test context
• 1c22402 Decreased time interval for status updates (10 -> 5 seconds)
• 75d03ae Fixed error gobbling when parsing pg and tsdb versions
• b2a7eff Only providing applied LSN (processed and pushed) to PG
• b4bf054 ReplicationContext is an interface now (increases testability)
• 7b75887 Abstracted handling of task and event management to a new sliced out interface TaskManager (increases testability)
• 9900be8 Fixed unit tests that broke due to the relocation of functions and properties.
• 04284d4 Abstracted handling of schema management to a new sliced out interface SchemaManager (increases testability)
• 86d5341 Abstracted handling of state handling to a new sliced out interface StateManager (increases testability)
• 8f1b39c Abstracted handling of publications to a new sliced out interface PublicationManager (increases testability)
• 619f35c Removed unused variables. The used version is in the replication context.