Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support XA distributed transactions in CockroachDB #22329

Closed
knz opened this issue Feb 2, 2018 · 3 comments
Closed

Support XA distributed transactions in CockroachDB #22329

knz opened this issue Feb 2, 2018 · 3 comments
Assignees
@nvanbenschoten
Labels
A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic A-sql-pgcompat Semantic compatibility with PostgreSQL C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@knz
Copy link
Contributor

knz commented Feb 2, 2018 •
edited by cockroach-jira-scripts
Loading

X/Open XA is a standard for distributed transaction processing across multiple database engines. This is used in enterprise scenarios where data is spread across different technologies but applications need to preserve ACID semantics for transactions that span multiple database engines.

PostgreSQL supports XA transactions by a combination of two mechanisms:

  • internally it supports the PREPARE TRANSACTION statement which enables a SQL transaction manager (TM) to "take over" transaction management
  • the official pg ODBC driver contains an implementation of the standard MSDTP API (Microsoft's view on what XA should look like), so that pg becomes interoperable with other MSDTP-aware databases.
  • ditto JDBC

For enterprise users CockroachDB should provide a path towards XA compatibility:

  • provide a mechanism / API so that an external TM can take over transaction control in CockroachDB
  • provide one or more driver extension that interoperates with one or more industry standard API (perhaps MSDTP or the java equivalent)

Strict compatibility with pg's PREPARE TRANSACTION would enable providing a XA API via pg's own ODBC driver.

cc @awoods187 for prioritization. cc @bdarnell for technical insights.

Also mentioned here: https://forum.cockroachlabs.com/t/xa-distributed-transactions/1336

Jira issue: CRDB-5863
@knz knz added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Feb 2, 2018
@andreimatei andreimatei mentioned this issue Feb 4, 2018
Open
@awoods187 awoods187 added this to the Later milestone Feb 5, 2018
@knz knz mentioned this issue Jul 9, 2018
Closed
@knz knz added A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic labels Jul 9, 2018
@petermattis petermattis removed this from the Later milestone Oct 5, 2018
jordanlewis added a commit to jordanlewis/cockroach that referenced this issue Oct 2, 2019
@jordanlewis
roachtest: add test that aggregates orm blacklist failures
cebc811 
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: cockroachdb#5807   (sql: Add support for TEMP tables)
 151: cockroachdb#17511  (sql: support stored procedures)
  86: cockroachdb#26097  (sql: make TIMETZ more pg-compatible)
  56: cockroachdb#10735  (sql: support SQL savepoints)
  55: cockroachdb#32552  (multi-dim arrays)
  55: cockroachdb#26508  (sql: restricted DDL / DML inside transactions)
  52: cockroachdb#32565  (sql: support optional TIME precision)
  39: cockroachdb#243    (roadmap: Blob storage)
  33: cockroachdb#26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: cockroachdb#27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: cockroachdb#12123  (sql: Can't drop and replace a table within a transaction)
  24: cockroachdb#26443  (sql: support user-defined schemas between database and table)
  20: cockroachdb#21286  (sql: Add support for geometric types)
  18: cockroachdb#6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: cockroachdb#22329  (Support XA distributed transactions in CockroachDB)
  16: cockroachdb#24062  (sql: 32 bit SERIAL type)
  16: cockroachdb#30352  (roadmap:when CockroachDB  will support cursor?)
  12: cockroachdb#27791  (sql: support RANGE types)
   8: cockroachdb#40195  (pgwire: multiple active result sets (portals) not supported)
   8: cockroachdb#6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: cockroachdb#23468  (sql: support sql arrays of JSONB)
   5: cockroachdb#40854  (sql: set application_name from connection string)
   4: cockroachdb#35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: cockroachdb#32610  (sql: can't insert self reference)
   4: cockroachdb#40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: cockroachdb#35897  (sql: unknown function: pg_terminate_backend())
   4: cockroachdb#4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: cockroachdb#27796  (sql: support user-defined DOMAIN types)
   3: cockroachdb#3781   (sql: Add Data Type Formatting Functions)
   3: cockroachdb#40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: cockroachdb#35882  (sql: support other character sets)
   2: cockroachdb#10028  (sql: Support view queries with star expansions)
   2: cockroachdb#35807  (sql: INTERVAL output doesn't match PG)
   2: cockroachdb#35902  (sql: large object support)
   2: cockroachdb#40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: cockroachdb#18846  (sql: Support CIDR column type)
   1: cockroachdb#9682   (sql: implement computed indexes)
   1: cockroachdb#31632  (sql: FK options (deferrable, etc))
   1: cockroachdb#24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: cockroachdb#36215  (sql: enable setting standard_conforming_strings to off)
   1: cockroachdb#32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: cockroachdb#36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: cockroachdb#26732  (sql: support the binary operator: <int> / <float>)
   1: cockroachdb#23299  (sql: support coercing string literals to arrays)
   1: cockroachdb#36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: cockroachdb#26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: cockroachdb#21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: cockroachdb#36179  (sql: implicity convert date to timestamp)
   1: cockroachdb#36118  (sql: Cannot parse '24:00' as type time)
   1: cockroachdb#31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None
@rafiss rafiss added the A-sql-pgcompat Semantic compatibility with PostgreSQL label Oct 23, 2019
jordanlewis added a commit to jordanlewis/cockroach that referenced this issue Oct 24, 2019
@jordanlewis
roachtest: add test that aggregates orm blacklist failures
4010eac 
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: cockroachdb#5807   (sql: Add support for TEMP tables)
 151: cockroachdb#17511  (sql: support stored procedures)
  86: cockroachdb#26097  (sql: make TIMETZ more pg-compatible)
  56: cockroachdb#10735  (sql: support SQL savepoints)
  55: cockroachdb#32552  (multi-dim arrays)
  55: cockroachdb#26508  (sql: restricted DDL / DML inside transactions)
  52: cockroachdb#32565  (sql: support optional TIME precision)
  39: cockroachdb#243    (roadmap: Blob storage)
  33: cockroachdb#26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: cockroachdb#27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: cockroachdb#12123  (sql: Can't drop and replace a table within a transaction)
  24: cockroachdb#26443  (sql: support user-defined schemas between database and table)
  20: cockroachdb#21286  (sql: Add support for geometric types)
  18: cockroachdb#6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: cockroachdb#22329  (Support XA distributed transactions in CockroachDB)
  16: cockroachdb#24062  (sql: 32 bit SERIAL type)
  16: cockroachdb#30352  (roadmap:when CockroachDB  will support cursor?)
  12: cockroachdb#27791  (sql: support RANGE types)
   8: cockroachdb#40195  (pgwire: multiple active result sets (portals) not supported)
   8: cockroachdb#6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: cockroachdb#23468  (sql: support sql arrays of JSONB)
   5: cockroachdb#40854  (sql: set application_name from connection string)
   4: cockroachdb#35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: cockroachdb#32610  (sql: can't insert self reference)
   4: cockroachdb#40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: cockroachdb#35897  (sql: unknown function: pg_terminate_backend())
   4: cockroachdb#4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: cockroachdb#27796  (sql: support user-defined DOMAIN types)
   3: cockroachdb#3781   (sql: Add Data Type Formatting Functions)
   3: cockroachdb#40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: cockroachdb#35882  (sql: support other character sets)
   2: cockroachdb#10028  (sql: Support view queries with star expansions)
   2: cockroachdb#35807  (sql: INTERVAL output doesn't match PG)
   2: cockroachdb#35902  (sql: large object support)
   2: cockroachdb#40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: cockroachdb#18846  (sql: Support CIDR column type)
   1: cockroachdb#9682   (sql: implement computed indexes)
   1: cockroachdb#31632  (sql: FK options (deferrable, etc))
   1: cockroachdb#24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: cockroachdb#36215  (sql: enable setting standard_conforming_strings to off)
   1: cockroachdb#32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: cockroachdb#36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: cockroachdb#26732  (sql: support the binary operator: <int> / <float>)
   1: cockroachdb#23299  (sql: support coercing string literals to arrays)
   1: cockroachdb#36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: cockroachdb#26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: cockroachdb#21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: cockroachdb#36179  (sql: implicity convert date to timestamp)
   1: cockroachdb#36118  (sql: Cannot parse '24:00' as type time)
   1: cockroachdb#31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None
craig bot pushed a commit that referenced this issue Nov 7, 2019
@jordanlewis
Merge #41252
27fd417 
41252: roachtest: add test that aggregates orm blacklist failures r=jordanlewis a=jordanlewis

The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: #5807   (sql: Add support for TEMP tables)
 151: #17511  (sql: support stored procedures)
  86: #26097  (sql: make TIMETZ more pg-compatible)
  56: #10735  (sql: support SQL savepoints)
  55: #32552  (multi-dim arrays)
  55: #26508  (sql: restricted DDL / DML inside transactions)
  52: #32565  (sql: support optional TIME precision)
  39: #243    (roadmap: Blob storage)
  33: #26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: #27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: #12123  (sql: Can't drop and replace a table within a transaction)
  24: #26443  (sql: support user-defined schemas between database and table)
  20: #21286  (sql: Add support for geometric types)
  18: #6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: #22329  (Support XA distributed transactions in CockroachDB)
  16: #24062  (sql: 32 bit SERIAL type)
  16: #30352  (roadmap:when CockroachDB  will support cursor?)
  12: #27791  (sql: support RANGE types)
   8: #40195  (pgwire: multiple active result sets (portals) not supported)
   8: #6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: #23468  (sql: support sql arrays of JSONB)
   5: #40854  (sql: set application_name from connection string)
   4: #35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: #32610  (sql: can't insert self reference)
   4: #40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: #35897  (sql: unknown function: pg_terminate_backend())
   4: #4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: #27796  (sql: support user-defined DOMAIN types)
   3: #3781   (sql: Add Data Type Formatting Functions)
   3: #40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: #35882  (sql: support other character sets)
   2: #10028  (sql: Support view queries with star expansions)
   2: #35807  (sql: INTERVAL output doesn't match PG)
   2: #35902  (sql: large object support)
   2: #40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: #18846  (sql: Support CIDR column type)
   1: #9682   (sql: implement computed indexes)
   1: #31632  (sql: FK options (deferrable, etc))
   1: #24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: #36215  (sql: enable setting standard_conforming_strings to off)
   1: #32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: #36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: #26732  (sql: support the binary operator: <int> / <float>)
   1: #23299  (sql: support coercing string literals to arrays)
   1: #36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: #26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: #21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: #36179  (sql: implicity convert date to timestamp)
   1: #36118  (sql: Cannot parse '24:00' as type time)
   1: #31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None

Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>
@github-actions
Copy link

github-actions bot commented Jun 7, 2021

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

@jlinder jlinder added the T-kv KV Team label Jun 16, 2021
@SebTardif
Copy link

While looking if I 'could' use CockroachDB instead of PostgreSQL, I found out this was not yet implemented. It's a big deal. Without it you end-up with convoluted solution like -> https://senoritadeveloper.medium.com/microservices-transaction-management-patterns-46ef2df9a9c4

This would probably help thinking about how to test it -> https://docs.hazelcast.com/hazelcast/5.2/pipelines/xa

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: add max_prepared_transactions variable
38034c3 
Informs cockroachdb#22329.

This commit adds the max_prepared_transactions session variable, returning a
value of math.MaxInt32 to indicate that there is no limit on the number of
prepared transactions that can be created.

This change has the effect of unblocking the pgjdbc XADataSourceTest test
suite, which was previously failing during initialization. Is also includes
a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so
that each test in the XA test suite actually runs (and fails).

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Merged
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Merged
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: add max_prepared_transactions variable
e892b8e 
Informs cockroachdb#22329.

This commit adds the max_prepared_transactions session variable, returning a
value of math.MaxInt32 to indicate that there is no limit on the number of
prepared transactions that can be created.

This change has the effect of unblocking the pgjdbc XADataSourceTest test
suite, which was previously failing during initialization. Is also includes
a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so
that each test in the XA test suite actually runs (and fails).

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Dec 13, 2024
Merged
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 13, 2024
@nvanbenschoten
multitenant: add can_prepare_txns tenant capability
f7f7e69 
Informs cockroachdb#22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we
don't allow secondary tenants to prepare transactions by default. Allowing
an untrusted tenant to prepare transactions would allow it to block the
progress of system-wide backups, so it is too dangerous to allow by default.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 16, 2024
@nvanbenschoten
roachpb: introduce PREPARED transaction status
88021e1 
Informs cockroachdb#22329.

This commit adds TransactionStatus.PREPARED.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 16, 2024
@nvanbenschoten
kv: support EndTransaction.Prepare request
5e23d4d 
Informs cockroachdb#22329.

This commit adds support for preparing a transaction record
with an EndTransaction request that caries the Prepare flag.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 16, 2024
@nvanbenschoten
kv: support conflict resolution with prepared transactions
7233ad9 
Informs cockroachdb#22329.

These transactions cannot have their timestamp bumped or be aborted.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 16, 2024
@nvanbenschoten
kv/kvserver: misc additional support for prepared transactions
2fdc190 
Informs cockroachdb#22329.

Adds support for prepared transactions to MVCC GC, rangefeeds, and a few
other places.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 16, 2024
@nvanbenschoten
kv/kvclient: add support for prepared transactions
a2e23ff 
Informs cockroachdb#22329.

This commit adds support for `Txn.Prepare`, `DB.CommitPrepared`, and
`DB.RollbackPrepared` APIs to KV. These methods allow clients to prepare
a transaction for commit, commit a prepared transaction, and rollback a
prepared transaction, respectively.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
multitenant: add can_prepare_txns tenant capability
cb48e27 
Informs cockroachdb#22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we
don't allow secondary tenants to prepare transactions by default. Allowing
an untrusted tenant to prepare transactions would allow it to block the
progress of system-wide backups, so it is too dangerous to allow by default.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
kv: support conflict resolution with prepared transactions
780cc19 
Informs cockroachdb#22329.

These transactions cannot have their timestamp bumped or be aborted.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
kv/kvserver: misc additional support for prepared transactions
f24721a 
Informs cockroachdb#22329.

Adds support for prepared transactions to MVCC GC, rangefeeds, and a few
other places.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
kv/kvclient: add support for prepared transactions
b01f778 
Informs cockroachdb#22329.

This commit adds support for `Txn.Prepare`, `DB.CommitPrepared`, and
`DB.RollbackPrepared` APIs to KV. These methods allow clients to prepare
a transaction for commit, commit a prepared transaction, and rollback a
prepared transaction, respectively.

Release note: None
craig bot pushed a commit that referenced this issue Dec 17, 2024
@nvanbenschoten
Merge #137409
5eefbce 
137409: kv: support two-phase commit transactions r=arulajmani a=nvanbenschoten

Informs #22329.

This commit adds support for Txn.Prepare, DB.CommitPrepared, and DB.RollbackPrepared APIs to KV. These methods allow clients to prepare a transaction for commit, commit a prepared transaction, and rollback a prepared transaction, respectively.

When a transaction is prepared, we send an EndTxnRequest as usual, but a new flag indicates the transaction should be moved to status PREPARED instead of COMMITTED. In state PREPARED, a transaction record is updated to include all of its intents, and awaits a subsequent EndTxnRequest which will move the transaction either to COMMITTED or ABORTED. Transaction in state PREPARED cannot be pushed, regardless of isolation level, ensuring that a subsequent commit or rollback is guaranteed to succeed.

Release note: None

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
craig bot pushed a commit that referenced this issue Dec 17, 2024
@andy-kimball @nvanbenschoten
@fqazi @andrewbaptist
Merge #137374 #137409 #137627 #137634
1203a72 
137374: vecstore: disable deadlock linting for inMemoryLock r=drewkimball a=andy-kimball

Do not use syncutil.RWMutex in the inMemoryLock class, because deadlock detection reports spurious failures. Different partitions in the vector index can be locked in different orders by merge, split, format and other operations. In all these cases, we first acquire the in-memory store's structure lock to prevent deadlocks. But the deadlock detection package is not smart enough to realize this and reports false positives.

Epic: CRDB-42943
Fixes: #136958.
Fixes: #136960.

Release note: None

137409: kv: support two-phase commit transactions r=arulajmani a=nvanbenschoten

Informs #22329.

This commit adds support for Txn.Prepare, DB.CommitPrepared, and DB.RollbackPrepared APIs to KV. These methods allow clients to prepare a transaction for commit, commit a prepared transaction, and rollback a prepared transaction, respectively.

When a transaction is prepared, we send an EndTxnRequest as usual, but a new flag indicates the transaction should be moved to status PREPARED instead of COMMITTED. In state PREPARED, a transaction record is updated to include all of its intents, and awaits a subsequent EndTxnRequest which will move the transaction either to COMMITTED or ABORTED. Transaction in state PREPARED cannot be pushed, regardless of isolation level, ensuring that a subsequent commit or rollback is guaranteed to succeed.

Release note: None

137627: pgwire: parseClientProvidedSessionParameters can leak entire ReadBuffer r=fqazi a=fqazi

Previously, when parsing the client session parameters, we would fetch option value via ReaderBuffer.GetString. This function would return a reference to the original read buffer, so any session parameters would cause the entire message buffer to stay allocated. To address this, this patch copies the values from the read buffer, so that no references cause the entire message buffer to be kept alive.

Fixes: #137623

Release note (bug fix): Address potential memory leak parsing client session parameters for new connections.

137634: roachtest: add io_listener_logs to pertubation/* tests r=kvoli a=andrewbaptist

Epic: none

Release note: None

Co-authored-by: Andrew Kimball <andyk@cockroachlabs.com>
Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
Co-authored-by: Andrew Baptist <baptist@cockroachlabs.com>
@nvanbenschoten nvanbenschoten mentioned this issue Dec 17, 2024
Open
2 tasks
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
sql: implement XA transaction SQL statements
81e3ecf 
Informs cockroachdb#22329.

This commit implements the statement handling for `PREPARE TRANSACTION
<gid>`, `COMMIT PREPARED <gid>`, and `ROLLBACK PREPARED <gid>`, for
parity with Postgres support. The implementation is based on the
Postgres documentation and the Postgres source code.

Release note (sql change): XA transaction support allows CockroachDB to
participate in distributed transaction with other resources (e.g.
databases, message queues, etc) using a two phase commit protocol.
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
sql: implement pg_catalog.pg_prepared_xacts
3041570 
Informs cockroachdb#22329.

`pg_catalog.pg_prepared_xacts` reads from `system.prepared_transactions`.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
roachtest: fix pgjdbc XA test suite
f2acda2 
Informs cockroachdb#22329.

This commit fixes the pgjdbc blocklist to reflect the current state of
support for XA transactions in CockroachDB. With the newly added support
for two-phase commit transactions, all tests in the XADataSourceTest
test suite now pass except for `mappingOfConstraintViolations`, which is
blocked by cockroachdb#31632 (DEFERRABLE foreign keys).

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
sql: implement pg_catalog.pg_prepared_xacts
bca2022 
Informs cockroachdb#22329.

`pg_catalog.pg_prepared_xacts` reads from `system.prepared_transactions`.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 17, 2024
@nvanbenschoten
roachtest: fix pgjdbc XA test suite
59b4dfd 
Informs cockroachdb#22329.

This commit fixes the pgjdbc blocklist to reflect the current state of
support for XA transactions in CockroachDB. With the newly added support
for two-phase commit transactions, all tests in the XADataSourceTest
test suite now pass except for `mappingOfConstraintViolations`, which is
blocked by cockroachdb#31632 (DEFERRABLE foreign keys).

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
multitenant: add can_prepare_txns tenant capability
0f93871 
Informs cockroachdb#22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we
don't allow secondary tenants to prepare transactions by default. Allowing
an untrusted tenant to prepare transactions would allow it to block the
progress of system-wide backups, so it is too dangerous to allow by default.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
sql: implement XA transaction SQL statements
503d914 
Informs cockroachdb#22329.

This commit implements the statement handling for `PREPARE TRANSACTION
<gid>`, `COMMIT PREPARED <gid>`, and `ROLLBACK PREPARED <gid>`, for
parity with Postgres support. The implementation is based on the
Postgres documentation and the Postgres source code.

Release note (sql change): XA transaction support allows CockroachDB to
participate in distributed transaction with other resources (e.g.
databases, message queues, etc) using a two phase commit protocol.
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
sql: implement pg_catalog.pg_prepared_xacts
bdb670d 
Informs cockroachdb#22329.

`pg_catalog.pg_prepared_xacts` reads from `system.prepared_transactions`.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
roachtest: fix pgjdbc XA test suite
6e2aea5 
Informs cockroachdb#22329.

This commit fixes the pgjdbc blocklist to reflect the current state of
support for XA transactions in CockroachDB. With the newly added support
for two-phase commit transactions, all tests in the XADataSourceTest
test suite now pass except for `mappingOfConstraintViolations`, which is
blocked by cockroachdb#31632 (DEFERRABLE foreign keys).

Release note: None
craig bot pushed a commit that referenced this issue Dec 18, 2024
@nvanbenschoten
Merge #129448
694ca19 
129448: sql: implement XA transaction SQL statements r=rafiss a=nvanbenschoten

Informs #22329.
Closes #137325.

This PR implements the statement handling for `PREPARE TRANSACTION <gid>`, `COMMIT PREPARED <gid>`, and `ROLLBACK PREPARED <gid>`, for parity with Postgres support. The implementation is based on the Postgres documentation and the Postgres source code.

The PR then implements the `pg_catalog.pg_prepared_xacts` system catalog table, which reads from `system.prepared_transactions`. This adds the remaining PG compatibility, so that XA in CockroachDB looks near-identical to XA in PostgreSQL.

Finally, the PR fixes the `pgjdbc` blocklist to reflect the current state of support for XA transactions in CockroachDB. With the newly added support for two-phase commit transactions, all tests in the `XADataSourceTest` test suite now pass except for `mappingOfConstraintViolations`, which is blocked by #31632 (DEFERRABLE foreign keys).

Release note (sql change): XA transaction support allows CockroachDB to participate in distributed transaction with other resources (e.g. databases, message queues, etc) using a two phase commit protocol.

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
multitenant: add can_prepare_txns tenant capability
d24646c 
Informs cockroachdb#22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we
don't allow secondary tenants to prepare transactions by default. Allowing
an untrusted tenant to prepare transactions would allow it to block the
progress of system-wide backups, so it is too dangerous to allow by default.

Release note: None
@nvanbenschoten nvanbenschoten self-assigned this Dec 18, 2024
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Dec 18, 2024
@nvanbenschoten
multitenant: add can_prepare_txns tenant capability
7b23988 
Informs cockroachdb#22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we
don't allow secondary tenants to prepare transactions by default. Allowing
an untrusted tenant to prepare transactions would allow it to block the
progress of system-wide backups, so it is too dangerous to allow by default.

Release note: None
@nvanbenschoten
Copy link
Member

Closing as complete, now that the following PRs have merged:

Remaining work will be tracked in more specific issues(#137636 and #137549).

craig bot pushed a commit that referenced this issue Dec 18, 2024
@nvanbenschoten @yuzefovich
Merge #137457 #137580
88b6c94 
137457: multitenant: add can_prepare_txns tenant capability r=nvanbenschoten a=nvanbenschoten

Informs #22329.

This commit adds a new `can_prepare_txns` tenant capability, so that we adon't allow secondary tenants to prepare transactions by default. Allowing aan untrusted tenant to prepare transactions would allow it to block the aprogress of system-wide backups, so it is too dangerous to allow by default.

Release note: None

137580: sql: include DB name in create_statement in create_type_statements r=yuzefovich a=yuzefovich

This commit updates how we populate `crdb_internal.create_type_statements` virtual table so that `create_statement` column included the DB name (previously only the schema and the UDT name were included). This will make it easier to reproduce stmt bundles with cross-DB references.

Also, this commit fixes how we use this virtual table during bundle creation - we now properly use `"".crdb_internal` search path and cast the type oid to `OID` type. The same fix is applied for `create_function_statements` and `create_procedure_statements` vtables.

Epic: None
Release note: None

Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic A-sql-pgcompat Semantic compatibility with PostgreSQL C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jlinder @knz @rafiss @SebTardif @nvanbenschoten @petermattis @awoods187