Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix segfault when setting chunk replica identity #7434

Conversation

fabriziomello
Copy link
Contributor

@fabriziomello fabriziomello commented Nov 11, 2024

The segfault happened executing AlterTableInternal for setting the REPLICA IDENTITY during chunk creation (introduced by #5512) that is dispatched by a trigger invoked by a DDL statement.

Fixed it by using our internal ts_alter_table_with_event_trigger handler function to setup the event trigger context.

Fixes #7406

@fabriziomello fabriziomello self-assigned this Nov 11, 2024
@fabriziomello fabriziomello force-pushed the fix_segfault_when_setting_replica_identity_for_chunk branch from 2951ac2 to eea4b71 Compare November 11, 2024 21:14
Copy link

codecov bot commented Nov 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.04%. Comparing base (59f50f2) to head (2ba3df8).
Report is 588 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7434      +/-   ##
==========================================
+ Coverage   80.06%   81.04%   +0.97%     
==========================================
  Files         190      229      +39     
  Lines       37181    42514    +5333     
  Branches     9450    10625    +1175     
==========================================
+ Hits        29770    34455    +4685     
- Misses       2997     3749     +752     
+ Partials     4414     4310     -104     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fabriziomello fabriziomello force-pushed the fix_segfault_when_setting_replica_identity_for_chunk branch 2 times, most recently from 5897b74 to 33b1722 Compare November 12, 2024 15:54
@fabriziomello fabriziomello enabled auto-merge (rebase) November 12, 2024 18:53
Copy link
Contributor

@mkindahl mkindahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit comment is very focused on the code, so you might want to re-write it to explain the context so your future self can understand what the problem was.

This problem can occur when you insert into a hypertable from inside a trigger, so starting there might be a good idea, that is:

Insert into hypertable calls AlterTableInternal [why?]... if called from inside a trigger, this will not have currentCommand set up [why and why does it work if not in a trigger?]... we need to wrap the call in ...Start and ...End.

A few suggestions for adding comments and one more call to AlterTableInternal that you might want to deal with (but fine to submit as separate PR if you prefer that).

Approving since the code looks fine and I see no issues with the fix.

src/chunk.c Outdated
Comment on lines 902 to 916
ts_catalog_database_info_become_owner(ts_catalog_database_info_get(), &sec_ctx);
EventTriggerAlterTableStart((Node *) &cmd);
AlterTableInternal(chunk->table_id, list_make1(&cmd), false);
EventTriggerAlterTableEnd();
ts_catalog_restore_user(&sec_ctx);
table_close(ht_rel, NoLock);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be called in a nested fashion since the start and end commands remember the previous value of currentCommand. You might want to add a comment explaining why the start and end are necessary though, for your future self.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we have a function ts_alter_table_with_event_trigger to wrap the event trigger context and execute the AlterTableInternal that have the enough comments about why it is necessary.

src/chunk.c Show resolved Hide resolved
src/chunk.c Outdated
@@ -900,7 +900,9 @@ chunk_set_replica_identity(const Chunk *chunk)
}

ts_catalog_database_info_become_owner(ts_catalog_database_info_get(), &sec_ctx);
EventTriggerAlterTableStart((Node *) &cmd);
AlterTableInternal(chunk->table_id, list_make1(&cmd), false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first argument to AlterTableInternal should be an OID and it indeed looks like table_id is taken from the ObjectAddress for the chunk. You might want to double-check, but it seems to make no difference for this particular case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well the chunk->table_id is the OID of the chunk relation returned by the ts_chunk_create_table that internally returns the ObjectAddress->objectId returned by DefineRelation so we're fine here.

@fabriziomello
Copy link
Contributor Author

The commit comment is very focused on the code, so you might want to re-write it to explain the context so your future self can understand what the problem was.

Ok.

This problem can occur when you insert into a hypertable from inside a trigger, so starting there might be a good idea, that is:

Will recheck but it was not so simple like this to reproduce the case.

Insert into hypertable calls AlterTableInternal [why?]...

Because of this PR #5515. But looking now at the implementation looks like we didn't did a good job because we always call the AlterTableInternal, so I think we should improve it a bit:

➜ git diff                                                         
diff --git a/src/chunk.c b/src/chunk.c
index c2535d368..89105ba4a 100644
--- a/src/chunk.c
+++ b/src/chunk.c
@@ -875,6 +875,14 @@ static void
 chunk_set_replica_identity(const Chunk *chunk)
 {
    Relation ht_rel = relation_open(chunk->hypertable_relid, AccessShareLock);
+
+   /* Do nothing if hypertable is using the default replica identity */
+   if (ht_rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT)
+   {
+       table_close(ht_rel, NoLock);
+       return;
+   }
+
    ReplicaIdentityStmt stmt = {
        .type = T_ReplicaIdentityStmt,
        .identity_type = ht_rel->rd_rel->relreplident,

if called from inside a trigger, this will not have currentCommand set up [why and why does it work if not in a trigger?]... we need to wrap the call in ...Start and ...End.

Will recheck!

A few suggestions for adding comments and one more call to AlterTableInternal that you might want to deal with (but fine to submit as separate PR if you prefer that).

The other one need to investigate if it is possible to raise the same problem.

@fabriziomello fabriziomello force-pushed the fix_segfault_when_setting_replica_identity_for_chunk branch 3 times, most recently from 49d64fd to 138d415 Compare November 13, 2024 20:19
The segfault happened executing `AlterTableInternal` for setting the
REPLICA IDENTITY during chunk creation (introduced by timescale#5512) that is
dispatched by a trigger invoked by a DDL statement.

Fixed it by using our internal `ts_alter_table_with_event_trigger`
handler function to setup the event trigger context.

Fixes timescale#7406
@fabriziomello fabriziomello force-pushed the fix_segfault_when_setting_replica_identity_for_chunk branch from 138d415 to 2ba3df8 Compare November 13, 2024 20:46
@fabriziomello fabriziomello merged commit 5ae1d40 into timescale:main Nov 14, 2024
48 checks passed
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Nov 20, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#7271 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Nov 22, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#7271 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7455: Support DROP NOT NULL on compressed hypertables

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 13, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 16, 2024
This release contains performance improvements and bug fixes since
the 2.18.0 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 16, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 16, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 16, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Dec 17, 2024
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901 Add hypertable support for transition tables
* timescale#7104 Hypercore table access method
* timescale#7271 Push down ORDER BY in real time continuous aggregate queries
* timescale#7295: Support ALTER TABLE SET ACCESS METHOD on hypertable.
* timescale#7390 Disable custom hashagg planner code
* timescale#7411 Change parameter name to enable Hypercore TAM
* timescale#7412 Add GUC for hypercore_use_access_method default
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7443 Add Hypercore function and view aliases
* timescale#7455: Support DROP NOT NULL on compressed hypertables
* timescale#7458 Support vecorized aggregation with aggregate FILTER clauses that are also vectorizable
* timescale#7486 Prevent building against postgres versions with broken ABI

**Bugfixes**
* timescale#7378 Remove obsolete job referencing policy_job_error_retention
* timescale#7409 Update bgw job table when altering procedure
* timescale#7410 "aggregated compressed column not found" error on aggregation query.
* timescale#7426 Fix datetime parsing error in chunk constraint creation
* timescale#7432 Verify that heap tuple is valid before using
* timescale#7434 Fixes segfault when internally set the replica identity for a given chunk
* timescale#7488 Emit error for transition table trigger on chunks
* timescale#7514 Fix error: invalid child of chunk append

**Thanks**
* @bharrisau for reporting the segfault when creating chunks
* @pgloader for reporting an issue an internal background job
* @uasiddiqi for reporting the "aggregated compressed column not found" error.
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Jan 16, 2025
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901: Add hypertable support for transition tables.
* timescale#7104: Hypercore table access method.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7412: Add GUC for `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7433 Add support for merging chunks
* timescale#7436 Add index creation on orderby columns
* timescale#7443: Add hypercore function and view aliases.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7486: Prevent building against postgres versions with broken ABI.
* timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate`
* timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565 Add hint when hypertable creation fails
* timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517 Fixes performance regression on `cagg_migrate` procedure
* timescale#7527 Restart scheduler on error
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566 Improve transaction check in CAgg refresh
* timescale#7584 Fix NaN-handling for vectorized aggregation

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting the improvement
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending PR to improve transaction check in CAgg refresh
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
@pallavisontakke pallavisontakke mentioned this pull request Jan 16, 2025
pallavisontakke added a commit to pallavisontakke/timescaledb that referenced this pull request Jan 17, 2025
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* timescale#6901: Add hypertable support for transition tables.
* timescale#7104: Hypercore table access method.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7412: Add GUC for `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.
* timescale#7433 Add support for merging chunks
* timescale#7436 Add index creation on orderby columns
* timescale#7443: Add hypercore function and view aliases.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7486: Prevent building against postgres versions with broken ABI.
* timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate`
* timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565 Add hint when hypertable creation fails
* timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517 Fixes performance regression on `cagg_migrate` procedure
* timescale#7527 Restart scheduler on error
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566 Improve transaction check in CAgg refresh
* timescale#7584 Fix NaN-handling for vectorized aggregation

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting the improvement
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending PR to improve transaction check in CAgg refresh
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
@pallavisontakke pallavisontakke mentioned this pull request Jan 17, 2025
svenklemm pushed a commit that referenced this pull request Jan 23, 2025
This release contains performance improvements and bug fixes since
the 2.17.2 release. We recommend that you upgrade at the next
available opportunity.

**Features**
* #6901: Add hypertable support for transition tables.
* #7104: Hypercore table access method.
* #7271: Push down `order by` in real-time continuous aggregate queries.
* #7295: Support `alter table set access method` on hypertable.
* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column
* #7390: Disable custom `hashagg` planner code.
* #7411: Change parameter name to enable hypercore table access method.
* #7412: Add GUC for `hypercore_use_access_method` default.
* #7413: Add GUC for segmentwise recompression.
* #7433 Add support for merging chunks
* #7436 Add index creation on orderby columns
* #7443: Add hypercore function and view aliases.
* #7455: Support `drop not null` on compressed hypertables.
* #7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable.
* #7482: Optimize recompression of partially compressed chunks.
* #7486: Prevent building against postgres versions with broken ABI.
* #7521 Add optional `force` argument to `refresh_continuous_aggregate`
* #7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* #7565 Add hint when hypertable creation fails
* #7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API

**Bugfixes**
* #7378: Remove obsolete job referencing `policy_job_error_retention`.
* #7409: Update `bgw_job` table when altering procedure.
* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
* #7426: Fix `datetime` parsing error in chunk constraint creation.
* #7432: Verify that the heap tuple is valid before using.
* #7434: Fixes the segfault when internally setting the replica identity for a given chunk.
* #7488: Emit error for transition table trigger on chunks.
* #7514: Fix the error: `invalid child of chunk append`.
* #7517 Fixes performance regression on `cagg_migrate` procedure
* #7527 Restart scheduler on error
* #7557: Fix null handling for in-memory tuple filtering.
* #7566 Improve transaction check in CAgg refresh
* #7584 Fix NaN-handling for vectorized aggregation

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting the improvement
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending PR to improve transaction check in CAgg refresh
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit that referenced this pull request Jan 23, 2025
This release improves `group by` performance of compressing data and filters,
introduces the ability to add secondary indexes, and contains the highly upvoted
community request of transition table support. We recommend that you upgrade at the
next available opportunity.

**Highlighted features in this release**
* TAM
* Significant performance improvements for aggregations using a `group by` with one column
and/or using a filter clause. SIMD
* Transition tables

**Dropping support for Bitnami images**
After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671),
we are no longer building Bitnami images for TimescaleDB.

**Features**
* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* #7104: Hypercore table access method.
* #6901: Add hypertable support for transition tables.
* #7482: Optimize recompression of partially compressed chunks.
* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* #7433: Add support for merging chunks.
* #7271: Push down `order by` in real-time continuous aggregate queries.
* #7455: Support `drop not null` on compressed hypertables.
* #7295: Support `alter table set access method` on hypertable.
* #7411: Change parameter name to enable hypercore table access method.
* #7436: Add index creation on `order by` columns.
* #7443: Add hypercore function and view aliases.
* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* #7565: Add hint when hypertable creation fails.
* #7390: Disable custom `hashagg` planner code.
* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* #7486: Prevent building against PostgreSQL versions with broken ABI.
* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* #7413: Add GUC for segmentwise recompression.

**Bugfixes**
* #7378: Remove obsolete job referencing `policy_job_error_retention`.
* #7409: Update `bgw_job` table when altering procedure.
* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
* #7426: Fix `datetime` parsing error in chunk constraint creation.
* #7432: Verify that the heap tuple is valid before using.
* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
* #7488: Emit error for transition table trigger on chunks.
* #7514: Fix the error: `invalid child of chunk append`.
* #7517: Fix the performance regression on the `cagg_migrate` procedure.
* #7527: Restart scheduler on error.
* #7557: Fix null handling for in-memory tuple filtering.
* #7566: Improve transaction check in CAGG refresh.
* #7584: Fix NaN-handling for vectorized aggregation.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit that referenced this pull request Jan 23, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.17.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* #7104: Hypercore table access method.
* #6901: Add hypertable support for transition tables.
* #7482: Optimize recompression of partially compressed chunks.
* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* #7433: Add support for merging chunks.
* #7271: Push down `order by` in real-time continuous aggregate queries.
* #7455: Support `drop not null` on compressed hypertables.
* #7295: Support `alter table set access method` on hypertable.
* #7411: Change parameter name to enable hypercore table access method.
* #7436: Add index creation on `order by` columns.
* #7443: Add hypercore function and view aliases.
* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* #7565: Add hint when hypertable creation fails.
* #7390: Disable custom `hashagg` planner code.
* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* #7486: Prevent building against PostgreSQL versions with broken ABI.
* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* #7413: Add GUC for segmentwise recompression.

**Bugfixes**
* #7378: Remove obsolete job referencing `policy_job_error_retention`.
* #7409: Update `bgw_job` table when altering procedure.
* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
* #7426: Fix `datetime` parsing error in chunk constraint creation.
* #7432: Verify that the heap tuple is valid before using.
* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
* #7488: Emit error for transition table trigger on chunks.
* #7514: Fix the error: `invalid child of chunk append`.
* #7517: Fix the performance regression on the `cagg_migrate` procedure.
* #7527: Restart scheduler on error.
* #7557: Fix null handling for in-memory tuple filtering.
* #7566: Improve transaction check in CAGG refresh.
* #7584: Fix NaN-handling for vectorized aggregation.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @pgloader for reporting the issue in an internal background job.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 23, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 23, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 23, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 24, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 24, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit to pallavisontakke/timescaledb that referenced this pull request Jan 24, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* timescale#7104: Hypercore table access method.
* timescale#6901: Add hypertable support for transition tables.
* timescale#7482: Optimize recompression of partially compressed chunks.
* timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* timescale#7433: Add support for merging chunks.
* timescale#7271: Push down `order by` in real-time continuous aggregate queries.
* timescale#7455: Support `drop not null` on compressed hypertables.
* timescale#7295: Support `alter table set access method` on hypertable.
* timescale#7411: Change parameter name to enable hypercore table access method.
* timescale#7436: Add index creation on `order by` columns.
* timescale#7443: Add hypercore function and view aliases.
* timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* timescale#7565: Add hint when hypertable creation fails.
* timescale#7390: Disable custom `hashagg` planner code.
* timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* timescale#7486: Prevent building against PostgreSQL versions with broken ABI.
* timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* timescale#7413: Add GUC for segmentwise recompression.

**Bugfixes**
* timescale#7378: Remove obsolete job referencing `policy_job_error_retention`.
* timescale#7409: Update `bgw_job` table when altering procedure.
* timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query.
* timescale#7426: Fix `datetime` parsing error in chunk constraint creation.
* timescale#7432: Verify that the heap tuple is valid before using.
* timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk.
* timescale#7488: Emit error for transition table trigger on chunks.
* timescale#7514: Fix the error: `invalid child of chunk append`.
* timescale#7517: Fix the performance regression on the `cagg_migrate` procedure.
* timescale#7527: Restart scheduler on error.
* timescale#7557: Fix null handling for in-memory tuple filtering.
* timescale#7566: Improve transaction check in CAGG refresh.
* timescale#7584: Fix NaN-handling for vectorized aggregation.
* timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
svenklemm pushed a commit that referenced this pull request Jan 24, 2025
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity.

**Highlighted features in TimescaleDB v2.18.0**

* The ability to add secondary indexes to the columnstore through the new hypercore table access method.
* Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore.
* Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests.
* Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression.

**Dropping support for Bitnami images**

After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha)

**Deprecation Notice**

We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below:

| Deprecated | Replacement | Type |
| --- | --- | --- |
| decompress_chunk | convert_to_rowstore | Procedure |
| compress_chunk | convert_to_columnstore | Procedure |
| add_compression_policy | add_columnstore_policy | Function |
| remove_compression_policy | remove_columnstore_policy | Function |
| hypertable_compression_stats | hypertable_columnstore_stats | Function |
| chunk_compression_stats | chunk_columnstore_stats | Function |
| hypertable_compression_settings | hypertable_columnstore_settings | View |
| chunk_compression_settings | chunk_columnstore_settings | View |
| compression_settings | columnstore_settings | View |
| timescaledb.compress | timescaledb.enable_columnstore | Parameter |
| timescaledb.compress_segmentby | timescaledb.segmentby | Parameter |
| timescaledb.compress_orderby  | timescaledb.orderby | Parameter |

**Features**
* #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types).
* #7104: Hypercore table access method.
* #6901: Add hypertable support for transition tables.
* #7482: Optimize recompression of partially compressed chunks.
* #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable.
* #7433: Add support for merging chunks.
* #7271: Push down `order by` in real-time continuous aggregate queries.
* #7455: Support `drop not null` on compressed hypertables.
* #7295: Support `alter table set access method` on hypertable.
* #7411: Change parameter name to enable hypercore table access method.
* #7436: Add index creation on `order by` columns.
* #7443: Add hypercore function and view aliases.
* #7521: Add optional `force` argument to `refresh_continuous_aggregate`.
* #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases.
* #7565: Add hint when hypertable creation fails.
* #7390: Disable custom `hashagg` planner code.
* #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API.
* #7486: Prevent building against PostgreSQL versions with broken ABI.
* #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default.
* #7413: Add GUC for segmentwise recompression.

**Bugfixes**
* #7378: Remove obsolete job referencing `policy_job_error_retention`.
* #7409: Update `bgw_job` table when altering procedure.
* #7410: Fix the `aggregated compressed column not found` error on aggregation query.
* #7426: Fix `datetime` parsing error in chunk constraint creation.
* #7432: Verify that the heap tuple is valid before using.
* #7434: Fix the segfault when internally setting the replica identity for a given chunk.
* #7488: Emit error for transition table trigger on chunks.
* #7514: Fix the error: `invalid child of chunk append`.
* #7517: Fix the performance regression on the `cagg_migrate` procedure.
* #7527: Restart scheduler on error.
* #7557: Fix null handling for in-memory tuple filtering.
* #7566: Improve transaction check in CAGG refresh.
* #7584: Fix NaN-handling for vectorized aggregation.
* #7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables.

**Thanks**
* @bharrisau for reporting the segfault when creating chunks.
* @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables.
* @k-rus for suggesting that we add a hint when hypertable creation fails.
* @staticlibs for sending the pull request that improves the transaction check in CAGG refresh.
* @uasiddiqi for reporting the `aggregated compressed column not found` error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: PG Segfault when ETL upsert triggers insert to empty hypertable
4 participants