-
Notifications
You must be signed in to change notification settings - Fork 901
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove autovacuum setting from compressed chunks #6752
Conversation
61a12b9
to
5d8b62f
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #6752 +/- ##
==========================================
+ Coverage 80.06% 81.00% +0.94%
==========================================
Files 190 191 +1
Lines 37181 36352 -829
Branches 9450 9434 -16
==========================================
- Hits 29770 29448 -322
- Misses 2997 3158 +161
+ Partials 4414 3746 -668 ☔ View full report in Codecov by Sentry. |
5d8b62f
to
43fcc74
Compare
sql/updates/latest-dev.sql
Outdated
BEGIN | ||
FOR chunk IN | ||
SELECT format('%I.%I', schema_name, table_name)::regclass | ||
FROM _timescaledb_catalog.chunk WHERE compressed_chunk_id IS NOT NULL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm i would remove the IS NOT NULL check, otherwise i think it might not get properly reset in certain situation
consider the following:
- chunk was compressed with vacuum disabled
- upgrade to version that removed the vacuum disable code
- chunk decompressed
the chunk might now still be flagged as vacuum disabled
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe the check should be for chunks of public hypertables, but i dont see problem removing it from all
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. The check has been removed, and the autovacuum_enabled
setting will be reset on all chunks now.
43fcc74
to
84894ec
Compare
84894ec
to
72ce21c
Compare
In old TSDB versions, we disabled autovacuum for compressed chunks to keep the statistics. However, this restriction was removed in timescale#5118, but no migration was performed to reset the custom autovacuum setting for existing chunks.
72ce21c
to
28bc79a
Compare
Automated backport to 2.14.x not done: cherry-pick failed. Git status
|
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg
In old TSDB versions, we disabled autovacuum for compressed chunks to keep the statistics. However, this restriction was removed in #5118, but no migration was performed to reset the custom autovacuum setting for existing chunks.