Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alter_distributed_table seems to OOM with many tables #6503

Closed
onderkalaci opened this issue Nov 21, 2022 · 2 comments · Fixed by #6726
Closed

alter_distributed_table seems to OOM with many tables #6503

onderkalaci opened this issue Nov 21, 2022 · 2 comments · Fixed by #6726

Comments

@onderkalaci
Copy link
Member

CREATE TABLE  time_series_events (event_time timestamp, event int, user_id int)
PARTITION BY RANGE (event_time);

SELECT create_time_partitions(table_name:= 'time_series_events',
    partition_interval:= '1 day',
    end_at:= '2025-10-30',
    start_from:= '2021-10-10');

SELECT create_distributed_table('time_series_events', 'user_id');

SELECT alter_distributed_table('time_series_events', shard_count:=512, cascade_to_colocated:=yes);

After some time, this gives OOM

@onderkalaci
Copy link
Member Author

One potential problem could be that we keep

def ConvertTable:
 
    foreach_oid(partitionRelationId, partitionList)
        attachPartitionCommands = lappend(attachPartitionCommands, attachPartitionCommand);

and then keep attachPartitionCommands until all the partitions are converted.

This is NOT the only problem, but one problem. We should aggressively pfree (or better MemoryContextCreate/Delete) per table conversion

@hanefi
Copy link
Member

hanefi commented Nov 26, 2022

For mitigation, we considered the following list of steps:

  1. Detach all the partitions.
  2. Change shard counts for the detached partitions
  3. Change shard count for the partitioned table
  4. Attach all the partitions.

Edit: I got a transient error that vanished on a clean Citus build. I used to have an error message that is not possible to reproduce, so I deleted it.

aykut-bozkurt added a commit that referenced this issue Feb 28, 2023
…tion and shards (#6726)

2 improvements to prevent memory leaks during altering or undistributing
distributed tables with a lot of partitions and shards:

1. Free memory for each call to ConvertTable so that colocated and partition tables at
`AlterDistributedTable`, `UndistributeTable`, or
`AlterTableSetAccessMethod` will not cause an increase
in memory usage,
2. Free memory while executing attach partition commands for each partition table at
`AlterDistributedTable` to prevent an increase in memory usage.

DESCRIPTION: Fixes memory leak issue during altering distributed table
with a lot of partition and shards.

Fixes #6503.
aykut-bozkurt added a commit that referenced this issue Feb 28, 2023
…tion and shards (#6726)

2 improvements to prevent memory leaks during altering or undistributing
distributed tables with a lot of partitions and shards:

1. Free memory for each call to ConvertTable so that colocated and partition tables at
`AlterDistributedTable`, `UndistributeTable`, or
`AlterTableSetAccessMethod` will not cause an increase
in memory usage,
2. Free memory while executing attach partition commands for each partition table at
`AlterDistributedTable` to prevent an increase in memory usage.

DESCRIPTION: Fixes memory leak issue during altering distributed table
with a lot of partition and shards.

Fixes #6503.

(cherry picked from commit e2654de)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants