[ENHANCEMENT] Improve RDS cluster search latency while indexing large datasets

[JeremyJonas]

Describe the feature

When performing tasks like indexing on large dataset (1.6M rows) takes 20-30 minutes the complete (without concurrently), and the drastically slows down query/app response times (reader). We have concurrently enabled, but that causes the indexing task to take even longer.

Investigate the right setup for RDS Cluster writer/reader configs

• should we have separate writer and reader definitions on the cluster?
• Create read-only proxy endpoint? Does that need separate reader on cluster or does cluster manage that already?

References:

• https://aws.amazon.com/blogs/database/use-amazon-rds-proxy-with-read-only-endpoints/
• https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_rds.DatabaseCluster.html
• RDS: add support for creating Proxy endpoints aws/aws-cdk#14186 (comment)

Use Case

Indexing large dataset (>1M rows) without degrading the search performance.

Proposed Solution

• add "reader(s)" to cluster
• add "read-only" endpoint to proxy (inference engine should use this)

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[github-actions]

Closing this issue as it hasn't seen activity for a while. Please add a comment @mentioning a maintainer to reopen. If you wish to exclude this issue from being marked as stale, add the "backlog" label.