Project Restructure: Service Level Packages

[breathingdust]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Closely related #20000

Description

The structure of the Terraform AWS provider codebase is not scaling well as it grows. This explosive growth is fuelled by the AWS provider’s popularity and AWS’s ever increasing services. As a result, maintainers and community contributors face an unnecessary layer of difficulty maintaining the code base such as non-responsive IDEs and the inability to see all project files on GitHub.

In order to better follow Go best practices and address these growing pains, we are planning significant code refactoring of the Terraform AWS Provider. The refactoring will position the provider for better maintainability now and into the future as AWS ever expands its offerings.

Code for every AWS service we support, in 2,000+ Go files, lives in one massive package. A single line change requires the entire package to be recompiled, which at best, can take several minutes. Contributors, maintainers, IDEs, Go compilers, and security scanning products, have unnecessary overhead dealing with such a massive package. At the same time, many tasks require jumping back and forth between different directory structures.

Proposal

The primary goal is simply stated: move everything related to a single AWS service, such as ec2, into one package. This includes resources, data sources, associated tests, and support code. During the refactor, we will also undertake housekeeping to simplify things like naming and imports. Further details are given below.

This solution provides several benefits:

• A reduction in defect risks since service packages are independent. Unintended consequences of changes in one service are less likely to affect other services.
• Having everything you need in one compact package for new development and maintenance tasks reduces the mental burden of scrolling and jumping around in the code.
• Not having unrelated code in a package makes it easier to grasp what is in the package.
• Go is optimized to compile the least number of packages necessary. Go will often be able to compile a single service package rather than the entire provider.
• Service Level Package structure will require a better-defined interface and division of responsibilities between specific service packages and general support packages. Clear divisions lead to uncoupling and help ensure proper testing.
• Simplicity in the structure makes the project more readable and reduces the challenge of submitting contributions. A clearer understanding of the project leads to better quality contributions.

Cutover Process

This refactoring will occur as a "big-bang" change, internally these changes will be automated and tested against our acceptance test suite before rollout. The consequence of this is that most if not all existing Pull Requests will become unmergeable "as-is" due to the large amount of structural changes.

The date of the rollout will be communicated publicly, and guides on how to update your Pull Request will also be made available closer to the confirmed date.

We realize that many Pull Requests have been sitting for some time without being addressed. We want to make it very clear that the mergability of a Pull Request due to structural changes like this do not affect its prioritization for review. Once a Pull Request is prioritized for review, the maintainers will either make those changes directly or work with the PR author to do so. There is no expectation from the maintainers that the community updates their PR's en-masse, but we realize that some authors will choose to do so in order to explore the new package structure. We will make these requirements explicit both in the contributing guide, our linter suite and from this issue.

[breathingdust]

We expect that this change will occur in the next few weeks as our automation to accomplish the restructuring is reaching its final stages.

[github-actions]

This functionality has been released in v3.64.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.