helper/schema: Provide Functionality to Reduce Resident Memory Usage from Schemas #1217
Comments
bflad
added a commit
that referenced
this issue
Jun 23, 2023
Reference: #1217 This change introduces a new `SchemaFunc` field and `SchemaMap` method to the `Resource` type. Currently, the `Schema` field data of all `Resource` is held in memory for the lifecycle of the provider server, which is problematic for providers with many resources and/or larger schemas in resources. The new field enables provider developers to swap pieces of resident memory usage for the slight additional computation time of reconstituting the data when necessary. Callers directly referencing the exported `Schema` field should switch to referencing the `SchemaMap` method, which returns the result of `SchemaFunc` or `Schema` in that preference order. To ensure internal usage was migrated to the new method, this change was performed by temporarily commenting out the `Schema` field itself with broken references in non-testing code migrated to the method. The `Schema` field is not marked as deprecated via Go documentation comment as this would introduce a major ecosystem burden to migrate with generally little benefit for most use cases. The `Resource` type `InternalValidate` method has been updated to return an error if both `Schema` and `SchemaFunc` are defined. Provider developers are encouraged to migrate resources to the terraform-plugin-framework, as it already behaves in a manner similar to `SchemaFunc` by nature of resource schema data being behind a method call, amongst many of the other benefits outlined at https://developer.hashicorp.com/terraform/plugin/framework-benefits.
bflad
added a commit
that referenced
this issue
Jun 28, 2023
Reference: #1217 This change introduces a new `SchemaFunc` field and `SchemaMap` method to the `Resource` type. Currently, the `Schema` field data of all `Resource` is held in memory for the lifecycle of the provider server, which is problematic for providers with many resources and/or larger schemas in resources. The new field enables provider developers to swap pieces of resident memory usage for the slight additional computation time of reconstituting the data when necessary. Callers directly referencing the exported `Schema` field should switch to referencing the `SchemaMap` method, which returns the result of `SchemaFunc` or `Schema` in that preference order. To ensure internal usage was migrated to the new method, this change was performed by temporarily commenting out the `Schema` field itself with broken references in non-testing code migrated to the method. The `Schema` field is not marked as deprecated via Go documentation comment as this would introduce a major ecosystem burden to migrate with generally little benefit for most use cases. The `Resource` type `InternalValidate` method has been updated to return an error if both `Schema` and `SchemaFunc` are defined. Provider developers are encouraged to migrate resources to the terraform-plugin-framework, as it already behaves in a manner similar to `SchemaFunc` by nature of resource schema data being behind a method call, amongst many of the other benefits outlined at https://developer.hashicorp.com/terraform/plugin/framework-benefits.
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
SDK version
Use-cases
This Go module caches the entire provider definition within the provider server, including all data and managed resource schema data via the
Resourcetype having aSchemafield directly referencing themap[string]*schema.Schemadata. At sufficient provider scale, by nature of many resources and/or large schemas within resources, resident memory usage permanently increases without a way for provider developers to avoid the issue.Attempted Solutions
Migrate resources to terraform-plugin-framework, which schema data is already behind a method call. This ensures the provider server can treat the schema data as eligible for garbage collection immediately after usage (minus any internal caching the framework itself may perform).
Proposal
Introduce a
ResourcetypeSchemaFuncfield with afunc() map[string]*Schematype, which would store a function pointer for the schema data in memory rather than the data itself. To prevent consumers from needing to check both the old and new schema data fields, introduce a newResourcetypeSchemaMapmethod which returnsmap[string]*Schemaand handles either calling theSchemaFunc, if defined, or returning the priorSchema` data.The
ResourcetypeSchemafield should NOT be marked as deprecated, as this would create a major ecosystem burden with little benefit for most use cases.The
ResourcetypeInternalValidatemethod should be updated to return an error if bothSchemaandSchemaFuncare defined.References
The text was updated successfully, but these errors were encountered: