Exporter: generate references that are matching only to a prefix #1890

alexott · 2023-01-02T09:33:52Z

There are cases when reference isn't a full match of a given value but a prefix of it. For example, user's notebook has a path of /Users/user@domain/notebook. To make it working correctly we need to emit user or service principal, and then make a correct reference with dependency on emitted user or notebook, like, ${databricks_user.user_domain.home}/notebook.

This PR adds a new field to dependency specification: MatchType that may have following values:

prefix - value of attribute of resource is a prefix of a given attribute
exact (or empty string) - must have exact matching (previous behaviour)

this fixes #1777

codecov-commenter · 2023-01-02T09:38:33Z

Codecov Report

Merging #1890 (ac3f1fa) into master (ab3df33) will decrease coverage by 0.14%.
The diff coverage is 65.45%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1890      +/-   ##
==========================================
- Coverage   90.18%   90.04%   -0.15%     
==========================================
  Files         143      143              
  Lines       11427    11456      +29     
==========================================
+ Hits        10305    10315      +10     
- Misses        712      730      +18     
- Partials      410      411       +1

Impacted Files	Coverage Δ
exporter/model.go	`88.88% <ø> (ø)`
exporter/importables.go	`89.32% <31.25%> (-1.18%)`	⬇️
exporter/context.go	`85.78% <65.21%> (-1.75%)`	⬇️
exporter/util.go	`87.63% <100.00%> (+0.71%)`	⬆️

Exporter now detects Spark conf and environment variables values that are references to secrets (the `{{secrets/scope/secret}}` syntax) and emit a secret scope referenced there. P.S. It doesn't put explicit reference into the value, but it could be added after #1890 is merged (it will require addition of `substring` match type, but it's doable).

exporter/context.go

nfx · 2023-01-03T10:00:44Z

exporter/context.go

 if traversal != nil {
+ if d.MatchType == "prefix" { // we're replacing the found prefix with the HCL reference using the "${reference}rest" syntax


is it more complex if ic.Find returns prefixed traversal?

you mean to generate tokens directly in the ic.Find ? yes, maybe it's good idea to isolate it there

exporter/model.go

nfx · 2023-01-03T10:16:37Z

@alexott #1897 secrets probably also should be matchable by prefix (maybe not right now, but when people start asking for it)

Exporter now detects Spark conf and environment variables values that are references to secrets (the `{{secrets/scope/secret}}` syntax) and emit a secret scope referenced there. P.S. It doesn't put explicit reference into the value, but it could be added after #1890 is merged (it will require addition of `substring` match type, but it's doable).

There are cases when reference isn't a full match of a given value but a prefix of it. For example, user's notebook has a path of `/Users/user@domain/notebook`. To make it working correctly we need to emit user or service principal, and then make a correct reference with dependency on emitted user or notebook, like, `${databricks_user.user_domain.home}/notebook`. This PR adds a new field to dependency specification: `MatchType` that may have following values: - `prefix` - value of attribute of resource is a prefix of a given attribute - `exact` (or empty string) - must have exact matching (previous behaviour) this fixes #1777

# Version changelog ### 1.8.0 * Added support for GCP (Public Preview) with state upgraders for [databricks_mws_workspaces](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) ([#1871](#1871), [#1886](#1886), [#1748](#1748), [#1879](#1879)). * Added documentation for [databricks_permission_assignment](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/permission_assignment) ([#1880](#1880)). * Added `azure_attributes` to clusters in [databricks_pipeline](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/pipeline) resource ([#1854](#1854)). * Added [databricks_instance_pool](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/instance_pool) data source ([#1907](#1907)). * Added [databricks_provider](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/provider) resource for Delta Sharing ([#1572](#1572)). * Added acceptance tests for [databricks_cluster](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/cluster) data and [databricks_sql_endpoint](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/sql_endpoint) data ([#1882](#1882)). * Added search by name for [databricks_cluster](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/cluster) data source ([#1901](#1901)). * Added support for sparse checkouts in the [databricks_repo](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/repo) ([#1869](#1869)). * Added `config_reference` attribute to [databricks_secret](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret) for easier reference from Spark configuration ([#1898](#1898)). * Added `datarbicks_directory` data source ([#1902](#1902)). * Extended [databricks_catalog](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) resource to support Delta Sharing Catalog ([#1887](#1887)). * Improved [databricks_metastore_assignment](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment) resource ([#1900](#1900)). * Improved [databricks_grants](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) error messages ([#1888](#1888)). * Improved [databricks_group](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group) documentation ([#1816](#1816)). * Improved `data_security_mode` documentation for [databricks_cluster](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster) ([#1830](#1830)). * Replace the incorrect `id` attribute with the correct `sp_id` in service principal data source docs example ([#1873](#1873)). * Use NVMe disk size for selection in the [databricks_node_type](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/node_type) data source ([#1856](#1856)). * Allowed to override resource ID for specific data source ([#1885](#1885)). Experimental Resource Exporter * Added command-line flag to export all users and service principals ([#1893](#1893)). * Added emit for secret scopes from Spark Conf and Env Vars ([#1897](#1897)). * Fixed issue with exporting of resources with simple name ([#1891](#1891)). * Generate references that are matching only to a prefix ([#1890](#1890)). * Updated list of supported services and associated resources ([#1894](#1894)). * Use [databricks_group_role](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_role) instead of deprecated [databricks_group_instance_profile](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group_instance_profile) ([#1905](#1905)). * Use `dbfs_path` as reference for init scripts in clusters ([#1892](#1892)). * Logging improvements ([#1895](#1895)). Updated dependency versions: * Bump google.golang.org/api from 0.104.0 to 0.105.0 ([#1863](#1863)).

…#1897) Exporter now detects Spark conf and environment variables values that are references to secrets (the `{{secrets/scope/secret}}` syntax) and emit a secret scope referenced there. P.S. It doesn't put explicit reference into the value, but it could be added after databricks#1890 is merged (it will require addition of `substring` match type, but it's doable).

…abricks#1890) * Exporter: generate references that are matching only to a prefix There are cases when reference isn't a full match of a given value but a prefix of it. For example, user's notebook has a path of `/Users/user@domain/notebook`. To make it working correctly we need to emit user or service principal, and then make a correct reference with dependency on emitted user or notebook, like, `${databricks_user.user_domain.home}/notebook`. This PR adds a new field to dependency specification: `MatchType` that may have following values: - `prefix` - value of attribute of resource is a prefix of a given attribute - `exact` (or empty string) - must have exact matching (previous behaviour) this fixes databricks#1777 * improving test coverage & emit user/sp for repos * Address PR feedback

alexott added the exporter TF configuration generator label Jan 2, 2023

alexott requested a review from nfx January 2, 2023 09:33

alexott marked this pull request as ready for review January 2, 2023 10:19

alexott requested a review from a team January 2, 2023 10:19

alexott mentioned this pull request Jan 3, 2023

Exporter: emit secret scopes from Spark Conf and Env Vars #1897

Merged

nfx requested changes Jan 3, 2023

View reviewed changes

alexott added 3 commits January 3, 2023 15:32

improving test coverage & emit user/sp for repos

80535d1

Address PR feedback

3dca34d

alexott force-pushed the generic-dependencies branch from 2fbc1f7 to 3dca34d Compare January 3, 2023 16:12

alexott requested a review from nfx January 3, 2023 16:13

nfx approved these changes Jan 4, 2023

View reviewed changes

nfx enabled auto-merge (squash) January 4, 2023 13:33

Merge branch 'master' into generic-dependencies

8f0d3f4

nfx merged commit 6a00182 into master Jan 4, 2023

nfx mentioned this pull request Jan 10, 2023

Release v1.8.0 #1915

Merged

nfx deleted the generic-dependencies branch January 25, 2023 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exporter: generate references that are matching only to a prefix #1890

Exporter: generate references that are matching only to a prefix #1890

alexott commented Jan 2, 2023

codecov-commenter commented Jan 2, 2023

nfx Jan 3, 2023

alexott Jan 3, 2023

nfx commented Jan 3, 2023

		if traversal != nil {
		if d.MatchType == "prefix" { // we're replacing the found prefix with the HCL reference using the "${reference}rest" syntax

Exporter: generate references that are matching only to a prefix #1890

Exporter: generate references that are matching only to a prefix #1890

Conversation

alexott commented Jan 2, 2023

codecov-commenter commented Jan 2, 2023

Codecov Report

nfx Jan 3, 2023

Choose a reason for hiding this comment

alexott Jan 3, 2023

Choose a reason for hiding this comment

nfx commented Jan 3, 2023