Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable use of provider functions in the Terraform language #34394

Merged
merged 10 commits into from
Dec 21, 2023

Conversation

jbardin
Copy link
Member

@jbardin jbardin commented Dec 11, 2023

Enable the use of provider-defined functions within the provider:: scope.

Providers can now implement functions which can be used from within the Terraform configuration language. The schema for each function is defined via the provider schema, using a layout similar to that of cty function definitions already in use internally. All provider functions found via their schemas are added to the provider:: namespace, using the provider's local name as defined in the module's required_providers. This means that a noop function declared by the aws provider would be access as provider::aws::noop(). The use of the :: delimiter as a function scoping mechanism was already added to the updated hcl package.

While providers must be trusted to ensure their functions only operate "offline", Terraform helps enforce this by only calling provider functions on an un-configured provider instance. This special provider instance will be started on demand, and cached for the remainder of the evaluation run. Having a separate provider is also required because normal resource providers have a lifecycle tied to their respective provider graph node, however function evaluation can happen from any node in the configuration so a separate instance must be waiting for any possible calls.

One important validation is not yet implemented, which is checking the purity of the function implementations. Checking for side effects of course is not possible, so again we must trust the providers implementations, but we do intend to validate the result values for consistency.

@jbardin jbardin force-pushed the jbardin/provider-functions-lang branch from 8267098 to 789f31c Compare December 11, 2023 20:27
@jbardin jbardin requested a review from a team December 12, 2023 20:03
@jbardin jbardin force-pushed the jbardin/provider-functions-lang branch from 789f31c to 95763a2 Compare December 12, 2023 20:13
@jbardin jbardin marked this pull request as ready for review December 12, 2023 20:29
Base automatically changed from jbardin/provider-functions-impl to main December 20, 2023 20:15
apparentlymart and others added 9 commits December 20, 2023 15:20
This essentially doubles up the registrations for all of the built-in
functions to have both non-namespaced and core::-namespaced versions of
each function.

This is in preparation for later commits introducing other namespaces,
such as a provider:: namespace which could hold functions that were
contributed by providers that are currently in scope.
The jsonfunction package has special cases for some internal functions,
and wee need to add the namespaced equivalents to those lists.
This is intended as a bridge to functions defined outside of the core
language, which will live in other namespaces in order to avoid ambiguity
with any built-in functions.

For now we only accept provider-contributed external functions. The lang
package doesn't really know anything special about providers, so its only
responsibility here is to enforce the naming scheme that any
provider-contributed functions always live under the namespace prefix
"provider::NAME::", where NAME is a name decided by whatever codepath
instantiated the state as a unique identifier for a particular provider.

This commit also includes updates to the core package to make it possible
to pass in ExternalFuncs in principle, although in practice we don't yet
have any codepaths which do so, and so this is effectively a no-op commit.
We need to be able to reuse providers which are used for function calls,
so cache them in the EvalContext.

This does require a way to close the provider however, which we can do
via the normal root module close node since it already does the same for
provisioners. Update the `CloseProvisioners` method to be a more general
`ClosePlugins` method, and use that to call Close on the cached
providers too.
This establishes all of the wiring necessary for a provider's declared
functions to appear in the hcl.EvalContext when we evaluate expressions
inside a module that depends on a provider that contributes functions.

This does not yet make any attempts to guarantee that the
provider-contributed functions correctly honor the contracts such as
behaving as pure functions. Properly checking this is important because
if a function doesn't uphold Terraform's expectations then it will cause
confusing errors reported downstream, incorrectly blaming other
components for the inconsistency.
Now that we dynamically add scoped versions to the global functions we
can't check the static Descriptions list. This should be OK since the
function descriptions are taken directly from Descriptions list, and if
any other scoped functions were added in the future, they may not be
coming from the current global set of functions anyway.
Copy link
Contributor

@alisdair alisdair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I'm surprised we have quite so many instances of conditional logic based on function name.

@@ -12,7 +12,7 @@ import (
)

var (
ignoredFunctions = []string{"map", "list"}
ignoredFunctions = []string{"map", "list", "core::map", "core::list"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting here because this change reminded me of this command: do we intend in a later PR to expand metadata functions to include the functions offered by installed providers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metadata functions doesn't require an initialized working directory, so unfortunately provider functions are not going to be part of that output. Since that is mainly for the language server integration, it's yet to be determined if a separate command to list providers functions is needed (they are fetching schemas anyways if providers are available, which include functions), or maybe having inconsistent output from metadata is actually OK.

internal/terraform/context_functions_test.go Outdated Show resolved Hide resolved
condition = (local.result == 3)
error_message = "Wrong number of Es in my cheese."
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good example of a useful provider function.

@jbardin jbardin merged commit 8f6b13a into main Dec 21, 2023
6 checks passed
@jbardin jbardin deleted the jbardin/provider-functions-lang branch December 21, 2023 16:45
Copy link
Contributor

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

Copy link
Contributor

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants