return spec field for entities retrieved using backstage_entities data source

[crivetechie]

We are currently retrieving Components and Resources from Backstage using backstage_entities datasource.

data "backstage_entities" "services" {
  filters = [
    "kind=component,spec.lifecycle=production",
    "kind=resource,spec.lifecycle=production",
  ]
}

We would need to have access to the spec field in order to identify the type of the component/resource.
Is it possible to return the spec field for the entities retrieved with backstage_entities datasource ?

Thanks for building this awesome provider!

[tdabasinskas]

Hi,

I think the initial reason I haven't added support for spec to backstage_entities was that this field highly depends on the entity type. As per the documentation:

The precise structure of the spec depends on the apiVersion and kind combination, and some kinds may not even have a spec at all. See further down in this document for the specification structure of specific kinds.

I'm not sure how to implement such dynamic value properly in the provider.

Would it be an option for you to use backstage_resource and backstage_component individually?

[crivetechie]

Hi, thanks for getting back to me, I thought that would have been the challenge.

I had a look at using backstage_resource and backstage_component but they fetch a single entity and we need to load hundreds of entities so I'd keep it as last resort.
I wonder if it would make sense to introduce backstage_resources and backstage_components datasource allowing to filter resources and components? With these we could just query Backstage twice: to collect components and resources separately.

[tdabasinskas]

I think having both backstage_resource and backstage_resources might be a bit too confusing…

I did some research — it seems we might be able to support dynamic attributes in the near future, once hashicorp/terraform-plugin-framework#931 is merged and released (part of v1.7.0 of the framework?). This would allow implementing spec properly.

I'm not sure if it's worth implementing some workaround until then.

[tdabasinskas]

Just a small update:

Since the v1.7.0 framework is released with included Dynamic attribute support, I played with it a bit, but it faced the following limitation:

Dynamic types inside of collections are not currently supported in terraform-plugin-framework

Since spec is nested in entities collection, it cannot be dynamic..

I will come back to this in the future, but right now I can't think of an easy and proper way to implement this.

[crivetechie]

thank for the update!

[brianphillips]

(caveat: I know very little about building terraform modules, apologies if this is a foolish suggestion) Would it make sense to just return the spec as a map[string]string similar to how metadata.annotations are represented in the absence of the ability to fully support it for now? Or perhaps that would just result in a breaking change in the future if/when dynamic attributes are more adequately supported?

[tdabasinskas]

(caveat: I know very little about building terraform modules, apologies if this is a foolish suggestion) Would it make sense to just return the spec as a map[string]string similar to how metadata.annotations are represented in the absence of the ability to fully support it for now? Or perhaps that would just result in a breaking change in the future if/when dynamic attributes are more adequately supported?

What makes it tricky, compared to metadata.annotations, is that's the elements of spec are not necessary strings.

Based on the current documentation, a spec element can also be a list of strings (e.g., spec.members). Also, even though there are no current entity types that have something other than than a string or a list of string, I would assume that spec being dynamic, there could also be entities that have numbers or even more complex types for its spec elements.

[brianphillips]

Supporting a subset of the most commonly used fields that are consistently typed might be a worthwhile compromise. For instance, spec.type is present on many (although not all) of the builtin backstage kinds and is always a string. Similarly, spec.system and spec.owner are also always strings when they are used on one of the officially supported kinds (aside: in the case of spec.owner, that value can be found in the relations field so is not as important as spec.system and spec.type, IMO).

I suppose it's entirely possible that someone might extend the model with a new kind and use one of those fields for a different data type, so perhaps there's still some risk that things will not be consistently typed.

[tdabasinskas]

@crivetechie , @brianphillips ,

Would you mind giving 2.4.0-rc1 a try (based on the example provided in #138) and let me know if it covers your requirements?

[tdabasinskas]

As there was no feedback for a week, I will assume #138 resolves this, as it allows you to access spec for the entities retrieved via backstage_entities.