Cross-test unknown values #1858
Labels
area/testing
The internal testing setup of the bridge
kind/engineering
Work that is not visible to an external user
Comments
t0yv0
added a commit
that referenced
this issue
Apr 12, 2024
Fixes #1790 by building a rapid generator for schemas and associated values. Large-ish problem 1: I do not have it figured out how to test unknown values. TF literals as unknown values are forbidden and do not make sense. We might need a helper resource so that testing unknown values generates references to an output of the helper resource. This is logged for future work. Large-ish problem 2: iteration is pretty slow (x-proc). Normal n=100 rapid tests can take up to 10min. Could try batching so several resources are tried in one shot say 100 resources. Large-ish problem 3: I'm not sure if no-op Update and Create implementations are acceptable. There is something to testing Computed attributes where provider has to set values. Possibly Update also needs to set values? Possibly not. Small problems: - [x] Using TF JSON syntax didn't handle null/empty correctly; that is now discarded, using actual HCL syntax - [x] TF representations are difficult to visualize in failing tests and difficult to assert against - [x] Lots of lost-in-translation papercuts possible between representations (cty.Value, resource.PropertyValue, tftypes.Value) - [x] this requires a change to providertest to abstract from testing.T so we can pass rapid.T - [x] it's very hard to disable annoying TF logging, using env vars for now We are starting to find bugs and discrepancies from this work: - #1856 panic corner-case - #1852 need to InternalValidate - #1828 Future work: - #1856 - #1857 - #1858 - #1859 - #1860 - #1861 - #1862 - #1863 - #1864 - #1865 - #1866 - #1867
iwahbe
added
kind/engineering
|
This might be interesting to investigate once we add unknowns to cross-tests: #2032 |
This was referenced Oct 30, 2024
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The cross-testing in #1801 is as yet unable to verify handling unknown values. The limitation here is that the drivers
use HCL and Pulumi YAML to feed data into the respective CLIs for an end-to-end test, but there are no literals for
unknown values. To emulate this one would need to emit an additional helper resource reference to simulate an
unknown-at-preview output in the provider.
In addition, it is worth verifying if Block-typed TF schemas can ever accept unknown values at runtime; if not,
randomized generation should not try to sample them probably, and we would need to confirm that Pulumi bridged providers
reject them for blocks.
Tasks
The text was updated successfully, but these errors were encountered: