Discussion: improving the validation experience

[williamdclt]

This is a summary of a Discord discussion started here.

The "official" way to test permissions with SpiceDB seems to be using zed validate. There's other options (eg spin up a server and send grpc requests to it), but I'm talking about developer experience using zed validate in this discussion.

zed validate takes a "self-contained" YAML containing the schema, assertions, and expected relations, looking like this. A developer experience must include the generation of such YAML in some way: I see a few possible development experience, neither being really satisfying.

Workflow 1: develop in the Playground
I've been recommended to use the YAML as a source of truth for both schema and tests. That means running zed validate on the YAML for testing, and somehow extracting the schema part of it to load into SpiceDB.
This is straightforward enough, and allows a good integration with the Playground: to work on my permission system, I can simply upload this file in the playground and work from there, having all my tests within reach.

• Problem 1: I don't really want to use the Playground as my editor. I have an editor I use daily and which is fine-tuned to my needs, I use Vim keybindings, I have a consistent color scheme, shortcuts etc etc: having to use an arbitrary in-browser editor isn't a better DevX. Note that I'm not debating that the Playground offers very useful functionalities, I'd love to have it in my editor.
• Problem 2: this workflow means that all my tests are defined in a single place (the "assertions" and "expected relations" tabs), and all seed data for these tests in another place (the "test relationships" tab). I have >150 permissions: having only one big test file cannot realistically work, it would be near-impossible to read, keep tests decoupled and to figure out what relationships each test is depending on.

Workflow 2: create one YAML per permission
To avoid having all tests defined in a single file, we can "just" create one YAML per permission. "Running the tests" means running zed validate on each of these files, which a few lines of bash can do very well.

• Problem 1: the schema is duplicated between all these files, and need to be updated everywhere at every change. What is even the source of truth for the schema in this case?
• Problem 2: we still have multiple test cases's data and assertions being mixed together (a permission usually needs a least 2 tests). It's a bit messy, although I don't find it to be a deal-breaker. We could create a YAML file per test case, but that ends up being just as messy I think (many more file)

Workflow 3: define the schema once, and have one YAML per test
This is what I'm doing at the moment: I have my schema defined in a schema.zed file and I have one YAML per test, these YAML do not have the schema attribute. To run the tests, I have some bash script reconstructing a YAML with the schema attribute, injecting the schema.zed in it.

• Problem 1: it's a lot of ugly custom bash code.
• Problem 2: I have YAMLs that aren't loadable in the playground, as they don't have a schema attribute. I also can't just download a YAML from the playground as test: I need to get rid of the schema attribute.
• Problem 3: we still have multiple test cases's data and assertions being mixed together (same as workflow 2).

---

It would be great to have a supported way to define test files, separately from the schema :) And ideally, being able to define independant test cases within a test file.