[CT-119] [Bug] run_results.json schema validation error with oneOf constraint on status

[indyyyyy]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Trying to validate the run_results.json artifact produced by the dbt test command, I used the jsonschema python package and the schema you expose here (run-results/v3.json).
The validation fails with the error below.

Expected Behavior

As the artifact and the json schema are produced by dbt, I expect the schema validation to be OK.
"oneOf" constraint seems to be unadapted as a status could appears in several enum.

Steps To Reproduce

1. Run dbt test
2. In a python3 console, run the following lines:

import jsonschema
import requests

dbt_runresults_schema = requests.get(url='https://schemas.getdbt.com/dbt/run-results/v3.json').json()
runresults_rawjson=json.load(open('path/to/target/run_results.json'), encoding="UTF-8"))

validate(instance=runresults_rawjson, schema=dbt_runresults_schema)

Relevant log output

>>> validate(instance=runresults_rawjson, schema=dbt_runresults_schema)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/jsonschema/validators.py", line 934, in validate
    raise error
jsonschema.exceptions.ValidationError: 'warn' is valid under each of {'type': 'string', 'enum': ['pass', 'warn', 'error', 'runtime error']}, {'type': 'string', 'enum': ['pass', 'error', 'fail', 'warn', 'skipped']}

Failed validating 'oneOf' in schema['properties']['results']['items']['properties']['status']:
    {'oneOf': [{'enum': ['success', 'error', 'skipped'], 'type': 'string'},
               {'enum': ['pass', 'error', 'fail', 'warn', 'skipped'],
                'type': 'string'},
               {'enum': ['pass', 'warn', 'error', 'runtime error'],
                'type': 'string'}]}

On instance['results'][0]['status']:
    'warn'

### Environment

```markdown
- OS:Debian GNU/Linux 10 (buster)
- Python:3.9.6
- dbt:0.21.1
- jsonschema:3.1.1

What database are you using dbt with?

redshift

Additional Context

As a workaround I changed in the schema the "oneOf" constraint of

"properties": {
        "status": {
          "oneOf": [
          ...

by a "anyOf" constraint which seems more relevant.

[iknox-fa]

👋 @indyyyyy Sorry it's taken a min to get back to you on this. I was able to replicate the issue-- it seems our jsonschema could use some love. I'll categorize this as a bug and we'll get it prioritized ASAP.

[iknox-fa]

For future reference here's a slight update to fix some typos in the reproduction code posted above:

import jsonschema
import requests
import json

breakpoint()
dbt_runresults_schema = requests.get(url='https://schemas.getdbt.com/dbt/run-results/v3.json').json()
runresults_rawjson=json.load(open('/path/to/run_results.json'), encoding="UTF-8")

jsonschema.validate(instance=runresults_rawjson, schema=dbt_runresults_schema)

[jtcohen6]

We identified some limitations of the library (hologram) that we're using to generate JSONSchemas from internal dbt objects. As such, while we could make this fix as one-off, what we'd really need is a more rigorous conversion library and validation testing.

I hesitate to prioritize that larger effort over a continuing investment in structured logging, which we believe can offer a more powerful, reliable, and real-time metadata interface going forward.

[jtcohen6]

Improvement to the accuracy of our jsonschema generation isn't something we're able to prioritize currently. We should document that the JSONSchemas at schemas.getdbt.com are better understood as descriptive of the information contained in the JSON artifacts, rather than usable for rigorous validation of every artifact. In the meantime, we'll continue discussion in #4617 about how to create some healthy distance between dbt-core's internal classes and the artifacts it produces for external consumption.