making dbt-loom work with dbt-osmosis?

[rognerud]

Hello I am just curious.

Is it possible to make dbt-loom work with dbt-osmosis?
Ie inserting the nodes with all of the documentation so that the documentation can be inherited?

Is is something that has not been developed or is there a more fundamental problem making it impossible?

[nicholasyager]

Hey @rognerud 👋🏻 Unfortunately, this is a limitation in dbt-core's implementation of ModelNodeArgs. Whereas a typical node would have all manner of documentation fields like description and columns, the ModelNodeArgs¹ type does not support those fields. This intuitively makes sense, given that dbt Labs is generating all of their documentation using their metadata service.

For the rest of us, there may be a possible approach using the artifact plugin hook, but this would be out-of-band documentation and would require tweaks to dbt-osmosis. There may also be some experimentation possible where we totally ignore typing and see if we can get dbt-core to serialize extra fields if we provide them, but this is a bit less likely to work IMHO.

Having all that been said, dbt-loom is open to contributions if you have a particular approach in mind!

Footnotes

1. https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/contracts/graph/node_args.py#L10 ↩

[rognerud]

Hi @nicholasyager

I have tinkered with the artifact plugin hook a little.

It seems to work in that it adds metadata to the "loom" models, by updating them using the get_manifest_artifacts() hook
And that again keeps the manifest "updated" as this happens every time the manifest gets made.

I tried using dbt-osmosis on top of this, and even with the depends-on and the models being there it wouldnt work.

link to diff

[rognerud]

I see, osmosis parses the manifest using dbts ManifestLoader and that strips away the added metadata in the manifest.json file as the nodes created by loom are parsed as external nodes and has to conform to the ModelNodeArgs definition when read by the dbt ManifestLoader.

[rognerud]

A possible solution if inheriting using osmosis is the only goal would then be to make no changes to dbt-loom so that loom only supports referencing external models in models, and instead make osmosis be able to search through other manifests for documentation.

[nicholasyager]

@rognerud Thank you for the deep digging on this! I agree with your final takeaway: a "good" scenario is likely for dbt-osmosis to parse and merge together multiple manifests for documentation.