[CT-960] Configurable Color-Coding of Schemas/Nodes in Docs

[talagluck]

Similar to here.
Essentially, we are interested in being able to color code either schemas or individual models in the dbt docs in a similar fashion to the way you can configure materialization of individual models.

e.g.

models:
  rad_warehouse:
    materialized: view
    staging:
      schema: staging
      materialized: view
      color: #453453
    analytics:
      schema: analytics
      materialized: table
      color: #589593

We really like the difference in coloration between source and models, and it would be awesome to carry this further to make it easier to distinguish between different parts of the dag.

A colleague (Taylor M.) and I are interested in working on this. We can imagine implementing this in a couple of different ways, either through the dbt_project.yml, or through configurable tags in models or schemas.

Do you have any suggestions, tips, warnings, or words of caution?

Thanks!

[drewbanin]

Thanks @talagluck! Check out this related issue in dbt-core: dbt-labs/dbt-core#1671

This is one of those issues that spans the core + docs repos, so some coordination is required! Maybe a good approach is to hammer our that docs: section for the schema.yml (+dbt_project.yml?) file contract. Once dbt-core supports that, we can add in configs like show/hide, colors, logos (for sources in loaders??), etc.

Super into this, happy to help however I can :)

[Aylr]

@drewbanin Another thought we were bouncing around was extending the built-in tags so you could do something like this:

  filtering:
      schema: filtering
      materialized: view
      tags: ['filtering', 'orange']

This way we could be in better control of the color scheme by providing a handful of nicely coordinated color tags for users to use at will.

[talagluck]

Some really interesting conversation in that issue! Good things to think about.

One thing that occurs to me is that we could probably make good use of tags for this.
For instance:

events:
      tags: ["events", "hourly", "blue"]
      materialized: table

Pros:

• Restricts the number of colors to ensure matching color combinations, maintain the overall aesthetic.
• Piggybacks on existing functionality

Cons:

• Restricts the number of colors which limits customization
• Colors would wind up being reserved words
• Need to figure out logic if a user tags a model with more than one color

Any thoughts? Does this work against the intended use of tags?

[talagluck]

Looks like Taylor and I were working at the same time :) What he said!

[drewbanin]

Hey guys! Check out this PR - this is still pretty rough, but should give you an idea of the (minimal) code changes required in dbt-core: dbt-labs/dbt-core#1692

I like the general idea behind using tags, but I'd opt for maybe classes + some sort of stylesheet instead? Maybe that's bonkers, but I don't think it's a good idea to repurpose tags for both rich graph selection + node coloring. If anything, we can add classes to nodes for each of the specified tags, and then use a user-supplied stylesheet to style nodes as required? That would be a little further out, but it sounds pretty cool to me!

[Aylr]

This is starting to click for me! I'm now sold on not overloading the tags in favor of the proposed docs portion. That makes way more sense.

I think as an MVP we could have a set of pre-set colors then in the future make it extensible to custom styles? (This isn't a strong opinion, just looking for the shortest reasonable path there).

[drewbanin]

I'm getting started on this over here: #45

this uses artifacts generated by running dbt with this branch: dbt-labs/dbt-core#1692

the schema.yml file I'm using looks like:

version: 2

models:
    - name: incr
      docs:
          color: "blue"
          hidden: true

    - name: incr2
      docs:
          color: "red"
          hidden: false

    - name: incr3
      docs:
          color: "green"
          hidden: false

    - name: incr4
      docs:
          color: "pink"
          hidden: false

So, lots more work to do here for sure, but wanted to do the worst possible version of this to help inform our opinions / future work!

I'm going to put this down until after the dbt v0.14.1 release is complete, but looking forward to digging back in!

[mike-weinberg]

the schema.yml file I'm using looks like:

version: 2

models:
    - name: incr
      docs:
          color: "blue"
          hidden: true

    - name: incr2
      docs:
          color: "red"
          hidden: false

    - name: incr3
      docs:
          color: "green"
          hidden: false

    - name: incr4
      docs:
          color: "pink"
          hidden: false

I love this as a solution to both this and dbt-labs/dbt-core#1671

Thank you @drewbanin ! #grateful

[Aylr]

FYI @drewbanin @talagluck and I are slowly pushing this forward. We are currently working on making the custom colors not interfere w/ the user selection colors.

[rarup1]

Is there any updated on whether the custom colours is going to be worked on for a future release? Would be super useful for explaining to stakeholders which models are "safe" to use and which models are intermediate calculations.

[sungchun12]

Love the idea of using node colors for "approved" consumption layers of the DAG

[leahwicz]

We did it! Coming out in v1.3 🎉

[sungchun12]

@rarup1 Here's a demo: https://www.loom.com/share/f7161efd14fa4e209e74702d7caf33c9

[rarup1]

This is great news @sungchun12. Will be upgrading to latest dbt asap!

[Aylr]

@talagluck Our color-coded dreams have come true! 🥲

[sungchun12]

It's comments like the above that make all the hard work so worth it. Thank you @Aylr @rarup1 !