Metadata search tooling

[jdangerx]

Problem

We have 227 tables and several thousand columns in our data warehouse. While we've attempted to provide some structure via the raw/core/out naming scheme, it's still confusing and overwhelming to figure out whether we have data that's relevant to someone's needs.

Currently the only real option to answer "do you have data I want?" is to go into the Data Dictionary page and do a browser-based search.

Options

3rd party data catalog

We could spin up a service like Open Metadata or DataHub, which have many delicious features that help us:

• metadata search
• sample data display
• data lineage display
• data quality metrics

etc.

These mostly operate on the following model:

• run a data pipeline to ingest metadata about all your datasets
• run elasticsearch across that metadata
• run a webapp that interfaces with elasticsearch, as well as the non-searchable metadata (e.g. data lineage, sample data, etc.)

Client-side search

We could also throw some client-side search at the data dictionary. This lets people do a full-text search of our data dictionary & do various filtering, without spinning up any new services.

The model would be:

• generate the docs as usual
• have some Javascript that parses the documentation into some sort of search index, with text as well as structured metadata fields
  • or, do this work at generation time so that the client-side code just needs to load a JSON from somewhere
• expose some interface to the search engine

Here's a prototype video & the code lives on the docs-search branch.

Analysis

Here's a very simplistic analysis. Overall, I think the client-side search will meet our needs and be simpler to set up as well as maintain - but open to further discussion!

3rd party data catalog

Pros:

• lots of cool features!
• designed to scale up!
• we don't have to maintain the codebase

Cons:

• maybe we don't need these features/scale? our metadata will not get even close to the size/complexity that these tools are designed to handle
• ingesting metadata into their system requires a whole new data pipeline - lots of operational load
• we don't maintain the codebase, but do have to maintain our system's interaction with their system

Client-side search

Pros:

• fast to set up!
• no infrastructure to run!
• metadata generation is straightforward! no data pipeline needed, we just build it at doc generation time.
• we maintain the codebase, so we can change it easily

Cons:

• we maintain the codebase, so we have to do work
• doesn't have all the fancy features in the 3rd party data catalog
• won't scale to massive amounts of metadata (but... will that ever happen?)

[zschira]

I'm definitely in favor of the easier option for now. It looks like the 3rd party tools are really sleek with some potentially super useful features, but we also have lots of other big infrastructure work to do right now and the client side search seems "good enough" for the time being.

[zaneselvans]

I agree that the in-page filtering is a big and apparently simple step toward making the data dictionary usable for at least a while longer. I have absolutely no idea how JavaScript / web stuff works and would love a really brief orientation just so I can understand what's easy / possible / hard with this kind of solution. Like how is it doing this at all? What kinds of search or filtering operations can we have it do? What is the data structure that it's operating on and how can we populate it?