[CT-1222] [CT-1220] [Bug] dbt documentation does not list macros that were migrated from dbt-utils

[ali-infostrux]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

dbt documentation currently has no mention of the macros that were migrated from dbt-utils in March (dbt-labs/dbt-core#4813) and subsequently released.

For example for date-diff, dbt-utils doc mentions: "DEPRECATED: This macro is now provided in dbt Core. It is no longer available in dbt_utils and backwards compatibility will be removed in a future version of the package."

Expected Behavior

Having list of macros included in dbt-core and expected behaviour documented like they were in dbt-utils

Steps To Reproduce

Search for macros in the dbt documentation and find nothing. https://docs.getdbt.com/
Search for macros in dbt-utils and see that they are deprecated. https://github.com/dbt-labs/dbt-utils#datediff-source

Relevant log output

No response

Environment

dbt 1.2

Which database adapter are you using with dbt?

No response

Additional Context

No response

[ali-infostrux]

Also I filed this as a bug and it might be the wrong category, couldn't think what missing doc would fall under. Apologies in advance if that wasn't right.

[jtcohen6]

Thanks @ali-infostrux! I'm going to move this one over to the dbt-docs repo.

Here's the spot where we explicitly filter out any macros defined in the dbt (global project) namespace:

dbt-docs/src/app/services/project_service.js

Lines 337 to 339 in aa128ea

	if (package_name == 'dbt' || package_name == 'dbt_' + adapter) {
	return
	}

Should we keep doing that? Perhaps not! There's good reason to want easy access and documentation to these utilities — especially if we added description, args hints, ...

[ali-infostrux]

I think if at least the list of macros was available in official dbt documentation, then no need for it in generated project doc, but the list of available macros should be available somewhere aside from diving deep into the dbt-core package code to know what's there.

[jtcohen6]

Fair! We do have this: https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros

But that page is not automatically updated today. Whereas, if we included them here, they would be

@dbeatty10 Curious if you have a thought here. Relevant to our convo earlier re: the expanded user persona for cross-db macros as utilities / building blocks

[dbeatty10]

@jtcohen6 auto-generated documentation for our cross-db macros à la Sphinx docs could be handy for the expanded user persona for cross-db macros as utilities / building blocks.

@ali-infostrux thanks for reaching out about this!

Did you happen to try the search bar in the upper right corner of https://docs.getdbt.com ?

When I typed "datediff" I got the following results. The first four results are all Developer Blog posts, but the next two link to our list of macros.

Admittedly, this page might be a bit hard to find within the navigation section on the left:

[jtcohen6]

@jtcohen6 auto-generated documentation for our cross-db macros à la Sphinx docs could be handy for the expanded user persona for cross-db macros as utilities / building blocks.

What I was realizing, thanks to this issue, is that we already have auto-generated docs for macros — it's dbt-docs! In our codebase, instead of adding docstrings a la Python/Sphinx, we'd need to add description + args: https://docs.getdbt.com/reference/macro-properties

Bonus points if we could:

• Figure out a way to serve the same structured information (from manifest.json) in docs.getdbt.com, to avoid the need to repeat ourselves
• Use Jinja's parser / AST (?) to auto-detect arguments for macros, and record those in the manifest, in case users haven't documented themselves. Sorta relevant: [CT-459] [Feature] Extend Jinja macros to contain required arguments (and maybe type hints) dbt-core#4999

[jtcohen6]

The more immediate question to answer right now: Are there still good reasons to exclude "internal" macros from the dbt-docs site?

Especially now that we're adding in utilities that users might want to know about, beyond the internal guts that they should happily ignore.

This caused confusion for at least one user, and I imagine it's done the same for many more.

[dbeatty10]

+1 for dbt-docs 💪

I wonder if including "internal" macros for dbt docs generate would help make it more discoverable when there is a project macro that is overriding an "internal" macro?

[ali-infostrux]

Did you happen to try the search bar in the upper right corner of https://docs.getdbt.com ?

When I typed "datediff" I got the following results. The first four results are all Developer Blog posts, but the next two link to our list of macros.

Indeed, thanks!
I was initially trying to get the list of supported macros in dbt core (like the list in dbt-utils) so I searched everything macro related.
Now that I know what exact page I'm looking for, I see that a search for "macro" in the search bar returns the "Cross-database macros" in last after 16 other options. For a user looking for a straightforward list coming from dbt-utils, I must say I found it a tad confusing. That came after a colleague spent 15 minutes trying to find that same list of macros and couldn't. Maybe we should just be better at searching for things :)

Admittedly, this page might be a bit hard to find within the navigation section on the left:

I'm ashamed to say I was browsing the doc with 1.1 selected by default so even as I did look throughout the tree, I never found it. My bad there

I'm glad that popped a more interesting discussion in the process though!

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.