Running dbt with an invalid table reference returns a cryptic can't pickle ParserMacroCapture objects.

[GuillaumeLegoy]

Edit: originally a bug, but more of an improvement so labels might not be adequate anymore.

Describe the bug

When running a dbt command (dbt run or dbt seed) with an invalid reference table (no quote) as part of a model ({{ ref(keywords) }} instead of {{ ref('keywords') }} ) for instance, the terminal returns the following:

can't pickle ParserMacroCapture objects

Weirdly, the behavior seems to be present in dbt 0.15.0, 0.15.1, 0.15.2 but not in dbt 0.14.4.

Steps To Reproduce

In my project here the bug happens when running the command described above: pipenv run dbt seed --profile-dir=. from the root folder. Fixing the ref model by adding single quotes around its name fixes the issue.

Expected behavior

A better error message when a ref is not valid, for instance: Invalid ref able, missing single quotes around identifier.

Screenshots and log output

System information

Which database are you using dbt with?

• [X ] postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

dbt 0.15.2
dbt 0.15.1
dbt 0.15.0

Again works with dbt 0.14.4 and below.

The operating system you're using:
Linux PopOs

The output of python --version:
3.7.5

[drewbanin]

Thanks for the report @GuillaumeLegoy! I'm leaving this one as a bug, but I do appreciate your followup on the root cause here! If you're curious, the issue is that the keywords variable in your example code is a jinja2 Undefined object. dbt "pickles" these objects to pass them between threads, or to serialize them to disk in a .pickle file, for instance. Since the Undefined object is not pickleable, this raises an exception.

I'm think the fix here might be to make the ParserMacroCapture object pickable so that this particular error can be avoided (and a more helpful error can be surfaced instead).

Edit: just because it's worth saying: bad/unhandled error messages like this one are bugs IMO!

[beckjake]

@drewbanin I think that's actually a little bit hard - the ParserMacroCapture class is defined in a closure. From the pickle docs

Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

We'd have to change how we define the ParserMacroCapture and bind it to things.

A better path might be removing the idea of "capturing" macros at all, seeing as we don't use that code anyway. (I'm pretty sure if we were actually using it, we'd be not including macros in the model parsing env). I'm not sure what that would break, but probably a lot!

[beckjake]

100% agree on the error message being terrible though - we can do better than this, for sure.

[drewbanin]

@beckjake sure, I'm happy to rip out macro capturing if we can!

[cmcarthur]

@beckjake is this caused by post-parse pickling for partial parsing? would using JSON (or another format) fix this error?

we should still rip out ParserMacroCapture, just curious how this connects to other specific partial parsing work that we are prioritizing

[drewbanin]

Let's spend ~1 day on this, and if we don't feel like we have a good answer, we should reconvene about the best way to solve this problem @beckjake