Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLMesh+dbt+duckdb: Unable to read source declared in dbt #2988

Closed
cnstlungu opened this issue Aug 7, 2024 · 2 comments · Fixed by #2999
Closed

SQLMesh+dbt+duckdb: Unable to read source declared in dbt #2988

cnstlungu opened this issue Aug 7, 2024 · 2 comments · Fixed by #2999
Assignees

Comments

@cnstlungu
Copy link

I'm using sqlmesh+dbt+duckdb. I have declared my sources as follows:

image

In my model, upon using the source jinja function, I get an error:

image

Traceback (most recent call last):
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/web/server/api/endpoints/commands.py", line 69, in evaluate
    df = context.evaluate(
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/analytics/__init__.py", line 99, in wrapper
    return func(*args, **kwargs)
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/context.py", line 802, in evaluate
    df = self.snapshot_evaluator.evaluate_and_fetch(
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/snapshot/evaluator.py", line 164, in evaluate_and_fetch
    result = self._evaluate_snapshot(
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/snapshot/evaluator.py", line 578, in _evaluate_snapshot
    return self.adapter._fetch_native_df(query_or_df.limit(limit))
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/engine_adapter/base.py", line 1697, in _fetch_native_df
    self.execute(query, quote_identifiers=quote_identifiers)
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/engine_adapter/base.py", line 1831, in execute
    self._execute(sql, **kwargs)
  File "/home/costin/repos/portable-data-stack-sqlmesh/.venv/lib/python3.10/site-packages/sqlmesh/core/engine_adapter/base.py", line 1837, in _execute
    self.cursor.execute(sql, **kwargs)
duckdb.duckdb.CatalogException: Catalog Error: Table with name channels does not exist!
Did you mean "postcard_company_raw.raw_channels"?
LINE 1: ...T_TIMESTAMP AS "loaded_timestamp" FROM "datamart"."parquet_input"."channels" A...
                                                  ^

image

It is to be pointed out that the parquet_input is just a source name, not a database schema/dataset as indicated above. Also, this does work if I use the read_parquet directly

image

My environment:

duckdb==1.0.0
dbt-duckdb==1.8.1
sqlmesh==0.104.0
pydantic==2.4.0
@tobymao
Copy link
Contributor

tobymao commented Aug 7, 2024

@Themiscodes or @treysp can you take a look?

@Themiscodes
Copy link
Contributor

Sure I will look into it @tobymao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants