Tag selector not working with tests using the manifest json

[nicohein]

There is a bug with the parsing of tests (at lease when using load_from_dbt_manifest).

Dbt does not populate the tags on the tests, still I would expected the behavior like in dbt that tests for the models matched by a tag are selected. The current implementation in astronomer-cosmos however only considers nodes in the filtered-notes for tests.

astronomer-cosmos/cosmos/dbt/graph.py

Line 326 in 2e3e4fd

def load_from_dbt_manifest(self) -> None:

I would hope that this can be fixed by iterating through nodes not filtered_nodes in the following line:

astronomer-cosmos/cosmos/dbt/graph.py

Line 375 in 2e3e4fd

for _, node in self.filtered_nodes.items():

Alternatively there could be a way to do a second run through the nodes in the selector to catch all tests that depend on selected models:

astronomer-cosmos/cosmos/dbt/selector.py

Line 84 in 2e3e4fd

def select_nodes_ids_by_intersection(nodes: dict[str, DbtNode], config: SelectorConfig) -> set[str]:

I have not thought too much about possible side effects.

Current Workaround: Use a different selection method or explicitly tag tests.

--

astronomer-cosmos version 1.1.3
dbt-core vserion 1.5.7

[tatiana]

@nicohein This seems to be a limitation of Cosmos custom selector, which is used with LoadMethod.CUSTOM or LoadMethod.DBT_MANIFEST. If using load_via_dbt_ls, the selector behaviour would be just like dbt's.

Would you like to make a contribution to support this use-case in the custom selector?

[nicohein]

@tatiana while I would like to make a contribution I am currently unable to give commitments. I am happy to be around for testing also beta versions within our test environment.

[dosubot]

Hi, @nicohein,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, the issue "Tag selector not working properly with tests using manifest json in astronomer-cosmos" seems to have been resolved with a potential fix suggested in the code, along with a workaround provided. Additionally, there is an invitation for you to contribute to support this use-case, to which you expressed interest.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and contributions to the Cosmos repository.

Dosu

[nicohein]

HI @tatiana ,
I have written an alternative solution for dbt dags on airflow using offline parsing of the same. I will unfortunately not be able to contribute further.
Best
Nico

[dosubot]

🤖

Hi @nicohein,

Thank you for your response and for sharing your alternative solution. We appreciate your contributions to the Cosmos repository. We'll go ahead and close the issue now.

Best,
Dosu

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.