Tests folder is Included during os.walk Additionally with no Exception Handling during Model Queries

[z3z1ma]

Long title but the issue is interrelated.

The tests folder is included when building model catalog. This is a problem because tests are not manifest as tables in the database but they are still being added to models list and queried for while connected to the database. This cause a SQLAlchemy exception that the table does not exist and the program hard fails. In theory you can pass excluded folders: - tests
but that is suboptimal because everyone who uses data tests would need to do this.

I propose 2 things:

1. Lets ignore some directories by default leveraging the dbt_project yml instead of this exclusion pattern here because it has both the issue of the tests folder not being excluded but also it is not resilient to users with non standard directories

2. Lets add some basic exception handling during the loop that is building models and actually querying the database. Then we can decide if we want to fail gracefully or push past. Without that, users will be completely stuck if a model has not materialized or is dropped in a post run hook or who knows.

I'll put up a PR a little later today. LMK thoughts otherwise

[bastienboutonnet]

Hi @z3z1ma makes sense to me.

Regarding 1:
I think we probably still want to have the exclusion pattern here, but reading it from the dbt_profile.yml as well as sugar_config.yml should not be mutually exclusive. In the end, I would read from all of those places and compose the right exclusion pattern. How does that sound to you?

Regarding 2: Absolutely agree, it's not something I encountered during development but I think it would make total sense. Do you have an idea of how you want to write that or do you want to spitball a few ideas?

PR totally welcome whenever you feel like it. I'm a bit snowed under these days so if you have time do go for it, if not I'll get around to it as soon as I can dive deeper into this again.