sqllineage 1.4.3 dialect option producing unnecessary verbose logs even with verbose False

[gky249]

Hi,

Using sqllineage 1.4.3 version with dialect option for oracle, I noticed that it is writing unnecessary statements into my logfile when I write the LineageRunner object into logfile. With previous release, printing the LineageRunner object would only print the source and target tables, but now some unrequired statements like below are also printing and filling up the logfile -

2023-06-05 11:56:28,195 TEMPLATING RAW [jinja] (<string>)
2023-06-05 11:56:28,198 Slicing File Template
2023-06-05 11:56:28,200 LEXING RAW (<string>)
2023-06-05 11:56:28,203 Elements to Segments.
2023-06-05 11:56:28,206 Lexed tokens:
2023-06-05 11:56:28,206 PARSING (<string>)
2023-06-05 11:56:28,207 [PD:0  MD:0 ]   FileSegment                                             AnyNu.match PRN                 [ns=1, ps=1, ms=0, pruned=[<Ref: ExecuteFileSegment>], opts=[<Delimited: [<Ref: StatementSegment>]>]]
2023-06-05 11:56:28,207 [PD:0  MD:0 ]   FileSegment                                             AnyNu.match PRN                 [ns=1, ps=1, ms=0, pruned=[<Ref: ExecuteFileSegment>], opts=[<Delimited: [<Ref: StatementSegment>]>]]
2023-06-05 11:56:28,223 [PD:0  MD:6 ]   ......ExecuteFileSegment                                OneOf.match PRN                 [ns=0, ps=2, ms=0, pruned=[<Sequence: [<Ref: AtSignSegment>, <Ref: AtSignSegment [opt]>]>, <Ref: StartKeywordSegment>], opts='ALL']
2023-06-05 11:56:28,223 [PD:0  MD:6 ]   ......ExecuteFileSegment                                OneOf.match PRN                 [ns=0, ps=2, ms=0, pruned=[<Sequence: [<Ref: AtSignSegment>, <Ref: AtSignSegment [opt]>]>, <Ref: StartKeywordSegment>], opts='ALL']
2023-06-05 11:56:28,224 [PD:0  MD:6 ]   ......ExecuteFileSegment                                OneOf.match OUT                 [m 

This is how I'm doing it -

result = LineageRunner(sqlstmt,dialect='oracle',verbose=False)
logger.info(result)

When I remove the dialect option, it prints only the source and target tables like previous releases.
Please let me know if this is expected behavior or a bug and if so, when can we fix it ?

[reata]

This is more about how you configure your logger instead of how sqllineage works.

Since v1.4.3, sqllineage uses sqlfluff as underlying parser for other dialect options. sqlfluff has various logging message logged at different level (debug, info, warning, etc). sqllineage doesn't actively use logging module as of now except for cli scenario. But that can change in the future. LineageRunner is guaranteed to print information to stdout though.

That being said, if you configure your logger to handle INFO level message, then by default all the logging message with INFO level will show up in your log. You can disable all the message from sqlfluff though. As a library, we won't do that.

[gky249]

ok thank you.