Can't import python runner

[michael-berk]

I can't import the python lineage runner. The raw import sqllineage works fine and I can get lineage via subprocess.

Error

Stacktrace:

TypeError                                 Traceback (most recent call last)
File <command-1798185685866154>:1
----> 1 from sqllineage.runner import LineageRunner

File /databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py:171, in _create_import_patch.<locals>.import_patch(name, globals, locals, fromlist, level)
    166 thread_local._nest_level += 1
    168 try:
    169     # Import the desired module. If you’re seeing this while debugging a failed import,
    170     # look at preceding stack frames for relevant error information.
--> 171     original_result = python_builtin_import(name, globals, locals, fromlist, level)
    173     is_root_import = thread_local._nest_level == 1
    174     # `level` represents the number of leading dots in a relative import statement.
    175     # If it's zero, then this is an absolute import.

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-dc49e689-66e4-41c1-92ff-4e9ce5d33aa8/lib/python3.10/site-packages/sqllineage/runner.py:9
      7 from sqllineage.core.models import Column, Table
      8 from sqllineage.core.parser.sqlfluff.analyzer import SqlFluffLineageAnalyzer
----> 9 from sqllineage.core.parser.sqlparse.analyzer import SqlParseLineageAnalyzer
     10 from sqllineage.drawing import draw_lineage_graph
     11 from sqllineage.io import to_cytoscape

File /databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py:171, in _create_import_patch.<locals>.import_patch(name, globals, locals, fromlist, level)
    166 thread_local._nest_level += 1
    168 try:
    169     # Import the desired module. If you’re seeing this while debugging a failed import,
    170     # look at preceding stack frames for relevant error information.
--> 171     original_result = python_builtin_import(name, globals, locals, fromlist, level)
    173     is_root_import = thread_local._nest_level == 1
    174     # `level` represents the number of leading dots in a relative import statement.
    175     # If it's zero, then this is an absolute import.

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-dc49e689-66e4-41c1-92ff-4e9ce5d33aa8/lib/python3.10/site-packages/sqllineage/core/parser/sqlparse/__init__.py:34
     30     _patch_adding_builtin_type()
     31     _patch_updating_lateral_view_lexeme()
---> 34 _monkey_patch()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-dc49e689-66e4-41c1-92ff-4e9ce5d33aa8/lib/python3.10/site-packages/sqllineage/core/parser/sqlparse/__init__.py:31, in _monkey_patch()
     29 _patch_adding_window_function_token()
     30 _patch_adding_builtin_type()
---> 31 _patch_updating_lateral_view_lexeme()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-dc49e689-66e4-41c1-92ff-4e9ce5d33aa8/lib/python3.10/site-packages/sqllineage/core/parser/sqlparse/__init__.py:21, in _patch_updating_lateral_view_lexeme()
     19 def _patch_updating_lateral_view_lexeme() -> None:
     20     for i, (regex, lexeme) in enumerate(SQL_REGEX):
---> 21         rgx = re.compile(regex, re.IGNORECASE | re.UNICODE).match
     22         if rgx("LATERAL VIEW EXPLODE(col)"):
     23             new_regex = r"(LATERAL\s+VIEW\s+)(OUTER\s+)?(EXPLODE|INLINE|PARSE_URL_TUPLE|POSEXPLODE|STACK|JSON_TUPLE)\b"

File /usr/lib/python3.10/re.py:251, in compile(pattern, flags)
    249 def compile(pattern, flags=0):
    250     "Compile a regular expression pattern, returning a Pattern object."
--> 251     return _compile(pattern, flags)

File /usr/lib/python3.10/re.py:302, in _compile(pattern, flags)
    300     return pattern
    301 if not sre_compile.isstring(pattern):
--> 302     raise TypeError("first argument must be string or compiled pattern")
    303 p = sre_compile.compile(pattern, flags)
    304 if not (flags & DEBUG):

TypeError: first argument must be string or compiled pattern

[reata]

This is more of a usage thing, so I move this to discussion.

First of all, if import fails and subprocess works, then most probably they're not calling the same python environment. I see /local_disk0/.ephemeral_nfs/envs/pythonEnv-dc49e689-66e4-41c1-92ff-4e9ce5d33aa8/lib/python3.10/site-packages/sqllineage/ in the exception stack, this must be the python env for your import part (a standalone script or something else) are using. I was guessing in the subprocess call (maybe another script triggered else where using different shell PATH), the python env is different. It's up to you to debug.

As for the exception, this is something we fixed in sqllineage v1.4.3, where we enforcing people to upgrade to sqlparse>=0.4.4, because sqlparse==0.4.4 introduces a backward incompatible change. If somehow you're using sqllineage==1.4.3, while sqlparse is still in lower version, say 0.4.3 or 0.4.2, then you'll see this exception.

Hope this helps.

[michael-berk]

Got it, thanks. I just enforced sqlparse>=0.4.4 and still getting the error. Will continue debugging.