pip cells break code analyzer

[gpelouze]

Jupyter allows notebooks to contain cells with individual pip install ... commands. However, they break the code analyzer because they don’t contain valid Python code.

    Traceback (most recent call last):
      File "/venv/lib/python3.11/site-packages/pytype/io.py", line 336, in wrap_pytype_exceptions
        yield
      File "/venv/lib/python3.11/site-packages/pytype/tools/annotate_ast/annotate_ast.py", line 40, in infer_types
        return traces.trace(source, options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/tools/traces/traces.py", line 85, in trace
        ret = analyze.infer_types(
              ^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/analyze.py", line 103, in infer_types
        loc, defs = ctx.vm.run_program(src, filename, init_maximum_depth)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/vm.py", line 557, in run_program
        src_tree = directors.parse_src(src, self.ctx.python_version)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/directors/parser.py", line 462, in parse_src
        ast.parse(src, feature_version=python_version[1]), _process_comments(src))  # pylint: disable=unexpected-keyword-arg
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/ast.py", line 50, in parse
        return compile(source, filename, mode, flags,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<unknown>", line 899
        pip install minio
            ^^^^^^^
    SyntaxError: invalid syntax

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/venv/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/jupyterlab_vre/component_containerizer/handlers.py", line 116, in post
        extractor = PyExtractor(notebook)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/jupyterlab_vre/services/extractor/pyextractor.py", line 22, in __init__
        self.notebook_names = self.__extract_cell_names(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/jupyterlab_vre/services/extractor/pyextractor.py", line 183, in __extract_cell_names
        tree = self.__get_annotated_ast(cell_source)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/jupyterlab_vre/services/extractor/pyextractor.py", line 141, in __get_annotated_ast
        return annotate_ast.annotate_source(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/tools/annotate_ast/annotate_ast.py", line 20, in annotate_source
        source_code = infer_types(source, pytype_options)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/venv/lib/python3.11/site-packages/pytype/tools/annotate_ast/annotate_ast.py", line 39, in infer_types
        with io.wrap_pytype_exceptions(PytypeError, filename=options.input):
      File "/venv/lib/python3.11/contextlib.py", line 158, in __exit__
        self.gen.throw(typ, value, traceback)
      File "/venv/lib/python3.11/site-packages/pytype/io.py", line 348, in wrap_pytype_exceptions
        raise exception_type("Error reading file %s at line %s: %s" %
    pytype.tools.annotate_ast.annotate_ast.PytypeError: Error reading file dummpy_input_file at line 899: invalid syntax

The issue can be avoided by using !pip install ....

We should figure out how Jupyter lab identifies such cells and remove them before they reach the code analyzer.

[gpelouze]

We already have an exception for lines starting with ! in the PyExtractor.__init__