Should token errors be handled silently?

[krassowski]

Currently pyls-memestra raises on the following code:

sum(

as there is no closing ) bracket it is not a valid Python syntax, but it is a commonplace to see code like that in editors when users type. Should pyls-memestra know which errors to catch and ignore as harmless?

Example traceback:

Traceback (most recent call last):
  File "lib/python3.7/site-packages/frilouz.py", line 94, in parse
    return parser(code, *args, **kwargs), errors
  File "lib/python3.7/site-packages/gast/gast.py", line 307, in parse
    return ast_to_gast(_ast.parse(*args, **kwargs))
  File "lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    sum(
       ^
SyntaxError: unexpected EOF while parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "lib/python3.7/threading.py", line 1177, in run
    self.function(*self.args, **self.kwargs)
  File "lib/python3.7/site-packages/pylsp/_utils.py", line 33, in run
    return func(*args, **kwargs)
  File "lib/python3.7/site-packages/pylsp/python_lsp.py", line 299, in lint
    flatten(self._hook('pylsp_lint', doc_uri, is_saved=is_saved))
  File "lib/python3.7/site-packages/pylsp/python_lsp.py", line 156, in _hook
    return hook_handlers(config=self.config, workspace=workspace, document=doc, **kwargs)
  File "lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "lib/python3.7/site-packages/pluggy/manager.py", line 337, in traced_hookexec
    return outcome.get_result()
  File "lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "lib/python3.7/site-packages/pluggy/callers.py", line 52, in from_call
    result = func()
  File "lib/python3.7/site-packages/pluggy/manager.py", line 335, in <lambda>
    outcome = _Result.from_call(lambda: oldcall(hook, hook_impls, kwargs))
  File "lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/krassowski/jupyterlab-lsp/pyls-memestra/pyls_memestra/plugin.py", line 47, in pylsp_lint
    search_paths=search_paths)
  File "lib/python3.7/site-packages/memestra/memestra.py", line 421, in memestra
    module, syntax_errors = frilouz.parse(ast.parse, file_descriptor.read())
  File "lib/python3.7/site-packages/frilouz.py", line 101, in parse
    code = _build_safe_code(code, se.lineno)
  File "lib/python3.7/site-packages/frilouz.py", line 40, in _build_safe_code
    for tok in tokenize.tokenize(tmp.readline):
  File "lib/python3.7/tokenize.py", line 579, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (9, 0))