pylint checker: False unable to import because of implicit namespaces

[abadger]

Information

VIM version

NVIM v0.3.6
Build type: RelWithDebInfo

Also occurs with

VIM - Vi IMproved 8.1 (2018 May 18, compiled Jun  6 2019 11:29:32)
Included patches: 1-1471

Operating System: Fedora Linux 29

pylint-2.3.1

What went wrong

The pylint checker reports a False positive:

[pylint] relative-beyond-top-level: Attempted relative import beyond top-level package [E]

with code like the following:

 from .level2 import plugin1

when the toplevel is a PEP 420 implicit namespace package: https://www.python.org/dev/peps/pep-0420/

This is somewhat fragile in pylint itself as well.

• Running pylint on the directory gives the same error
• Running pylint on the filename gives the same error

However, if you run pylint on the module name instead of the filename, then it will detect the relative import correctly and will not find this false positive. (Examples in the reproducing section, below)

Reproducing the bug

hello-1.0.tar.gz

1. wget https://github.com/w0rp/ale/files/3302570/hello-1.0.tar.gz
2. tar -xzvf hello-1.0.tar.gz
3. cd hello-1.0/
4. vim hello/top.py

Line 1 is highlighted with a pylint error,

[pylint] relative-beyond-top-level: Attempted relative import beyond top-level package [E]

5. the same error should be seen if you do vim hello/level2/plugin1.py which has a slightly different relative path.

6. You can see this is a pylint limitation with filenames by doing:

• pylint --reports n -d all -e relative-beyond-top-level hello/top.py

  hello/top.py:1:0: E0402: Attempted relative import beyond top-level package (relative-beyond-top-level)
  --------------------------------------------------------------------
  Your code has been rated at -6.67/10 (previous run: -6.67/10, +0.00)
  

7. You can see that there is a way to work around this pylint limitation by using a module name instead:

• pylint --reports n -d all -e relative-beyond-top-level hello.top

  --------------------------------------------------------------------
  Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
  

Based on the above, I wrote a short pylint wrapper which can work around this problem. However, it is not generic and possibly fragile. It is probably better if ALE provided a way to pass a function to transform the path that is given to the checker. ALE could give the function the project root and the path which will probably solve the genericity issue. I'm not sure whether the fragility issue can be perfectly remedied. The wrapper is here: https://gist.github.com/abadger/cd4d90552406290733a8f6fa8bdc0db4

:ALEInfo

 Current Filetype: python
Available Linters: ['bandit', 'flake8', 'mypy', 'prospector', 'pycodestyle', 'pydocstyle', 'pyflakes', 'pylama', 'pylint', 'pyls', 'pyre', 'vulture']
  Enabled Linters: ['flake8', 'mypy', 'pylint']
 Suggested Fixers: 
  'add_blank_lines_for_python_control_statements' - Add blank lines before control statements.
  'autopep8' - Fix PEP8 issues with autopep8.
  'black' - Fix PEP8 issues with black.
  'isort' - Sort Python imports with isort.
  'remove_trailing_lines' - Remove all blank lines at the end of a file.
  'reorder-python-imports' - Sort Python imports with reorder-python-imports.
  'trim_whitespace' - Remove all trailing whitespace characters at the end of every line.
  'yapf' - Fix Python files with yapf.
 Linter Variables:

let g:ale_python_flake8_auto_pipenv = 0
let g:ale_python_flake8_change_directory = 1
let g:ale_python_flake8_executable = 'flake8'
let g:ale_python_flake8_options = ''
let g:ale_python_flake8_use_global = 0
let g:ale_python_mypy_auto_pipenv = 0
let g:ale_python_mypy_executable = 'mypy'
let g:ale_python_mypy_ignore_invalid_syntax = 0
let g:ale_python_mypy_options = '--ignore-missing-imports --namespace-packages'
let g:ale_python_mypy_use_global = 0
let g:ale_python_pylint_auto_pipenv = 0
let g:ale_python_pylint_change_directory = 1
let g:ale_python_pylint_executable = 'pylint'
let g:ale_python_pylint_options = ''
let g:ale_python_pylint_use_global = 0
let g:ale_python_pylint_use_msg_id = 0
 Global Variables:

let g:ale_cache_executable_check_failures = v:null
let g:ale_change_sign_column_color = 0
let g:ale_command_wrapper = ''
let g:ale_completion_delay = v:null
let g:ale_completion_enabled = 0
let g:ale_completion_max_suggestions = v:null
let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'E'
let g:ale_echo_msg_format = '[%linter%] %code: %%s [%severity%]'
let g:ale_echo_msg_info_str = 'Info'
let g:ale_echo_msg_warning_str = 'W'
let g:ale_enabled = 1
let g:ale_fix_on_save = 0
let g:ale_fixers = {'python': ['isort']}
let g:ale_history_enabled = 1
let g:ale_history_log_output = 1
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_filetype_changed = 1
let g:ale_lint_on_insert_leave = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'normal'
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let g:ale_linters_explicit = 0
let g:ale_list_vertical = 0
let g:ale_list_window_size = 10
let g:ale_loclist_msg_format = '[%linter%] %code: %%s [%severity%]'
let g:ale_lsp_root = {}
let g:ale_max_buffer_history_size = 20
let g:ale_max_signs = -1
let g:ale_maximum_file_size = v:null
let g:ale_open_list = 0
let g:ale_pattern_options = v:null
let g:ale_pattern_options_enabled = v:null
let g:ale_set_balloons = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 0
let g:ale_sign_error = '>>'
let g:ale_sign_info = '--'
let g:ale_sign_offset = 1000000
let g:ale_sign_style_error = '>>'
let g:ale_sign_style_warning = '--'
let g:ale_sign_warning = '--'
let g:ale_statusline_format = v:null
let g:ale_type_map = {}
let g:ale_use_global_executables = v:null
let g:ale_virtualtext_cursor = 0
let g:ale_warn_about_trailing_blank_lines = 1
let g:ale_warn_about_trailing_whitespace = 1
  Command History:

(executable check - success) flake8
(finished - exit code 0) ['/bin/zsh', '-c', '''flake8'' --version']

<<<OUTPUT STARTS>>>
3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0, radon: 3.0.1) CPython 3.7.3 on Linux
<<<OUTPUT ENDS>>>

(executable check - success) mypy
(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/home/badger/tmp/hello-1.0'' && ''mypy'' --show-column-numbers --ignore-missing-imports --namespace-packages --shadow-file ''/home/badger/tmp/hello-1.0/hello/top.py'' ''/var/tmp/nvimmhTbgG/2/top.py'' ''/home/badger/tmp/hello-1.0/hello/top.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - success) pylint
(finished - exit code 18) ['/bin/zsh', '-c', 'cd ''/home/badger/tmp/hello-1.0'' && ''pylint''  --output-format text --msg-template="{path}:{line}:{column}: {msg_id} ({symbol}) {msg}" --reports n ''/home/badger/tmp/hello-1.0/hello/top.py''']

<<<OUTPUT STARTS>>>
************* Module top
hello/top.py:1:0: C0111 (missing-docstring) Missing module docstring
hello/top.py:1:0: E0402 (relative-beyond-top-level) Attempted relative import beyond top-level package
hello/top.py:4:0: C0111 (missing-docstring) Missing function docstring

---------------------------------------------------------------------
Your code has been rated at -13.33/10 (previous run: -6.67/10, -6.67)

<<<OUTPUT ENDS>>>

(finished - exit code 0) ['/bin/zsh', '-c', 'cd ''/home/badger/tmp/hello-1.0/hello'' && ''flake8'' --format=default --stdin-display-name ''/home/badger/tmp/hello-1.0/hello/top.py'' - < ''/var/tmp/nvimmhTbgG/3/top.py''']

<<<NO OUTPUT RETURNED>>>

[w0rp]

This could be difficult to handle. I wonder if this might be something that could be fixed in pylint itself.

[abadger]

Yeah, I opened up a bug report there, pylint-dev/pylint#2967 , but I don't know how quickly things get fixed in their project. In the past I had other import-related things that pylint handled better with module-name instead of file-name but I haven't seen those undder ALE... perhaps because ALE changes to the project root directory before running.