Add hidden --extension to override inference of source type from file extension

[felix-cw]

Summary

This PR addresses the incompatibility with jupyterlab-lsp + python-lsp-ruff arising from the inference of source type from file extension, raised in #6847.

In particular it follows the suggestion in #6847 (comment) to specify a mapping from file extension to source type.

The source types are

• python
• pyi
• ipynb

Usage:

ruff check --no-cache --stdin-filename Untitled.ipynb --extension ipynb:python

Unlike the original suggestion, : instead of = is used to associate file extensions to language since that is what is used with --per-file-ignores which is an existing option that accepts a mapping.

Test Plan

2 tests added to integration_test.rs to ensure the override works as expected

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no linter changes.

[charliermarsh]

Thank you! I'm comfortable with this approach but can we try moving extension onto settings?

[dhruvmanila]

Thanks for doing this! And sorry for the back and forth in the issue :)

[dhruvmanila]

Sorry for not noticing this in my first review 😓

How will this work if the cell contained IPython escape commands (!pwd, %matplotlib inline)? The parser mode is tied up with PySourceType. This mode is then used to allow the escape commands.

ruff/crates/ruff_python_parser/src/lib.rs

Lines 325 to 332 in 2d5ce45

	impl AsMode for PySourceType {
	fn as_mode(&self) -> Mode {
	match self {
	PySourceType::Python | PySourceType::Stub => Mode::Module,
	PySourceType::Ipynb => Mode::Ipython,
	}
	}
	}

Now, if the cell contained escape commands and the override flag tells that this is a Python language (--extension-override ipynb:python), then it'll throw a syntax error.

[felix-cw]

jupyterlab-lsp can replace notebook magics with equivalent python code in most cases, so in this use case these syntax errors are avoided (I think the code is here).

Any python code affected by magic ends up in a string so doesn't get checked, for example with magics like %time and %%time.

[dhruvmanila]

Thanks for letting us know about that.

Coming from a public API perspective, this might still be a problem as, for example, if someone's trying to use it in a similar manner with an integration that doesn't perform those transformations, we'd throw an error.

I would love any other opinion on this. I'm fine with including the flag but if we include it in the config, then it's part of the public API which is where I'm unsure of.

[felix-cw]

FWIW jupyter nbconvert --to script also seems to convert notebook magics to python in the same way. nbdev has a routine callled scrub_magics which removes magics altogether, though it seems to be optional.

The docs could make it clear that python means python without notebookisms. Maybe something like:

Users who use this option to lint code extracted from notebooks as python should ensure that the code should be free of IPython magics, as these will cause a syntax error.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

[charliermarsh]

I pushed some nits, and I'll let @dhruvmanila approve and merge ultimately, but this looks good to me. I opted to remove the setting from Options for now, since @dhruvmanila is right that we only need to support it on the CLI -- but I do like that it's on Settings rather than a standalone argument, so I think that refactor was necessary anyway :)

[dhruvmanila]

Thanks @felix-cw for contributing and welcome to the repo!