Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AssertionError when opening with chardet #2785

Merged
merged 2 commits into from
Mar 15, 2023
Merged

Fix AssertionError when opening with chardet #2785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
689d952
Fix AssertionError when opening with chardet
n-thumann Mar 13, 2023
91903d8
Add test for unknown encoding using chardet
n-thumann Mar 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions codespell_lib/_codespell.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,6 @@ def open_with_chardet(self, filename: str) -> Tuple[List[str], str]:
break
self.encdetector.close()
encoding = self.encdetector.result["encoding"]
assert encoding is not None # noqa: S101
Copy link
Collaborator

@DimitriPapadopoulos DimitriPapadopoulos Mar 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assertion was introduced by 8d0d82b / #2588 in an attempt to please mypy. That's not a proper fix, especially since Python can optimise assertions away, but mypy did give a good hint to look into this part of the code. And we now have the proper fix!


try:
f = open(filename, encoding=encoding, newline="")
Expand All @@ -245,7 +244,7 @@ def open_with_chardet(self, filename: str) -> Tuple[List[str], str]:
lines = f.readlines()
f.close()

return lines, encoding
return lines, f.encoding
Copy link
Collaborator

@DimitriPapadopoulos DimitriPapadopoulos Mar 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, we return io.TextIOBase.encoding, as actually set by open in text mode (checked it's UTF-8 for empty files).


def open_with_internal(self, filename: str) -> Tuple[List[str], str]:
encoding = None
Expand Down
10 changes: 10 additions & 0 deletions codespell_lib/tests/test_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,16 @@ def test_encoding(
assert "WARNING: Binary file" in stderr


def test_unknown_encoding_chardet(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Test opening a file with unknown encoding using chardet"""
fname = tmp_path / "tmp"
fname.touch()
assert cs.main("--hard-encoding-detection", fname) == 0


def test_ignore(
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
Expand Down