Redact basic authentication passwords from log messages #5773

orf · 2018-09-10T12:16:04Z

This PR #5590 had merge conflicts, so I made a new one with the changes suggested. Fixes #4746

cjerdonek · 2018-09-11T08:33:29Z

src/pip/_internal/utils/misc.py

@@ -880,15 +880,29 @@ def split_auth_from_netloc(netloc):
 return netloc, user_pass


+def redact_password_from_url(url):
+ def _redact_netloc(netloc):


I would pull this function out to module level so it can be tested separately. You can use the same parametrized test cases as test_split_auth_from_netloc(). This will combinatorially cut down the number of test cases you need to properly test redact_password_from_url(), since redact_password_from_url() will be a simple combination of redact_netloc() and transform_url().

cjerdonek · 2018-09-11T08:37:50Z

src/pip/_internal/utils/misc.py


+def transform_url(url, transform_netlock):


I would move this function to be above redact_password_from_url() and remove_auth_from_url() (and after redact_netloc()), since it is used in both of them, and it is nice to have a progression so that functions within a module depend only on functions that precede them (so it can be read from top to bottom).

cjerdonek · 2018-09-11T08:38:20Z

src/pip/_internal/utils/misc.py


+def transform_url(url, transform_netlock):
+ purl = urllib_parse.urlsplit(url)
+ netloc = transform_netlock(purl.netloc)


Spelling: transform_netloc()

cjerdonek · 2018-09-11T08:41:58Z

tests/unit/test_utils.py

+ ('https://user:pass@domain.tld/project/tags/v0.2',
+ 'https://domain.tld/project/tags/v0.2'),
+ ('https://user:pass@domain.tld/svn/project/trunk@8181',
+ 'https://domain.tld/svn/project/trunk@8181'),


I would leave these test cases alone in the same order, especially if you're not changing them. Otherwise, it makes it needlessly harder to tell whether the tests have been changed or not.

cjerdonek · 2018-09-11T08:45:36Z

tests/unit/test_utils.py

+ ('git+ssh://git@pypi.org/something',
+ 'git+ssh://git@pypi.org/something'),
+ ('git+https://user:pass@pypi.org/something',
+ 'git+https://user:****@pypi.org/something'),


This seems like more test cases than you need. If you break out redact_netloc() like I suggested above and test it separately, you should be able to reduce the number of test cases you need for combinatorial reasons. You can focus on testing redact_netloc() well (which is a simpler function). And then test_redact_password_from_url() can be limited to making sure that redact_password_from_url() is "wired up" correctly from redact_netloc() and transform_url().

orf · 2018-09-11T09:47:44Z

Thanks for the prompt review @cjerdonek. I've made the changes you requested. Regarding testing, I just did a mock.patch to assert that redact_password_from_url is hooked up correctly. Is this OK?

cjerdonek · 2018-09-11T15:18:43Z

src/pip/_internal/utils/misc.py

- # and are not recognized in the url.
+def redact_netloc(netloc):
+ netloc, (user, passw) = split_auth_from_netloc(netloc)
+ if not user:


This should check for None it seems.

cjerdonek · 2018-09-11T15:22:19Z

Thanks. Looks a lot better. I’m not a huge fan of mocks, especially when not needed. I think it would be just as good or better to do one or two test cases to confirm that it’s being called (e.g. one with and one without password).

Also, one reason the mock test isn’t so great is that it’s not checking the return value, which is the most important thing to be checking.

cjerdonek · 2018-09-11T15:25:18Z

tests/unit/test_utils.py

+ # Test the password containing a : symbol.
+ ('user:pass:word@example.com', 'user:****@example.com'),
+])
+def test_redact_netloc(netloc, expected):


I would move this test to be after the split-auth test (so same order as in the module).

src/pip/_internal/utils/misc.py

cjerdonek · 2018-09-11T15:44:35Z

tests/unit/test_index.py

@@ -153,4 +153,4 @@ def test_get_formatted_locations_basic_auth():
 finder = PackageFinder([], index_urls, session=[])

 result = finder.get_formatted_locations()
- assert 'user' not in result and 'pass' not in result
+ assert 'user' in result and 'pass' not in result


How about also checking that “****” is in the result to check / make it more obvious that redaction should be happening?

BrownTruck · 2018-09-27T16:45:03Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

orf · 2018-10-10T09:51:46Z

Thanks for the review @cjerdonek, I've made the changes as requested.

cjerdonek

Some more comments.

cjerdonek · 2018-10-10T10:06:26Z

tests/unit/test_utils.py

+
+
+@patch('pip._internal.utils.misc.transform_url')
+def test_redact_password_from_url(mocked_transform_url):


It seems like trying to mock here just obscures the test.

cjerdonek · 2018-10-10T10:07:23Z

tests/unit/test_utils.py

+
+@patch('pip._internal.utils.misc.transform_url')
+def test_redact_password_from_url(mocked_transform_url):
+ redact_password_from_url('user@example.com')


"user@example.com" isn't a full URL. I would test with an actual URL.

cjerdonek · 2018-10-10T10:07:48Z

tests/unit/test_utils.py

@@ -373,8 +374,6 @@ class Test_normalize_path(object):
 # it's easiest just to skip this test on Windows altogether.
 @pytest.mark.skipif("sys.platform == 'win32'")
 def test_resolve_symlinks(self, tmpdir):
- print(type(tmpdir))
- print(dir(tmpdir))


This isn't related to this PR, so I would leave it alone.

cjerdonek · 2018-10-10T10:08:17Z

src/pip/_internal/utils/misc.py

@@ -907,6 +911,18 @@ def remove_auth_from_url(url):
 return surl


+def redact_password_from_url(url):
+ """Replace the password in a given url with ****"""


You need a period at the end of the sentence.

cjerdonek · 2018-10-10T10:08:51Z

src/pip/_internal/utils/misc.py

@@ -907,6 +911,18 @@ def remove_auth_from_url(url):
 return surl


+def redact_password_from_url(url):


I would move this function to be after remove_auth_from_url() (since it is a "fancier" version).

cjerdonek · 2018-10-10T10:09:13Z

src/pip/_internal/utils/misc.py

- # username/pass params are passed to subversion through flags
- # and are not recognized in the url.
+def redact_netloc(netloc):
+ netloc, (user, passw) = split_auth_from_netloc(netloc)


I would just write out "password" since it's not that much longer.

cjerdonek

Some more.

cjerdonek · 2018-10-10T10:20:35Z

src/pip/_internal/utils/misc.py

+ if user is None:
+ return netloc
+ password = '' if password is None else ':****'
+ return '{user}{passw}@{netloc}'.format(user=user,


"passw" -> "password"

cjerdonek · 2018-10-10T10:22:28Z

tests/unit/test_utils.py

+
+
+def test_redact_password_from_url():
+ with patch('pip._internal.utils.misc.transform_url') as mocked_transform_url:


I was saying before that I think mocking obscures the test. Why can't you just pass in a few real URL's like the other tests (e.g. one with a user-pass, one with no user-pass, and one with a user but no pass).

Yeah good idea, I dislike mocks as well. Done!

cjerdonek · 2018-10-10T10:35:53Z

Looking a lot better. Thanks!

cjerdonek

Some final(?) minor comments.

src/pip/_internal/index.py

cjerdonek · 2018-10-11T03:24:03Z

news/4746.bugfix

@@ -0,0 +1 @@
+Redact password from the URL in various log messages.


"Redact the password"

src/pip/_internal/utils/misc.py

cjerdonek · 2018-10-11T03:28:01Z

src/pip/_internal/utils/misc.py


+def transform_url(url, transform_netloc):


I would mark this private: _transform_url.

cjerdonek · 2018-10-11T03:29:01Z

src/pip/_internal/utils/misc.py

+ # Return a copy of url with 'username:password@' removed.
+ # username/pass params are passed to subversion through flags
+ # and are not recognized in the url.
+ return transform_url(url, lambda netloc: split_auth_from_netloc(netloc)[0])


I would define the lambda on its own line:
transform_netloc = lambda netloc: split_auth_from_netloc(netloc)[0]

The linter dislikes this, and I also try not to do this. I think it's fine as it is?

What does the linter not like? I’m not a fan of cramming a lot into one line. You can also define the function using def syntax.

Lambdas are supposed to be anonymous - if you bind them to a name it kind of defeats the point. The linter points this out and tells you to use def.

I'll do whatever you suggest (I don't care that much), but a lambda seems perfect here. A whole named function for this seems overkill.

I just think the code will be easier to understand if you give the function a name (whether that be a lambda or def syntax). It will also have a more parallel / symmetric look with redact_password_from_url below.

cjerdonek · 2018-10-11T03:29:51Z

tests/unit/test_utils.py

+ ('user:pass:word@example.com', 'user:****@example.com'),
+])
+def test_redact_netloc(netloc, expected):
+ result = redact_netloc(netloc)


"result" -> "actual" to match the earlier test.

cjerdonek · 2018-10-11T03:31:08Z

tests/unit/test_utils.py

+ ('https://example.com', 'https://example.com')
+])
+def test_redact_password_from_url(auth_url, expected_url):
+ result = redact_password_from_url(auth_url)


"result" -> "actual" (or "url") to match previous tests.

cjerdonek · 2018-10-11T03:34:50Z

tests/unit/test_utils.py

@@ -681,3 +701,14 @@ def test_split_auth_from_netloc(netloc, expected):
 def test_remove_auth_from_url(auth_url, expected_url):
 url = remove_auth_from_url(auth_url)
 assert url == expected_url
+
+
+@pytest.mark.parametrize('auth_url,expected_url', [


Add space to match previous test: auth_url, expected_url

cjerdonek · 2018-10-19T08:12:38Z

Closing / opening to trigger CI to re-run.

cjerdonek · 2018-10-19T09:08:42Z

Thanks a lot for all your work and follow-through on this, @orf.

orf · 2018-10-19T10:15:01Z

No problem, thank you for the prompt and detailed reviews @cjerdonek 🎉

haizaar · 2018-11-05T03:35:00Z

Does anyone know when this fix gets released?

benoit-pierre · 2018-11-05T03:41:30Z

@haizaar : https://github.com/pypa/pip/blob/master/docs/html/development/release-process.rst#release-cadence

pradyunsg · 2018-11-06T09:27:14Z

It'll be a part of 19.0, scheduled early next year.

pip has a predictable release cadence, which @benoit-pierre linked to.

lock · 2019-05-31T04:21:14Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

orf mentioned this pull request Sep 10, 2018

Redact password from various log messages #5590

Closed

cjerdonek suggested changes Sep 11, 2018

View reviewed changes

cjerdonek reviewed Sep 11, 2018

View reviewed changes

src/pip/_internal/utils/misc.py Outdated Show resolved Hide resolved

cjerdonek reviewed Sep 11, 2018

View reviewed changes

BrownTruck added the needs rebase or merge PR has conflicts with current master label Sep 27, 2018

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Oct 10, 2018

cjerdonek suggested changes Oct 10, 2018

View reviewed changes

cjerdonek added this to the Print Better Error Messages milestone Oct 10, 2018

cjerdonek suggested changes Oct 11, 2018

View reviewed changes

Tom Forbes and others added 10 commits October 17, 2018 20:56

Redact basic authentication passwords from URLs

18b9c0b

Review comments

ba906c7

Review fixes

679a655

Linting fixes

061c07e

Review fixes

c8e25b5

Review fixes 2, electric boogaloo

f4eab41

Remove whitespace to fix tests.

d4f459a

Review fixes

ea7ebb9

Linting fixes

6070faf

Split lambda into a named function

d2bd248

cjerdonek closed this Oct 19, 2018

cjerdonek reopened this Oct 19, 2018

cjerdonek merged commit 78371cc into pypa:master Oct 19, 2018

orf deleted the redact-auth branch October 19, 2018 10:11

This was referenced Oct 25, 2018

Hide passwords in the output #5917

Closed

Support credentials in configuration for private PyPI #4315

Closed

uranusjr mentioned this pull request Nov 4, 2018

Suppressing expansion of environment variables pypa/pipenv#3138

Closed

pradyunsg modified the milestones: Print Better Error Messages, 19.0 Jan 3, 2019

atugushev mentioned this pull request May 3, 2019

pip-compile replaces password in URL with **** jazzband/pip-tools#801

Closed

anlutro mentioned this pull request May 9, 2019

redact passwords in output when using --find-links #6489

Merged

lock bot added the auto-locked Outdated issues that have been locked by automation label May 31, 2019

lock bot locked as resolved and limited conversation to collaborators May 31, 2019



		@patch('pip._internal.utils.misc.transform_url')
		def test_redact_password_from_url(mocked_transform_url):

		@@ -907,6 +911,18 @@ def remove_auth_from_url(url):
		return surl


		def redact_password_from_url(url):



		def test_redact_password_from_url():
		with patch('pip._internal.utils.misc.transform_url') as mocked_transform_url:

		@@ -0,0 +1 @@
		Redact password from the URL in various log messages.

Redact basic authentication passwords from log messages #5773

Redact basic authentication passwords from log messages #5773

Conversation

orf commented Sep 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orf commented Sep 11, 2018

Choose a reason for hiding this comment

cjerdonek commented Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BrownTruck commented Sep 27, 2018

orf commented Oct 10, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek commented Oct 10, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Oct 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek commented Oct 19, 2018

cjerdonek commented Oct 19, 2018

orf commented Oct 19, 2018

haizaar commented Nov 5, 2018

benoit-pierre commented Nov 5, 2018

pradyunsg commented Nov 6, 2018

lock bot commented May 31, 2019

cjerdonek commented Sep 11, 2018 •

edited

Loading

cjerdonek Oct 11, 2018 •

edited

Loading