Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip freeze does case sensitive comparsion #5716

Closed
abs51295 opened this issue Aug 19, 2018 · 15 comments · Fixed by #6857
Closed

pip freeze does case sensitive comparsion #5716

abs51295 opened this issue Aug 19, 2018 · 15 comments · Fixed by #6857
Labels
auto-locked Outdated issues that have been locked by automation C: freeze 'pip freeze' related state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior

Comments

@abs51295
Copy link
Contributor

Environment

  • pip version: 18.0
  • Python version: 3.6
  • OS: Mac OS X

Description

Running pip freeze -l -r requirements.txt does a case sensitive comparison with dependencies mentioned in requirements.txt.

Expected behavior
Running pip freeze -l -r requirements.txt should do case insensitive comparison with dependencies mentioned in requirements.txt or at least allow that via a command line option.

How to Reproduce

  1. Create a virtualenv.
  2. Create a requirements.txt file.
  3. Run pip install -r requirements.txt.
  4. Run pip freeze -l -r requirements.txt.

Output

#
# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --output-file requirements.txt requirements.in
asn1crypto==0.24.0
attrs==18.1.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
colorama==0.3.9
coverage==4.5.1
cryptography==2.2.2
flake8-polyfill==1.0.2
flake8==3.5.0
Requirement file [requirements.txt] contains flask-cors==3.0.4, but that package is not installed
Requirement file [requirements.txt] contains flask==1.0.2, but that package is not installed
gevent==1.3.1
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
itsdangerous==0.24
Requirement file [requirements.txt] contains jinja2==2.10, but that package is not installed
mando==0.6.4
Requirement file [requirements.txt] contains markupsafe==1.0, but that package is not installed
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psycopg2==2.7.4
py==1.5.3
pycodestyle==2.3.1
pycparser==2.18
pycrypto==2.6.1
pyflakes==1.6.0
Requirement file [requirements.txt] contains pyjwt==1.6.3, but that package is not installed
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest==3.5.1
radon==2.2.0
requests-futures==0.9.7
requests==2.18.4
semantic-version==2.6.0
six==1.11.0
Requirement file [requirements.txt] contains sqlalchemy==1.2.7, but that package is not installed
urllib3==1.22
Requirement file [requirements.txt] contains werkzeug==0.14.1, but that package is not installed
## The following requirements were added by pip freeze:
Flask==1.0.2
Flask-Cors==3.0.4
Jinja2==2.10
MarkupSafe==1.0
PyJWT==1.6.3
SQLAlchemy==1.2.7
Werkzeug==0.14.1
@cjerdonek cjerdonek added the C: freeze 'pip freeze' related label Aug 19, 2018
@abs51295
Copy link
Contributor Author

@cjerdonek I will like to work on this issue. Let me know, when we have classified this as a bug or a feature request.

@cjerdonek cjerdonek added the S: needs triage Issues/PRs that need to be triaged label Aug 19, 2018
@pradyunsg pradyunsg added type: bug A confirmed bug or unintended behavior and removed S: needs triage Issues/PRs that need to be triaged labels Aug 20, 2018
@pradyunsg
Copy link
Member

Hey @abs51295!

I'd consider this a bug since this is essentially a case where pip is not normalizing the given package names correctly.

@abs51295
Copy link
Contributor Author

Thanks @pradyunsg I will send a fix soon! 😄

@pradyunsg pradyunsg added the state: awaiting PR Feature discussed, PR is needed label Aug 20, 2018
@abs51295
Copy link
Contributor Author

@pradyunsg how should the fix look like? Should we change the dependency names to lower case explicitly?

@pradyunsg
Copy link
Member

There's a normalize_name or a canonical_name (or similarly named) function in pip's codebase, which should be used to normalize names.

(sorry I missed you on IRC - I slept off with the client open)

@abs51295
Copy link
Contributor Author

Thanks @pradyunsg will look into this.

( I don't see you on #pypa-dev anymore on IRC)

@abs51295
Copy link
Contributor Author

@pradyunsg
Copy link
Member

Hey @abs51295! I was away from a PC for most of the time over the weekend.

I've not looked carefully but you'd want to start looking here: https://github.com/pypa/pip/blob/b13cee6388adea5ddaf1ddb982beeec37d07ffda/src/pip/_internal/commands/freeze.py

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name.

I found this: /src/pip/_vendor/packaging/requirements.py@master#L86-L89.

That's a vendored library (packaging) and while I do think that change needs discussion, the change in pip doesn't and they're fairly separate.

@abs51295
Copy link
Contributor Author

abs51295 commented Sep 4, 2018

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name

@pradyunsg I did that, but why I was referring to the https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89, is because of the following:

  1. Canonicalizing the name when pip's getting the information, which can be achieved by canonicalizing the name here, solves only half the problem.

  2. The other half of the problem lies when the requirement line is parsed from requirements.txt file and compared with existing installations as obtained in step 1.

  3. During the above comparsion, if we don't canonicalize the name obtained from the requirements.txt file, then we will again run into the same problem of case sensitive comparison. To mitigate that, we need to canonicalize the name of this requirement which leads to this and hence the comment.

@abs51295
Copy link
Contributor Author

@pradyunsg any updates on this?

@pradyunsg
Copy link
Member

Sorry, I haven't taken a deeper look yet. Real life has kept me more than busy lately.

I'll try to take a look today or this weekend. Feel free to ping me again in that time.

@abs51295
Copy link
Contributor Author

Thanks. Will wait for your reply 🙂

@abs51295
Copy link
Contributor Author

abs51295 commented Oct 4, 2018

@pradyunsg any updates?

@Chetabahana
Copy link

Chetabahana commented May 6, 2019

any updates? hope it badly to regenerate requirements.txt
all the warned packages were actually been added by pip already

(virtual-env)$ pip --version
pip 19.1 from /env/lib/python3.7/site-packages/pip (python 3.7)

(virtual-env)$ pip install -r requirements.txt
... done
Successfully installed all

(virtual-env)$ pip install django-storages[google]
... done
Successfully installed 
cachetools-3.1.0 google-api-core-1.9.0 google-auth-1.6.3 google-cloud-core-0.29.1 
google-cloud-storage-1.15.0 google-resumable-media-0.3.2 googleapis-common-protos-1.5.9 
protobuf-3.7.1 pyasn1-0.4.5 pyasn1-modules-0.2.5 rsa-4.0

(virtual-env)$ pip freeze -r requirements.txt
-i https://pypi.python.org/simple
amqp==2.4.2
aniso8601==3.0.2
autopep8==1.4.4
WARNING: Requirement file [requirements.txt] contains babel==2.6.0, but package 'babel' is not installed
billiard==3.6.0.0
bleach==2.1.4
boto3==1.9.138
botocore==1.12.138
braintree==3.49.0
cairocffi==1.0.2
WARNING: Requirement file [requirements.txt] contains cairosvg==2.3.1, but package 'cairosvg' is not installed
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
cssselect2==0.2.1
defusedxml==0.6.0
dj-database-url==0.5.0
dj-email-url==0.2.0
django-appconf==1.0.3
django-babel==0.6.2
django-bootstrap4==0.0.8
django-cache-url==3.0.0
django-celery-results==1.0.4
django-countries==5.3.3
django-debug-toolbar-request-history==0.0.10
django-debug-toolbar==1.11
django-elasticsearch-dsl==0.5.1
django-filter==2.1.0
django-graphql-jwt==0.2.1
django-impersonate==1.4
django-js-asset==1.2.2
django-measurement==3.0.0
django-mptt==0.10.0
django-phonenumber-field==2.3.1
django-prices-openexchangerates==1.0.1
django-prices-vatlayer==1.0.2
django-prices==1.0.4
django-recaptcha==1.5.0
django-redis==4.10.0
django-render-block==0.5
django-silk==2.0.0
django-storages==1.7.1
django-templated-email==2.3.0
django-versatileimagefield==1.10
django-webpack-loader==0.6.0
WARNING: Requirement file [requirements.txt] contains django==2.2, but package 'django' is not installed
docutils==0.14
elasticsearch-dsl==6.0.1
elasticsearch==6.3.1
WARNING: Requirement file [requirements.txt] contains faker==1.0.5, but package 'faker' is not installed
freezegun==0.3.11
google-i18n-address==2.3.4
google-measurement-protocol==1.0.0
gprof2dot==2016.10.13
graphene-django-optimizer==0.4.0
graphene-django==2.2.0
graphene==2.1.3
graphql-core==2.1
graphql-relay==0.4.5
gunicorn==19.9.0
html5lib==1.0.1
idna==2.8
ipaddress==1.0.22
WARNING: Requirement file [requirements.txt] contains jinja2==2.10.1, but package 'jinja2' is not installed
jmespath==0.9.4
jsonfield==2.0.2
kombu==4.5.0
WARNING: Requirement file [requirements.txt] contains markdown==3.1, but package 'markdown' is not installed
WARNING: Requirement file [requirements.txt] contains markupsafe==1.1.1, but package 'markupsafe' is not installed
maxminddb-geolite2==2018.703
maxminddb==1.4.1
measurement==2.0.1
mpmath==1.1.0
oauthlib==3.0.1
phonenumberslite==8.10.10
WARNING: Requirement file [requirements.txt] contains pillow==5.4.1, but package 'pillow' is not installed
prices==1.0.0
promise==2.2.1
psycopg2-binary==2.8.2
purl==1.5
pycodestyle==2.5.0
pycparser==2.19
WARNING: Requirement file [requirements.txt] contains pygments==2.3.1, but package 'pygments' is not installed
WARNING: Requirement file [requirements.txt] contains pyjwt==1.7.1, but package 'pyjwt' is not installed
WARNING: Requirement file [requirements.txt] contains pyphen==0.9.5, but package 'pyphen' is not installed
python-dateutil==2.8.0
python3-openid==3.1.0
pytz==2019.1
raven==6.9.0
razorpay==1.2.0
redis==3.2.1
requests-oauthlib==1.2.0
requests==2.21.0
WARNING: Requirement file [requirements.txt] contains rx==1.6.1, but package 'rx' is not installed
s3transfer==0.2.0
singledispatch==3.4.0.3
six==1.12.0
social-auth-app-django==3.1.0
social-auth-core==3.1.0
sqlparse==0.3.0
stripe==2.27.0
sympy==1.4
text-unidecode==1.2
tinycss2==1.0.2
typing==3.6.6
urllib3==1.24.2
WARNING: Requirement file [requirements.txt] contains uwsgi==2.0.18 ; platform_system != 'Windows', but package 'uwsgi' is not installed
vine==1.3.0
WARNING: Requirement file [requirements.txt] contains weasyprint==47, but package 'weasyprint' is not installed
webencodings==0.5.1

The following requirements were added by pip freeze:

astroid==2.2.5
Babel==2.6.0
cachetools==3.1.0
CairoSVG==2.3.1
configparser==3.7.4
Django==2.2
django-fsm==2.6.0
Faker==1.0.5
google-api-core==1.9.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-storage==1.15.0
google-resumable-media==0.3.2
googleapis-common-protos==1.5.9
isort==4.3.17
Jinja2==2.10.1
lazy-object-proxy==1.3.1
Markdown==3.1
MarkupSafe==1.1.1
mccabe==0.6.1
Pillow==5.4.1
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pyflakes==2.1.1
Pygments==2.3.1
PyJWT==1.7.1
pylint==2.3.1
Pyphen==0.9.5
rsa==4.0
Rx==1.6.1
typed-ast==1.3.5
uWSGI==2.0.18
WeasyPrint==47
wrapt==1.11.1

@abs51295
Copy link
Contributor Author

@pradyunsg @cjerdonek I have created #6857 in respect to this. Please have a look. Thanks.

abs51295 added a commit to abs51295/pip that referenced this issue Aug 11, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Aug 31, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Aug 31, 2019
abs51295 added a commit to abs51295/pip that referenced this issue Sep 2, 2019
@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Nov 17, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Nov 17, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation C: freeze 'pip freeze' related state: awaiting PR Feature discussed, PR is needed type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants