Pipenv lock overwrites its output in verbose mode

[colincadams]

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

When running pipenv lock --verbose, it outputs verbose messages (starting round, adding requirement, etc.) but then carriage returns and overwrites it with the next message. Because of that you can't actually see the sequence of steps that it is taking. When debugging an infinity lock issue this can be annoying.

Expected result

It should leave all of the steps in the output, or at least have an option to log them to a file.

Actual result

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv lock --dev --verbose --clear

https://github.com/Recidiviz/pulse-data/blob/01d29e570a255b78ca0c7c2ab304c1a34d1f7dde/Pipfile

You'll notice in support that I have updated sklearn to scikit-learn in the Pipfile. It happens for both.

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2022.11.25'

Pipenv location: '/Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/lib/python3.9/site-packages/pipenv'

Python location: '/Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python'

OS Name: 'posix'

User pip version: '22.3'

user Python installations found:

• 3.10.8: /usr/local/bin/python3
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python3
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python3.9
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python3
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python
• 3.9.14: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin/python3.9
• 3.9.14: /usr/local/bin/python3.9
• 3.8.14: /usr/local/bin/python3.8
• 3.8.2: /usr/bin/python3
• 3.7.14: /usr/local/opt/python@3.7/bin/python3.7m
• 3.7.14: /usr/local/opt/python@3.7/bin/python3.7
• 3.7.14: /usr/local/bin/python3.7m
• 3.7.14: /usr/local/bin/python3.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.14',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '21.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT '
                     '2022; root:xnu-8020.140.49~2/RELEASE_X86_64',
 'python_full_version': '3.9.14',
 'python_version': '3.9',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_SESSION_ID
• SSH_AUTH_SOCK
• LC_TERMINAL_VERSION
• COLORFGBG
• ITERM_PROFILE
• SQLITE_EXEMPT_PATH_FROM_VNODE_GUARDS
• XPC_FLAGS
• LANG
• PWD
• SHELL
• __CFBundleIdentifier
• SECURITYSESSIONID
• TERM_PROGRAM_VERSION
• TERM_PROGRAM
• PATH
• LC_TERMINAL
• COLORTERM
• COMMAND_MODE
• TERM
• HOME
• TMPDIR
• USER
• XPC_SERVICE_NAME
• LOGNAME
• LaunchInstanceID
• __CF_USER_TEXT_ENCODING
• ITERM_SESSION_ID
• SHLVL
• OLDPWD
• ZSH
• PAGER
• LESS
• LSCOLORS
• LC_ALL
• EDITOR
• NVM_DIR
• NVM_CD_FLAGS
• NVM_BIN
• NVM_INC
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PIPENV_ACTIVE
• VIRTUAL_ENV
• PS1
• _
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

• PIPENV_ACTIVE: 1

Debug–specific environment variables:

• PATH: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt/bin:/usr/local/opt/postgresql@13/bin:/usr/local/opt/redis@4.0/bin:/Users/colin/.nvm/versions/node/v14.15.1/bin:/Users/colin/google-cloud-sdk/bin:/Users/colin/dev/flutter/bin:/usr/local/opt/python@3.7/bin:/Users/colin/Library/Python/3.7/bin:/usr/local/bin:/usr/local/opt/postgresql@13/bin:/usr/local/opt/redis@4.0/bin:/Users/colin/.nvm/versions/node/v14.15.1/bin:/Users/colin/google-cloud-sdk/bin:/Users/colin/dev/flutter/bin:/usr/local/opt/python@3.7/bin:/Users/colin/Library/Python/3.7/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
• SHELL: /bin/zsh
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /Users/colin/code/recidiviz-data
• VIRTUAL_ENV: /Users/colin/.local/share/virtualenvs/recidiviz-data-QdL6rflt

---

Contents of Pipfile ('/Users/colin/code/recidiviz-data/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]
# TODO(#4287): Unpin once the latest version of google-api-core supports protobuf
#  version >= 3.18.0 (might be resolved by upgrading other google libraries > 2.0.0).
mypy-protobuf = "<3.0.0"
"autopep8" = "*"
mock = "*"
pytest = "*"
pylint = "*"
pytest-cov = "*"
mypy = "*"
bandit = "*"
coveralls = "*"
# This should always be pinned to the latest version officially supported by Dataflow: https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#python
apache-beam = {extras = ["gcp", "test"],version = "==2.41.0"}
fakeredis = "*"
freezegun = "*"
ipdb = "*"
jupyter = "*"
pytest-alembic = "*"
pipenv = "*"
pygit2 = "*"
PyGithub = "*"
parameterized = "*"
google-api-python-client = "*"
google-auth-httplib2 = "*"
google-auth-oauthlib = "*"
google-cloud-build = "*"
gspread = "*"
pytablewriter = "*"
# Note: Update the `black` version in `.pre-commit-config.yaml` whenever this is updated.
# If we remove the restriction, then add `pre-commit autoupdate` to the Pipenv Github action to keep them in sync.
black = {extras = ["jupyter"],version = "==22.3.0"}
pre-commit = "*"
# TODO(#4287): Unpin/remove explicit dependency once the latest version of
#  google-api-core supports protobuf version >= 3.18.0 (might be resolved by upgrading
#  other google libraries > 2.0.0). Version proto-plus==1.19.7 requires protobuf >= 3.19.0.
proto-plus = "==1.19.6"
pygtrie = "*"
rope = "*"
"ruamel.yaml" = "*"
types-beautifulsoup4 = "*"
types-dateparser = "*"
types-freezegun = "*"
types-html5lib = "*"
types-httplib2 = "*"
types-itsdangerous = "*"
types-jsonschema = "*"
types-mock = "*"
types-paramiko = "*"
types-protobuf = "*"
types-psycopg2 = "*"
types-python-dateutil = "*"
types-pytz = "*"
# TODO(#11759): Remove version pin once fakeredis supports 4.2.0+
types-redis = "<4.2.0"
types-requests = "*"
types-PyYAML = "*"
msal = "*"
types-pysftp = "*"
Faker = "*"
# TODO(https://github.com/getsentry/responses/issues/556): Unpin this version once the
#  responses package ships a version that doesn't break mypy.
responses = "==0.20.0"
shellcheck-py = "*"

[packages]
callee = "*"
"html5lib" = "*"
numpy = "*"
tabula-py = "==2.2.0"
us = "*"
dateparser = "*"
pandas = "*"
more-itertools = "*"
lxml = "*"
opencensus = ">=0.7.0"
opencensus-context = "*"
opencensus-correlation = "*"
# TODO(census-instrumentation/opencensus-python#1072): Move to current version once this
# is merged.
# Context: Opencensus arbitrarily limits us to less than Flask 2, which now is quite old
# and was holding back a lot of dependencies. An open PR shows that all of their tests
# pass on Flask 2, so we include their package with that PR for now to unblock our
# upgrade.
opencensus-ext-flask = {ref = "93dbd9043d6a367ab9866dd8ae28aebcc45baf2d",git = "https://github.com/census-instrumentation/opencensus-python.git", editable = true, subdirectory = "contrib/opencensus-ext-flask"}
opencensus-ext-google-cloud-clientlibs = "*"
opencensus-ext-grpc = "*"
opencensus-ext-requests = "*"
# TODO(#9593): Version 0.1.3 introduced a constraint on sqlalchemy requiring versions
#  sqlalchemy<1.3.24, however we use features from sqlalchemy>=1.4, so cannot downgrade.
opencensus-ext-sqlalchemy = "<0.1.3"
opencensus-ext-stackdriver = "*"
# TODO(#9157): Unpin alembic -- blocked by sqlalchemy/alembic#900 and sqlalchemy/alembic#903
alembic = "==1.6.5"
sqlalchemy = ">=1.4.25"
flask = ">=2.0.0"
Flask-Limiter = "*"
Flask-SQLAlchemy-Session = "*"
Flask-WTF = "*"
iteration-utilities = "*"
"pypdf2" = "*"
pyyaml = "*"
# TODO(#4287): Upgrade google-cloud-* to 2.0.0 now that we have removed apache-airflow from our Pipfile
google-cloud-datastore = "<2.0.0"
google-cloud-tasks = "<2.0.0"
google-cloud-pubsub = "*"
google-cloud-monitoring = "*"
pyjwt = "*"
cssselect = "*"
grpcio = "*"
gunicorn = "*"
gevent = ">=20.5.1"
google-cloud-logging = "<2.0.0"
google-cloud-trace = "<2.0.0"
wrapt = "*"
attrs = "*"
requests = "*"
pytz = "*"
# TODO(#9732): Upgrade to 4.0 introduces dateparsing errors to ingest controller tests
tzlocal = "<4.0"
python-dateutil = "*"
google-api-core = "*"
# If the protobuf version changes be sure to update the pinned version in dataflow_setup.py
protobuf = "*"
"psycopg2" = "*"
cryptography = "*"
# TODO(#4287): Versions >= 3.0.0 create namespace conflicts with other cloud libraries
#  (and resulting mypy errors). We should revisit upgrade when we've updated all
#  google-cloud* libraries to >= 2.0.0.
google-cloud-bigquery = "<3.0.0"
google-cloud-bigquery-datatransfer = "<3.0.0"
google-api-python-client = "*"
google-auth = "*"
deepdiff = "*"
"mmh3" = "*"
aenum = "*"
yq = "*"
oauth2client = "*" #TODO(#2922): Remove deprecated oauth2client dependency
google-cloud-storage = "<2.0.0"
google-cloud-firestore = "<2.0.0"
progress="*"
google-cloud-secret-manager = "<2.0.0"
pandas-gbq = "*"
google-cloud-bigquery-storage = "*"
guppy3 = "*"
sendgrid = "*"
zope = {event = "*"}
psutil = "*"
psycogreen = "*"
marshmallow = "*"
marshmallow_enum = "*"
cattrs = "*"
# If the dill version changes be sure to update the pinned version in dataflow_setup.py
dill = "==0.3.1.1"
matplotlib = "*"
pandas_gbq = "*"
scipy = "*"
seaborn = "*"
scikit-learn = "*"
statsmodels = "*"
openpyxl = "*"
analytics-python = "*"
astroid = "*"
# TODO(#11759): Remove version pin once fakeredis supports 4.2.0+
redis = "<4.2.0"
pyarrow = "*"
auth0-python = "*"
sentry-sdk = {extras = ["flask"], version = "*"}
blinker = "*"
beautifulsoup4 = "*"
paramiko = "*"
tqdm = "*"
linearmodels = "*"
jsonschema = "*"
# TODO(#4287): Unpin/remove explicit dependency once the latest version of
#  google-api-core supports protobuf version >= 3.18.0 (might be resolved by upgrading
#  other google libraries > 2.0.0). Version proto-plus==1.19.7 requires protobuf >= 3.19.0.
proto-plus = "==1.19.6"
nltk = "*"
thefuzz = "*"
# Needed for thefuzz to avoid "Using slow pure-python SequenceMatcher" warning
python-Levenshtein = "*"
ratelimit = "*"
ratelimit-stubs = "*"
# TODO(flask-sqlalchemy-session#14): Unpin/remove explicit dependency once the latest
# version of Flask-SQLAlchemy-Session stops relying on __ident_func__ from Locals
werkzeug = "<2.1.0"
pysftp = "*"
nbconvert = ">=6.5.1"

[requires]
# If you change this, please update https://github.com/Recidiviz/recidiviz-research at the same time!
python_version = "3.9"

[scripts] 
# Run via `pipenv run <name>`
pylint = "./recidiviz/tools/lint/run_pylint.sh"
docker-build = "docker build . -t us.gcr.io/recidiviz-staging/appengine/default:latest"
docker-build-dev = "docker build . -t us.gcr.io/recidiviz-staging/appengine/default:latest --build-arg DEV_MODE=True"
cloudsql = "./recidiviz/tools/postgres/access_cloudsql_instance.sh"
docker-admin = "docker-compose -f docker-compose.yaml -f docker-compose.admin-panel.yaml up"

# Specific to Justice Counts team
# For `docker-build-jc` commands, append build args FRONTEND_APP and FRONTEND_URL at the end
# e.g. `docker-build-jc --build-arg FRONTEND_URL=https://github.com/Recidiviz/justice-counts/archive/main.tar.gz`
#      `docker-build-jc --build-arg FRONTEND_URL=https://github.com/Recidiviz/justice-counts/archive/refs/tags/v1.0.0.tar.gz`
docker-build-jc-publisher = "docker build . -f Dockerfile.justice-counts -t us.gcr.io/recidiviz-staging/justice-counts/publisher:latest --platform=linux/amd64 --build-arg FRONTEND_APP=publisher"
docker-build-jc-dashboard = "docker build . -f Dockerfile.justice-counts -t us.gcr.io/recidiviz-staging/justice-counts/agency-dashboard:latest --platform=linux/amd64 --build-arg FRONTEND_APP=agency-dashboard"
docker-jc = "docker-compose -f docker-compose.yaml -f docker-compose.justice-counts.yaml up"
fixtures-jc = "docker exec recidiviz-data_control_panel_backend_1 pipenv run python -m recidiviz.tools.justice_counts.control_panel.load_fixtures"
migrate-jc = "python -m recidiviz.tools.migrations.autogenerate_migration --database JUSTICE_COUNTS --message "

# Scripts for working with the BigQuery emulator
# Run this script once to download the emulator and again if you want to update the version.
pull-bq-emulator="docker pull ghcr.io/goccy/bigquery-emulator:latest"
# Run this script to launch the BQ emulator locally
start-bq-emulator="docker run -p 9050:9050 --rm ghcr.io/goccy/bigquery-emulator:latest /bin/bigquery-emulator --project=recidiviz-bq-emulator-project --log-level=info"

Contents of Pipfile.lock ('/Users/colin/code/recidiviz-data/Pipfile.lock'):

See lock file here: https://github.com/Recidiviz/pulse-data/blob/01d29e570a255b78ca0c7c2ab304c1a34d1f7dde/Pipfile.lock
(Github was mad it was so big)

</details>

[matteius]

@oz123 I believe this started with the change to rich.

[m3d]

Note, that this is surely bug as it is impossible to debug cases when the lock fails (using latest 2022.12.19)

[erdnaxeli]

This makes it impossible to debug anything with the verbose mode :(

The first version to introduce this bug is 2022.11.23.

[matteius]

I was chatting with Oz and we know what the fix should be -- hopefully a release end of this month or beginning of next to address.