up/down arrow history doesn't work on pipenv shell on windows

[crioux]

In the cmd shell, usually you can hit 'up arrow' to get your previous command. Once you run 'pipenv shell', this no longer works. if you run some commands inside the pipenv shell and then hit 'up' it does nothing.

[crioux]

FWIW, it works in 'ConEmu', just not the normal 'Windows Console Host' process that you get by running cmd.exe directly, so whatever they're doing in there to pass along the arrow keys is what you'd need. Not sure it's worth it to implement seperately in pipenv, but at least this is some kind of a workaround.

[erinxocon]

This is an issue with pew since it spawns the subshell on windows. Can you open an issue over there? I'm a maintainer of that project too and I'll make sure to address it on that project!
https://github.com/berdario/pew/issues

[askpatrickw]

This is still not in the patched PEW in this repo or in Pew directly, but I wanted to let others who end up here that if you use Powershell instead of the command prompt it does work.

[techalchemy]

(FYI in the next release -- which will be today -- we are no longer using pew for managing subshells).

[askpatrickw]

I tried installing from the repo and after I did pipenv shell, command history still didn't work. This is how I installed:

pip install git+https://github.com/pypa/pipenv
Collecting git+https://github.com/pypa/pipenv
  Cloning https://github.com/pypa/pipenv to c:\<snip>\appdata\local\temp\pip-req-build-yv19jpwg
  Installing build dependencies ... done
Requirement already satisfied: pip>=9.0.1 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (10.0.1)
Requirement already satisfied: certifi in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (2018.4.16)
Requirement already satisfied: setuptools>=36.2.1 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (39.2.0)
Requirement already satisfied: virtualenv-clone>=0.2.5 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (0.3.0)
Requirement already satisfied: virtualenv in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (16.0.0)
Installing collected packages: pipenv
  Running setup.py install for pipenv ... done
Successfully installed pipenv-2018.7.1.dev0

I tried it with an existing virtualenv and a newly recreated one. Let me know if I should try another branch.
Thanks!

[techalchemy]

Right this is a limitation of subshells in windows. There isn’t a fix for that and nobody is actively implementing one (this would effectively mean reimplementing subshells as far as I understand). Just sharing that pew is no longer involved in this aspect

[askpatrickw]

I keep seeing that said that, but with virtualenvwrapper-win workon <env> this works fine and I think its just batch files on top of virtualenv.

Virtualenvwrapper-win workon: https://github.com/davidmarble/virtualenvwrapper-win/blob/master/scripts/workon.bat

Worth noting: virtualenvwrapper-win does not support powershell it is mentioned there are versions which do, but I'm not sure which is the best so I won't link to one.

Maybe a pipenvwrapper-win needs to appear. But that would be sad I think.

If there are suggestions on approaches to fixing, I'm happy to take a swing, I just don't know this area well enough to do without a little guidance.

Thanks!

[uranusjr]

Just want to mention there is always pipenv run. Subshell-omg is by nature difficult to handle and has many platform restrictions. run is a better model unless you have a very subshell-friendly shell (e.g. fish).

[askpatrickw]

I'm intrigued, but I don't understand.

How would you use pipenv run to have your dev or run time environment available at the windows CLI or from an IDE launched from same?

[uranusjr]

I’m afraid I don’t quite understand the use cases you mentioned :( Could you be more specific what you think is the problem with run?

[askpatrickw]

I I see your point now (tried a few more things). When you pipenv run cmd you will have your environment available to you and you can launch your IDE from there as well. (VS Code: code .) and within VS Code in terminal you will also see all your pipenv installed resources as well.

AND your command history work (up arrow).

That seems like how it should work to begin with. I wonder if there are any downsides to this approach.

[techalchemy]

I can’t speak for everyone but I very rarely use subshells myself and I know @uranusjr doesn’t either. Since we are the main devs on the project following our workflow is most likely a safe bet

[askpatrickw]

I had assumed pipenv shell was the analog to workon <env>. If pipenv run cmd is the better answer then I'm totally stoked!

Thanks!

[mungojam]

@w7pea

When you pipenv run cmd
AND your command history work (up arrow)

up arrow still doesn't work for me in this scenario. I'm using latest version 2018.7.1

[uranusjr]

Okay, now we purged Pew, we should track this on our own :p

From the discussion in pew-org/pew#165, I think the simplest way to do this right now is to perform DLL-injection. Vendor the Clink executable, and when we are about to launch cmd.exe, instead of running it directly, run Clink instead. We can probably reuse Clink’s startup BAT file, or clink.exe inject to achieve the same effect.

[uranusjr]

Semi check semi todo list:

• Clink breaking DOSKEY could be a problem. We should likely need to provide a toggle for Clink injection. Should we offer the injection as an opt-in, or opt-out, however?
• Could GPL be a problem? We will probably only vendor the binaries, so maybe not.
• Clink in binary form is about 1.5MB—quite significant—but there is not much we can shed. What do we do?
• There’s clink inject --pid to inject into an arbitrary PID. Sounds very useful for us.

[uranusjr]

Seems like there is a fork to make DOSKEY work. Cmder also uses the fork (if I read the thread correctly), so that seems like well-proofed work.

mridgers/clink#464 (comment)

[techalchemy]

We need to find a way to be clever about this. I guess we can build platform dependent wheels.

[uranusjr]

I’m thinking maybe we can perform a one-time download at runtime, to avoid vendoring altogether. Only plain cmd.exe users would need to make that download once (unless the user clears cache).

[uranusjr]

Note: The above comment is outdated. Clink released 0.4.9 which included the DOSKEY fix, and Cmder is using that instead of the fork.

https://github.com/cmderdev/cmder/blob/44aadccbecc6903918d749c693771c211ebb8fce/vendor/sources.json

[askpatrickw]

@mungojam this may no longer be relevant with the new changes down the road, but for me on Windows 10, this worked:

pip install git+https://github.com/pypa/pipenv.git
cd \someemptyfolder
pipenv install requests
pipenv run cmd

Now you're at a new prompt run a command like DIR and then hit the up arrow... bingo!
NOTE: It doesn't access the command history from before you launched the new CMD. I think that is to be expected.

[mungojam]

@w7pea
pip install git+https://github.com/pypa/pipenv.git
Unfortunately I fail at this first hurdle, my computer is just hanging when I try to install this from GitHub. I'm only able to install the latest from pypi which still exhibits the issue.

One thing I have realised is that pipenv run cmd works as a workaround when in a standard command prompt, but not within VS Code terminal using either cmd terminal or powershell.

[askpatrickw]

@mungojam Are you starting VS Code from inside the virtualenv? So if I start a clean CMD and do this...

mkdir test
cd test
pipenv install requsts
pipenv run cmd
code .
CTL+` (Open VS Code Terminal)
pip list 

You will see requests is installed because VS Code was started inside your Virtualenv.
Now open a new CMD prompt

cd test
code .
CTRL+` (Open Terminal)
pip list

You only see what you have installed in your OS, not the virtualenv.
This all worked for me from the released PyPi version.

As for the hanging install from the Git Repo. I can't reproduce that sorry.

[uranusjr]

VS Code can be configured to automatically pick up the virtual environment. You don’t even need to pipenv run cmd in it.

https://code.visualstudio.com/docs/python/environments

Also the conversation is slowly drifting away. Further discussion should probably happen elsewhere.

[mungojam]

Thanks for the tips, useful for me to be able to get it working for myself, but I look forward to when I can point my colleagues at generic pipenv guides and things work as they are used to with pipenv shell. Good luck getting it all working, sounds complex!

[asweigart]

I I see your point now (tried a few more things). When you pipenv run cmd you will have your environment available to you and you can launch your IDE from there as well. (VS Code: code .) and within VS Code in terminal you will also see all your pipenv installed resources as well.

AND your command history work (up arrow).

I am on Windows 10 Pro (10.0.17134 Build 17134) with Pipenv 2018.11.14 and the pipenv run cmd workaround does not fix the up/down arrow issue for me.

[techalchemy]

If you start by using pipenv shell in a cmd.exe shell, nothing anyone does will help with the up and down arrow history.

If however you simply use pipenve run <cmd> instead of first activating pipenv shell and then running <cmd> separately, then if you are still having problems with up and down arrow history it has nothing to do with pipenv because you didn't activate pipenv's subshell.

[clragon]

This is still a thing over here on my windows 10 machine running the latest windows 10 and the latest pipenv.

[mikedrawback]

I can access my cmd shell history using pipenv run cmd but the interactive python shell history does not work. I'm on windows 10 running pipenv 2018.11.26

[PetterS]

This happens for me with latest pipenv on Windows 10 1903. I don't recall it happening a year or so ago.

[LanetheGreat]

Hi guys, having the same issue with pipenv on my machine (Windows 10 1903/pipenv 2018.11.26) but I played around with it today and noticed some things while trying to get command history working for my personal scrapy project for when I use scrapy's shell from within a pipenv (pipenv scrapy shell "[url]"). I also included some snapshots of the process hierarchy during my testing to help explain.

I found that if instead of invoking pipenv through its .exe (pipenv shell) and did it using python's module execution (python -m pipenv shell) I'm able to use my command history correctly inside the pipenv shell, though I don't have command history for python if I invoke it from within the pipenv.

Non-working Before:

Working After:

So I did some more digging and testing and noticed that when the pipenv shell is started it's done so as "C:\Windows\System32\cmd.exe" /c "cmd.exe" with cmd.exe as a child process so I dug through the code and when I edited the function pipenv.shell._handover replacing:
sys.exit(subprocess.call(args, shell=True, universal_newlines=True))
with
sys.exit(subprocess.call(args, universal_newlines=True))
It'd start it as just "cmd.exe" without the parent cmd.exe and when I invoked python from the pipenv shell then my command history worked for python! And across multiple starts and stops from within the same shell instance! Not sure if that's the best way to invoke the shell yet, but it worked.
After code change:

So I'm guessing from my testing that if a process sits between the invoking process and the currently active child process it screws up command history, because I'd also loose my command history if I used scrapy's native scrapy.exe from Python's Scripts folder (like pipenv.exe) inside the pipenv shell so I also have to use python -m scrapy shell in order to test the shell, same would happen for pipenv run so making me have to use pipenv run python -m scrapy shell.
Non-working Scrapy from pipenv shell:

Working Scrapy from pipenv shell:

For the TL;DR:

• python -m pipenv shell worked for me (had to not use "pipenv.exe" at all for command history in the shell)
• Removing shell=True in pipenv.shell._handover alongside using:
• pipenv run python -m [library] [args] worked for me for starting sub-shells provided by python libraries. (bypasses using [pipenv folder]\Scripts\[library].exe when launching)

This just worked for me and I'm not sure if this works for anybody else yet, so if anyone else could double check and test this on their Windows systems that would be much appreciated to know. (Just remember to edit using a program started with admin privileges for editing pipenv.shells._handover if you installed python under "C:\Program Files" or "C:\Program Files (x86)")

[jackdewinter]

Given LanetheGreat's work, would it be possible to introduce an environment variable that would control whether or not the new shell was spawned with sys.exit(subprocess.call(args, shell=True, universal_newlines=True))
i.e. if the env var, say "PIPENV_WIN_SPAWN_WITH_SHELL" was anything other than false, use the above line, otherwise use sys.exit(subprocess.call(args, shell=False, universal_newlines=True))

[fireattack]

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

[MausamGaurav]

Up arrows working on my VS Code in Windows after python -m pipenv shell

[fireattack]

It still doesn't here. Tested in both Win 7 and Win 10.

It (up arrow) doesn't work in powershell, either.

[RZarifov]

Writing here to draw attention to the issue as it still not solved. Yes - I'm facing the same problem. Looking forward for possible solution(s).

[HerbinZYH]

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

it works for me, thank you

[jcf-dev]

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

worked like a charm on Pycharm terminal. I will use this workaround for now.

[LucianoBAF]

Confirmed: it doesn't work in VS Code's built-in console but works on external consoles.
I just got it to work when using cmder with Powershell.

[edonghyun]

I have a same problem in powershell and cmd even with python -m pipenv shell or python -m pipenv run powershell.

[HowManyOliversAreThere]

Interestingly for me, I can't traverse history at all when running pipenv shell in VS Code's console, when using python -m pipenv shell in VS Code's console I can traverse history using up / down, however if I enter python from within this shell, my python console then doesn't let me traverse history.

[snphan]

Not sure if anyone mentioned already, but I tried opening another cmd from the pipenv shell using "start cmd.exe" and it seemed to preserve the environment while enabling up/down arrow history...

[bblanchon]

I see this issue got closed.
Is it finally fixed?

[rocketmonkeys]

I just tried this on Win 10 from cmd.exe using pipenv 2022.5.2, doesn't seem to be fixed.

@matteius - could you share which version it this was fixed/closed in? It'd be great to start using that.

[matteius]

@rocketmonkeys Apparently it (the up and down arrows) does not work in cmd.exe when you run pipenv shell, but it does work in PowerShell as well as git shell which are both arguably better experiences than cmd.exe -- so I seldom ever use that except to triage these reports. If you want an open ticket for cmd.exe up/down arrows, you can open a new ticket, but I won't be able to fix it for cmd.exe as I am not sure what the issue is nor have time to invest into this particular issue. Feel free to investigate it and open a PR though that would improve support for cmd.exe -- but I recommend using PowerShell or Git bash for Windows.

[mevimo]

This stopped working for me even in Git bash and PowerShell.

[gregcowell]

No Python REPL history is also an issue on Windows 11 / Powershell / pipenv shell. A workaround is to activate the virtual environment created by pipenv using the Activate.ps1 script. Not very convenient but I guess okay if you only need REPL history occasionally but still want to stick with pipenv on Windows.

[rlleshi]

Doesn't work in either Powershell or git bash. In the latter nothing happens, in the former the cursor literally moves up or down the screen.

Also doesn't work in any VSCode consoles. @matteius