Using psh magics

pshnb adds %psh and %%psh functions to Jupyter and IPython, which execute expressions in a persistent shell.

Installation

Install pshnb with:

pip install pshnb

Once that’s complete, you can install the magics to all IPython and Jupyter sessions automatically by running in your terminal:

pshnb_install

What’s the point?

In jupyter and ipython, you can run a shell command using the ! prefix:

!pwd

/Users/jhoward/Documents/GitHub/pshnb

However, each time you run this command, a new shell is created and then removed, so for instance, cd doesn’t actually do anything if you run another command afterwards:

!cd ..
!pwd

/Users/jhoward/Documents/GitHub/pshnb

As you see from the !pwd output, the directory hasn’t actually changed!

%psh, on the other hand, creates a persistent shell, which solves this problem:

%psh pwd

/Users/jhoward/Documents/GitHub/pshnb

%psh cd ..
%psh pwd

/Users/jhoward/Documents/GitHub

With %psh, you can implement, and document in notebooks, multi-step stateful shell interactions, including setting environment variables, sourcing scripts, and changing directories.

Features

Cell magic

You can use the %%psh cell magic to run multi-line shell commands, such as here-docs. For instance:

%%psh
cat > tmp << EOF
hi
there
EOF

This creates a file called tmp containing two lines. Let’s check it worked, and then remove it – as you see, you can also use the cell magic to run multiple commands:

%%psh
cat tmp
rm tmp

hi
there

Variable expansion

You can pipe commands together just like in a regular shell, and use standard unix utilities like head to process the output. For instance, here we show just the first 3 lines of the directory listing:

%psh ls | head -3

You can use Python variables in your shell commands by prefixing them with @{}. For instance, here we create a variable n and then display it using echo:

n = 2

%psh echo @{n}

2

Here we use n to show just the first two entries from the directory listing:

%psh ls | head -@{n}

ContextKit
FastHTML-Gallery

Background tasks

You can run commands in the background in the shell by adding & at the end of a command. The parentheses (...) group commands together to run as one unit. In this example, we first print “starting”, and then create a background process that will wait for 1 second (using sleep 1) and then print “finished”. The shell immediately shows us “starting” and tells us it created background process number 1 (with a process ID):

%%psh
echo starting
(sleep 1; echo finished) &

starting
[1] 99418

For this demonstration, we wait for 1.1 seconds (slightly longer than the background process needs). During this time, the background process will complete in the background. But we won’t see any output from it yet.

from time import sleep
sleep(1.1)

The next time we run any psh magic we will also see any output that has occurred in our persistent shell since the last command. Run %psh by itself to only see those updates, e.g here we see that “finished” was printed, and the shell tells us that background job 1 completed successfully.

%psh

finished

[1]+  Done                    ( sleep 1; echo finished )

Flags

You can get help on the %psh magic’s options using -h.

%psh -h

::

  %psh [-h] [-r] [-o] [-x] [-X] [-s] [-S] [-t TIMEOUT] [command ...]

Run line or cell in persistent shell

positional arguments:
  command               The command to run

options:
  -h, --help            Show this help
  -r, --reset           Reset the shell interpreter
  -o, --obj             Return this magic object
  -x, --expand          Enable variable expansion
  -X, --no-expand       Disable variable expansion
  -s, --sudo            Enable sudo
  -S, --no-sudo         Disable sudo
  -t TIMEOUT, --timeout TIMEOUT
                        Set timeout in seconds

You can reset the shell to its initial state using the -r flag. Let’s first check our current directory:

%psh pwd

/Users/jhoward/Documents/GitHub

Now let’s reset the shell:

%psh -r

As you can see, after resetting we’re back in our starting directory:

%psh pwd

/Users/jhoward/Documents/GitHub/pshnb

The -s flag enables sudo mode, which runs commands as the root user, and -S disables it. For instance, here we first enable sudo mode:

%psh -s

Then we can check which user we’re running as:

%psh whoami

root

As you can see, we’re now running as root. We can disable sudo mode:

%psh -S

And when we check again, we’re back to our regular user:

%psh whoami

jhoward

You can set a timeout (in seconds) using the -t flag, which will raise a TIMEOUT exception if a command takes too long. For instance, here we set a 1-second timeout:

%psh -t 1

Then we try to run a command that sleeps for 2 seconds – since this is longer than our timeout, we’ll get a timeout error:

try: get_ipython().run_line_magic('psh', 'sleep 2')
except TIMEOUT: print("timed out")