New shell beginner lessons

[gvwilson]

Work in progress: this PR brings in a half day lesson on the Unix shell aimed at people who've never seen it before. What it doesn't bring in is:

• diagrams
• styling
• a single make-style command to build the pretty versions of things
• metadata (e.g., instructor's notes and summaries for students)

Please comment on the content while we work on those things.

[ptone]

… command and then hits the enter/return key,

(helps introduce the notion of statements)

[ptone]

This is solid stuff. Is there something about your editor that gets in the way of more sane re-wrapping of lines?

In vim I just highlight blocks I've edited and then type 'gq' to reflow - there has to be something similar in other editors.

There is NO good convention out there, but I wonder if it is worth clarifying the difference between the command:

pwd

and

ls | wc -l

In the sense that pwd is actually a small program you can run (proven to students by doing which pwd

and ls | wc -l is a "command" that strings together several commands/programs.

I'm not aware of any terminology for talking about this distinction of compound command statements out there.

Do you think it would be possible to create simple mock versions of goostat etc as programs, and then actually create and distribute Nelle's Pipeline environment as a zip file that you could work through this exercise with?

[stevenkoenig]

s/that/than/
IMHO

[wking]

On Mon, Nov 11, 2013 at 09:55:49AM -0800, Steven Koenig wrote:

+(so-called because it's derived from a shell written by Stephen Bourne—this

I haven't looked at the patches directly (just following the
comments), but for Markdown in order of decreasing preference:

1. UTF-8 characters (U+2014 EM DASH).
2. ASCII equivalent (---), ideally with some SmartyPants 1 rendering
   downstream to convert to HTML entities for webpage output.
3. Source HTML entities (—).

I'm not sure about declaring encodings for Markdown source if we can't
rely on UTF-8 $LANGs everywhere, but including HTML entities directly
just sounds like a way to confuse students reading the source
directly. If we must use HTML entities in the source (for some reason
I don't understand), we should have a way to convert them to UTF-8 or
ASCII equivalents before we pass these notes out to students.

[gvwilson]

On 2013-11-11 12:51 PM, Preston Holmes wrote:

This is solid stuff. Is there something about your editor that gets in
the way of more sane re-wrapping of lines?

I break on phrase boundaries, since that reduces "to to" style errors
when moving text around.

[stevenkoenig]

As far as I can see this is the first time we are passing arguments to a programme. From my experience when learning the shell, the rules you need to follow for passing arguments can be quite difficult to comprehend. Most important here is that you need at least one space between "ls" and "-F". When I showed people arguments for the first time, I made this explicit by showing them which error they get when trying to run "ls-F". With arguments that can have arguments (--max-count=4) and escaping it quickly gets way more complex, so it is important to make sure that people understand this very basic requirement for passing simple arguments to a programme.

[DamienIrving]

I found the summaries in the v4 shell content pretty useful, so I've collected them up into a single .md file: https://github.com/DamienIrving/teaching/blob/master/swc-shell/summary.md

[lexnederbragt]

@DamienIrving first suggestion: that was what I was going to suggest too. I teach shell using a small set of files and explain students how to make a script that loops through them summarising some of the input. See https://github.com/swcarpentry/boot-camps/tree/2013-07-oslo/shell/data, which is based on the gapminderDataFiveYear.txt file at https://github.com/swcarpentry/bc/tree/gh-pages/lessons/misc-r/data.

[lexnederbragt]

@DamienIrving second suggestion: YES! Perhaps reformat like this? https://github.com/swcarpentry/2013-10-09-canberra/tree/master/05-shell

[ahmadia]

@stevenkoenig @ptone - Awesome job 🤘 Feel free to add any summarizing thoughts about how the lesson flowed, it's pacing, or content in the contents here :)

[stevenkoenig]

Not sure how well this fits here, but IMHO this should be included in beginner material.

During the Weihenstephan bootcamp this January I noticed my neighbour having severe problems with one thing:
Default language on his computer was Arabic. Every time he opened Bash (in Cygwin) he had to change the language for that window to en_US. The keyboard layout printed on his laptop was German. That is quite different, especially with regards to special characters, compared to the US American layout.

So, the cognitive burden was enormous: not only did he have to recognise the character used, he also had to try and find this one on his keyboard. But on his keyboard, the different layout gave wrong hints about which key to press... When I noticed what was going on, I fixed it by setting the keyboard layout to German. Still, I think he was pretty much lost.

The bottom line is that IMHO for beginners we should tell them which key(s) to use on their keyboard. And not only that, we also need to make sure that they are able to identify the correct symbol: it's a huge difference if you use ' or ` and the difference can be pretty hard to see using projectors. We need to make them aware of this: while a human can identify a misspelt wôrd, shells/interpreters cannot.

I think that this also is a reason why most people liked the virtual keyboard I used to enter stuff.

[ACharbonneau]

Up until about here, plus a handful sections below, this is the same lesson I recently rewrote that's still in PR land. The reason I decided to re-write it, and I have the same comment here: this is not zero entry. It's front-loaded with a lot of unnecessary history and technical talk about how things were created, with lots of notes about how incomprehensible it all is...It's not that history is unimportant, but IME with real zero entry people, this is all terrifying.

[DamienIrving]

One thing this set of lessons is missing is a box/note about the differences when on a Windows machine. This probably needs to come early on, so that learners who are following along during the lesson don't get confused (I'm thinking when the root directory is introduced in the Files and Directories lesson). The version 4 materials (http://swcarpentry.github.io/bc/lessons/swc-shell/tutorial.html) have a nice little section on Windows, so this could probably just be cut and paste. This only change needed is that instead of talking about Cygwin, it might be better to refer to Git Bash, since that appears to be the most popular option amongst instructors these days.

[DamienIrving]

Text vs. Whatever

We usually call programs like Microsoft Word or LibreOffice Writer "text
editors", but we need to be a bit more careful when it comes to
programming. By default, Microsoft Word uses .doc files to store not
only text, but also formatting information about fonts, headings, and so
on. This extra information isn't stored as characters, and doesn't mean
anything to the Python interpreter: it expects input files to contain
nothing but the letters, digits, and punctuation on a standard computer
keyboard. When editing programs, therefore, you must either use a plain
text editor, or be careful to save files as plain text.

I'm as little confused as to why this section in the Shell Scripts lesson mentions the Python interpreter (most of the audience won't have ever used a Python interpreter at this point).

[DamienIrving]

Regarding this example in the Shell Scripts lesson:

Second, suppose we have just run a series of commands that did something useful—for example, that created a graph we'd like to use in a paper. We'd like to be able to re-create the graph later if we need to, so we want to save the commands in a file. Instead of typing them in again (and potentially getting them wrong) we can do this:

$ history | tail -4 > redo-figure-3.sh

The file redo-figure-3.sh now contains:

 297 goostats -r NENE01729B.txt stats-NENE01729B.txt
 298 goodiff stats-NENE01729B.txt /data/validated/01729.txt > 01729-differences.txt
 299 cut -d ',' -f 2-3 01729-differences.txt > 01729-time-series.txt
 300 ygraph --format scatter --color bw --borders none 01729-time-series.txt figure-3.png

After a moment's work in an editor to remove the serial numbers on the commands, we have a completely accurate record of how we created that figure.

This is a great example, however it's a little hard to do while you're live coding because that history doesn't actually exist...

[DamienIrving]

Regarding this content from the loops lesson:

Using the left arrow key, Nelle backs up and changes the command echo to goostats:

$ for datafile in *[AB].txt; do goostats $datafile stats-$datafile; done

When she presses enter, the shell runs the modified command. However, nothing appears to happen—there is no output. After a moment, Nelle realizes that since her script doesn't print anything to the screen any longer, she has no idea whether it is running, much less how quickly. She kills the job by typing Control-C...

To be able to do this example while live coding the program goostats will actually need to exist. Does it exist?

[rhiever]

Hey @gvwilson, goostats doesn't exist, as noted by @DamienIrving. What do you do when teaching the loops section in the classroom?

[ahmadia]

@wltrimbl - I remember discussing this with you. Any thoughts?

[gvwilson]

I usually use:

for filename in *.txt
do
head -3 $filename | cut -d , -f 1 | sort | uniq
done

because I've built up something like that pipeline earlier in the
lesson, so the cognitive load of figuring out what's going on is just
that of figuring out the loop.

[wltrimbl]

Only that we noticed that it didn't exist, and that @BernhardKonrad may have implemented dummy verisons of them in December. I like @gvwilson's use of head, sort, uniq, though.

[BernhardKonrad]

My dummy versions are here:
https://github.com/wltrimbl/2013-12-03-sesync/blob/9c7871ac909e853d19e901b430f626cbd4b3ee3a/lessons/shell-advanced/loops/goodiff
https://github.com/wltrimbl/2013-12-03-sesync/blob/9c7871ac909e853d19e901b430f626cbd4b3ee3a/lessons/shell-advanced/loops/goostat

I like the sleep command to drive home that not having output for time consuming tasks can be frustrating. However, please don't use decimals (1.5 was a bad choice) as windows didn't like that. sleep 1 or sleep 2 should do the trick.

[gvwilson]

+1 to merging the dummy versions for those who want 'em.
Thx,
G

[rhiever]

Materials likely need to be updated to have "bash goostats" etc. instead of just "goostats" as well.

[rhiever]

I have made the fixes we discussed in this PR: #254