Medley Interlisp project use of git

[masinter]

Discussed in #102

Converting back to an Issue so we can close it (spawning new issues if there are things undone).

^{Originally posted by masinter December 21, 2020}
There are several problems with the way the project has and continues to misuse GitHub.
Fixing these requires more Git expertise than I have, and concerted effort. Let me assure you that each of these has a longer explanation than given here.

1. Storing old versions as separate files in the repo. This refers to seeing FOO and FOO.~1~ in the repo. Doing this enables some workflows that are important, now, to be able to pluck out a previous definition with a simple `GETDEF(FOO FILE;3). There might be some way of doing the same with GiT but that doesn't matter unless we hook into the Git API.
2. This is a minor artifact of 1 but it has an easier fix. The problem is that the VM doesn't exactly follow EMacs' way of numbering versions. Emacs you see foo and foo.~3~ then foo with no version is really version 4. Lisp instead makes an explicit hard link between foo.~4~ and foo which is fine. Except Git knows nothing about hard links and treats them as two separate files, doubling the space (and Git hashes file names with contents so they don't help). A simple fix would be to write a quick script that scans through looking for foo and foo.~nn~ same size and content then replace one with a hard link with the other. (This also would remove the annoyance of having an extra version pop up.
3. We're storing derived files that should be rebuilt -- compiled files (John had code to batch compile everything), sysouts, whereis.hash, exports.all. There are good reasons for each of these but it should be a goal for someone with no experience could rebuild from source. It's the only way That's separate from releases (which we're not using now).
4. I didn't understand the git model (I still don't, but I've learned a few things): Move a bunch of files in. Add and commit. Then move them out. Git RM and then commit that. Then move them back in. etc. Multiple copies.
5. A diff that worked for lisp. And things like that. shellcommand git commands that work with lisp file names
6. LFS of sysouts might help a little, but 1 and 2 and 3 and 4

[lassik]

On point 1:

Git can checkout multiple worktrees from the same clone, but people rarely do this. There probably isn't a standard feature to check out multiple copies of the same file into the same worktree using a running number in the filename.

I would recommend:

• Storing only the latest version of the file in git at any given time, i.e. normal usage of git.
• Relying on git history to keep the old versions.
• Writing a separate shell script that examines the Git history and copies old versions of files from there into the worktree. (EDIT: You already have a very nice script to do just that, added in commit 966b837.)
• Writing a .gigignore file which tells git to ignore filenames with the pattern *.~[0-9]*~ instead of suggesting to commit them. (Seems this has been done in commit 2f41e9e.)

[masinter]

Sorry for not being clearer. These are mainly issues that have been addressed. I wanted to not lose the bits that were still open.

See scripts/restore-versions.sh

[masinter]

@rmkaplan @nbriggs @ecraven

@rmkaplan

I’ve been playing around with a medley interface to the worktree
commands, systematically making worktrees for specified branches in
named subtrees in the file system.
One issue that comes up is that the default seems to be that you can’t
make a worktree for the currently check out branch. And if you have
made a worktree for a non-checked-out branch, then you can’t switch to
that branch in the git desktop.
There are some options listed in the documentation that might override
these defaults, but I’m not sure what the consequences are. —force
seems to allow the worktree even if is the current branch, and
—nocheckout seems to do the add without actually checking it out.
Is it dangerous to specify git worktree add —force —nocheckout <path> <branch>

As long as you do not commit anything, nothing bad can happen. So just don't run git add .. && git commit ..
;)

It seems that this should be safe if all Medley wants to do is to
compare files across branches and with the separate Medley working
directory.
But would things get out of step if (by mistake) Medley copied a file
into one of these unchecked-out working trees or deleted a file there?
Best would be if these trees were conceptually read-only.

No, as long as you don't commit anything. Ideally, after you are "done", just delete all the directories. I'm not sure why git is designed the way it is, it would seem much better to me if the main repository directory didn't keep track of the worktrees, but only the worktrees pointed back to the main directory, but alas, that's just the way it is...

I'd suggest creating some sort of temp directory (I'll call it ./tmp), then just check out the "main" repo into ./tmp/medley, and create the worktrees in ./tmp/lmm14, ./tmp/ron25, etc. When all the comparing (and adding things to "main" and committing and pushing) is done, just delete all of ./tmp entirely. That way, nothing is left over that could create any problems.

[masinter]

and here is a previous attempt in the wiki:

https://github.com/Interlisp/medley/wiki/How-to-avoid-massive-PRs-when-developing-Medley-Interlisp

[rmkaplan]

Is there a command line way of finding out what branch corresponds to a particular PR? Or do you have to go to the website and click in the PR itself to see the branch name?

[rmkaplan]

The git desktop lists a whole bunch of branches that have not been pulled down to my repo, origin/lmm9, origin/lmm10 etc which presumably have PR's associated with them.

Can I build a worktree for the remote branch without first checking it out and maybe getting confused about what's actually checked out? All I want it for is to do comparisons with other branches.

[nbriggs]

That's a case where you don't have a tracking branch. There's no requirement that there be a PR associated with a branch (PRs are github, not git). git can deal with remote branches that it knows about, which are the remote branches that existed at the time you last did a "git fetch".

You can "git worktree add /tmp/xxx origin/lmm14" and it will check out origin/lmm14 into /tmp/xxx just as it would with a local branch -- but it's not necessarily what's on the server, it's what's known in your local repository from the last time you did a "git fetch".

[nbriggs]

In maiko, for example, I have multiple branches that are nominally "master" --

• gitlab/master
• origin/master
• master

At least two of them have quite different things in them.

[nbriggs]

Actually, if you want to get all your remotes updated so that your local copy of the repo matches what's in the remotes, you'd do (pseudo code):
for r in "git remote" do "git remote update" r

[rmkaplan]

What does it mean to be “nominally master”? Master branches in different places, or just different branches that you think will continue independently into the future—no particular plan to merge?

…

On Nov 17, 2021, at 5:53 PM, Nick Briggs ***@***.***> wrote: In maiko, for example, I have multiple branches that are nominally "master" -- gitlab/master origin/master master At least two of them have quite different things in them. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#495 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJM4Z3PK3T6F6XKJN43UMRMDFANCNFSM5ESADMFA>.

[nbriggs]

They're the branch named "master" in some repository, maybe they contain the same stuff, or related stuff, or maybe they don't. In my case the remote named "gitlab" (which happens to be stored on gitlab.com) has a branch named "master" which represents the state of maiko at some time in the past:

% git log gitlab/master
commit cd4c59dbb3114e4d26260705cb3f10e7b4b67d31
Author: Nick Briggs <nicholas.h.briggs@gmail.com>
Date:   Fri Aug 21 12:18:03 2020 -0700
[...]
% git log origin/master
commit 987cf4c7c6305c07f12e95b80349a2ac45c185a5
Author: Nick Briggs <nicholas.h.briggs@gmail.com>
Date:   Sat Nov 6 18:36:25 2021 -0700
[...]
% git log master
commit 987cf4c7c6305c07f12e95b80349a2ac45c185a5
Author: Nick Briggs <nicholas.h.briggs@gmail.com>
Date:   Sat Nov 6 18:36:25 2021 -0700
[...]

Here origin/master and master happen to be the same because I've not committed (or merged anything into) master before pushing it to origin/master.