Add ABCL support

[rkoeninger]

Fixes #51

Discussion

Working on this raised a question about how this would be distributed for Armed Bear. ABCL runs on Java so it would distributed as a .jar file?

This also reminds me about source distribution (rendering a shen.lisp with the source in this repo including the translated kernel) that would be multi-lingual. Looking in the asdf.lisp file including in some CL projects, it might be able to handle multi-platform related stuff for us.

Testing Notes

Running make build-abcl might work depending on if you have abcl.jar in the repo checkout root (and java installed) (and it might have to be java 8). build-abcl compiles the .lsp files for abcl, compiles a .java file that has the main method used for entrypoint (to collect args and start shen.abcl) and puts it in a bin/abcl/shen.jar file.

Running make run-abcl will try to run java -jar bin/abcl/shen.jar which isn't working for me right now. I get a ClassDefNotFoundError for the abcl stuff. Probably can't find abcl. Don't know why.

CC: @tizoc @junkerjoe

[tizoc]

Looks like Travis is going to queue this forever.

@rkoeninger I'm planning on switching CI on this repo (and Shen/Scheme's too) to GitHub actions, Travis is sucking lately. Just saw this too https://ropensci.org/technotes/2020/11/19/moving-away-travis/

[rkoeninger]

It finally started and than failed because I needed some sudo's.

@tizoc Actions looks like the way to go. I'll probably start moving my projects there as well. (And don't forget Shen-Language/shen-source - or I'll do that one once I see how's done)

[tizoc]

Yes, I have been using actions for a while and it works quite well (even today I switched a repo from travis to github actions because of how bad travis is behaving lately).

[rkoeninger]

Running make build-abcl might work depending on if you have abcl.jar in the repo checkout root (and java installed) (and it might have to be java 8). build-abcl compiles the .lsp files for abcl, compiles a .java file that has the main method used for entrypoint (to collect args and start shen.abcl) and puts it in a bin/abcl/shen.jar file.

Running make run-abcl will try to run java -jar bin/abcl/shen.jar which isn't working for me right now. I get a ClassDefNotFoundError for the abcl stuff. Probably can't find abcl. Don't know why.

javac -cp abcl.jar -d bin/abcl src/ShenABCL.java
cp src/ShenABCLManifest.txt bin/abcl
cd bin/abcl; jar cfm shen.jar ShenABCLManifest.txt ShenABCL.class shen.abcl
java -jar ./bin/abcl/shen.jar -- eval -e "(cd \"kernel/tests\")" -l README.shen -l tests.shen
Error: Unable to initialize main class ShenABCL
Caused by: java.lang.NoClassDefFoundError: org/armedbear/lisp/ProcessingTerminated
Makefile:189: recipe for target 'test-abcl' failed
make: *** [test-abcl] Error 1
The command "make $MAKE_BUILD_TARGET" exited with 2.

---

My goal was to have build-abcl spit out an "uberjar" containing abcl.jar, abcl-contrib.jar and the joined, compiled kernel in the form of the shen.abcl fasl file - into a jar called shen.jar or shen-abcl.jar.

This has gotten a little tiring so if someone else wants to take a crack it at, go ahead.

I'm continuing to question the need to have an executable form for each CL impl, i think it would be enough to have a shen.lisp library and a single sbcl executable.

[tizoc]

@rkoeninger this is more than good enough, seems like you already did >90% of the work here. I am not familiar with ABCL (didn't even know it existed until now) nor have any experience with the JVM, so there isn't much I can help with here.

@junkerjoe if you are familiar enough with ABCL maybe you could provide some ideas here to complete what is missing, I think everything that is related to the Shen/CL side has been covered by @rkoeninger already.

[tizoc]

I'm continuing to question the need to have an executable form for each CL impl, i think it would be enough to have a shen.lisp library and a single sbcl executable.

@rkoeninger can you elaborate on this? we only release SBCL binaries, right?

I'm all for simplifying things.

[rkoeninger]

can you elaborate on this?

@tizoc My thinking (as seen in ShenScript and ShenSharp) is that a Shen implementation should be thought of as more of a library that gets imbedded in other applications than a foundational language that gets distributed as a binary.

There's a build step in the Makefile that pre-compiles the kernel to lisp; I'm thinking in the future that step becomes the main build step to render a "shen.lisp" library, maybe using ASDF and published to Quicklisp (see #27 - asdf could help with targeting multiple CL impls and QL would make usage more accessible). Such a library could then be embedded into whatever lisp program, one such example being a SBCL-based program that loads up shen.lisp and calls shen-cl.toplevel.

This would mean we wouldn't have the need to build artifacts for every CL impl. Just so long as shen.lisp library is usable from each one; so maybe we'd still have optional test steps.

I started thinking this back when porting to ECL (a couple years ago I think) when I realized ECL isn't about building lisp executables but embedding lisp in native programs (Embeddable Common Lisp).

I'm having this thought again now since I'm having trouble rendering a build artifact (a jar file) specifically for ABCL. Taking the approach described, the make build-abcl step would be removed but maybe a test-abcl step would remain.

[tizoc]

@tizoc My thinking (as seen in ShenScript and ShenSharp) is that a Shen implementation should be thought of as more of a library that gets imbedded in other applications than a foundational language that gets distributed as a binary.

Ah yes, I remember this. But Shen doesn't force you to do anything in particular, it is up to the port authors. Shen/Scheme for example didn't work as a runnable binary until I ported it to Chez, it was just a library for Chibi and Gauche.

I thought you were talking specifically about Shen/CL. IMO the best option would be to care only about the SBCL binary, and then do as you say for everything else. I don't even know if anyone uses anything other than Shen/SBCL, so it makes no sense to waste energy with those binaries -- specially so when, as you say, it is a PITA in some of the Common Lisp implementations.

[rkoeninger]

Alright, so for this PR: what's a good way to wrap this up? I don't think we can say shen-cl "supports" ABCL unless there's some kind of artifact that this repo produces that ABCL can use. So that means either:

• a library/executable - a java .jar file in either case
• a distributable library in some standard form - zipping up or concat'ing the contents of the ./compiled/ folder?

I was trying to get option 1 working and gave up on that for the moment. Now I'm thinking option 2.

[tizoc]

@rkoeninger from my point of view, only SBCL binaries are required, 1) because it is the reference implementation, 2) I think it is the only one people use, 3) it is worth the cost/benefit of being maintained.

We can even say explicitly that for now at least, although code that is specific to other implementations is included, only SBCL is officially supported and the rest are "best effort". And then do as you say and produce a precompiled lisp file at release time (btw I started the Github Actions stuff for Shen/Scheme the other day, needs a little bit more of work, most of it should be reusable in this repo I think).

Anyone can then use that precompiled .lisp file to produce (and maintain if they want) binaries/packages/whatever for other Common Lisp implementations.

So as far as wrapping this one up, I propose you leave as a CI test for ABCL to the minimum thing that works (just being able to compile the thing?), and then the generation of the compiled .lisp file can be handled separately.

Does this sound good to you?

[rkoeninger]

I'm settling for rendering a combined fasl file called shen.abcl.

(this is now a liveblog comment)

Trying to load it in abcl repl and run |shen-cl.toplevel| causes

!!! FATAL error:
Exiting Shen.

with no other explanation.

The error occurs when shen-cl.toplevel is called. I just re-used the CCL version of that method. Could the problem be there or is it farther down the call chain? Maybe a stack overflow or out-of-stack-space error?

SHEN(11): |*hush*|
#<THREAD "interpreter" {B843732}>: Debugger invoked on condition of type UNBOUND-VARIABLE
  The variable SHEN::|*hush*| is unbound.
Restarts:
  0: TOP-LEVEL Return to top level.

|shen.initialise-environment| isn't getting called... I know I loaded shen.abcl manually, but I would think that things would get initialised anyway...

SHEN(15): (|shen.repl|)
exit REPL with (cl.exit)
#<THREAD "interpreter" {B843732}>: Debugger invoked on condition of type STORAGE-CONDITION
  Stack overflow.
Restarts:
  0: TOP-LEVEL Return to top level.

other functions are working....

SHEN(18): (|map| (|lambda| X (+ 1 X)) (|cons| 1 (|cons| 2 (|cons| 3 NIL))))
(2 3 4)

---

ABCL doesn't support tail calls:

https://0branch.com/notes/tco-cl.html#sec-2-9

The following implementations were found to provide full TCO:

CMUCL
SBCL
CCL
Allegro
LispWorks

…only optimise self-calls in tail position:

CLisp
GCL
ECL (when explicitly compiled)

…neither:

ABCL

https://abcl.org/trac/ticket/353

@tizoc ABCL might be a no-go anyway

so I/O is not being done correctly...

SHEN(27): (|trap-error| (|shen.read-evaluate-print|) (|lambda| E (format nil "~A" E)))
"READ-BYTE: unsupported element type CHARACTER."

[rkoeninger]

read-byte and write-byte now working

SHEN(10): (|shen.read-evaluate-print|)
(+ 1 1)
2
SHEN(11): (|shen.read-evaluate-print|)
(output "~A" "hi")
"hi"
SHEN(12): (|print| "hi")
"hi"

shen.read-evaluate-print works, shen.prompt works, shen.initialise_environment works but shen.loop does not, it just keeps reading input

SHEN(14): (|shen.initialise_environment|)
NIL
SHEN(15): (|shen.prompt|)
"

(2-) "
SHEN(16): (|shen.loop|)
(+ 1 1)
^
^D

Terminate batch job (Y/N)?