What should we do with TCP and other networking tools

[masinter]

Network: Medley has full protocol stacks, in Lisp, for three families of network protocols: PUP, XNS, and TCP/IP. Can we demonstrate?

[nbriggs]

At the moment, we have access to the (raw ethernet) network only on a Solaris system. In order to demonstrate network access on other systems we should look to the Berkeley Packet Filter interface, rather then the DLPI or NIT interfaces that are/were available on SunOS/Solaris.

[tuexen]

Where is the TCP/IP source code located?

[nbriggs]

@tuexen -- The Lisp implementation of TCP/IP was deleted from the repo -- better to use the underlying OS implementation for anything we can, I think. For PUP and XNS we have to use the Lisp implementation and the underlying OS raw ethernet interface (e.g., the /dev/bpf)

[nbriggs]

@tuexen -- In the medley repo: % git whatchanged | grep TCP

[tuexen]

@nbriggs Thanks for the information. I was looking for the TCP code and did not find it... I have not idea what PUP and XNS is, I was just wondering if there is interest in using the TCP stack from the Lisp code base. I'm somewhat familiar with TCP and have an interest in Lisp. So I was wondering if there is interest in getting the old codebase to work... But using the host stack gives you a much more modern TCP implementation...

Thanks for the quick response!

[nbriggs]

@tuexen -- PUP => PARC Universal Packet, the packets on the original 3 Mb ethernet (http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-79-10_Pup_An_Internetwork_Architecture_Jul79.pdf), and XNS => Xerox Network Services, the packets on the 10 Mb ethernet. Novell Netware copied the XNS Internet Datagram Protocol (IDP) for their IPX protocol. The old Lisp TCP/IP implementations were deprecated by the original maintainers after Interlisp started to be run on the emulator. On the original D-machines Interlisp was the OS, and had to implement everything including the raw ethernet packet handling, the page fault handlers, the disk controller operations, the file system etc., ... It became a lot easier to maintain when that could all be delegated to the OS.

[Anzus]

It followed pretty closely the design of the BSD stack from the early ‘80s, but implemented in Interlisp-D to work on 10-Mb Ethernet. With the transition to non-Xerox hardware the decision was made to use the stack provided by the host OS since the support and improvements would come from those OS maintainers. TCP/IP’s come a pretty long way since then and I wouldn’t be surprised if there aren’t some incompatibilities against current stacks.

…

On Dec 25, 2021, at 3:25 PM, Michael Tüxen ***@***.***> wrote: @nbriggs <https://github.com/nbriggs> Thanks for the information. I was looking for the TCP code and did not find it... I have not idea what PUP and XNS is, I was just wondering if there is interest in using the TCP stack from the Lisp code base. I'm somewhat familiar with TCP and have an interesting in Lisp. So I was wondering if there is interest in getting the old codebase to work... But using the host stack gives you a much more modern TCP implementation... Thanks for the quick response! — Reply to this email directly, view it on GitHub <#573 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBP2DFZQNP6X5KKJHZYZSTUSZAERANCNFSM5IITCKUQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.

[tuexen]

It followed pretty closely the design of the BSD stack from the early ‘80s, but implemented in Interlisp-D to work on 10-Mb Ethernet. With the transition to non-Xerox hardware the decision was made to use the stack provided by the host OS since the support and improvements would come from those OS maintainers. TCP/IP’s come a pretty long way since then and I wouldn’t be surprised if there aren’t some incompatibilities against current stacks.

That is what I'm interested in. I'm working on today's TCP stack of FreeBSD and would expect that it can interoperate with a stack just doing the initial specification (RFC 793)...

[tuexen]

@tuexen -- PUP => PARC Universal Packet, the packets on the original 3 Mb ethernet (http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-79-10_Pup_An_Internetwork_Architecture_Jul79.pdf), and XNS => Xerox Network Services, the packets on the 10 Mb ethernet. Novell Netware copied the XNS Internet Datagram Protocol (IDP) for their IPX protocol. The old Lisp TCP/IP implementations were deprecated by the original maintainers after Interlisp started to be run on the emulator. On the original D-machines Interlisp was the OS, and had to implement everything including the raw ethernet packet handling, the page fault handlers, the disk controller operations, the file system etc., ... It became a lot easier to maintain when that could all be delegated to the OS.

Thanks for the clarification. Unfortunately, I have no experience with PUP and XNS. My interest would have been the TCP stack in lisp...

[nbriggs]

You're more than welcome to check out the old code that you'll find from the git whatchanged command I mentioned above. I am looking to get the BPF interface for raw ethernet packets working, and that should allow the Lisp TCP/IP code to run again.

[tuexen]

You're more than welcome to check out the old code that you'll find from the git whatchanged command I mentioned above. I am looking to get the BPF interface for raw ethernet packets working, and that should allow the Lisp TCP/IP code to run again.

That should work... You have to make sure that the host stack does not interfere with the stack in the Lisp VM. Drop me a note once it is finished or if you have questions related to this...

[masinter]

you might want to check in the interlisp/history repository; the TCP files were moved to library/Obsolete/ by 2016.
history-tcp-files.txt
There might be some possibility of running Medley 2 on DOS, but the state of emulating old network cards in FreeDOS or DosBox is probably a problem.

What I remember was that trying to implement a full TCP protocol stack by reading RFCs was a learning experience.
The XNS and PUP documents were a lot clearer (or maybe the implementors were down the hall).

[masinter]

The work on "dodo" (XNS in Java) noted above might provide an opportunity to revive Medley networking?

[masinter]

This issue has more useful information than #456 so I'll mark that one duplicate and close it.

[devhawala]

Hello,

the branch dodo-nethub-support in my clone of Maiko
has a version of Maiko allowing to connect directly to a Dodo Nethub for
using Dodo XNS services.
(for details about building and usage
see Using Dodo-networking with Maiko)

The Dodo Nethub support was implemented and tested with x86_64-Linux, so the
following information and tour through a demo apply to this platform.

The file dodo-and-medley.zip attached to this post contains the following items:

• dodo-demo.tgz: this is a demo Dodo setup (see below)
• run-medley: a version of the standard script from the Medley site,
  adapted for the additional options of the Dodo-networking capable Maiko
• job_0183_7090_DD36.pdf: a sample PDF of a .tedit-file loaded
  from the medley file service in the Dodo-demo opened through the FileBrowser
  with See and printed from Medley

Unpacking the dodo-demo.tgz archive creates a directory dodo-demo with
a prepared Dodo setup, based on the standard Dodo distribution with an additional
file service medley having the file drawer internal holding some documentation
files taken from the subdirectory docs/internal of the Medley github project.
The print service poseidon is prepared to produce PDF files with support for
standard Xerox fonts (Classic, Modern fonts).
Remark: producing PDF for printouts requires that the program ps2pdf is available
through the shell PATH, so make sure that Ghostcript is installed on your system
and the PATH environment variable is set up accordingly.

For running the Dodo demo system, open 2 shells, change to the dodo-demo directory
in each and:

• in the first shell, start the Dodo Nethub with:
  ./run-nethub.sh
• in the second shell, start the Dodo server with:
  ./run-dodoserver.sh

For shutting down the Dodo demo system, simply stop the programs with Ctrl-C in reverse
order.

The Medley environment i used for my experiments was the release medley-211122 (loadups
and runtime, downloaded 2021-11-24), using the full sysout. Newer releases also worked
but were not my "daily" test environment. In any cases, XNS networking must be initialized
by running the function \NSINIT when a fresh "loadup" sysout is started resp. XNS networking
was not used so far (without \NSINIT Lisp will try forever to reach a Clearinghouse).

For a quick tour through the combined Medley/Dodo demo environment:

• start the Dodo demo system (see above)
• start Medley with the attached run-medley script and connecting to the Dodo Nehub
  with:
  run-medley -nh-host localhost
  (possibly adding more options like your preferred image or -nocroll)

And then in the Lisp environment:

• if not already done or perviously saved with the image: initialize XNS networking by

  • opening an old Interlisp-D Exec window
  • running the function (\NSINIT) there

• open a FileBrowser through the background context menu and let it open the following
  target directory:
  {medley:}<internal>

• a prompt window requesting to login should open, enter the user motu and the password
  master

• the FileBrowser should now list the files of the above directory and allow the usual
  operations there (see, recompute etc.)

• to view a document, select a .tedit file in the list
  attention: for some documents, the content cannot be displayed due to some missing Lisp function,
  so possibly close the TEdit window and choose a different document

• use the printer poseidon: (note the colon!) for printing, the print results will be created
  in the subdirectory dodo-demo/prt-poseidon by the Dodo print service (.ip,
  .interpress, .ps, .ps.log and (if ps2pdf is present) .pdf files);
  printing was tested with the Hardcopy > To a printer function (background context menu) and with TEdit
  for .tedit files (including embedded bitmap- and Sketch-images)

Experienced Interlisp-D users probably know more use-cases for using XNS services, the above were
my standard tests for extending Dodo to support the little Medley-specialities at XNS filing and
printing level.

Greetings
Hans

Attachment:
dodo-and-medley.zip

[masinter]

Astounding!

[devhawala]

Hello,

looking again at the file i uploaded yesterday, i've seen that the filenames in the archive were messed up...

So please rename the files after unpacking as follows to match the names mentioned in my previous post:

from:	to:
dodo-demo.TGZ	dodo-demo.tgz
dodo-demo.sh	run-medley
dodo-demo.pdf	job_0183_7090_DD36.pdf

(Github does not accept all extensions for file uploads to a post, e.g. insisted on an uppercase .tgz extension and run-medley (or run-medley.sh) was simply rejected; so before putting them into a single ZIP-archive, i wanted to rename the .TGZ back to lower case: that's when scrambling the filenames must have happened, as it seems i had all 3 files selected when renaming with the Windows10-Explorer, which fired this super-Windows feature where all 3 got the same name and the .TGZ extension kept being upper case)

Sorry for the inconvenience
Hans

[masinter]

Success!

I'm not sure what I did differently -- I was dying with a stack overflow trying to CH.FINDSERVER -- but today, after several similar failures, it all worked!

debuglog.txt

demoing CoLab and things like it (didn't Notecards have some networking possibilities?)
maybe TELERAID?

[devhawala]

Congratulations!

Yes, i also had stack overflows with some of the fresh loadup images, but others worked fine. Not sure if some of the failing ones have less things preloaded and one crucial thing is missing.

The important thing is to call \NSINIT first! Can this be included in the loadup generation so this can't be forgotten?

BTW: is there a mail client in Interlisp to check if exchanging mails with Star or Viewpoint works?

Hans

[nbriggs]

Larry can look back at how the network initialization used to work -- he removed the Lisp code that used to get the ethernet properly initialized in order to speed things up when the network wasn't present. It's NOT as simple as calling \NSINIT in the loadup.

Yes, Lafite is the mail client that understands XNS mail (and Grapevine on a PUP network).

[nbriggs]

Could you rebase onto the current maiko master branch -- all the Addr68k_from_LADDR etc. have been replaced by NativeAligned{2,4}FromLAddr to reduce the compiler warnings for incorrect alignment.

[nbriggs]

I'm looking at trying to reduce the entropy in the code, and this seems like a good time to look at splitting the ethernet handling code up in to separate files so that you don't have to guess which definition of something is active for which set of #ifdef options.

[devhawala]

just pushed a new Maiko version in my branch:

• calls to Addr68k_from_LADDR are replaced with NativeAligned{2,4}FromLAddr in ether.c
• timer and async emulation got 2 definable constants MAIKO_EMULATE_TIMER_INTERRUPTS and MAIKO_EMULATE_ASYNC_INTERRUPTS, with the 2nd implying the 1st
• both constants are defined in platform.h, yesterday's definition disappeared from the cygwin makefile
• the instruction count for constants can optionally be defined with MAIKO_TIMER_ASYNC_EMULATION_INSNS_COUNTDOWN, defaulting to 20.000
• if one of the MAIKO_EMULATE_* is defined, a new command line parameter can be given as -intr-emu-insns to specify an override for the interrupt countdown value

Compiles and works on Linux-x86_64 and Cygwin, the new command line option is untested however

ok, that's all for today...

[nbriggs]

OK, I'll look. It should probably reuse the "-t" flag that is currently (supposed to...) set the timer frequency.

The style for options should be '-x' (or '-xx') for short options or '--longer-option' (double dash). We're going to rationalize the option processing though, so you probably don't need to deal with that.

[masinter]

I'm going to this issue into a Discussion about possible networking goals (for TCP) since it's not clear what the goal is
and open an issue to document the 'dodo' networking for PUP and XNS access via emulation

[nbriggs]

Dodo does XNS, but not PUP.

[pamoroso]

As a user my goal would be a TCP/IP implementation conforming to the Lisp API, behavior, and tools described in the documentation of the Lisp library modules and LispUsers. Any extensions would be welcome.

[masinter]

PR #1283 PR #1284 bring back TCP files (to 'obsolete').
This discussion should turn into an Issue to get them to work again.