Change EOL convention to LF?

[masinter]

@rmkaplan reported:

Windows may still be the outlier, Mac OSX and Unix/Linux are LF.

It’s really a question of what the default EOL convention should be when output streams are created. It shouldn’t matter for input streams, at least for the operations that read characters and not bytes. The character reading functions should all recognize any of the EOL conventions on files and map them into the internal CR (the value of (CHARCODE EOL). (I remember setting it up that way—silly to have to know where or how a file was created before you could read it).

I thought that Unicode might have something to say about this, but they aren’t very helpful. They point out (in the Unicode 3.0 book that I have) that the CR/LF/CRLF conventions are confused…and then they add to the confusion. They define a new code U+2028 as the unambiguous “line separator” (also an unambiguous “paragraph separator" U+2029).

My Xerox XCCS book doesn’t say anything about this, so I’m not sure what the representation is in XCCS-compliant files (which would have run-codes for mixed character-set strings, but control characters are unique in any run).

My temptation is to change the default, so that we are more compatible with Unix/Mac files. We were never compatible with Windows/CRLF. If not for all files, then at least for UTF8 and UTF16 files.

In prowling around, I have also discovered that some of the low-level files got corrupted by the Japanese. A substantial part of the LLREAD file, for example, is filled with conversion tables for various Japanese coding systems, and this stuff is mixed in in a number of other places. Should have been in separate and later files—hard to imagine that these would be needed in the INIT.SYSOUT.

[masinter]

Should we make sure the files are all LF before we push the Interlisip source code into the GitHub repo?

[johnwcowan]

Windows can mostly cope with LF-terminated text files nowadays. For a long time the last holdout was WIndows Notepad, but that's finally been fixed; in addition, there are many third-party plain text editors that have no problem with LF.

[masinter]

more @rmkaplan notes:
But there is an ancient EOL issue that I thought had been resolved many years ago, but seems to still be there in the code.

I did a lot of the character reading stuff in the early days, and apparently I also implemented the notion of an external format in the mid 90’s (which I don’t remember at all). But that’s the interface that makes it easy to add the UTF8 stuff.

The issue has to do with the low-level character reading macros. They all go through a macro \inchar that basically wraps a call to \nsin, that then does xccs/ns stuff inline but otherwise calls out to the external format character function.

But \inchar wraps the \nsin call inside another macro \checkeolc, which triggers on CR and LF, and does coercion to internal EOL (which happens to be CR) when the byte sequence matches the EOLCONVENTION of the file.

If it sees a CR and the EOLCONVENTION is CRLF, it peeks at the next byte and reads it if it sees LF. If the macros are told to decrement the byte count, then the CRLF cause the necessary extra decrement. If it doesn’t see the LF, then it returns CR by itself, which is OK because that’s the internal. But if the convention is CR (which it defaults to now) and the file has CRLF, the LF is left in the file, and that screws things up. And if sees a naked LF, it doesn’t get converted to EOL.

In aboriginal times—before text files moves back and forth between operating system environments with different conventions—there was a lot of fussiness about properly interpreting the EOL. This is still the correct thing to do for output files, so that files will look good in their home environment.

But at one point it became apparent that this is a mistake for input files. Since you don’t know the provenance of a file, if you are operating on a file—or a region of a file—with text or character input functions, then any of the 3 eol indicators that happen to appear in the file should be mapped to the internal EOL.

In fact, that is what PFCOPYBYTES is doing—it calls \NSIN directly instead of \INCHAR because it doesn’t want the accidental EOL convention of the file to get in the way.

I thought I had cleaned this up a long time ago, but apparently not. I’m tempted to take another crack at it: to change the \CHECKEOLC macro to scoop up all the options, and then to recompile the relatively few functions that contain the macros (which may run into the problem in LLREAD).

[masinter]

@rmkaplan @nbriggs re comments on #128
Isn't reading just a matter of having the right entry for LF in the readable? To treat it the same as CR?
Why would you need to patch LLREAD?

[nbriggs]

Have a look at the source for PFCOPYBYTES in PRINTFN

[masinter]

In fact, could you just add CHARSET of xccs vs. Unicode to the READTABLE too?
Instead of a different independent property?

[masinter]

Have a look at the source for PFCOPYBYTES in PRINTFN

looks like at a minimum PFCOPYBYTES is misnamed or the patch was put into the wrong place.

[rmkaplan]

What patch? I wonder if there is a simpler way of doing this, for the common case where input and output file have the same character encoding, just do the right number of bin/bouts, don’t get into character interpretation (except for ^F and the EOL coercion). If the external formats differ and you do need to do the character interpretation, well, that’s where you get the multiple-value stack-smashing.

…

On Jan 4, 2021, at 5:26 PM, Larry Masinter ***@***.***> wrote: Have a look at the source for PFCOPYBYTES in PRINTFN looks like at a minimum PFCOPYBYTES is misnamed or the patch was put into the wrong place. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJNOAA3GVBD6XIWQPTTSYJTDLANCNFSM4PICVDKQ>.

[masinter]

i'm confused. This is the thread to talk about "Change EOL convention to LF".
There's another bug #19 about stack overflow with multiple values.
It's best to keep these discussions separate by choosing "View it on GitHub" rather than replying to the email notification.

[rmkaplan]

It is PFCOPYBYTES (or SEE) that provokes the stack crash. If PFCOPYBYTES didn't do what it is doing, then it wouldn't be the provoker...but the bug would still be there

[johnwcowan]

CR by itself was last used on Mac Classic, and although WIndows still likes to write CRLFs, as I said above it can handle either LF or CRLF formats: Cygwin writes LF by default. As for those Unicode line and paragraph terminators, absolutely nobody uses them: they were a fine example "There are so many standards to choose from, and if you don't like any of them, wait till next year." So switching from CR-only to LF-only would be the Right Thing for usability's sake.