Handle windows carriage returns (CRLF) in Filter form

[alexiswl]

Hello,

Whenever I copy multiple lines from my IDE into the jqplay services url, I would get this error

unexpected INVALID_CHARACTERm expecting $end

I did a bit of debugging locally and discovered that when I copy from my clipboard on Windows 10 into a Microsoft Edge browser (version 105.0.1343.50) that the line endings are ascii characters 10 + 13, as opposed to just 13.

If I remove the line endings and then recreate them inside the Filter form, everything is fine

This doesn't happen when I do the same thing in Firefox so seems to be OS/browser specific.

I did try stripping the CRLF line endings in the GO source code but don't know enough about the GO language to know exactly where this snippet needs to be put.

[generalmimon]

I've just hit this issue in Chrome. The worst part about it is that if you don't know about it, you have absolutely no clue what's happening and what's wrong with the perfectly valid-looking jq command (because the error is in an invisible character). So I guess this may be quite puzzling for many people.

And I don't think it occurs that rarely - yes, it doesn't happen when you write the jq command directly in the "Filter" field in the browser. But once you're writing some more complicated jq filter, chances are you'll want to start by copying a prepared jq filter from somewhere (some text editor, etc.), which is where you're quite likely to run into this issue, because the CR LF line endings are default on Windows.

You can check the presence of CRLF endings for example by pasting the clipboard contents to a hex editor like HxD. If you see 0d 0a somewhere in the hex view, that's a CR+LF sequence; an isolated 0a is a LF ending.

This doesn't happen when I do the same thing in Firefox so seems to be OS/browser specific.

I can confirm this doesn't happen in Firefox - apparently Firefox sanitizes the input you copy in the "Filter" field by replacing the CR LF endings to just LF. Chrome doesn't do this and sends the jq expression unchanged (with \r\n, if it was present in "Filter" field), which isn't accepted by the jq on the server (and it responds with the error error: syntax error, unexpected INVALID_CHARACTER, expecting $end as we saw).

---

I did a bit of debugging locally and discovered that when I copy from my clipboard on Windows 10 into a Microsoft Edge browser (version 105.0.1343.50) that the line endings are ascii characters 10 + 13, as opposed to just 13.

You have the ASCII codes the other way around - you actually meant 13 + 10 (13 is CR and 10 is LF; it can be written as 0d 0a in hexadecimal, or in many programming languages in string literals using escape sequences as \r\n), as opposed to just 10 (0a, \n, LF).

[owenthereal]

Do you have an example to reproduce the issue? It would be helpful to see a specific case where this occurs. Could you also check if the problem persists with the latest rewrite, which now uses the Monaco Editor? Let us know if you find anything!

[generalmimon]

@owenthereal:

Could you also check if the problem persists with the latest rewrite

No, it does not. I've just checked at https://play.jqlang.org/ (I made extra sure to prepare a query in a text editor set to CRLF endings and verified in HxD that it indeed has them) and I can no longer reproduce it, so it seems to have been fixed.

I see that so many things have changed since this was reported. Not only the editor as you mentioned, but also the overall architecture of the jq playground. Perhaps an even bigger change than the editor is that the jq queries are now handled locally in the browser. When I was writing the above comment, I remember seeing HTTP requests in Chrome DevTools to the server sending the multi-line jq query with CRLF endings (\r\n), and the server responded with the error syntax error, unexpected INVALID_CHARACTER as we know.

Also, given that many major versions of Chrome have been released since 2023, is is also quite possible that some version has started normalizing CRLF to LF endings in text pasted from the clipboard (which would also fix the problem - as I mentioned in #127 (comment), Firefox was doing this already back then, so it was not affected), but I'm not sure about that. The important thing is that it's no longer a problem 🙂

Do you have an example to reproduce the issue?

Actually yes, I've just locally reproduced exactly what happened back then. If you feed a jq query with CRLF endings to a Linux jq binary, you still get the same error:

$ file "$(which jq)"
/usr/local/bin/jq: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6f9bc081254ee90d45dbcfa87457de2e93c456e4, for GNU/Linux 3.2.0, stripped
$ sha256sum "$(which jq)"
5942c9b0934e510ee61eb3e30273f1b3fe2590df93933a93d7c58b81d19c8ff5  /usr/local/bin/jq
$ jq --version
jq-1.7.1
$ file test.jq
test.jq: cannot open `test.jq' (No such file or directory)
$ echo '31 20 2b 0d 0a 32 0d 0a' | xxd -r -p > test.jq
$ hd test.jq
00000000  31 20 2b 0d 0a 32 0d 0a                           |1 +..2..|
00000008
$ cat test.jq
1 +
2
$ jq -n -f test.jq
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at <top-level>, line 1:

jq: 1 compile error

Which is to be expected, since there is no reason for a Linux binary to recognize CRLF endings.