Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't normalize CRLF to LF in Junk #193

Merged
merged 1 commit into from
Oct 30, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
test/fixtures/crlf.ftl eol=crlf
test/fixtures/cr.ftl eol=cr
2 changes: 1 addition & 1 deletion spec/fluent.ebnf
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Term ::= "-" Identifier blank_inline? "=" blank_inline? Value Att
CommentLine ::= ("###" | "##" | "#") ("\u0020" /.*/)? line_end

/* Adjacent junk_lines are joined into FTL.Junk during the AST construction. */
junk_line ::= /.*/ line_end
junk_line ::= /[^\n]*/ ("\u000A" | EOF)

/* Attributes of Messages and Terms. */
Attribute ::= line_end blank? "." Identifier blank_inline? "=" blank_inline? Pattern
Expand Down
10 changes: 6 additions & 4 deletions syntax/grammar.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,14 @@ let CommentLine = defer(() =>

/* ------------------------------------------------------------------------- */
/* Adjacent junk_lines are joined into FTL.Junk during the AST construction. */
let junk_line = defer(() =>
let junk_line =
sequence(
regex(/.*/),
line_end)
regex(/[^\n]*/),
either(
string("\u000A"),
eof()))
.map(join)
.chain(into(FTL.Junk)));
.chain(into(FTL.Junk));

/* --------------------------------- */
/* Attributes of Messages and Terms. */
Expand Down
1 change: 1 addition & 0 deletions test/fixtures/cr.ftl
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
### This entire file uses CR as EOL.err01 = Value 01err02 = Value 02err03 = Value 03 Continued .title = Titleerr04 = { "strerr05 = { $sel -> }
Expand Down
10 changes: 10 additions & 0 deletions test/fixtures/cr.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"type": "Resource",
"body": [
{
"type": "Junk",
"annotations": [],
"content": "### This entire file uses CR as EOL.\r\rerr01 = Value 01\rerr02 = Value 02\r\rerr03 =\r\r Value 03\r Continued\r\r .title = Title\r\rerr04 = { \"str\r\rerr05 = { $sel -> }\r"
}
]
}
11 changes: 9 additions & 2 deletions test/fixtures/crlf.ftl
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@

key01 = Value 01
key02 =

Value 02
Continued

# ERROR (Missing value or attributes)
key03
.title = Title

# ERROR Unclosed StringLiteral
err03 = { "str

# ERROR Missing newline after ->.
err04 = { $sel -> }
32 changes: 29 additions & 3 deletions test/fixtures/crlf.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,43 @@
}
]
},
"attributes": [],
"attributes": [
{
"type": "Attribute",
"id": {
"type": "Identifier",
"name": "title"
},
"value": {
"type": "Pattern",
"elements": [
{
"type": "TextElement",
"value": "Title"
}
]
}
}
],
"comment": null
},
{
"type": "Comment",
"content": "ERROR (Missing value or attributes)"
"content": "ERROR Unclosed StringLiteral"
},
{
"type": "Junk",
"annotations": [],
"content": "err03 = { \"str\r\n"
},
{
"type": "Comment",
"content": "ERROR Missing newline after ->."
},
{
"type": "Junk",
"annotations": [],
"content": "key03\n"
"content": "err04 = { $sel -> }\r\n"
}
]
}