Fix the spec deviation in Unicode

[KavinduZoysa]

Purpose

$subject

Fixes #13180

Approach

Change the lexer to add the curly braces to Unicode.

Check List

• Read the Contributing Guide
• Updated Change Log
• Checked Tooling Support (#)
• Added necessary tests
  • Unit Tests
  • Spec Conformance Tests
  • Integration Tests
  • Ballerina By Example Tests
• Increased Test Coverage
• Added necessary documentation
  • API documentation
  • Module documentation in Module.md files
  • Ballerina By Examples

[codecov-io]

Codecov Report

Merging #20714 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #20714   +/-   ##
=======================================
  Coverage   14.59%   14.59%           
=======================================
  Files          51       51           
  Lines        1398     1398           
  Branches      214      214           
=======================================
  Hits          204      204           
  Misses       1178     1178           
  Partials       16       16

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1f82b1e...fd1c7b8. Read the comment docs.

[rdhananjaya]

do we need [{] and [}] instead we should be able to use \\{ and \\}

[rdhananjaya]

And please check what happens if an empty string is matched here and too large number matches here.
I think it will be a number format exception when parsing the int.
Shall we therefore use something like \\\\u\\{([a-fA-F0-9]{1,6})\\}

[KavinduZoysa]

do we need [{] and [}] instead we should be able to use \\{ and \\}

+1

[KavinduZoysa]

And please check what happens if an empty string is matched here and too large number matches here.
I think it will be a number format exception when parsing the int.
Shall we therefore use something like \\\\u\\{([a-fA-F0-9]{1,6})\\}

According to the spec, there should be at least one hex value. So if the user enters an empty string, parse throws an error. Also, the Unicodes which are out of range(greater than 0x10FFFF) are handled here, because we need to throw a compile error.

[Shan1024]

I think it is better to merge these two lines in this PR as well -

String text = node.getText().substring(1, text.length() - 1);

[KavinduZoysa]

We cannot do that because we need to get text.length()

[hasithaa]

Should we change HexDigit* here to avoid a syntax error?

[KavinduZoysa]

According to the spec, Unicode is defined as,

StringNumericEscape := \u{ CodePoint }
CodePoint := HexDigit+

[rdhananjaya]

Shall we generate the parser from compiler/ballerina-lang/src/main/resources/grammar/ directory

Navigate to grammar directory:
java -jar ~/Downloads/antlr-4.5.3-complete.jar *.g4 -package org.wso2.ballerinalang.compiler.parser.antlr4 -o ../../java/org/wso2/ballerinalang/compiler/parser/antlr4/

[hasithaa]

This will get fixed, Once I combine All PRs. But it is a good practice to generate parser as @rdhananjaya mentioned.

[KavinduZoysa]

ACK.