Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support typed array in parser #1796

Merged
merged 32 commits into from
Jan 27, 2019
Merged

support typed array in parser #1796

merged 32 commits into from
Jan 27, 2019

Conversation

jerch
Copy link
Member

@jerch jerch commented Nov 23, 2018

This PR moves the parser to typed array support instead of JS strings.
Changes the internal API for the parse method, the print handler and the DCS handler.

Parser speedup: runtime from 120 ms down to 45 ms (ls -lR /use/lib)

With a later change of the BufferLine API this enables the possibility to completely remove any string processing from InputHandler.print, which will give the input chain another nice speedup (>50%). This is not possible until we removed the JSArray variant of BufferLine.
It also opens the possibility to evaluate other array buffer based input encodings later on.

Part of #791.

@Tyriar
Copy link
Member

Tyriar commented Nov 24, 2018

A very impressive freak out happens when you open vim:

kapture 2018-11-24 at 5 39 48

@jerch
Copy link
Member Author

jerch commented Nov 24, 2018

@Tyriar At least it runs fast 😊

@jerch
Copy link
Member Author

jerch commented Nov 29, 2018

@Tyriar Should be fixed now, problem was that I mixed up the input data array with the result concat array.

src/common/TypedArrayUtils.ts Show resolved Hide resolved
src/common/TypedArrayUtils.ts Outdated Show resolved Hide resolved
Copy link
Member

@Tyriar Tyriar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after the String.fromCharCode.apply change is done. I think we should hold off on this until 3.9.0 goes out as it hasn't had any real testing yet.

@Tyriar Tyriar added this to the 3.10.0 milestone Nov 29, 2018
@jerch
Copy link
Member Author

jerch commented Nov 29, 2018

@Tyriar Yeah we can hold it back, also it will not show its real potential until we switched over to typed array only buffer.

@jerch
Copy link
Member Author

jerch commented Dec 12, 2018

I think we should hold this back until 3.11, it does not make much sense with the JS Array based buffer lines.

@jerch jerch removed this from the 3.10.0 milestone Dec 12, 2018
@Tyriar Tyriar added this to the 3.11.0 milestone Dec 12, 2018
@Tyriar
Copy link
Member

Tyriar commented Dec 12, 2018

@jerch sounds good, try to periodically merge in changes so the branches don't diverge too much

@jerch
Copy link
Member Author

jerch commented Jan 3, 2019

Part one - the transition to UTF32 parser buffer is done. Here are some speed numbers from xterm-benchmark:

  • master
   Context "lib/xterm_perfcases/parser.js"
      Context "Parser performance - 50MB data"
         Context "print - a"
            Case "throughput" : 1 - throughput: 52.04 MB/s
            Case "throughput" : 2 - throughput: 53.34 MB/s
            Case "throughput" : 3 - throughput: 53.04 MB/s
         Context "execute - \n"
            Case "throughput" : 1 - throughput: 39.98 MB/s
            Case "throughput" : 2 - throughput: 43.69 MB/s
            Case "throughput" : 3 - throughput: 44.55 MB/s
         Context "escape - ESC E"
            Case "throughput" : 1 - throughput: 29.48 MB/s
            Case "throughput" : 2 - throughput: 35.94 MB/s
            Case "throughput" : 3 - throughput: 36.81 MB/s
         Context "escape with collect - ESC % G"
            Case "throughput" : 1 - throughput: 11.70 MB/s
            Case "throughput" : 2 - throughput: 13.80 MB/s
            Case "throughput" : 3 - throughput: 13.88 MB/s
         Context "simple csi - CSI A"
            Case "throughput" : 1 - throughput: 28.55 MB/s
            Case "throughput" : 2 - throughput: 34.29 MB/s
            Case "throughput" : 3 - throughput: 34.54 MB/s
         Context "csi with collect - CSI ? p"
            Case "throughput" : 1 - throughput: 31.80 MB/s
            Case "throughput" : 2 - throughput: 37.38 MB/s
            Case "throughput" : 3 - throughput: 37.95 MB/s
         Context "csi with params - CSI 1;2 m"
            Case "throughput" : 1 - throughput: 41.37 MB/s
            Case "throughput" : 2 - throughput: 54.89 MB/s
            Case "throughput" : 3 - throughput: 56.10 MB/s
         Context "osc - OSC 0;hi ST"
            Case "throughput" : 1 - throughput: 27.59 MB/s
            Case "throughput" : 2 - throughput: 28.94 MB/s
            Case "throughput" : 3 - throughput: 29.04 MB/s
  • UTF32 typed array
   Context "lib/xterm_perfcases/parser_typedarray.js"
      Context "Parser performance - 50MB data"
         Context "print - a"
            Case "throughput" : 1 - throughput: 340.81 MB/s
            Case "throughput" : 2 - throughput: 413.12 MB/s
            Case "throughput" : 3 - throughput: 447.50 MB/s
         Context "execute - \n"
            Case "throughput" : 1 - throughput: 70.81 MB/s
            Case "throughput" : 2 - throughput: 84.29 MB/s
            Case "throughput" : 3 - throughput: 84.04 MB/s
         Context "escape - ESC E"
            Case "throughput" : 1 - throughput: 47.19 MB/s
            Case "throughput" : 2 - throughput: 55.65 MB/s
            Case "throughput" : 3 - throughput: 55.97 MB/s
         Context "escape with collect - ESC % G"
            Case "throughput" : 1 - throughput: 14.79 MB/s
            Case "throughput" : 2 - throughput: 15.33 MB/s
            Case "throughput" : 3 - throughput: 15.49 MB/s
         Context "simple csi - CSI A"
            Case "throughput" : 1 - throughput: 31.11 MB/s
            Case "throughput" : 2 - throughput: 39.01 MB/s
            Case "throughput" : 3 - throughput: 38.56 MB/s
         Context "csi with collect - CSI ? p"
            Case "throughput" : 1 - throughput: 43.34 MB/s
            Case "throughput" : 2 - throughput: 51.73 MB/s
            Case "throughput" : 3 - throughput: 51.43 MB/s
         Context "csi with params - CSI 1;2 m"
            Case "throughput" : 1 - throughput: 66.76 MB/s
            Case "throughput" : 2 - throughput: 111.24 MB/s
            Case "throughput" : 3 - throughput: 112.88 MB/s
         Context "osc - OSC 0;hi ST"
            Case "throughput" : 1 - throughput: 39.81 MB/s
            Case "throughput" : 2 - throughput: 43.46 MB/s
            Case "throughput" : 3 - throughput: 43.33 MB/s

Almost every testcase doubles the throughput rate, PRINT is even 8 times faster. The lame ducks are those sequences that do an interim string conversion, esp. ESC % G, which uses '%G' for a property lookup, seems multi char strings are not any good as property keys?

Up for review.

Next steps (in separate PRs):

@jerch jerch removed the work-in-progress Do not merge label Jan 3, 2019
@jerch jerch mentioned this pull request Jan 3, 2019
@jerch
Copy link
Member Author

jerch commented Jan 25, 2019

@Tyriar Did some more cleanup, ready for review.

.gitignore Outdated Show resolved Hide resolved
src/core/input/TextDecoder.ts Outdated Show resolved Hide resolved
src/core/input/TextDecoder.ts Outdated Show resolved Hide resolved
src/core/input/TextDecoder.ts Outdated Show resolved Hide resolved
src/core/input/TextDecoder.ts Outdated Show resolved Hide resolved
Copy link
Member

@Tyriar Tyriar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Onwards!

src/common/Types.ts Outdated Show resolved Hide resolved
@jerch jerch merged commit cc531f1 into xtermjs:master Jan 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants