New performance graphs for a couple of grammars

[kaby76]

I've been composing some graphs of grouped parse time vs target type to better understand performance. The graphs are computed using two scripts using the latest Trash Toolkit.

The first script written for Bash computes the raw data describing runtimes for a grammar, for all the tests in examples/, and with grouped parsing, e.g., java/java/ and sql/postgresql/. te.sh.txt. It simply loops through all assumed targets, builds the target driver, runs a standardized performance script 5 times using "group" parsing each time. "Grouped" parsing performs a warm-up of the parser using the first file, and accumulates additional cache information with each subsequent parse. The output generated by the test also contains the names of the grammar and target, and information about the environment and machine. The PHP target is not tested because it is extremely slow, but I couldn't update my environment for PHP. The CSharp target does an extra step to determine the ambiguities of the grammar. Raw data: data.zip

The second script, written for Bash and Octave, extracts the runtimes and computes a bar graph with standard deviation error bars, and a legend that describes the targets for the graph. gr.sh.txt.

Here are the graphs of the performance for the two grammars.

Some observations about the tests.

• Dart is the fastest of all targets tested.
• Python3 is the slowest of all targets tested. (We haven't tests for PHP, which is probably the slowest, and Swift, which I can't test.)
• Most targets are more or less similar in speed with each other as compared to the Python3 target.
• It is not clear why the standard deviation for Java is so large compared to the other targets.

[kaby76]

For comparison, I've been porting the mysql/Oracle grammar to the rest of the targets. Here's a preliminary graph of the performance of the grammar.

On the large SD of the Java ports, the first run of Java for the driver app incurs a large penalty. This, apparently, is caused by anti-virus checking and disk caching.

[mike-lischke]

Funny, you list antlr4ng as own language. It's TypeScript! What you have as TypeScript target is the JS target. The only TS code there is the generated files. I'm surprised there are differences in the performance of the two columns JavasScript and TypeScript.

btw. can you make the graphs larger please?

[kaby76]

There's a difference between Antlr4ng vs. "Official Antlr with TypeScript" vs Antlr4ts vs. "Official Antlr with JavaScript." It just a name to identify the pairs of tool and runtime. We all know that Antlr4ng is in TypeScript and it uses a patched version of the .jar. Otherwise what would you have me call it? "Mike's patched tool .jar of Official Antl 4.13.2 with a rewritten TypeScript runtime Antlr4ng" and "Official unaltered tool .jar Antlr 4.13.2 with unaltered Official Antlr TypeScript 4.13.2 runtime?" You yourself call Antlr4ng "Antlr4ng."

The diff between Official TypeScript and Official JavaScript is attributed to the way it's called from the command line. I had to use npx tsx because ts-node stopped working. The TypeScript and JavaScript environments are evolving constantly asnd I can't seem to keep a stable environment because old releases are being deleted. All the scripts are in the Trash trgen directory.

[kaby76]

Actually, thinking about it, "target" really is the wrong phrase. I should use "port." A "target" refers to the programming language used in the generated code. A "port" refers to the programming language targeted, the tool and runtime.

[mike-lischke]

Otherwise what would you have me call it?

I would actually not mention that "TypeScript" target from ANTLR4 at all. It's mostly the same as the JavaScript target. Given that the decision would be easy: use TS for antlr4ng. The only place were it makes sense to use the name of the library is, when you compare multiple libraries for the same target language (like I did for antlr4ng, antlr4ts, antlr4 and antlr4wasm).

Anyway, you can name it like you want, it's your submission :-)

... because ts-node stopped working. The TypeScript and JavaScript environments are evolving constantly ...

It's surprising to me you have so much trouble with that (I probably would have the same with Java...). To make it easier for you I can only recommend to look at the NPM scripts in package.json, for example "run-benchmarks": "node --no-warnings --experimental-vm-modules --loader ts-node/esm tests/benchmarks/run-benchmarks.ts", it shows how to use Node.js directly with ts-node as loader to execute a TS script. This works since a long time and does the job very well. This stuff didn't change (much) over the last 1-2 years. You can even use it with older Node.js vesions (18.x, 20.x).

When you need help with that just open an issue or a discussion in the relevant repo and I'll be there to support you.