Report on the creation of extensions to support AutoHotkey development

[zero-plusplus]

I use a DeepL translator.

---

2021/02/09

Currently I am developing another extension that supports all AutoHotkey versions (including H).
I have put the addition of features to this extension on hold in order to prioritize the development of that extension.

I was planning to release the first version of new extension in December last year, but I had to abandon it because the amount of work was much larger than expected.

Even though it is still in the development stage, there are more than 3,000 lines of syntax highlighting definitions (10 times more than the existing ones), more than 5,000 lines of API definitions. Eventually there will be more than 10,000 lines. These are the reasons for the significant delay in development.

I hope to release it by June if possible, but it may be postponed again.
However, I will definitely finish it, so I would appreciate it if you could wait a little longer.

As soon as that is completed, I'll implement the new features of this extension.

---

2021/04/13

I've been trying to detect symbols (classes, functions, etc.) for the past few days. As a result I realized that it is difficult to solve this without using a parser.

It takes a lot of time and knowledge to create a parser. To alleviate this, I considered using a parser generator, but gave up when I realized that it could not parse incomplete code (e.g., code when the user is typing).

So I decided to create a parser from scratch.
The reason for this decision was that there were other good extensions available, so I expected that releasing it early with few features would not increase the number of installations.
Increasing the number of installs (i.e. users) is important because feedback is necessary for development.

Once the parser is complete, I can implement many very attractive features, but as explained above, it will take a great deal of time and the release date will be far off.
Initially, I was planning to take about a year to complete the project, but my current estimate is that it may take 3 to 6 months. In other words, I aim to release it by the end of this year.

---

2021/07/27

I decided to replace the parser with a parser generator called tree-sitter.

tree-sitter is a parser generator used by Atom and other editors.
Because it is designed to be used in editors, it has features such as incremental parsing and error recovery when an unknown token appears.

The definition file can also be written in JavaScript, with support for relatively easy parsing of expressions, which can be complicated.

It has syntax highlighting capabilities, which, if supported by VSCode in the future, will eliminate the double management of syntax highlighting and parser definitions.

As the next generation parser, I am very hopeful that it will become a future standard and decided that taking the time to learn it would be the best way to proceed with this project.

---

2022/03/02

It's been a while since I've updated, but I haven't missed a work session.

I have not been able to work on the AutoHotkey parser because I have been busy updating this extension, but I have been working on it since 12/24 when I updated 1.10.0.

The work has been very difficult due to the difficulty of handling tree-sitter and the quirky syntax of AutoHotkey.
Currently, I am working on reorganizing the code because it was very unstable, although it is about 50% complete.

I hope to have it done by April, but there will easily be delays, as it can take up to a week to resolve a single issue.

---

2022-03-11

The parsing rate of my own library (10k lines) for AutoHotkey v1 is now 100%.

It is not yet complete, as it needs to implement some syntax, such as hotkey label statements that are not used in the library, but it is very close to completion.

After completion, I plan to implement syntax highlighting and intellisense using the parser.

After that, I will create a v2 parser.

---

2022-04-09

Added missing syntax such as hotkey labels.
In addition, a non-error but mis-parsed section has been corrected, and tests and parsing samples have been added.

After four months of work, the parser work can finally be completed.

My plans for the future are as follows

1. adjusting the previously created syntax highlighting. Also add more detailed highlighting (semantic highlighting) using the parser's syntax tree.
2. create minimum API (built-in functions, etc.) data (using previously created data)
3. public preview release
4. add API data while creating new features, etc.

There is still more work to be done, but this is a multi-year plan, so please be patient.

---

2022-09-30

I am currently suspending development of this extension due to a problem that cannot be resolved.

Therefore, I have spent the last 5 months or so creating a previously planned AutoHotkey library to easily define complex hotkeys.

This is a project that I had planned for several years and had been put on the back burner by the creation of vscode extensions.

This is unfortunate news for those of you who were hoping for an extension, but since I needed a large sample library to create syntax highlighting and parsers, I took this opportunity to create one since development had stopped.

The library is almost complete, with only proofreading of the documentation and final testing remaining.

I plan to release it within the next month and get back to updating vscode-autohotkey-debug, but my plans tend to lag behind, so don't hold your breath.

---

2023-07-23

Development is still ongoing, and with the parser nearly complete, the type system can now be implemented.

This type system takes into account Include statements. That is, if you include a file with class B in a block of class A, the type system will understand the structure A.B.
However, in order for the completion process to work correctly with a file of class B, the file with class A must be configured as root.

Thanks to treesitter's incremental analysis, the analysis speed is 30 ms for 3000 lines. (Only the first analysis takes about 5 times longer.)

If the number of lines is 8000, it will take a little longer, about 100 ms, but this can be solved by splitting the file.

The split file will not be reanalyzed unless it is modified. Therefore, including large libraries will not slow down the parsing speed of your scripts.

Since it takes an enormous amount of time to complete this type system, I will try to speed up the release of the extension by setting as small a goal as possible.

So far I have been able to implement automatic type determination for expressions and functions and type determination by jsdoc-like some annotations, which is quite useful.

If there are any features you would like to see incorporated, please let us know here.
I will report back here if anything else comes up.

[SAbboushi]

Thanks for the update. Looking forward to it!

[SAbboushi]

I think this is good news! This looks like a tremendous amount of work - I wish that you will be rewarded and that lots of AHK users will adopt your new extension.

Can you share more details about your extension's Built-in API and User-defined API? What is the purpose? Is this vscode-specific?

[zero-plusplus]

Thank you.

I used API as a generic term for variable, function, etc. I've changed the description to be more clear.

Using those definitions, can do something like the following image. This is a demo of Autocomplete and Signature Help.

[CA10C3SAR]

Hello, does this auto complete exist in your current extension?

[zero-plusplus]

The above features are not exist in this extension. This is part of an extension that is currently under development (non-public). Development is going well, but due to its large scale, it will be released in a few months at the earliest.

[SAbboushi]

Wow! The demo above looks great! Keep up the good work!

[FuPeiJiang]

hi, I just discovered your extension
I've been trying to write an AHK_L v1 parser too : https://github.com/FuPeiJiang/ahk_parser.js
demo here: https://fupeijiang.github.io/ahk_parser.js/
, enter AHK_L v1 and you will see the parsed array of objects
feel free to ask any questions

this parser is not following any standards, I was too lazy to see how javascript parser worked
this parser is not complete, I just made it complete enough to convert to AHK_H v2

I have not done Gui,
I have no clue how to deal with continuation sections like this:

msg
(
box
) hello world

I think this REQUIRES preprocessing
but then I wouldn't be able to show line number and column number

can you tell me what features you need in a parser ?
I can try to add them

---

I just learned about incremental parsing, this should be fast, if it is possible to do so on autohotkey

I'll look into tree-sitter

[zero-plusplus]

I use a DeepL translator.

Hi, I've seen your project before.
The reason I'm working on the current extension is to create a large library for v1 and v2.
So I am very hopeful that I can automate the conversion for v2 as it will greatly shorten my planning.

---

It looks like your issue can be solved by adding look-ahead processing.
That is, after parsing the msg, it checks if the next section is a newline + opening parenthesis.
If find that combination, can treat it as a continuation section.

This solves the issue in many cases, but sometimes it breaks down when the next line is a parenthesized expression.

For example, in the following case.

MsgBox,
(123 + 456)

This is basically uncommon source code, so you can ignore it.

---

I plan to complete the parser using the tree sitter in about a month, but it may take longer due to lack of documentation.
I think the release date will be done at the same time as the extension, so it will be much later.

---

I would also like to introduce lezer, which was influenced by tree-sitter.
If you are interested, you may want to look here.

[helsmy]

Hi! I am also wirting a autohotkey supporting extension Autohotkey Simple Support. And ,you may be seen before, a debugger extension Autohotkey debug.
After complete all most all features that regular expression can do, I am going into development syntax parser for autohokey. I have all most completed the parser(in pure typescript around 3k-4k lines). I am now testing it and plan to go into semantic phase. If you are interested in it, you can check the dev-parser branch of my extension.

[helsmy]

Hi, it is me agian. I have wrote the parser for around 60-70% and I thought it could be in alpha stage. You can try it by download the newest extension.

[zero-plusplus]

Thank you for your hard work.

Of course I was looking forward to it and tried it right after the update. With the recent update, the parser seems to be working well.
The best thing is that it runs fast even with 10,000 lines of source code.

I'm tempted to restart the parser creation process. I'll start working on it after I finish the current work.

[helsmy]

I am happy to see that. Code I am writing is not aiming to high preformence. My goal is to run it fast enough on files around 2000-3000 lines. As a result, javascript is fast enough to handle a parser for Autohotkey even with quite a bit of waste on preformence. I am looking forward to your tree-sitter parser, good luck and happy coding.

[helsmy]

By the way, if there are any api needed for debugger, just open an issue for it, I will try to write code for it.

[zero-plusplus]

I'll do my best.

I don't plan to update it for a while after the next release, so I don't need the API for now. thank you.