Elm Oracle 2.0

[ajhager]

There are a few issues with the current elm-oracle program:

• Requires an internet connection + time to download documentation.json files, which might not always work and currently doesn't work behind a proxy.
• Can only use information and data provided in documentation.json files, so no file position for 'jump to navigation', etc.
• It is currently impossible to provide insight into the user's local codebase at all, only packages in elm-stuff.
• Every time elm-oracle is queried, it rebuilds data from scratch without using a cache.
• Loading up a node vm for each query can be a bit slow.
• Other functionality, like previewing docs, might make sense in a tool that is meant to provide insight into an elm codebase.
• Parse elm files instead of downloading documentation.json files.
  • Parse module (name, comment, location)
  • Parse imports (name, alias, exposing, location)
  • Parse functions (name, type, comment, location)
  • Parse types (name, args, cases, comment, location)
  • Parse aliases (name, args, type, comment, location)
  • Filter out all private values unless they are in the user's local project.
• Parse elm-package.json files
  • Use "source-directories" to gather the correct .elm files to parse
  • Also parse the project's elm-package.json file so we can provide data on the user's local code
• Store gathered data in a time-accessed based cache
• Using the cache + the imports data of the current file, generate needed data.
• Add a daemon mode that keeps running and responds to queries
• Add a command that will generate a docs preview of your project that can be opened in your browser
• Using the cache, generate tags files
  • ctags
  • etags

[mrmurphy]

This sounds super! If you find a task that seems pretty self-contained and could use some help, please let me know.

[rundis]

Great stuff !

Just wanted to chip in with regards to parsing. As I'm sure you can imagine, parsing elm code isn't all that trivial. Just something as trivial as comments caused quite a bit of work for the elm-format project. Quite a bit of work has gone into parsing in that project and I'm just wondering if it wouldn't be better if parsing and AST representation could be coordinated/reused across tooling related projects ?
I've submitted an issue to elm-format regarding this subject already avh4/elm-format#124. I'm not sure what you guys think, but in my mind it would be really awesome if tooling projects like elm-oracle and elm-format had a bit of coordination. There is merit in them being totally separate, but I think the long term benefits of cooperation/coordination would outweigh the short term benefits.

[ajhager]

UPDATE

I am still thinking about and working on elm-oracle 2.0. I have a prototype parser that is 100% regex based and even does nested comments, without needing to build an ast. This allows me to do completion on everything inside elm-stuff and your own project, without the code currently in your buffer needing to be 100% complete or correct at the time of processing. I am exploring other options as well though.

I am convinced that I need to rewrite in a different language since the elm native / elm node stuff is a bit of a hack and unstable. I would also like to have better lightweight concurrency.

Finally, I have done a lot of research on how similar tools in the haskell, elixir, purescript, and idris ecosystems function. There are so many great ideas and rooms of improvement for us.

[mrmurphy]

@ajhager I'd be very interested in helping out with this in the limited time I have available. I'm curious to know the results of your research into other tools.

What language are you thinking you'll re-write in? Haskell? OCaml?

[rundis]

@ajhager I like the some of the conceptual ideas behind : https://github.com/haskell/haskell-ide-engine
Like having "tooling server" which you can reach over tcp with a json api (also nice idea with a pluggable architecture, but that might be a tad ambitious to start off with). If it's light-weight enough it's no problem having one per elm project. I'm agnostic to implementation language, but obviously curious about what you see as the best fit. Depending on choice I might be able to contribute :-)

I'll watch this space :-)

[avh4]

What information is needed from an Elm file to support elm-oracle? My intent for elm-format is to have it read/write an AST in json format to support other tools. Would that be useful for elm-oracle? The major limitation is that it currently doesn't to type checking.

[ajhager]

That would be very helpful indeed. Right now I am using the data from the documentation.json files, so function or type name, type signature, and docs for all exposed values. It would also be nice to have the position in the file so we can easily do "jump to definition".

Will elm-format be included in the elm install in the future? I know most people are going to install elm-format and elm-oracle for editing, but the only downside I can see is elm-oracle depending on elm-format to be installed. I am writing elm-oracle 2.0 in Haskell. Perhaps I can reuse the parsing code from elm-format directly?

[avh4]

elm-format doesn't do type-checking (it might in the future, but I'd like to avoid that if possible), so types would only available for things that have explicit annotations.

But yes, elm-format will be included in the elm install--should be in the first Elm release after elm-format has a stable release.

[ajhager]

I think its reasonable to only have types for things that have explicit annotations. I would love to do some more sophisticated stuff like ghc-mod and psc-ide provide in the future, but I would need to interopt with the compiler at that point.

[rundis]

@ajhager I'm thinking of posting a suggestion to the elm-discuss list about initiating a coordinated effort to create something like a tooling server (akin to https://github.com/purescript/purescript/tree/master/psc-ide-server)

I wanted to check with you about what the status is with elm-oracle and whether you are still working on it or plan to continue doing so.

I recently moved away from using elm-oracle and made my own parser in https://github.com/rundis/elm-light

In https://github.com/halohalospecial/atom-elmjutsu @halohalospecial has gone down the regex route for parsing (and Elm to represent stuff and drive features).

It's feels wasteful that each editor should do this stuff. The parsing bits and AST representation I feel/hope would be handled by @avh4 's parsing and AST stuff in elm-format. But it would still be super useful to expose stuff on top of AST stuff. With a running server we can do stuff like the indexers implemented in both elmjutsu and elmlight which can provide an asynchronously update set of asts for your project at all times. And of course there are lots of other services we could add that would be generic to any editor.

Anyways I'm just babbling now, just wanted to hear were you are at before proceeding to see if I can contribute to the Elm tooling space in general rather than just satisfying my own desires in Light Table !

[ajhager]

Hi @rundis, I first want to say that I love watching your work with elm-light. I look forward to bringing the elm-vim plugin up to that level of quality! Your thoughts on a new tool, including using @avh4's parsing and ast code, switching to Haskell, and making the tool a server, all align with the vision I had for elm-oracle's future.

I am still interested in helping make elm editor tooling the best it can be in the future, but I certainly do not want to be a road block to development. I give you my full hearted endorsement to move forward with your new tool. Elm-oracle 1.0 was always just a hacky stop gap in order to get some IDE features into elm editors as quickly as possible. I look forward to seeing what you come up with.

Sincerely,
@ajhager

[rundis]

tx for the kind words !

I posted this thread on elm-dev: https://groups.google.com/forum/#!topic/elm-dev/uDGGTEh8ZYg
Before doing any coding (coding is the Easy part right !?), I guess there is value in doing some work in terms of

• researching what all/most editors are actually able to handle
• consider pros/cons about a server vs standalone executable
• once the AST work in elm-format/elm-compiler materializes, consider whether an incremental approach improving elm-oracle might be a better solution before going the server route

I wouldn't mind at all if we used elm-oracle as the arena to continue and coordinate this work. I like the name and if you are still interested in keeping elm-oracle going forward I see little long-term value in creating a "competing" thing.

[ajhager]

I am definitely interested in continuing elm-oracle, and I appreciate your work. I have followed the discussion on elm-dev, read the issues you posted to elm-format about the ast, and dug into elm-light a bit. I have also been playing around with https://github.com/Bogdanp/elm-ast as a simple incremental improvement to the current elm-oracle. It is probably not a long term solution.

[ajhager]

Current Goals

My current plan is to refactor oracle in Elm removing all the current pain points while staying backwards compatible.

• Remove the need to download and use documentation.json files by doing regex base statement parsing.
• Use the above to also provide info on the files in your project.
• Build json cache for everything in elm-stuff

Todo

• Refactor for Elm 0.17 and remove elm-node. All IO is done on the javascript side.
• Refactor Imports parser, including a comprehensive test suite.
• Create module declaration parser with comprehensive test suite.
• Finish porting Function parser to Elm (name, comment, signature, args) and add tests
• Port Type parser to Elm and add tests
• Add position information to each data type.
• Add toString to each data type.

Future Work

Once that work is done, I think it is important to start looking at what similar tools can do, and what we as a community need. It is possible that in the future we will use elm-ast, the parser in elm-light, a ghcjs compiled version of the elm-format parser, or just move to Haskell and use that parser directly. The right choice will come with both research and experimentation, guided by real world needs.

There is a lot we can accomplish with just the parser I am currently wrapping up. I have done some research about the functionality that similar tools can do, as well as features I have seen people ask for specifically for Elm. Below I mark the features with whether I think they are possible now or in the future. We can then discuss and come up with a reasonable feature set for oracle.

Personal Idea List

• Ctag and Etag generation: There seems to be some interest here, and it is 100% doable with what we have.
• Jump to definition: We have all the values and their position. Just need to keep track of where the modules are on disk.
• Find Usages: Doable.
• Transform all imports to Qualified: Maybe.
• Sort imports: Doable.

Quick import feature

I think this one is definitely doable. We already have the import database of the file that is being edited, as well as all the types and values that are exported from every module in the project. Since we know where in the file the imports are, how to construct a new import, and a way to turn an import into a string, we can build the new import and insert it into the file.

Context sensitive auto-completions for exposed imports

This is also doable for the same reasons as above. We already have the information needed about all the modules used by the project. We also have the source of the currently edited file. All we need is the cursor position.

Context sensitive auto-completions for import module names

Once again, we get this in a similar way to the above.

Context sensitive auto-completions in module declaration

We have all the types and values in the file, so this one is pretty simple to implement as well. It might also be possible to add a @doc reference to the module comment and make sure the newly exported value has a doc comment as well.

Purescript IDE Commands

• Load: We do this automatically.
• Type: This is possible with types and functions, but not with variables since we don't do full expression parsing or type inference.
• Complete: Already possible.
• CaseSplit, AddClause: Not sure these are possible yet.
• addImplicitImport: This is definitely possible.
• addImport: This is equivalent to the "Quick Import Feature" above.
• Pursuit: On the surface this would depend on a 3rd party service, but I have an idea about build a database using elm-oracle of everything on the package store.
• List, et. al.: All of these are possible.

ghc-mod

• Displays a list of modules: Yes.
• Displays symbols of : Yes.
• Displays information about the expression: Not yet.
• Displays the types of all expressions including the expression: Not yet.
• Finds all module names exporting : Yes.
• Finds the root directory for the Haskell file: Yeah, but not sure why?
• Finds the html document for the module: Already done.

Server?

Many editor tools have the ability to run as a server, so that all information is cached in memory and the tool can provide near instant feedback.

What else?

Let me know!

[BrianHicks]

That's a really ambitious feature set. It seems like it grows the scope of oracle significantly. I think you could split the feature set along those that require AST modification and those that don't and have a more reasonable list. The jump to definition feature also feels like an editor concern, not oracle's.

[ajhager]

Thanks for the feedback @BrianHicks! Just to be clear, I am not saying that oracle should provide all of those features, I was just posting the results of my research about other similar tools. I will update the comment to reflect that. And with jump to definition, I meant that since the data returned by oracle will now have position information in it, jump to definition would be easier to implement.

[rundis]

I like ambitious !

As a side note now RUST has also gone the language-/tooling server route: https://internals.rust-lang.org/t/introducing-rust-language-server-source-release/4209

[shamansir]

The list looks very-very cool in general, the main thing needed in my case is to know the type of any expression in user code, and preferably in a very fast way, using no internet connection, just the data available locally, and with one elm-make cycle for several expressions. And in Javascript :).

For this purpose I was required to re-implement what elm-repl does using node.js, or instead it was very slow and almost unusable. So, this repository has somewhat like a grammar for types definitions but actually it parses binary .elmi files: https://github.com/shamansir/node-elm-repl.

[AntouanK]

@ajhager Is there any update on this?

I was looking if there's a way to get elm-oracle to give me types and auto-completion for my code, not just the standard modules.

[tylerthehaas]

I would like to see this work moved forward. I understand if @ajhager doesn't have the time to work on this. I'm just wondering if I and others would be able to step in and help complete this. Thanks!

[ajhager]

I do not use Elm day to day anymore, so it has been difficult to find time to work on this. I have done some exploration though. Perhaps this will help someone else to get started on a new version.

Elm AST (https://github.com/Bogdanp/elm-ast) has matured quite a bit lately, and Elm got Platform.program in 0.18. You should be able to tear out the use of elm-node, and move the IO parts of elm-oracle to pure javascript, using ports to communicate with the Elm side. (See the README of https://github.com/lukewestby/worker that shows how you might go about that.)

Finding the current open and closed imports in the file you are completing from

A problem with elm-ast is that it chokes if there are errors in your file at the top level (it doesn't care if expressions inside functions are not perfect.) This will cause issues if you are wanting to complete modules exports or imports. You can reuse the regex code in elm-oracle for finding the imports in the current file or dig into elm-ast to write a custom parser that fails more gracefully.

Finding all exported values in elm-stuff

You can use the elm-stuff/exact-dependencies.json file to know the directories under elm-stuff/packages that are used directly by your program, then read their elm-package.json file's source-directories to find where the code will be, then use node's readdir to find all the elm files. Push them through a port and run them through elm-ast.

Finding all exported values in your project

You can use elm-package.json and its source-directories to find where code will be, then use node's readdir to find all the elm files. Push them through a port and run them through elm-ast.

Using the data returned by elm-ast

Compare your import list with the values exported by all elm packages (see the elm-oracle code now to get an idea about what this looks like. The code will be different obviously because you will be using elm-ast.) Keep in mind that the file you are trying to complete from will have some imports that are just the module name. This is the easiest case, ie. List.m(ap). You might also import values directly or all exported values from a module. You must test against these exported values directly because the user can use them without the module before it.

Completions will be the easiest thing to get going, and you may choose to stop there. It will be more work to provide the same data as elm-oracle does today, like comments, types, and links to docs. The benefits are of course that elm-oracle will be able to complete values from the modules in your own project, as well as not needing to download documentation files for packages under elm-stuff.

Future technical improvements

To speed things up, you could even build a database under the user's home directory (in json perhaps) of the packages you find under elm-stuff after you parse them with elm-ast, storing them by version. Not sure if parsing a json file oof data will be faster than elm-ast'ing packages anew each time.

You could also add an option to run elm-oracle as a process and allow the ability to send commands. This would speed up things by allowing elm-oracle to not need to startup on each request and to keep already parsed modules in memory.

Conclusion

If anyone starts working on this, I will be happy to answer any questions.

[okkero]

Could this be split up into multiple smaller, more focused issues, so that it would be easier to contribute?