Elm Oracle 2.0

[ajhager]

There are a few issues with the current elm-oracle program:

• Requires an internet connection + time to download documentation.json files, which might not always work and currently doesn't work behind a proxy.
• Can only use information and data provided in documentation.json files, so no file position for 'jump to navigation', etc.
• It is currently impossible to provide insight into the user's local codebase at all, only packages in elm-stuff.
• Every time elm-oracle is queried, it rebuilds data from scratch without using a cache.
• Loading up a node vm for each query can be a bit slow.
• Other functionality, like previewing docs, might make sense in a tool that is meant to provide insight into an elm codebase.
• Parse elm files instead of downloading documentation.json files.
  • Parse module (name, comment, location)
  • Parse imports (name, alias, exposing, location)
  • Parse functions (name, type, comment, location)
  • Parse types (name, args, cases, comment, location)
  • Parse aliases (name, args, type, comment, location)
  • Filter out all private values unless they are in the user's local project.
• Parse elm-package.json files
  • Use "source-directories" to gather the correct .elm files to parse
  • Also parse the project's elm-package.json file so we can provide data on the user's local code
• Store gathered data in a time-accessed based cache
• Using the cache + the imports data of the current file, generate needed data.
• Add a daemon mode that keeps running and responds to queries
• Add a command that will generate a docs preview of your project that can be opened in your browser
• Using the cache, generate tags files
  • ctags
  • etags

[mrmurphy]

This sounds super! If you find a task that seems pretty self-contained and could use some help, please let me know.

[rundis]

Great stuff !

Just wanted to chip in with regards to parsing. As I'm sure you can imagine, parsing elm code isn't all that trivial. Just something as trivial as comments caused quite a bit of work for the elm-format project. Quite a bit of work has gone into parsing in that project and I'm just wondering if it wouldn't be better if parsing and AST representation could be coordinated/reused across tooling related projects ?
I've submitted an issue to elm-format regarding this subject already avh4/elm-format#124. I'm not sure what you guys think, but in my mind it would be really awesome if tooling projects like elm-oracle and elm-format had a bit of coordination. There is merit in them being totally separate, but I think the long term benefits of cooperation/coordination would outweigh the short term benefits.

[ajhager]

UPDATE

I am still thinking about and working on elm-oracle 2.0. I have a prototype parser that is 100% regex based and even does nested comments, without needing to build an ast. This allows me to do completion on everything inside elm-stuff and your own project, without the code currently in your buffer needing to be 100% complete or correct at the time of processing. I am exploring other options as well though.

I am convinced that I need to rewrite in a different language since the elm native / elm node stuff is a bit of a hack and unstable. I would also like to have better lightweight concurrency.

Finally, I have done a lot of research on how similar tools in the haskell, elixir, purescript, and idris ecosystems function. There are so many great ideas and rooms of improvement for us.

[mrmurphy]

@ajhager I'd be very interested in helping out with this in the limited time I have available. I'm curious to know the results of your research into other tools.

What language are you thinking you'll re-write in? Haskell? OCaml?

[rundis]

@ajhager I like the some of the conceptual ideas behind : https://github.com/haskell/haskell-ide-engine
Like having "tooling server" which you can reach over tcp with a json api (also nice idea with a pluggable architecture, but that might be a tad ambitious to start off with). If it's light-weight enough it's no problem having one per elm project. I'm agnostic to implementation language, but obviously curious about what you see as the best fit. Depending on choice I might be able to contribute :-)

I'll watch this space :-)

[avh4]

What information is needed from an Elm file to support elm-oracle? My intent for elm-format is to have it read/write an AST in json format to support other tools. Would that be useful for elm-oracle? The major limitation is that it currently doesn't to type checking.

[ajhager]

That would be very helpful indeed. Right now I am using the data from the documentation.json files, so function or type name, type signature, and docs for all exposed values. It would also be nice to have the position in the file so we can easily do "jump to definition".

Will elm-format be included in the elm install in the future? I know most people are going to install elm-format and elm-oracle for editing, but the only downside I can see is elm-oracle depending on elm-format to be installed. I am writing elm-oracle 2.0 in Haskell. Perhaps I can reuse the parsing code from elm-format directly?

[avh4]

elm-format doesn't do type-checking (it might in the future, but I'd like to avoid that if possible), so types would only available for things that have explicit annotations.

But yes, elm-format will be included in the elm install--should be in the first Elm release after elm-format has a stable release.

[ajhager]

I think its reasonable to only have types for things that have explicit annotations. I would love to do some more sophisticated stuff like ghc-mod and psc-ide provide in the future, but I would need to interopt with the compiler at that point.

[rundis]

@ajhager I'm thinking of posting a suggestion to the elm-discuss list about initiating a coordinated effort to create something like a tooling server (akin to https://github.com/purescript/purescript/tree/master/psc-ide-server)

I wanted to check with you about what the status is with elm-oracle and whether you are still working on it or plan to continue doing so.

I recently moved away from using elm-oracle and made my own parser in https://github.com/rundis/elm-light

In https://github.com/halohalospecial/atom-elmjutsu @halohalospecial has gone down the regex route for parsing (and Elm to represent stuff and drive features).

It's feels wasteful that each editor should do this stuff. The parsing bits and AST representation I feel/hope would be handled by @avh4 's parsing and AST stuff in elm-format. But it would still be super useful to expose stuff on top of AST stuff. With a running server we can do stuff like the indexers implemented in both elmjutsu and elmlight which can provide an asynchronously update set of asts for your project at all times. And of course there are lots of other services we could add that would be generic to any editor.

Anyways I'm just babbling now, just wanted to hear were you are at before proceeding to see if I can contribute to the Elm tooling space in general rather than just satisfying my own desires in Light Table !

[ajhager]

Hi @rundis, I first want to say that I love watching your work with elm-light. I look forward to bringing the elm-vim plugin up to that level of quality! Your thoughts on a new tool, including using @avh4's parsing and ast code, switching to Haskell, and making the tool a server, all align with the vision I had for elm-oracle's future.

I am still interested in helping make elm editor tooling the best it can be in the future, but I certainly do not want to be a road block to development. I give you my full hearted endorsement to move forward with your new tool. Elm-oracle 1.0 was always just a hacky stop gap in order to get some IDE features into elm editors as quickly as possible. I look forward to seeing what you come up with.

Sincerely,
@ajhager