Provide REST API explorer

[alanz]

The fledgeling HIE has a JSON API. This includes some introspection as to what plugins are available, and what commands they provide, together with their parameters.

Provide an application that can interact with the API, and present the results in a meaningful way.

i.e.

$ stack build
$ stack exec hie -- --repl -d

Then access http://localhost:8001/ as in

curl -v http://localhost:8081/req/base -X POST -H Content-Type:application/json --data-binary '{"ideParams":{},"ideCommand":"version","ideContext":{"ctxEndPos":null,"ctxCabal":null,"ctxStartPos":null,"ctxFile":null}}'

Possibly consider using http://swagger.io, especially as there is now servant-swagger

Or any other appropriate technology. It needs to be able to run locally though.

cc @kritzcreek

[gracjan]

Pulling in a web server is a big dependency. I would advise against it.

[alanz]

HIE's dependency footprint is going to be massive anyway.

Also, this can be a frontend plugin eventually, to be configured in per user if wanted.

See #25

[kjameslubin]

Hi all, @alanz. We are interested in getting involved with this project as we have time. Commenting here because I'm in favor of a REST API. It can help with a local remote symmetry in development. As my laptop's compile times are fairly long, there's a scenario in which I could see preferring to develop on a more powerful virtualized machine. It also makes it easy to integrate in build / stage / deploy pipelines, which would likely be cloud based.

[alanz]

Sounds good, shall I add you to the project?

[kjameslubin]

Please. No promises on pushing code soon, but we want an IDE! It was nearly at the point where we were considering rolling our own - not that the existing alternatives are bad, but they weren't enough of a gain over just using emacs. It would be much better to contribute to a community project.

[alanz]

Ok, great.

You know that this is not an IDE? That said, one of the first integrations
is into emacs, and there is a lot to be done on that front still.

On Sun, Dec 6, 2015 at 10:59 PM, Kieren James-Lubin <
notifications@github.com> wrote:

Please. No promises on pushing code soon, but we want an IDE! It was
nearly at the point where we were considering rolling our own - not that
the existing alternatives are bad, but they weren't enough of a gain over
just using emacs. It would be much better to contribute to a community
project.

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

[kjameslubin]

Yes - but I am still trying to understand exactly what it is. But I think the approach of not building an impenetrable monolith is completely right. Once I put a project into something like Eclipse, it's stuck there forever, and I have to find an Eclipse plugin to use e.g. CLI tools that I'm already using.

I think the vision could be flushed out a bit though - I have skimmed the docs but I'm not 100% clear what the organizing principle would be, in a sentence.

My priorities would be:

• Organize and make it easy to use existing Haskell ecosystem tooling
• Integrate with other popular tooling as much as possible (Git(hub), Docker, Travis..)
• Separate concerns so that it is easy to pick and choose which pieces you want in your personal "IDE environment"

[alanz]

That sounds like a good agenda.

I would summarise the project as a set of mutually reinforcing goals

1. Be a focal point for the haskell ecosystem wrt IDEs / tooling
2. Provide a separation of concerns, so that experts in their various
   fields can focus on their specialities.

This means the people who can do IDE integration can do that, the tool
writers do that, the guys sorting out the nitty-gritty details of managing
a project on the disk do that.

On Sun, Dec 6, 2015 at 11:15 PM, Kieren James-Lubin <
notifications@github.com> wrote:

Yes - but I am still trying to understand exactly what it is. But I think
the approach of not building an impenetrable monolith is completely right.
Once I put a project into something like Eclipse, it's stuck there forever,
and I have to find an Eclipse plugin to use e.g. CLI tools that I'm already
using.

I think the vision could be flushed out a bit though - I have skimmed the
docs but I'm not 100% clear what the organizing principle would be, in a
sentence.

My priorities would be:

• Organize and make it easy to use existing Haskell ecosystem tooling
• Integrate with other popular tooling as much as possible (Git(hub),
  Docker, Travis..)
• Separate concerns so that it is easy to pick and choose which pieces
  you want in your personal "IDE environment"

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

[kjameslubin]

Sounds great to me, thanks.

[tobiasgwaaler]

Maybe this is old news for you, @alanz, but If we go the swagger-route, the actual API-browser could be the Swagger UI

[alanz]

Yes, I agree that swagger is probably the best approach here, in terms of
documentation too.

On Mon, Dec 7, 2015 at 12:00 AM, Tobias G. Waaler notifications@github.com
wrote:

Maybe this is old news for you, @alanz https://github.com/alanz, but If
we go the swagger-route, the actual API-browser could be the Swagger UI
http://swagger.io/swagger-ui/

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

[tobiasgwaaler]

I've looked into servant-swagger lately, and have a crude POC ready, so I'll assign myself to this issue unless @kjameslubin is also working on this?

[kjameslubin]

Please go ahead @tobiasgwaaler. I do have one comment, though - it would be nice to have a RAML based spec, as RAML is pretty lightweight and can be useful to generate live docs. See an example here - this is generated from a yesod-raml routes file and it mostly just works though we need to fix some stuff. But if you are going to actually do the work, don't let me get in the way. Can swagger take RAML as input? I seem to remember servant being able to output RAML fairly easily.

[dmjio]

Just an FYI. Servant swagger is going to release a 1.0 version soon to generically infer schema. Cc @fizruk

Sent from my iPhone

On Dec 13, 2015, at 2:01 PM, Tobias G. Waaler notifications@github.com wrote:

I've looked into servant-swagger lately, and have a (crude POC ready)[https://github.com/haskell/haskell-ide-engine/tree/servant-swagger], so I'll assign myself to this issue unless @kjameslubin is also working on this?

—
Reply to this email directly or view it on GitHub.

[tobiasgwaaler]

That's great news, @dmjio! Will the 1.0 release break the code I've already written (which isn't a lot)? I guess my question is if I should wait for the 1.0 release or start working with the current version?

[fizruk]

@tobiasgwaaler yes, 1.0 release is going to bring a lot of changes, since it's spec model will change.
It also is going to bring a considerable amount of automation (with generic Schemas).

More specifically servant-swagger is going to use swagger2 for a complete Swagger model and generic Schemas. swagger2 is where the work is going on right now, that's why you don't see any progress in servant-swagger repo. I expect to finish it in a couple of days. Once that's done servant-swagger will be migrated/rewritten (we don't expect it to take a lot of time, the first usable version may be out in a week or sooner).

If you're are impatient and want to start working with something, you can probably help us out by reviewing swagger2 (here's a small Hackage API example and here are examples of generically derived schemas). A lot of late changes are not released yet, so you might want to check out HEAD. There is some documentation, but more explanatory/tutorial docs are yet to be written. Any feedback before we roll this out would be much appreciated!

If you have any questions, you can find us on #servant IRC channel.

[tobiasgwaaler]

@fizruk thank you for informing me :) I'm looking forward to swagger2. If I get impatient I'll try to use it to generically derive the schema for haskell-ide-engine. I'll let you know if I find the time

[fizruk]

@tobiasgwaaler we've just released servant-swagger-0.1! Happy New Year! 🎄

[cocreature]

@tobiasgwaaler servant-swagger is released and #152 is merged so there should be nothing stopping you now :)

[tobiasgwaaler]

Wonderful, I'll take a look as soon as I get the chance. If anyone else is eager to work on this, please go ahead. Although I'm motivated, I'm not sure when I'll get the time :/

[cocreature]

no pressure

[ankhers]

@tobiasgwaaler Out of curiosity, have you gotten anywhere with this ticket? If not, I was going to spend a bit of time this evening to see if I can get something working.

[tobiasgwaaler]

Hi, @ankhers. I started working on this branch a while ago, but haven't done anything since the release of servant-swagger-0.1 and after the commands were moved to the type level. So I would say no, I haven't gotten anywhere ;) And I probably won't in a good while, so please go ahead! 👍

[alanz]

@ankhers @tobiasgwaaler @cocreature Is anyone making progress on this? I have some time available and want to get my brain around the tech for my day job, so I can tackle it if not

[tobiasgwaaler]

@alanz I'm not working on this :)

[alanz]

FYI, I have started here https://github.com/alanz/haskell-ide-engine/tree/swagger2

I think I am making progress, but need to chase some instances through

[alanz]

I have constructed a function hieSwagger which results in a Swagger structure when called with our servant based API definition.

The problem is, I do not know how to create an instance of ToSchema for ParamValP, which is a data type with existentials in it. I suspect putting a ToSchema constraint on the existential may help.

@cocreature any ideas?

[fizruk]

@alanz ToSchema constraint won't help as the existentially quantified type variable does not actually affect the schema AFAICT.

One straightforward way to introduce ToSchema for an existential type like ParamValP is to provide an "untyped" variant and use a Generic-based instance of that. It may be the case that you already have such an untyped representation (I am not familiar with the codebase):

-- ------------------------------------------
-- This is copied for convenience
-- ------------------------------------------
data ParamType = PtText | PtFile | PtPos

data ParamValP = forall t. ParamValP { unParamValP :: ParamVal t }

data ParamVal (t :: ParamType) where
  ParamText :: T.Text -> ParamVal 'PtText
  ParamFile :: T.Text -> ParamVal 'PtFile
  ParamPos :: (Int,Int) -> ParamVal 'PtPos
-- ------------------------------------------

data UParamValP = UParamValP { unUParamValP :: UParamVal }
  deriving (Generic)

data UParamVal
  = UParamText T.Text
  | UParamFile T.Text
  | UParamPos (Int,Int)
  deriving (Generic)

instance ToSchema UParamVal where
  declareNamedSchema = genericDeclareNamedSchema defaultSchemaOptions
    { datatypeNameModifier   = drop 1   -- we drop the U prefix
    , constructorTagModifier = drop 1 } -- we drop the U prefix

instance ToSchema UParamValP where
  declareNamedSchema = genericDeclareNamedSchema defaultSchemaOptions
    { datatypeNameModifier  = drop 1    -- we drop the U prefix
    , fieldLabelModifier    = \s -> take 2 s ++ drop 3 s } -- we drop the U in unUParamValP

instance ToSchema (ParamVal t) where
  declareNamedSchema _ = declareNamedSchema (Proxy :: Proxy UParamVal)

instance ToSchema ParamValP where
  declareNamedSchema _ = declareNamedSchema (Proxy :: Proxy UParamValP)

Another straightforward approach would be to manually write the ToSchema instances:

instance ToSchema ParamValP where
  declareNamedSchema _ = do
    sParamVal <- declareSchemaRef (Proxy :: Proxy (ParamVal t))
    return $ NamedSchema (Just "ParamValP") $ mempty
      & type_ .~ SwaggerObject
      & properties . at "unParamValP" ?~ sParamVal
      & required .~ ["unParamValP"]

instance ToSchema (ParamVal t) where
  declareNamedSchema _ = do
    sText <- declareSchemaRef (Proxy :: Proxy T.Text)
    sPos <- declareSchemaRef (Proxy :: Proxy (Int, Int))
    return $ NamedSchema (Just "ParamVal") $ mempty
      & type_ .~ SwaggerObject
      & maxProperties ?~ 1
      & minProperties ?~ 1
      & properties . at "ParamText" ?~ sText
      & properties . at "ParamFile" ?~ sText
      & properties . at "ParamPos"  ?~ sPos

BTW, it should be possible to derive this instances with TH, so if you have lots of GADTs or existentials, feel free to file a feature request!

[alanz]

@fizruk thanks for the great pointers, I will give it a go some time this weekend

[cocreature]

Writing the instances manually seems like the best way to go.

[alanz]

The servant-swagger stuff seems to be going into an infinite loop which consumes all memory.

I think the best approach may be a custom generator based on the UntaggedPluginDescriptors, as is used in the hie-doc-generator.

[fizruk]

@alanz which part goes into an infinite loop?
Can you file an issue on https://github.com/haskell-servant/servant-swagger/issues?

[alanz]

If you run this test: https://github.com/alanz/haskell-ide-engine/blob/swagger2/test/JsonSpec.hs#L204 it will terminate (and fail the test), but if you uncomment https://github.com/alanz/haskell-ide-engine/blob/swagger2/test/JsonSpec.hs#L236 (instead of the RNil on the line above) it fails to terminate.

[alanz]

@fizruk FYI I have found using the swagger2 package directly that using

declareResponse (Proxy :: Proxy Object)

for Data.Aeson.Object causes an infinite loop.

See http://lpaste.net/7300947608627838976

[fizruk]

@alanz I have just fixed the infinite loop in swagger2-2.0.2.
Can you please try that?

[alanz]

@fizruk I just tested it, and it works, thanks for the update.

[alanz]

The swagger2 package provides

declareResponse :: ToSchema a => proxy a -> Declare (Definitions Schema) Response

I would like to declare a response schema for a CommandFunc, and have the following type signature

declareCmdResponse :: ToSchema a => CommandFunc a -> Declare (Definitions Schema) Response
declareCmdResponse _ = declareCmdResponse XXXX

Could this work, and any ideas of what XXX could be? my initial approach of trying

declareCmdResponse _ = declareCmdResponse (Proxy :: Proxy a)

does not typecheck. I am pretty sure I am missing something simple here.

[fizruk]

@alanz

declareCmdResponse :: ToSchema a => CommandFunc a -> Declare (Definitions Schema) Response
declareCmdResponse = declareResponse

This should work because proxy in proxy a in the type of declareResonse is a type variable.
So it should work for CommandFunc a just like for Proxy a.

Unless CommandFunc is a type family, of course.

[alanz]

yes indeed, thanks. Was pretty sure it would be something straightforward. I am still getting my head around all this type level programming.

[alanz]

This is turning out to be trickier than I thought. Some observations so far

1. The swagger ecosystem is not geared toward passing parameters in the body of a POST. This means that even if the generated swagger JSON file is correct, the online tooling does not generate commands properly, which is the whole point of the exercise.
2. The Pos type being compound makes it difficult to pass as a parameter, there will have to be some kind of wrapping/unwrapping to its component parts, which does not play nice with the swagger2 package.

So I am contemplating making the following changes to the HTTP API (with knock-on effects)

1. Split Pos into a Line and a Col variable. This is probably good anyway, I can envisage line-oriented commands that can then be supported.
2. Pass the HTTP API parameters as normal query parameters, so the swagger ecosystem tooling can support them easily.

Comments?