First class support for alternate backends

[andyarvanitis]

Has there been any discussion of being able to specify the options passed to purs in the config file? My particular use case (alternate backends) currently requires spago build -- -g corefn, which admittedly isn't bad, but it would be nice to just set it once for a project.

[f-f]

Hi @andyarvanitis! There has been no discussion about this yet, but so far the trend is that if there are commonly used compiler flags we try to take care of the more general use-case at the spago level rather than just passing them through (because the passthrough is not the best UX-wise and it introduces problems, e.g. see #353 or #216)

In this case I would be interested in investigating if we can have first-class support for alternate backends in spago, so e.g. in order to get the whole alternative build running you'd only have to add something like backend = "go" to your spago.dhall
I don't know what's the build workflow for the various backends, what are your thoughts on this?

[andyarvanitis]

That sounds fine to me, and would make it even easier for people to use the alternate backends. The Go backend would probably be a better candidate (than C++) to start with, since its tooling is simpler. I'm still making some adjustments, but I can let you know how it works and what the requirements would be.

[f-f]

@andyarvanitis thanks, looking forward to it! I'll rename this issue to track support for alternate backends then 🙂

[andyarvanitis]

Following up on my previous comment, I think I have the golang tooling tweaking done (for now, at least). In the end, the only thing you currently need to do a go-backend build (typical case) involves one more step than what I showed in this issue earlier, namely:

$ spago build -- -g corefn
$ psgo

If spago (via the project config file) can provide those purs flags and then execute psgo, the user would get a native build/executable.

FYI, psgo has a command switch of --run which doesn't build an executable, but instead runs the results of the compilation immediately. Maybe something to think about supporting for backends in general. Same goes for --no-build, which only generates intermediate language (e.g. Go) source files.

Let me know what you think.

[f-f]

Sounds great! A possible UX for all of this could be:

• adding an optional backend key to the spago.dhall, e.g. backend = "go"
• if that's set, spago build would call spago build -- -g corefn && psgo
• since psgo supports --run then we could make it so that spago run calls spago build -- -g corefn && psgo --run
• I don't have UI ideas for using the --no-build flag. We have such flag already in Spago, but that skips the build entirely. One thing I'd like to avoid is adding flags for certain backends only in Spago, so maybe we can skip the --no-build flag for now?

If the above makes sense then I have a few questions:

1. should Spago worry about fetching the psgo toolchain? And in general ensuring things like minimum psgo, purs versions?
2. is the default package set fine for use with psgo?
3. should we worry about bundling native go dependencies here or will psgo take care of that? (I'm afraid the go package management story has changed a lot since I last used it several years ago, so I have no context on this)

[f-f]

And cc @nwolverson: would this be interesting for the Erlang backend too?

[andyarvanitis]

• adding an optional backend key to the spago.dhall, e.g. backend = "go"
• if that's set, spago build would call spago build -- -g corefn && psgo
• since psgo supports --run then we could make it so that spago run calls spago build -- -g corefn && psgo --run

These all sound great to me.

• I don't have UI ideas for using the --no-build flag. We have such flag already in Spago, but that skips the build entirely. One thing I'd like to avoid is adding flags for certain backends only in Spago, so maybe we can skip the --no-build flag for now?

Definitely. My mention of it was mainly for information; skipping for now sounds good.

• should Spago worry about fetching the psgo toolchain? And in general ensuring things like minimum psgo, purs versions?

If spago is not responsible for installing purs, I'd say no for psgo too (which it seems is the case?). As for ensuring versions, maybe so. I'm happy to do what's needed to support it. For now, I think it would be fine not to check/ensure versions, especially if spago support for alternate backends can be marked "experimental" or something.

• should we worry about bundling native go dependencies here or will psgo take care of that? (I'm afraid the go package management story has changed a lot since I last used it several years ago, so I have no context on this)

Good question. psgo should be able to take care of it using Go's module system (which it now uses, and currently pulls in all needed go dependencies). It might help if I add support for something like psgo --package-set psc-0.13.0-20190626 if I start tagging the go foreign implementations of those packages.

[f-f]

@csicar congrats for the release of the Kotlin backend! 🙂

Cc'ing you to ask if you'd have any inputs about this thread, i.e. would the current description of "integration" work for pskt too?

[csicar]

First of all, I think adding support for alternative backends is a great idea.

adding an optional backend key to the spago.dhall, e.g. backend = "go"

I think just using the name of the executable there might even be easier: eg. for the go backend you would write psgo.
Spago could expect the commandline interface to be standardized: calling the executable without arguments compiles; calling with --run runs the program.
For spago that would mean, that no special support for any backend would need to be build.

The proposed changes would also work for pskt

should Spago worry about fetching the psgo toolchain? And in general ensuring things like minimum psgo, purs versions?

I agree with @andyarvanitis there: I think spago shouldn't have to care about that

Some other ideas:

• Support side-loading foreign files: PsKt and (if I remember correctly also PsGo) side-load foreign files for some standard purescript libraries (like prelude). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-set

[csicar]

btw: pskt now has the same cli interface as psgo, so no additional arguments need to be provided to compile

spago build -- --codegen corefn && pskt

[f-f]

@csicar thanks for the inputs, it all sounds great!

I have only one question:

Support side-loading foreign files: PsKt and (if I remember correctly also PsGo) side-load foreign files for some standard purescript libraries (like prelude). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-set

How would this work? Also why not just push the various .go and .cpp and .kt files to the core libraries so there wouldn't be the need for sideloading but we could just use the "official" versions?

Though I think the above is a detail, so I'd say at this point we have a pretty clear plan. Since I'm going to be short on time myself in the next weeks I'll detail here what I think it should be done, so anyone can eventually pick it up and implement it:

• add a backend key to the Config, of type Maybe Text, where Nothing is the default backend and if it's Just cmd then cmd is the alternate backend executable
• have spago build (and all the commands that build underneath) call spago build -- -g corefn && cmd
• same for spago run but passing the --run flag to the backend
• write down an "alternate backend CLI specification" in here, so that we have an interface that any backend can conform to if they want to work with Spago

[andyarvanitis]

It's been a bit of a hard sell to get non-js foreign code into the official core libraries. I can understand some of their reluctance, though, especially with more backends coming onto the scene. The approach I started taking with the second generation C++ backend and then continued with the Go backend works around this. But I wouldn't say it's a perfect solution, and some versioning provisions are still needed. See the comments on Aug 14 above for some (very basic) discussion on the matter.

As for conforming to an "alternate backend CLI specification," I'm all for it, and would be happy to modify my stuff to accommodate it. Spago has been really great to work with!

[csicar]

Having the foreign files in the original repos is of course preferable. In early development of the Backend and for alpha quality backends it is probably not desirable to upstream the foreign files to the official packages.

Though I think the above is a detail

Yes, absolutely. I don't want that to get in the way of progress. I'm looking forward to having `--watch` available by default :) Another question about the cli: I would also be interested in having a `no build` option: The kotlin compiler can be awfully slow, that means that recompilation is sometimes undesirable. Also when developing for Android, compilation is performed by the ide anyway, so that step can be skipped. I guess that's something to work out when defining the cli specification. Am 18. September 2019 19:59:25 MESZ schrieb Fabrizio Ferrai <notifications@github.com>:

…

@csicar thanks for the inputs, it all sounds great! I have only one question: > Support side-loading foreign files: PsKt and (if I remember correctly also PsGo) side-load foreign files for some standard purescript libraries (like prelude). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-set How would this work? Also why not just push the various `.go` and `.cpp` and `.kt` files to the core libraries so there wouldn't be the need for sideloading but we could just use the "official" versions? Though I think the above is a detail, so I'd say at this point we have a pretty clear plan. Since I'm going to be short on time myself in the next weeks I'll detail here what I think it should be done, so anyone can eventually pick it up and implement it: - add a `backend` key to the `Config`, of type `Maybe Text`, where `Nothing` is the default backend and if it's `Just cmd` then `cmd` is the alternate backend executable - have `spago build` (and all the commands that build underneath) call `spago build -- -g corefn && cmd` - same for `spago run` but passing the `--run` flag to the backend - write down an "alternate backend CLI specification" in here, so that we have an interface that any backend can conform to if they want to work with Spago -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #355 (comment)

[csicar]

I haven't tried upstreaming foreign code yet, but thst seems to confirm my suspicion. My idea with side loading foreign files would be to clone a foreign package set with the same tag as the primary package set that is being used. Say for example the `1.13` tag is used for the package set, then spago would also clone a foreign package set at tag `1.13` to some location in the current project - say `.spago/foreigns/psgo/`. The idea isn't refined, but maybe a starting point. @andyarvanitis could that be a solution? Am 19. September 2019 05:30:33 MESZ schrieb Andy Arvanitis <notifications@github.com>:

…

It's been a bit of a hard sell to get non-js foreign code into the official core libraries. I can understand some of their reluctance, though, especially with more backends coming onto the scene. The approach I started taking with the second generation C++ backend and then continued with the Go backend works around this. But I wouldn't say it's a perfect solution, and some versioning provisions are still needed. See the comments on Aug 14 above for some (very basic) discussion on the matter. As for conforming to an "alternate backend CLI specification," I'm all for it, and would be happy to modify my stuff to accommodate it. Spago has been really great to work with! -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #355 (comment)

[andyarvanitis]

Yes, I was thinking of something along those lines, but for the Golang backend, I'd probably need it to go through psgo itself, due to the way the go tools work. So something like a switch on psgo specifying the tag, and then it would take care of fetching the files. An example would be
psgo --ffi-tag 1.13. Do you think maybe pskt could also do the fetching in a similar way? It wouldn't be great, but it would at least make it easier on spago too.

[csicar]

That solution would also work for pskt. One think isn't clear to me yet: How would psgo know for which packages it should download the foreign files? If for example the packages "function" and "effect" are installed, would you still download the foreign files for other packages in the foreign files repo? Am 20. September 2019 07:01:36 MESZ schrieb Andy Arvanitis <notifications@github.com>:

…

Yes, I was thinking of something along those lines, but for the go backend, I'd probably need it to go through `psgo` itself, due to the way the `go` tools work. So something like a switch on `psgo` specifying the tag, and then it would take care of fetching the files. An example would be `psgo --ffi-tag 1.13`. Do you think maybe `pskt` could also do the fetching in a similar way? It wouldn't be great, but it would at least make it easier on `spago` too. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #355 (comment)

[andyarvanitis]

For now, the entire purescript-native/go-ffi repo is pulled in (which is what I was talking about tagging). If the user wants to pull in other ffi files, they would have to add them in the go module loader/config files for their project. If they want to omit some of them (in the standard libraries) from their build, they would need to comment them out in the same files.

[csicar]

Yeah, it's the same for me: pskt-foreigns contains all foreign files and all are compiled. Maybe that's fine for now, but I think installing only the packages requested by the user is a better solution long-term (at least for pskt).

Btw: --run is now also supported by pskt

[csicar]

I'm working on the implementation here: #426

[f-f]

Since the initial request was fixed by #426, I'll close this one and I spawned #434 and #435 to track support for the other things we discussed here.

Thanks for all the great input 😊

[andyarvanitis]

Sounds great, thanks!

[f-f]

You're welcome 🙂

[i-am-the-slime]

@f-f I think support for spago test is still missing, right? I don't think either of the tickets you spawned encapsulate this.

[nwolverson]

@i-am-the-slime right now spago test is ultimately following through the run workflow, which means it calls the backend with the test entry point (ie Test.Main). This works out just fine for purerl but some backends may benefit from an explicit test workflow

[f-f]

Yeah I'm totally in the dark wrt the need of other backends to have a custom test workflow, so please detail if you have more context - I think we'd be fine addind a separate path for test (rather than going through run), but we need to be careful about not hardcoding specific behaviour for this or that backend.