a wizard / quickstart a la oasis would be cool

[UnixJunkie]

yes, I'm lazy. ;)
But seriously, it helps newcomers to get started. It lowers the entry barrier to switching to jbuilder
existing projects.

[Khady]

There is some work going on Ohai, a small tool to bootstrap ocaml/reason projects: https://github.com/jaredly/ohai

It might complete/supersede a wizard in jbuilder. I guess it doesn't help with the switching use case for now, but it's only the beginning.

[rgrinberg]

https://github.com/dbuenzli/carcass is another general purpose alternative that can be used to solve this problem as well. I'd hesitate to introduce this bloat for this reason...

[UnixJunkie]

maybe outputting the right carcass command or pointing users to it is the right way to do

[rgrinberg]

I hesitate to recommend carcass just yet as it's unreleased. If I really wanted templates for this stuff, I'd create something using http://yeoman.io/

There's a good starting template for applications/libraries https://github.com/mjambon/jbuilder-starter/ so perhaps someone can create a yeoman generator out of it? 😉

[rgrinberg]

While this issue is important, it's outside the scope of a build system like jbuilder so I will close this. If someone does indeed create some templates using a project generator. I will include them in the manual.

cc @hcarty as well who also maintains some jbuilder templates.

[shonfeder]

Just wanted to check in on this, and confirm that it's still outside the scope of this build system. It is worth noting that many other venerable build systems include this functionality (including Haskell's stack, Rust's cargo.

If this is still outside of the scope of jbuilder, is it because jbuilder is intended to be an under-the-hood tool, that should perhaps be wrapped with more user-friendly cli? Or for some other reason?

[trefis]

If this is still outside of the scope of jbuilder, is it because jbuilder is intended to be an under-the-hood tool, that should perhaps be wrapped with more user-friendly cli?

I don't know if it's because of that, but I've heard rumors of such a thing yes.

[rgrinberg]

Jbuilder isn't completely analogus to cargo and rust. Those are package managers in addition to build system.

…

On Tue, Nov 21, 2017, 12:27 PM Shon Feder ***@***.***> wrote: Just wanted to check in on this, and confirm that it's still outside the scope of this build system. It is worth noting that many other venerable build systems include this functionality (including Haskell's stack <https://docs.haskellstack.org/en/stable/GUIDE/#stack-new>, Rust's cargo <http://doc.crates.io/guide.html#creating-a-new-project>. If this is still outside of the scope of jbuilder, is it because jbuilder is intended to be an under-the-hood tool, that should perhaps be wrapped with more user-friendly cli? Or for some other reason? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#159 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIe-3kiI-UuTqdciGH0ZDzn6kPZpJR7ks5s4lFIgaJpZM4ODPGG> .

[shonfeder]

I'm wondering if we can reopen this in light of @rgrinberg comment on the discuss forum. I'd be happy to help with this effort, and even drive it, if I seems suitable for that purpose. (As I noted on the forum, I've made a tiny, dead-simple, but fully functioning tool that I've been using for personal projects here https://github.com/shonfeder/nomad .)

[rgrinberg]

You're right, I've indeed reconsidered my position regarding this topic. @shonfeder, I don't think the solution is going to be complicated, but I think we should do some upfront design first.

Also, a pre-req for this working is going to be implementing $ dune fmt as we'll need to make sure the formatting is consistent after editing user files.

Some other concerns that we need to discuss:

• Which stanzas are we going to have templates for? libraries? executables?

• Do we allow for updating the stanzas with this mechanism?

• Do we update user files directly or do we write .corrected files for promotion?

• Do we allow for users to define their templates? We already have a template language in dune, so this shouldn't be too hard to implement.

[trefis]

Regarding the pre-req: not sure you really need this if you go with an "append only" approach (see below) which I believe would be the nicest experience.

1. libraries and executables seem like a good start. The set can always be extended later on.
2. I don't think you want to update stanzas.
   I think it should be "append only" if that makes sense, that is:
   • if a dune file already exists, you append to it, otherwise you create it
   • if you're asked to setup a stanza for a library named foo but one already exists in the project, you error out
   • etc.
3. I would update users files directly with the setup described above, that seems the most natural
4. That can probably be added later on, can't it?

My justification for being append only is: it seems easier for user to then go and update the stanza if they need to than to remember which cmd line flag to pass for the stanza to be updated. It's also a lot less work for the person implementing the tool if they don't have to support updates.
We can always reconsider this choice later on if people really push for it. But I think for a first version it should be kept as simple as possible.

[rgrinberg]

The append only restriction sounds fine to me.

That can probably be added later on, can't it?

Sure. Let's just make sure we don't do something that will make it more difficult than necessary to implement this feature.

[hcarty]

I do like the idea of having dune fmt and an editing version of this feature eventually.

From a human laziness/machine correctness standpoint it could be nice to have commands like dune dep add cmdliner otherlib --to=myapp.exe to add cmdliner and otherlib to the dependencies of the myapp.exe build target.

[shonfeder]

I think I see at least 3 distinct levels of feature-sets under discussion:

Level 1:

Simple project initialization with fixed, minimal default scaffolding

This is equivalent to Rust's cargo new command.

It is inflexible, but probably sufficient to drastically drop the learning curve for new users and significantly accelerate initialization even for experienced uses.

With such a tool, beginners can get up and going with one command and a paragraph explaining where to add the name of libraries they want to use in their projects.

An approximate example of what this might look like can be seen here: https://github.com/shonfeder/nomad/tree/master/templates (Of course, the implementation should use the dune templating engine rather than that adhoc approach.)

Level 2:

Project initialization parameterized around custom templates and user configs

At this level we can include features like specifying default library dependencies, pre-processors, and
specifying custom template files, testing libraries, etc

Level 3:

Project management and updating

At this level, we'll need to edit existing dune files. This seems to have at least two levels of complexity as well.

1. Simply appending new stanzas
2. Updating existing stanzas to handle features like dune dep {add,remove}, suggested by @hcarty.

At this level we might also consider handling things like automated build system migration.

iiuc, only level 3 has a fmt tool as a prerequisite (I think this feature is discussed in #940).

---

I am happy to tackle this any way you all advise, but my natural inclination is to begin with implementing level 1, and add the additional levels incrementally. This will allows us to learn from the needs that arise from usage of the first level, and facilitate quicker turnaround of the initial feature addition. (I suspect that the 80/20 rule will favor level 1, and max out midway through level 2.)

If we do start with realizing level 1, I believe the only immediate questions to answer are:

1. What should the scaffolding look like?
2. How can we be sure the implementation is easily extended to the other features?

Speaking to (1), (again, following cargo as an example) I would expect a minimal tool to provide the option to generate a scaffold for either a binary or a library project.

• The standard directories structures should be generated for the respective project types
• The template for a library-only project could include library and test stanzas
• The template for projects that will build a binary could include library, executable, and test stanzas
• Stub-files for other standard files should be provided

I don't have any very solid ideas for (2), but keeping things loosely coupled and parameterized around the key data should help.

---

If my suggestion to focus on level 1 sounds OK, I can start putting together a sketch. If not, I'm happy to take direction for a different approach (including working on fmt first 🙂 ).

[rgrinberg]

I'm happy with level 1. But which I'd assume cover the following use cases:

• Creating a new project
• Creating a library in a directory
• Creating executable(s) in a directroy
• Creating tests(s) in a directory

As for dune fmt, I believe @diml has a concrete plan for when/how this will be added. So let's for him to chime in on this first.

[shonfeder]

That's a very helpful further analysis of the functionality, and it makes perfect sense given the way dune works. I'll dig in this week, and hope to have a draft up by next weekend.

[ghost]

I agree that dune fmt shouldn't be a pre-requisite. Basically it just means that the formatting of files generated by this new command will improve over time, which seems fine.

For dune devs, the plan for dune fmt is the following:

1. finish converting the s-expression parsing code to use the let%map syntax and not use >>= anymore
2. remove Sexp.Of_sexp.(>>=)
3. turn Sexp.Of_sexp.t into a GADT
4. add the following api to Sexp:

module Augmented_sexp : sig
   type t =
     | Atom of string
     | Quoted_string of string
     | Template of Template.t
     | List of t list
     | Fields of (string * t list) list
     | Constructor of string * t list
end

module Of_sexp : sig
  val augmented_sexp : 'a t -> Sexp.t -> Augmented_sexp.t
end

The type Augmented_sexp.t will allow to easily produce nicely formatted dune files.

[shonfeder]

I've been thinking on this for sometime now and getting familiar with the code base. I have sketched out an approach, and would like to be sure it fits with the vision of the membership before I commit the time to an initial implementation.

The main design dilemma I've been facing is whether the templates should be

1. generated from static files that more-or-less resemble their end product, or
2. if they should just be generated from pure OCaml code (cargo takes the latter approach, in which the template content is just a few inline format strings).

So far, I've been leaning towards (2) and this sketch proposes an implementation along those lines. Please don't feel obligated to study the sketch in detail if it doesn't seem to tend in the right direction. My aim here is just to be sure I have the right orientation before I put together a pull request.

Thanks in advance for the additional guidance!

The CLI

• dune init [{--lib, --bin, --template PATH}] NAME [DIR]: Create a new project in DIR, which will be either a library, a binary, or based on the provided template. The kind defaults to lib and the DIR defaults to ..
• dune add {--lib NAME, --bin NAME, --test, --template PATH [PARAMS...]} [DIR]: Add a new directory of the indicated kind to DIR. DIR defaults to ..

Defining Templates

Principles

• Composable: New templates can reuse existing templates.
• Trivial to make (if you know dune): i.e., not a new DSL

Approach

• A template is a directory with the .template suffix
• A flat file in a template with the suffix .template will be replaced
  with a template at a path given by the first line of the file.
• Any occurrences of %%NAME%% will be replaced with the NAME supplied
  at the command line.

Example

The template directory for a library project would look like this:

$ tree projects/lib.template
projects/lib.template
├── %%NAME%%.opam
├── dune-project
├── src.template
└── test.template

The first line of each of the src.template and test.template files
would point to the paths of the respective templates. Which would be
structured like so:

$ tree parts/{src.template,test.template}
parts/src.template
├── %%NAME%%.ml
└── dune
parts/test.template
├── dune
└── tests.ml

The latter "parts" would set out the templates used when adding the
respective part to an existing project.

Configuration

It should be possible to specify default library, ppx, and testing
dependencies for the stock templates. I'm not sure where such configurations
would live.

[shonfeder]

Just checking in and giving an update: I've been a bit overwhelmed for the last couple weeks, but intend to redouble my efforts and push through the initial draft of this in the next week+.

I have it in mind constantly, but recent discussion on discuss reminded me I should check in. Any feedback on my sketch above would be welcome, but I think I've worked out a relatively clear way forward in the absence of feedback any how. :)

[rgrinberg]

So far, I've been leaning towards (2) and this sketch proposes an implementation along those lines.

Doing what you think is easier seems preferable. It's just an implementation detail that we can later change.

IMO, we should hold off on user definable templates for version 1 of this feature. It's not immediately obvious to me that there's demand for this, and it complicates the implementation a bunch. Your proposed CLI looks quite good, so I say you should go ahead with implementing this feature.

[jonsterling]

I just want to issue a word of encouragement --- something like this would help me so much. I find it completely impenetrable to figure out how to set up a new project with dune and opam, and really need some tool to automate that.

[shonfeder]

I'm bumping this way up my list of priorities. Will have updates in a few days. Thanks for the advice and encouragement, @rgrinberg and @jonsterling.

[shonfeder]

Now that dune init is in, it should be pretty straightforward to implement the desired quick start command.

The dune init command added in #1448 ended up being the dune add command I proposed in #159 (comment). Thus we need a new name for the command to generate a quick-start project.

I propose dune new. This is the project creation command in Elixir's mix, Rust's cargo, Haskell's stack, and probably other tools, so we should be in reasonably good company.

I propose the following:

1. The invocation dune new [--lib|--exe] NAME will create a basic project with a layout similar to that of https://github.com/ocaml/dune/tree/master/example/sample-projects/hello_world (which seems to be fairly standard in the community).
   • I suggest dune new NAME default to the --exe behavior: this will provide the quickest route to a "Hello, World!" program for newbies, and the definition of the bin's dependency on the lib will instruct them in the most critical core concept.
2. As a subsequent feature (regardless of our decision on (2)), dune new should support basic configuration, so users can customize what dependencies etc. they want when they start projects.

IMO, this is a really important feature, as it greatly lowers the bar to entry. It is arguably more important from the perspective of

[rgrinberg]

I do not see a need for a new subcommand. $ dune init exe-project or dune init lib-project should be enough to cover this use case. We could have even $ dune init opam-exe-project or $ dune init esy-exe-project for more variety.

[shonfeder]

I feel quite strongly that that is not a nice way to move forward.

1. It convolutes the semantics of the init command considerably. We have evolved the dune init command into a quite particular and specialized command with sub commands that specify how to construct individual components of a project (libraries, tests, executables).
2. Instead of providing a nice, simple one word command for new users dune new NAME, it will force them to confront all the complexity of the relatively subtle dune init [OPTION]... INIT_KIND NAME [PATH]. And what is gained?
3. exe-project/lib-project/opam-exe-project/esy-exe-project moves us towards the very unfortunate practice of encoding logic into stringly typed compound data structures. dune new myproj --pkg=[opam|esy] should be greatly preferred`
4. Why should we be resistant to having our tool be familiar to people coming from other, larger communities? That familiarity will make the transition smoother and consequently help us grow our community of users. Meanwhile, it costs us virtually nothing to add another command.

[shonfeder]

Perhaps we should solicit input from a wider audience, and in particular from quite new users to dune and OCaml generally. I think their input would be valuable.

[rgrinberg]

1. I don't see the convolution of the semantics. Yes, we have a builtin template for every stanza. Why can't we have templates for bigger things? Is the issue that $ dune init works on existing projects? Okay if this is such an issue we can get rid of that.

2. All of those options will be required in new as well. There will be options, different project templates, and certainly a path argument to determine where to initialize the project. What is gained is that we have one rather than two commands for a single purpose.

3. I agree here. $ dune init project --pkg=[opam|esy] --kind=[lib|exe] seems better.

4. 2 extremely similar but subtly different commands seems unfriendly to me.

[shonfeder]

This helps clarify the basis for our disagreement, which seems mostly to lie in some different understandings.

1. What are the current semantics of dune init? In accordance with our design, discussions, and implementation, the documentation of dune init reads:

   dune init {lib,exe,test} NAME [PATH] initialize a new dune component of
   the specified kind, named NAME, with fields determined by the supplied
   options.

   My reading of this: the direct object initialized by dune in this invocation is a component of a dune project. The options can be used to specify the fields of the component. In my understanding, a dune project is not a component of itself (or of any other project). Moreover, a dune project does not have fields which can be set by options. This is why I think it confuses the semantics.

   If we are going to overload init in the way you are advocating for, I'd like to see an alternative semantics proposed for the command first that will help ensure dune init doesn't devolve into an amorphous "generate some stuff of some kind according to a random set of options".

2. I am quite confident that the vast majority of uses of project initialization will have the form dune new myproject, because people almost always want to create a new project skeleton in a directory named after their project in their current working directory, and they don't want to have to set a bunch of options (which is why they are using a template). By contrast, almost every invocation of component addition will have the form dune init KIND PATH OPTIONS, because it will be invoked from the project root, making a PATH necessary, and because the command does virtually nothing if you don't specify fields to create in the options.

3. Your proposed CLI here is OK, but I think it also makes the semantic conflation quite apparent. Currently, we have the schema dune init INIT_KIND NAME PATH [OPTIONS], where INIT_KIND = lib | exe | test. Your proposed change now adds a new --kind option flag which takes the same keywords as our component kinds, but interprets them differently.

   If we went this way, then a more consistent interface would be dune init [project|component] --kind=[lib|exe|test] .... But this would mean reworking the newly implemented command (and I think it is significantly more verbose and clunky).

4. This is probably the crux of our disagreement. IMO, these commands are quite dissimilar, because a component of a project is very different from a project. Consider:

   i. An invocation of dune new myproject in its default behavior would be equivalent to something like

   mkdir myproject
   touch myproject/myproject.opam
   dune init lib myproject myproject/src --libs <from conifg> --ppx <from config> --public
   touch myproject/src/myproject.ml
   dune init test test myproject/test --libs myproject,<from config> --ppx <from config>
   

   The fact that a single invocation of dune new is equivalent to two specialized invocations of dune init plus some additional operations seems to me a good indication of dissimilarity.

   ii. dune init has been explicitly designed to be able to add new components to an existing project, and to eventually support updating fields of an existing component. Neither of these makes any sense for project creation. Do they?

   iii. The options to dune init [lib|exe|bin] all effect the fields of the components, and they all have related and relatively obvious senses. But this doesn't seem to hold for project creation. What would be done with the libraries supplied in dune init project myproject --libs foo,bar? Where are they supposed to go? What would be the sense of the argument to --pkg in dune init lib mylib src --pkg=opam? The fact the the options don't even seem to make across these different uses seems to be further evidence of the difference in these commands

Please also note that I am only trying to follow through with the CLI design which I proposed explicitly and in detail back in August at #159 (comment), and which you approved of in #159 (comment). I do not understand why two commands were acceptable at that time but suddenly seems too costly.

[rgrinberg]

Instead of thinking of dune components (stanzas?) and fields, how about we change the terminology to templates and parameters. It follows that lib, exe, bin are the simplest templates that correspond dune stanzas. The parameters to correspond to common field names - that should be intuitive to most users. If it helps, in the future we'd like it so that users are able to define their own templates (like stack).

Your proposed change now adds a new --kind option flag which takes the same keywords as our component kinds, but interprets them differently.

It's just a parameter to the project template, I don't see what relation it has to the other templates.

[shonfeder]

Seems like we should reopen this issue, as the feature it requests isn't implemented.

---

I'm afraid I don't see why we would want to generalize the CLI in a way that makes it indifferent to essential differences in the sole application domain of the utility, and I feel that many of my concerns haven't be addressed.

However, I think the importance of having this tooling available for growing the ecosystem far outweighs my concerns over the elegance of the UI. So I'm willing to disagree and move forward here.

---

I will implement the project creation following

dune init project --pkg=[opam|esy] --kind=[lib|exe]

With the exception of the CLI design, I'll proceed as described in #159 (comment), unless I hear otherwise.