parseopt: high level helper

[FedericoCeratto]

Summary

parseopt provides low-level procs and requires for+case to extract parsed values.
Nim could provide a more practical helper.

Features

• Parse args in one line and return a configuration object
• Integrate with parsecfg: Scan for <name>.cfg in known locations, read it and populate the configuration object. CLI options override the contents of the files
• Generate help automatically
• Generate help compatible with help2man (or generate a manpage)
• Support nested actions ("verbs") and contextual help e.g. foo delete cache; foo help delete
• Print application name and version as part of help
• Support parsing integers and floats
• Handle correctly 1-letter options followed by value: -o foo

[kaushalmodi]

/cc @c-blake as he might be one of the best experts in this field.

May be "something in-between" Nim stdlib and his cligen package can be ported to Nim stdlib?

(I would be happy if the whole cligen is ported to Nim stdlib :D)

[c-blake]

I think cligen (https://github.com/c-blake/cligen) covers every single requested feature (and many more!) except printing the application version as part of the auto-generated help. For versions, CL authors must add a line or two to activate and users must request via --version or whatever short option was selected. At some point I tested help2man compatibility, but it's been years. I do think just informing @FedericoCeratto of cligen is probably enough to close out this issue.

I'd be fine with inclusion in the stdlib instead of a Nimble package, but I think this runs counter to the zeitgeist of trimming the stdlib. I had originally hoped to at least get parseopt3 into some stdlib parseopt, but never pushed that hard for it. For what it's worth, I did get his 1-letter options working in the stdlib parseopt, though, but it is not "by default". The CL author must activate it by using a non-empty shortNoVal. See the module documentation.

[FedericoCeratto]

I'm aware of cligen. The main reason to have a (simpler) subset of its features in the stdlib is that CLI and config file parsing is a very common requirement for most developers and it hardly warrants adding a dependency on an external library.

[dom96]

Why not start this as a Nimble package? I 100% agree that we need something more friendly than parseopt, but I don't think whatever is developed should be included immediately in the stdlib without at least some testing as a Nimble package.

[c-blake]

I may be biased, but I feel like cligen features (and the inferred approach in general) "scale down" pretty nicely. If you only want to do something simple then you needn't know very much. As you want to do fancier things, you need to learn more. Also, FWIW, it started as a simpler package but people requested a ton of features. But if you want to take a stab at your own package, I don't want to discourage you.

[genotrance]

I'm against the idea of a parseopt3 when cligen is successful and popular. I'd rather see it added to the stdlib. It has proved itself as a nimble package.

I understand there's been discussions around removing older cruft but I don't see why we cannot continue to add mature packages into the stdlib. I've not seen any in the last two years and it honestly makes no sense.

[c-blake]

Minor point of clarification - when I said parseopt3 above, I meant what is now cligen/parseopt3.nim, and I only meant equipping the stdlib parseopt with everything cligen needs.

[Araq]

We need an RFC if and how to grow Nim's stdlib.

[luav]

I would prefer to see in the stdlib something like argparse rather then too high-level cligen because the purpose of the help is not just the API listing but rather a human-friendly explanation of the parameters, of their usage.

dispatch() of cligen is designed to generate an API for functions, which IMHO duplicates functionality of docstrings and the respective API manual, where the argument parser should provide an easy and uniform way to specify help for the console interface. argparse resembles standard and widely used Python argparse and GNU Gengetopt.

[c-blake]

I think you may misunderstand cligen. It is basically just like Gengetopt except that it's "specification language" is regular old Nim code/syntax that any Nim user ought to already know instead of some new micro-language they have to learn like a set of API calls/DSL like argparse.

Also, the most minimal usage of dispatch is not all there is. If parameter/option names are not self-explanatory, the API doc comment insufficiently descriptive, or automatic short options not quite right, the CL author has to learn dispatch(foo, help={"myOption": "human friendly explanation"}, doc="override API doc comment for help message", short={"myOption": 'z'}). Chances are, those are the only three dispatch parameters you need to know until you get fairly picky. And, yes, being an API itself, dispatch parameters are a little bit to learn, but I think less than the alternatives in the common case. Why, that single sentence above almost covers it if you already know Nim.

[luav]

@c-blake Probably, you are right, but there are some concerns related to the cligen approach. It extracts the function interface requiring to process all (mode) arguments in a single function rather than constructing the respective args object. Also, it seems to restrict some expected functionality including:

• mutually exclusive group(s) of arguments (in the same mode),
• hidden arguments (physically present arguments but excluded from the help),
• mutually exclusive modes (commands with groups of options), which seems do not support sub-commands in cligen,
• mandatory vs optional arguments (unspecified argument is not the same as the specified one but having an omitted default value),
• ...

cligen functionality is far beyond the actual CLI generation in many aspects but it also lacks some expected features and has the outlined restrictions. It provides API generation for functions being more convenient but arguably less readable and more heavy alternative to docstrings.
I would prefer to see docstrings (i.e., API manual) automation with cligen (-> apigen) rather than using it for the CLI help generation having some restrictions.

[c-blake]

Readability, convenience, preferences are subjective. Mandatory vs. optional is handled by the implicitDefault parameter to dispatch. I could probably interpret help={"param":""} as a way to "hide" the help row for param. Raise an issue over at cligen if you like.

The only main restriction you mention which is true is exclusivity/groups. Note, this is also beyond the syntax of usual procedure calls. I would say that "beyondness" calls its value into question. I'm open to possibly supporting it, but again such feature requests belong over on my git repo not here. You can always write a custom wrapper around some existing API that does exclusivity checks (which Nim importing/calling users might also appreciate!), but grouping is more syntax/parseopt behavior.

cligen does not generate a Nim callable API - the API author does that, the CL author maybe wraps that directly or writes their own wrapper. So, some of your descriptions confuse me. Anyway, you're under no obligation to like or even fully understand cligen. More power to you if you want do your own, or add to the argparse you linked to, get that hardened/more developed and do some RFC for Nim library extension both in general and for a specific module. I agree that a systematic procedure/RFC for expanding the stdlib is wise. Otherwise you wind up with 3 arg parsers in the stdlib like Python, just for example. I think most would agree that 3 is at least 1 too many.

You've mentioned doc strings a couple times when I think you mean doc comments. This suggests you are new to Nim coming from Python. Welcome! I kind of suspect what you may really be after is a kind of "Python -> Nim porting library/system" - a full suite of many modules and procs whose names, structure, and behavior mimic as closely as possible the Python stdlib/stdecosystem..This would let people change their mental models of various task decompositions as little as possible and port code/scripts as quickly as possible. Maybe even with a py2nim translator that translates to "readable/hackable Nim" code. Those sound like good ideas to me especially because 95+% of the Python I've seen does not use most of its dynamic features and is often even written more like Java. Someone has probably even already started these projects somewhere and maybe someone besides me can point you to them.

This thread is roughly of the form "stdlib parseopt not exactly what I want/expect, nor is cligen. Pick what commenter A,B,or C wants exactly and then force it on everyone via the stdlib". I think Araq is right to try to re-purpose the discussion into "how to best expand the stdlib" -- a discussion which will eventually need to be had over and over again. I don't know if he succeeded. :-)

[c-blake]

Well, I went ahead and added that requested feature of hidden/suppressed help rows over in c-blake/cligen@0cd1c5e . I agree that is common enough in other CLIs to support. One more feature to explain, of course.

[FedericoCeratto]

People are still complaining about the lack of better parseopt features in the stdlib: https://news.ycombinator.com/item?id=22401953

[c-blake]

I think the setup people were leaning toward is a distribution where certain popular packages might be "bundled" with Nim. For example, import cligen might work without people having to first learn nimble/be online/git clone& hack paths in nim.cfg/config.nims, but such packages also retain a life of their own on github/wherever. That seems to strike a good balance - making it easier for the Nim core devs to not be on the hook for maintaining these packages, retaining some package controlled primary documentation entry point, while also boosting visibility to newcomers and easing concerns about dependency availability.

If it matters, I'm planning on stamping cligen as 1.0 sometime in the next month or two unless some feature request derails that plan. It's been stable/backward compatible for a couple years now. Might still not be @FedericoCeratto's "ideal" amount of "high levelness", but it's also pretty hard to know how to please a very broad potential community about that { or most things, really :-) }.

[c-blake]

This issue thread should be cross-linked to the RFC discussion: nim-lang/RFCs#173

[ZoomRmc]

Current parseopt is a bit cumbersome, but it's workable and more or less idiomatic. However, I think something that behaves in a more common (POSIX-y) way would be a nice addition to the stdlib. Specifically:

• allowing --key val and -k val without requiring the {:,=} separators.
• short option bundling -abc == -a -b -c

Were there any efforts for inclusion of cligen/parseopt3 into the stdlib? I see it being suggested here and there but no sign of concrete actions to facilitate the inclusion.

[c-blake]

@genotrance's above two years has now become almost 6. AFAIK, fusion is long abandoned, nothing ever moved from there or stew to stdlib, and Araq still wants the stdlib on a slimming trajectory so that he need not oversee things he doesn't care much about, with plans only to shrink not grow (including his own recent malebogia -- so I am not calling him inconsistent on this). As a historical side note, I think TC & JuanCarlos pushed hard to get a few modules added since that 2019 comment, but slimming down (both manual imports of system/ stuff and just modules moved out) has probably been the main thrust and probably remains The Plan.

Is it a good plan? Dunno. I've said it before, but features/batteries included sell software and trusted lieutenants for subsystems is usually how people scale software, not delegation to a la carte package management or certain individuals managing every detail, BUT trust sure is tricky. As far as I know, not a single library package for Nim nor any "meta-installer" has made it into any Linux distro ecosystems as .rpm/.deb/.ebuild/.xps/whatever. Meanwhile, the Nim compiler/stdlib itself is commonly packaged and has been for 10+ years. So, to install anything written in Nim using beyond the stdlib, brand new users must learn nimble/atlas/nimph/nimp/NBS/whatever OR some bespoke installer for the app itself (and more significantly encounter installer failure modes) unless that installer" is just "nim c foo". This subtlely pressures program writers to be dependency allergic / make more monolithic packages with shallow dep trees with manageable manual interventions (maybe not the end of the world, but also maybe not ideal and I have personally gotten pushback). So, I think it's only a better plan for Nim adoption than growing the stdlib IF you have a rock solid build system/installer/pkg mgmt story which still feels very.. debatable? in NimLand.

NimScript is itself pretty capable for package-custom installer logic. AFAICT, the man page installation I recently did for lc using .nimble's NimScript superpowers is the first of its kind in The Nimbleverse (granted, ROFF is an archaic, hateful doc format, but the opening comment here does mention help2man which in my mind directly ties this to installation as well as ecosystem evolution). FWIW, I don't think other New Wave PLs have great stories either, but they're popular enough that system packagers pick up the slack on behalf of "ProgLang unaware/don't care users" (most users?).

To end on a more concrete note which may be more helpful, cligen/parseopt3 is small & mostly self-contained. People are, of course, free to only import cligen/parseopt3 if they want something very lightweight, or use cligen.dispatch with -d:cgCfgNone -d:cgNoColor as something intermediate weight/functionality, but still high-level like the request that opened this issue. parseopt3 is also basically backward compatible (or was/is close to it) with the std/parseopt stuff - just generalizing it and picking some different default values. So, it would not be crazy to just make cligen/parseopt3 (but with the defaults back.compat. defaults) become the new std/parseopt.

I am unaware of anyone besides @ZoomRmc who just uses that directly, though, and the request opening this was asking for almost all cligen (but with an object-oriented API w/named fields instead of named parameters). I can say, that if we do this, I would try to have newer cligen detect & use the stdlib version.

[Araq]

I'm still in favor of "batteries included" but at this point we cannot really provide it and the final nails in the coffin for me is CI and github integration in general. It's just so nice if you start with a new tiny project and it has a CI that gives you feedback in the minutes. Also nice if the github issues are all on the same topic and the issue numbers are not 5 digits long.