Semver integration in moduling / imports

[ozra]

I just saw #2819, and this is somewhat related issue I've had lying in a text file for a few days, so I figured I should post it now. I just got some ideas after the discussion on #2763, and this is what it is.

Tagging imports with semantic versioning

There's a thing one can borrow from npm, the node package manager (and likely a hoard of others, but since I know this one, I refer to it).
In a iojs/nodejs projects, one specifies a "package.json" file where all module dependencies are listed, with version matching patterns. Having an external file breaks DRY though, imo.
Since semantic versioning (http://semver.org/) (no matter what you think about the man behind the spec) is simple and ingeniously effective, it could be declared 'de-facto-standard' or 'strong recommendation' for Nim modules - which would be required for the matching scheme to make sense.

I propose that optional version-tagging/matching is made part of the import statement. It will not be nim's responsibility to verify it. It's up to the author And that is made easy, using Nimble (maybe? - I'm a bit in the dark about the tooling still). It could analyze the chosen nim app / root and resolve all dependencies and fetch them in accordance with specified requirements from the imports recursively.

Outdated deps

If one has deps on modules that has surpassed the version one depends on, warnings should be produced, so one can evaluate if one wants to update the code (or need to) and use the later version. This sticks to the DRY principle (the whole dependency graph can be resolved from the project source and its dependencies, so why the hell keep a package-file like for npm?)

Version Requirement Conflicts

If dependencies, and one's code, depend on the same modules with different version req's, it can be unified to an extent - as long as they stay within the same semantic version "barriers", the latest commonly acceptable version will be used, once again: helpful warnings.

When there are multiple dependencies within the project requiring incompatible versions of modules - well, that'll have to be manually resolved. It should be possible to override imported modules requirements, for dirty fixes when one know it's safe.

Optionally stating compiler version in source

Also, all languages grow, and sometimes "outgrow themselves". So why not think ahead and also allow a statement for nim version? This way Nim code from future, backwards breaking, versions can use older modules that might be stable but at that time unmaintained. The statement is voluntary, and if omitted, last version is assumed. It's up to the coders to ensure nimfix has been run. But when version is specified, nimfix can be implicitly run to generate intermediate source and knows from what version to upgrade - with strong warnings that it should be done 'permanently' by the coders of course.
Let me make a comparison: In C++ files can have a bunch of file endings, but they all mean "C++". C++11 is not compatible with older C++, still: it's just considered C++. The only way to get it right is to pass a command line flag. If the contents of a file are not compatible with what's going to process it, it should be evident from the content imo - not require knowledge from the user to figure out what version the code was written for, which might depend on a single comma changed somewhere.

The amount of work version notations incur is nothing compared to resolving dependency problems and "mysterious errors" introduced by dep-upgrades (including the compiler) - and - it's all optional!

The matching pattern

The version matching scheme could and should be made much simpler than npm's (for comparison), if semver is guaranteed standard for modules. We then know that import funkyModule@1.3.* will never break our code, even through the 'new overload introduced' problem (described in #2763 (comment)), (new features requires minor number to be increased), and we know that import funkyModule@1.* - will most most likely never break our code (breaking changes require major number to be increased).

Nim version spec, would best be done by just referencing the version used when coding, semantic versioning guarantees it will not break. Granted though, pre 1.0 is a special scenario, where the minor version, or even the patch version, can introduce breaking changes.

So, something like:

using nim 1.*   # Any _minor_ and _patch_ version of _major_ 1
import str_utils, json, someFunkyModule@1.* except foo
# the stdlib is trusted to go hand in hand with nim version

doSomeAwesomeShit()

[Varriount]

First off, I must mention that version tagging this can already be done using compile-time code. One can put a compile time constant which holds the version information, and use a static block and when statements to check that the correct version is available.

This being said, I do support the general idea that there should be some standard way to tag modules with version information. It would give external told a standard way of retrieving the version of a module. I would, however, like to see a slightly more detailed proposal of how such concepts could be integrated into the language (something like Python's PEPs), otherwise I fear that this issue will become too bogged down by minutia.

I've often wondered if such a mechanism could, along with a package manager, be used to version modules in the standard library in such a way as to lessen the impact that backwards-incompatible changes make. When such changes are made, one could simply store the previous versions of the changed standard library modules in the package repository, and have the package manager deal with things (Actually, I guess this could be done now, using nimble).

[dom96]

Please try out Nimble before making these types of suggestions. As far as I'm concerned Nimble takes care of it all :)

I could go over each point you make but it's likely best if I just let you read this: https://github.com/nim-lang/nimble/blob/master/developers.markdown

[ozra]

@dom96 - Is it to invasive of me to re-edit the whole issue post? I feel I should reduce it down to the essentials, the message gets lost in my ramblings...

[dom96]

@ozra Just post a comment here and then edit your original post to point to the comment.

[bluenote10]

Maybe I'm also missing something, but is it really possible to handle package versions individually for each project currently? When I switch between my projects, I currently also have to switch many nimble packages by doing nimble install <package>@<project-specific-version>, since they are installed globally and not locally in a project specific directory. If there is a better way to solve this, I at least could not find it in the documentation.

[ozra]

@bluenote10 - that's part of the point here - I wrote this issue way to unclear - Ill rewrite it in a few hours to clarify, need to attend som biz first..

[Varriount]

@dom96 The point here is to have a way to specify module versions that isn't dependant on another file format/program.

[dom96]

@bluenote10 @Varriount You are meant to specify package versions in your pkg.nimble file and then use nimble build to build your project. Specifying the module version per-import will be cumbersome and I don't see why you would need to do it.

[bluenote10]

@dom96 That pkg.nimble + nimble build mechanism is exactly what I was missing :). So maybe just a documentation issue. The docs mentions regarding nimble build: ... so there is rarely any reason to use this command directly. If nimble build does what I hope it does, it will become the solution for me to compile projects in Nim...

I totally agree that this is best solved on configuration level and not in the source code.

[dom96]

@bluenote10 That's good to hear. Yes, docs can always be improved, please create PRs as appropriate.

In addition to nimble build there is also nimble c which allows you to compile individual files (with the dependencies specified in your .nimble file). Admittedly I forgot to document that heh.

[ozra]

Yes, granted, the controversial part of this is obviously ditching a 'dependencies' file and typing out version requirements directly in source. You are definitely right: it is better to do the versioning of deps in a specific "project / package file" like currently - otherwise, one would have to repeat versions a thousand times in all ones files - which totally wrecks the DRY principle. So the 'versions at import' I proposed simply is a bad idea.

I still like the idea of unifiable requirements though - which requires local module copies rather than a ~/.nimble/ centralized approach. Nimble could of course cache modules in said directory, to avoid re-fetching from repos every time.

[dom96]

I still like the idea of unifiable requirements though

I'm not sure what you mean by that. But it sounds like something which others are proposing: nim-lang/nimble#114 (comment)

[bluenote10]

@ozra As far as I can tell, nimble already creates ~/.nimble/pkgs/<package>_<version>, where <version> can be both git tags or hashes, so you have full control of project-specific dependencies. As a result there really isn't a big difference between local and global: If your project requires a package version which does not exist yet, it will be created. If the specific version already exists, there really is no reason not to share it.

[dom96]

Indeed. That is precisely how Nimble works.

[ozra]

@bluenote10 - one pro, is the one of being able to have sub git-repos in the proj-dir, and just make symlinks from the local deps-mod dir to them, or to a another project one has, with a module, without out publishing, fetching via nimble. Tightly coupled development, sand-boxed is very ease, while still keeping them actually separate git-repos. If you understand what I mean?

[ozra]

@dom96 - thanks for the link - sounds very similar - reading...

[ozra]

Quickly qouting a bit from nim-lang/nimble#114 (comment), hope that's ok. Some of the exact same issues:

nimble checks out a package into /tmp and then installs in ~/.nimble/pkg/*. That is fine for things I simply depend on, but I want to develop both my own package and the packages I depend on. So I need git to manage those dependencies. nimble needs to use my git-directories for checked-out packages.

Furthermore, I might have several "contours". While working on one package, I might need one set of versions of dependencies. For another, I need another set.

[dom96]

I think that's a totally different issue which we need to discuss separately.

[ozra]

@dom96 - yes somewhat - I posted a rant in #2819 about the local pkg thingie.
The way I see it, this issue is basically closeable: the only part left of it is the need for proj-local deps - and that has more to do with #2819.
And that all modules should be required to be semantically versioned in the nimble sphere.

The rest of my suggestions have been proven moot imo.

Better to close and open a issue only for the put "down the law for semver" request, what do you think? Or is that a Nimble req already?

[dom96]

Don't think so. Open it in the Nimble repo.

[ozra]

Right - ofcourse! Thanks