Implement alternative module namespacing

[dom96]

The following proposal:

# module foo

proc bar(s: string) = echo(s)

proc goo() = echo 42

# module main

use foo

goo() # Error
foo.goo() # Works
var x = "test"
bar(x) # Error
foo.bar(x) # Works
x.bar() # Works

As discussed on IRC http://irclogs.nim-lang.org/19-05-2015.html

[refi64]

Does it have to use an extra keyword? I like Haskell's notion of a qualified import better.

[Varriount]

There was a more flexible approach mentioned on IRC - allow the use of filter macros to determine what to import, and how it should be imported.

[josephwecker]

Not sure I followed the whole IRC discussion- just scanned through, but fwiw, I feel namespaces in nim are easily misunderstood by people coming from most (all?) other languages due to the combination of:

• Unified call syntax
• Overloading instead of overriding, along with matching on greatest-specificity
• The sometimes orthogonal things that need to be imported: procs =/= types =/= macros...
• Nim's awesome ability to follow the "principle of least surprise" in resolving (much more rare than one would think due to the other items) namespace collisions.

On the surface, someone scanning the language for the first time might come to the conclusion that the namespace is more or less flat: more than the idea that you can't see where procedures have come from is the unsettling feeling (learned in other languages) that you'll reach some critical mass of imports where a later import has obliterated something in another module's namespace making it unstable or (from the developers point of view) runnable but ambiguous/nondeterministic etc.

As it turns out- first, collisions are surprisingly rare thanks to overloading. Second, in those rare cases that there is a genuine namespace collision (which I've only experienced by purposefully trying to construct them) you immediately know with an "ambiguous call" or "redefinition of '...'" compile-time error.

Having said all that, I wouldn't mind a use statement as syntactic sugar for "from __ import nil"-- as a kind of the equivalent of enum's {.pure.} pragma. I just wanted to point out that the reason this becomes less of an issue in nim when you get used to it is because of the elegance of the language makes it a non-issue. That's very different than getting used to it because you have no choice- like namespaces in C. It would be nice to come up with an effective way to communicate this better in the FAQ or manual.

(BTW, it's the same as when people have hangups w/ indent-sensitive syntax. most of the time they're having bad memories of fortran (especially) or makefiles, or even Python in olden times tabs/v/spaces, or whitespace sensitive languages where it becomes possible to do simple one-liners (for passing a command to be evaluated via command-line, for example): in other words where the whitespace sensitivity led to very frustrating workarounds or seemingly ambiguous behavior. Take away the actual concerns of frustrating and/or invisible behavior and they still have the bitter aftertaste for a time. Not to change the topic- just making the point that it's important to dig into and state the "real" concern people are having- not the concern they say they have.)

[dom96]

I think that people's main concern is the fact that when reading other people's code (or even their own), they cannot easily know which module (or file) a function that is called resides in.

@Araq argues that this is (or rather will be) solved by IDEs and I agree with him. But I don't think Nim's idetools is good enough to support that yet.

[josephwecker]

Ah. I guess I got used to ruby where sometimes methods aren't even 'grep-able.' I know it's a source of frustration when first learning the language there, so to a lesser degree I can imagine a beginner wondering in nim if echo exists because of the prelude or because of import strutils or import myIO (to come up with a simplistic minimal example).

On one hand, it's analogous to type inference-- the verbosity vs discoverability tradeoff. If the developer is ever worried about it being unclear they can of course simply use the module namespace, but it's occasionally nicer to say use module as a way to imply "trust me, when I use the stuff from this module you'll know about it- so you know echo didn't come from here..." I don't think it would be as useful as someone coming from C would initially imagine, but could be a quick way to occasionally communicate a convention. That said, IMO we would want to explain it simply in the manual- so it's clear that it's sugar and not another language feature. To that end, I'd almost rather have:

import myIO {. namespaced .} # or {. fullyQualify .} or something...
# ...
type MyEnum {. namespaced .} = enum A, B, C 
# and {. pure .} would still exist to affect structure but would imply {. namespaced .}.

Pros:

• Reinforces the idea that it is to help enforce a sometimes useful convention- not add new semantics.
• Can be unified with (e.g.,) enum namespace qualification etc.
• Self documenting- even more so than use

Cons:

• Might be perceived as a hack to someone with bad experiences with namespace pollution who hasn't learned why it doesn't happen in nim.
• (I assume more... I'm just thinking out-loud here...)

[dom96]

I think a pragma may be too verbose for something that users could potentially be writing for every module they import.

[josephwecker]

Hah, true, but they're writing it to force themselves to be more verbose in the code below, so maybe they'll like it ;-)

(seriously though - I have no strong opinions here).

[reactormonk]

@dom96 The idetools should be good enough once I get #2694 done

[Araq]

Filtering based on a macro has much higher chance of ending up in the language than this proposal which I'm growing to dislike. Again, what is the purpose here? Why is echo(a, b) a bad namespace pollution but a.echo(b) is not? This is completely detached from the reality in Nim.

[Araq]

I think that people's main concern is the fact that when reading other people's code (or even their own), they cannot easily know which module (or file) a function that is called resides in.

This proposal doesn't address this concern at all, and all the other proposals don't either. And the problem statement of "I want easy code navigation without IDE-like features" is stupid enough to begin with. Why not make the language use ALLCAPS keywords for the people who have no syntax highlighing? Why not spend a tremendous amount of work on creating manpages instead of HTML for the people who do not want to use browsers? Why not optimize the language for the people who want to program on their smartphones lacking a real keyboard?

[dom96]

I think our time (for now at least) is better spent improving idetools as well as support for Nim in Vim, emacs, IntelliJ, Visual Studio, Aporia and other text editors.

I say we leave this issue open but postpone discussion until later (possibly post-1.0).

[ozra]

I'll chip in a little from my primarily C++ & JS background. The only other languages I worked with for at least 2+ or so years are Blitz Basic, 68k Asm, E, Turbo Pascal, Perl, PHP and Delphi, but I honestly don't even remember much today (10+ years ago for all of them) - so I'll refer to the former.

Bear with my slightly lengthy reasonings or.. not.

If the above would be implemented, I agree with @kirbyfan64 - the Haskell way is neater.
Buuut... Namespaces is clutter in the code imho.

In C++ I use "using namespace..." to ditch the cruft pronto, unless there are hideous clashes (a real life problem regarding this comes further down).
In JS "namespaces" require a hash-lookup, so I ditch them too, and alias when necessary, for readability (imo), writability and performance.

In both these langs, there is nothing stopping Mr Z from extending namespace A, B or whatever in his module that I'm using, so I still won't know where something is defined from this info alone, so @dom96, I don't know about the other langs, where someone feels like that.

If I want to know which module X reside in, when I'm getting to know code, I like to do it with "ctrl+mouseclick-on-the-specific-identifier" or "context-menu-key-on-id-then-go-to-definition" - so yes: IDE tools! When that doesn't work (C++ is hard for many IDEs / cumbersome to conf right) - I just google it. If I'm working via ssh and vim (does happen!) I'll just google it. Or grep. @Araq - you're right on target!

The only thing I could think of is to be able to pick, choose and exclude at will from a mod to avoid a clash where one will always use X from A, never from B.

I can imagine a scenario of myself having a clash from importing A and B, using proc foo:

import A, B
foo "Hell yeah! Foo rules, etc!" # Boohoo! Foo fight between A & B!

• getting error for ambiguity - and my mental reaction would be to:

import A, B
disregard B.foo
foo "Hell yeah! Foo rules, etc!"  # Yeay! A.foo rules!

• (where "disregard", mentally most likely would be "fuck"). So, why not, something like:

import A, B (exclude foo)
foo "Hell yeah! Foo rules, etc!"  # Yeay! A.foo rules!

Now, there is one serious problem that can arise:
As @josephwecker touches upon:

Overloading instead of overriding, along with matching on greatest-specificity

Say I've been using mod A and B. I've been using proc foo, defined in A. Now suddenly (I keep updating the mods to later versions), B implements foo. It just so happens that the args I've been passing foo are implicitly converted. And it also just so happens that the new foo in B, which is completely unrelated to foo in A, matches my args closer. So now, baaam, my code fails weirdly at run time. If that call is made not too often, I might not even have a unit test covering it (we're not perfect after all), and shit-bananas a thousand customers (I wish) starts calling at the same time when it wrecks havoc. (Granted I've personally only had this problem once in C++, it was caught - after much confusion - so no customer had to cry)

My knee jerk reaction to this is to want to be able to:

import unique A, B
foo "Hell yeah! Foo rules, etc!"  # I hope A.foo rules, but noooo!

• Baam, B implemented foo - uniqueness broken - not: since overloading says they're different, it's not broken, and B took precedance. So...

import unique symbols A, B
foo "Hell yeah! Foo rules, etc!"  # I hope A.foo rules!

• Baam, B implemented foo - uniqueness broken for real! Because we demanded symbol-uniqueness (identifier..) - ambiguity check to our spec.
• How to solve?

import unique symbols A, B (symbol not in A)
foo "Hell yeah! Foo rules, etc!"  # I hope A.foo rules!

Well, it obviously look like shit, and I don't intend it as a proposed syntax, I just want to convey what ways I'd like to be able to safe guard what's let in to the world of my module with as little specificity as possible (not having to list a thousand procs and types specifically from A and B, which would be ridiculous). Also such safe guarding is not done whimsily, but rather when code has reached such a dependency that code hardening needs to be reeal strong. But then...

Another solution, and we're once again back to tooling: rather than messing with in source demands, would be to track such "module-switches" in what identifiers refers to between builds. A sort of build-to-build comparison and verification that could warn on such issues - that would be optimal indeed! This would obviously be an external tool, but would require to have the module-reference (which foo?) information available somewhere, somehow, I don't know enough of supplementary object info from Nim yet.

And then, of course, "the right way"(TM), is to make explicit type conversions for every arg in such live-or-die code and the problem could never arise in the first place.

I'm sorry if have completely missed the point of this issue - but these are my 2mBTC anyhoo.

[Araq]

Now, there is one serious problem that can arise

Yes, this is a real issue, but I know no programming language that doesn't have this issue in one way or another, I know Ada tries to but don't remember the details.

Another solution, and we're once again back to tooling: rather than messing with in source demands, would be to track such "module-switches" in what identifiers refers to between builds.

That's a brilliant idea. Absolutely brilliant. I'll think about a possible implementation.

[dom96]

The import A, B (exclude foo) syntax is already supported:

import b except foo

http://nim-lang.org/docs/manual.html#modules-import-statement

Now, there is one serious problem that can arise:

Is this really such a serious issue? Would it not only arise at compile-time (or am I missing something)?

[ozra]

@Araq - I'm glad to hear that :-) Granted, it is unresolved in most (all?) langs, and so the better reason to fix the programming world a bit if one can. I guess, maybe if one had a -dcandidate def for making a release build while finishing up, and then use -drelease for the final 'actual release' which would do heavier lifting and produce both the binary and a "release-verification-spec"-file or something, for reference for the next release. Then one could use the tooling in a git hook, or manually or whatever, before committing an actual release.

@dom96 - Nim is just too great! :) - thanks for the except tip - I'm still reading through the manual. Well this is why it's a serious issue - it only arises at run-time, and since it "crept" in to your code unknowingly (for instance, you could've just updated dependency mods, not changing a single line in your own code), it is a "hidden danger". But once again - it's only happened to me once in 16 years of C++. Far more often I've had the problem of C++ making weird overloading priorities, like picking overloads with "bool" over "int'ish" when passing some "unsigned". But those are ofc caught at compile time - and completely unrelated.

[timotheecour]

@dom96

x.bar() # Works

this would completely break the point of use / static import and would lead to confusion when 2 bar symbols are defined in use foo1 and use foo2

Instead how about:

x.(foo.bar)()
or:
x.foo::bar()

I actually quite like x.foo::bar() as it's easy to type (no parenthesis needed), self explanatory (reminds of C++ or rust) and doesn't break the modularity implied by static import

Furthermore, it avoids the confusing notation of . meaning both field access and fully qualified access, eg:
if bar is a proc,
foo.bar => does UFCS if foo is a variable/function call, but if foo is a module name it doesn't do UFCS and instead returns the proc.
foo::bar would make the distinction clearer.

NOTE: i'm not suggesting to change the meaning of foo.bar (still would continue to work), but I'm suggesting that foo::bar can be used for fully qualified access to bar from module foo so that it can be used in x.foo::bar

[andreaferretti]

Is this proposal still open? I think that after two years and a half with Nim going to stabilize for 1.0, alternative module meachanisms can be closed

[Araq]

Indeed.