[Design] import statement, for qualified access to symbols

[lydia-duncan]

This is a proposal to support enabling qualified access to symbols in a new scope.

Background

In Chapel today when a module is included in a program, its symbols are always available for qualified access. However, there are some inconsistencies about when a module is included in a program - modules with the same name as the file in which they are defined can be referenced and found without explicitly listing the file in the compilation command, if the file is in the same directory as a source file that is already listed. There's been a desire among the community for a way to ask for only qualified access.

Main Feature

This proposes an import statement for symbol visibility. When used with a module name, it allows the public symbols within that module to be accessed using full qualification, similarly to the behavior of use <module> only; or use <module> except *;. E.g. for a module Foo with a variable bar, import Foo; will allow bar to be accessible using Foo.bar.

import statements can list multiple modules, e.g. import A, B;. Import statements, like use statements, are all considered to occur at roughly the same time, regardless of their place in the scope (though an import can allow another import to occur, e.g. import Foo; import Foo.Bar;). A qualified access of a symbol could be written “before” the import statement that allows it in a scope, e.g. M.something(); import M;.

Unless the module being used for qualified naming is visible to the scope in which the naming is occurring, it is expected that import statements will be necessary for qualified naming going forward. This means that the addition of the import statement would be a backwards compatibility breaking change. (Edit: this is no longer a backwards breaking change if use statements still enable qualified access, as we have already disabled qualified access when no use statement is present)

• Note: this doesn't have to mean that use statements no longer enable qualified access. There is precedent for unqualified access being linked to qualified access (see Haskell chart).

Proposed Extensions

The following extensions are not necessary for an initial implementation and will not break backwards compatibility if they are added later. However, I believe we will want to implement them before considering this feature complete.

• imports should be able to be declared either public or private. When public, fully qualified naming will be available to any module that uses the module with the import statement. Fully qualified naming will not be available to modules that import the module with the import statement. (In Python, import chaining causes a hierarchy, so for Foo that imports Bar, you would need Foo.Bar.baz to utilize Bar.baz. This seems confusing to me given that we already use . for nested modules – looking at a qualified call, you wouldn’t be able to tell where the definition of Bar.baz was). If an import is private, modules that use the module with the private import will have to perform their own import to utilize the symbols with full qualification.
  • Note that if we do not add privacy settings in the initial implementation, we should determine the default visibility. Changing the default visibility when privacy settings are added would be a breaking change if we go from public to private.
• A module should be able to be renamed in an import statement. This will be done using the as keyword. For instance, import MyModule as M; will allow the symbols defined in MyModule to be accessed using only M. as the qualification - MyModule.foo will not be legal unless another import of MyModule is present that does not rename it. Every module being imported in the statement can be renamed, e.g. import MyModule as M, YourModule as Y;.
  • (Edit: this is no longer true as of Allow renaming a module on its use statement #14714, which added renaming to use statements) Note that use statements do not current support renaming the module itself. If we decide that the import feature means that use statements should no longer also enable qualified access, there is no point to enabling renaming of the module in a use statement. If use statements continue to enable both qualified and unqualified access, then I would move to add renaming to use at the same time.
    • Note also that users would still be able to accomplish the renaming via import MyModule as M; use M;, and as a result adding renaming to use would not necessarily be as high of a priority. It just seemed easy to accomplish at the same time.
• except and only lists would be an add-on feature not essential for the first implementation. If they are included, it is expected that they would limit the symbols that can be accessed using qualification - import Foo only bar; and import Foo except baz; would both prevent Foo.baz from resolving, while enabling Foo.bar to work. import Foo only; and import Foo except *; are not valid statements (we could alternatively allow them but warn that the operation is a no-op). only lists also allow renaming of the symbols being accessed in a qualified manner, similar to its use in use statements. It is possible to rename both the module and a symbol to access from it in the same statement, e.g. import MyModule as M only foo as f; allows M.f.
  • With except and only lists, it should be legal to import the same module multiple times, with different privacy settings or except/only lists. It might be nice to warn if something brings a symbol in multiple times in the same way.

Edge Cases

The following are cases to be aware of during implementation.

• Point of instantiation on symbols that are accessed using qualified naming only. If the function is pulled to a location due to fully qualified naming, it should still be able to call all functions that were local to its original definition (I don't expect this to be different than the current world but I'm not sure we have a test of point of instantiation and qualified naming)
• Importing a module with a class/record in it should still allow the initializer for that type to be called, like it does today (i.e. the path taken for import should not break this) Calling constructor of a class within a module requires explicit importing of the module #6483
  • Relatedly, methods should also still be callable: "use only" shades class methods #7857
  • And importing with just the type should not interfere with initializers, deinitializers, methods, writef instantiations using the type, etc. Error when importing a class definition with use <modname> only <typename> #12298

Open Questions

• Should the presence of import as an option mean that qualified access is not enabled by a use statement? Put another way, for a module M with a symbol x, should use M; mean we can write both M.x and x or should it mean we can only write x?
  • I do not feel strongly about this. On my own, I would probably not change the current behavior of use.
• Should the privacy of import and/or use impact the declared privacy of a module?
  • Given that there can be multiple import/uses of a module even in the same scope, I am against allowing the privacy to be altered by these statements.
• Should the import of a module be required prior to any use of it?
  • I think that would be annoying
• Should import be able to be applied to an enum, like use is?
  • This seems less useful than use - I would guess that in practice use myenum; happens more often on enums that are defined by the current scope, and qualified access of these enums is already possible. use M only myenum; would be equivalent to import M.myenum; and is slightly less characters, though.
    • As a result, I view this as much lower priority than any of the other extensions listed.

Precedence

• Python: The module being imported can be renamed using as. from a import b makes b available as b instead of a.b. from a import * brings all symbols from a into scope with unqualified access. Symbols that are imported can be referenced as if they were defined from the scope of the module being imported (even if they were made available in that module via an import).
  • When from is used, only the symbols specified are available (so from foo import bar doesn’t mean foo can be referred to, just bar)
• Julia’s using statement functions like our use statement, so far as I can tell. using Mod: x, y is equivalent to use Mod only x, y;, though Julia does not allow you to extend types obtained in this way (no secondary methods, though maybe you can still do that with the original module’s full path, e.g. Mod.x.whatever?). Julia also has an import keyword, which is used to bring one symbol into scope only, and that symbol can be extended. Explicitly bringing in a symbol with using or import means the module cannot define a symbol with that name.
  • I believe the fully qualified version of a symbol is only available when the module is made available via using or import without naming specific symbols. It seems the way to make just it available on its own is to import it.
• Rust allows symbols to be referenced in sibling modules via the full path. use serves to shorten a path for the referenced symbol - use a::b::c; allows unqualified access to c and qualified access to c’s symbols without a::b:: beforehand. You can add unqualified access to both b and c using use a::b::{self, c}; and unqualified access to all symbols in b using use a::b::*;. It allows relative uses as well (we allow you to access symbols in higher scopes for use statements, so this isn’t necessary).
• I had made a chart comparing Haskell's import command back when we first added only and except lists, at Brad's suggestion. This is an updated version of that chart for the current proposal (I don't want to try editing it for good formatting here)

[lydia-duncan]

Conversation with Michael:

Fully qualified naming will not be available to modules that import the module with the import statement.

This seems surprising to me. Why would this be enabled for use but not import? [...] I think my main objection here is that... there seems to be some criticism of re-exporting the module name, but we'd have that for use anyway? So why are we making anything much better by avoiding it for import?

Me:

The basic gist is that import brings in a single name and then allows you to look a little deeper using that name. But symbols that were made available to that module via an import aren’t accessible via that name - they’d be accessible via the name used to import them, or would follow Python’s pattern of <imported name>.<other module outside the hierarchy>.<symbol> and that seems really confusing to me

[lydia-duncan]

Following up a little more on that thought: I'm more against the idea of <imported name>.<other module outside the hierarchy>.<symbol> than I am against <other module outside the hierarchy>.<symbol> being made available due to the import of <imported name>, given the reason listed in the Python precedent.

[mppf]

Following up a little more on that thought: I'm more against the idea of .. than I am against . being made available due to the import of , given the reason listed in the Python precedent.

Doesn't this just push the problem down one level in the hierarchy (and so not really solve the problem so much as kick the can down the road)?

I would expect the author of A to control whether or not A.B.someFunction() is available by public import B; or private import B; rather than by using a use. If A had a use B; and it followed your suggestion, wouldn't we have the same problem, in that A.B.someFunction() would refer to the imported module rather than a submodule?

I think it's desireable than a module author can provide the same interface with a public submodule or by an public import/use.

[lydia-duncan]

If A had a use B; and it followed your suggestion, wouldn't we have the same problem, in that A.B.someFunction() would refer to the imported module rather than a submodule?

I'm generally against using . to reference to a hierarchy that doesn't involve the actual symbol definitions. If module Bar is not defined in module Foo, I don't want to be able to type Foo.Bar at all. It doesn't matter to me whether Bar was imported or used by Foo. If Bar was imported by Foo, I am okay with a use or import of Foo making Bar.someFunction() available, but not with making Foo.Bar.someFunction() available. I would probably prefer that import Foo; means that import Bar; is also necessary, but I can live with that being considered too restrictive.

I'm going to try writing some examples to help clarify that.

Case 1: Bar is used by Foo

module Foo {
   use Bar;
}
module Bar {
  proc someFunction() { ... }
}

1a) Another module imports Foo

import Foo;

// I don't like this at all
Foo.Bar.someFunction();
// I'm not a fan of this because there was never an explicit import of Bar,
// but I like it way better than the previous line.
Bar.someFunction();
// Obviously this shouldn't work.
someFunction();

1b) Another module uses Foo

use Foo;

// Are you arguing that this should work?  It does not work today, there's no
// true hierarchy between Foo and Bar.  If Bar also used Foo, would you expect
// to be able to write Foo.Bar.Foo.Bar.someFunction()?
Foo.Bar.someFunction();
// I believe this is okay because Bar is made in scope by Foo's use
Bar.someFunction();
// Obviously this should work.
someFunction();

Case 2: Bar is imported by Foo

module Foo {
   import Bar;
}
module Bar {
  proc someFunction() { ... }
}

2a) Another module imports Foo

import Foo;

// I don't like this at all
Foo.Bar.someFunction();
// Bar was imported by the module we imported so I guess I'd be okay with this being legal
// I would also be okay with needing an explicit `import Bar;` as well
Bar.someFunction();
// Obviously this shouldn't work.
someFunction();

2b) Another module uses Foo

use Foo;

// Are you arguing that this should work?  I don't see why we would prefix Bar with Foo in this
// circumstance - I don't know that a normal Chapel user would think to do so, though a Python
// user might.
Foo.Bar.someFunction();
// Since the import of Bar is public, it seems reasonable to me for this to work
Bar.someFunction();
// Obviously this shouldn't work, since there was no use of Bar
someFunction();

[ben-albrecht]

How would a user access secondary methods defined in an imported module?

E.g. Say I want to call the transpose _array.T method when importing the LinearAlgebra module.

[lydia-duncan]

That is a very good question. I would propose that importing a module makes the methods defined in that module available to use, similarly to how creating a class/record instance from a type that was defined in an imported module doesn't prevent the instance from being able to call its methods? I'm open to other suggestions as well

[mppf]

Responding to #13831 (comment)

Thanks @lydia-duncan for the examples:

If A had a use B; and it followed your suggestion, wouldn't we have the same problem, in that A.B.someFunction() would refer to the imported module rather than a submodule?

I'm generally against using . to reference to a hierarchy that doesn't involve the actual symbol definitions. If module Bar is not defined in module Foo, I don't want to be able to type Foo.Bar at all. It doesn't matter to me whether Bar was imported or used by Foo. If Bar was imported by Foo, I am okay with a use or import of Foo making Bar.someFunction() available, but not with making Foo.Bar.someFunction() available. I would probably prefer that import Foo; means that import Bar; is also necessary, but I can live with that being considered too restrictive.

This is an interesting discussion but it's clear we have different mental models. I'm coming from #13536 (comment) which @bradcray worked through in #13536 (comment) -

I'm saying the M in M.foo refers to a symbol that is created by the use statement. In other words, my viewpoint is that use creates local symbols for the imported things.

This viewpoint leads to different answers for most of your examples. However, I will focus on the two that seems the most surprising to me here:

I'm going to try writing some examples to help clarify that.

Case 1: Bar is used by Foo

module Foo {
   use Bar;
}
module Bar {
  proc someFunction() { ... }
}

1a) Another module imports Foo

import Foo;

// I don't like this at all
Foo.Bar.someFunction();
// I'm not a fan of this because there was never an explicit import of Bar,
// but I like it way better than the previous line.
Bar.someFunction();
// Obviously this shouldn't work.
someFunction();

I think we could choose not to make Foo.Bar.someFunction() available. I think it should depend on whether or not the use Bar in Foo is public or private, and whether or not use Bar makes Bar.something available at all, but I think we could end up in a reasonable place if it were never available.

If we chose not to make Foo.Bar.someFunction() work in this example, how could a module author arrange it to make it work? I would expect that one could "re-export" another module's symbols by creating a private submodule Bar, using the real Bar within that, and then public useing the private submodule. But, I think I prefer being able to just do it directly.

Bar.someFunction();

I really don't think this one makes sense in this pattern. As you said, there is no explicit import of Bar, and we had import foo (not use Foo) and so Bar, being a symbol within Foo, should not be available.

Case 2: Bar is imported by Foo

module Foo {
   import Bar;
}
module Bar {
  proc someFunction() { ... }
}

2a) Another module imports Foo

import Foo;

// I don't like this at all
Foo.Bar.someFunction();
// Bar was imported by the module we imported so I guess I'd be okay with this being legal
// I would also be okay with needing an explicit `import Bar;` as well
Bar.someFunction();
// Obviously this shouldn't work.
someFunction();

See my comments about Foo.Bar.someFunction(); above.
As with above, I really don't think Bar.someFunction(); should work without an import Bar.

[ben-albrecht]

I would propose that importing a module makes the methods defined in that module available to use, similarly to how creating a class/record instance from a type that was defined in an imported module doesn't prevent the instance from being able to call its methods? I'm open to other suggestions as well

That seems reasonable, although we lose the benefit of using import where we can identify the origin of all identifiers without leaving the source file.

The alternative is that secondary methods are not accessible when just using import, and an explicit import is required. That might look something like this:

// Import all the secondary methods on _array type
from LinearAlgebra import _array;

// Import all the _array.T secondary method overloads
from LinearAlgebra import _array.T;

I'm not sure which approach is a better fit for Chapel.

[lydia-duncan]

Bar.someFunction();

I really don't think this one makes sense in this pattern. As you said, there is no explicit import of Bar, and we had import foo (not use Foo) and so Bar, being a symbol within Foo, should not be available.

I find that convincing. I think I understand more now about why it would be okay to support Foo.Bar.someFunction(), but I'm still not a big fan of it due to the overlap with . when actual submodules are involved

[BryantLam]

I think I understand more now about why it would be okay to support Foo.Bar.someFunction()

I don't know the implementation details, but to provide some motivation, I require some way to re-export symbols as if they are in some other module hierarchy because of long-term code maintenance when undergoing refactor efforts. As I move symbols in my module hierarchy, I need the ability to re-export those symbols so my library's users do not perceive an API-breaking change.

[lydia-duncan]

I opened #14790 and #14799 with some alternatives to this proposal based on Rust and Python respectively. I'd be interested in hearing potential changes to those proposals as well as comparisons to this one. Thanks :)

[lydia-duncan]

I think at this point I can say that we're going to follow the Rust proposal for import statements, replacing only list syntax with more Rust-like .{} lists to avoid the clarity issue Greg pointed out. This means that:

• import M; will enable M.x (aka qualified access to M's symbols)
• import M.x; will enable x (aka unqualified access to the symbol x defined in M)
• import M.{x, y}; will enable x and y (aka unqualified access to the symbols x and y defined in M)
• import M.N.O; will enable O.x (aka qualified access to O's symbols without always having to use M.N.O at the front)
• import M as N; will enable N.x (aka qualified access to all of M's symbols but using the name N)
• import M.x as mx; will enable mx (aka unqualified access to x but using the name mx
• import M.{x as mx, y}; (aka unqualified access to M.x and M.y, calling them mx and y respectively)

It sounds as though we will not support import M.*; for the moment (this would mean unqualified access to all public symbols in M if we were to implement it). We may revisit this later if users request it, but will encourage use statements in its place.

We can support things like import M.{x, y, this}; to enable x, y, and M.<all public symbols in M> (aka unqualified access to x and y, and qualified access to M's symbols), but will wait to do so until there is user demand.

We're going to wait to decide on relative imports of nested modules (e.g. importing M.N from within M) based on discussion around submodules-as-files.

For re-exporting symbols, the general consensus is to use public import as opposed to alternate syntax or strategies.