Support include statement to permit code to be split across multiple files in arbitrary ways

[bradcray]

[this is a long-term request / intention that I'm only now getting around to filing]

As a Chapel programmer, I currently have no way to have one file incorporate the contents of several other files (possibly taken from varying search paths on different compiles) without breaking them up into distinct submodules / namespaces. This feature request is for an include statement that would behave like C's #include or LaTeX's \input by replacing the statement with the literal contents of the named file.

For example, this might look like:

module M {
  proc foo() { ... }

  include "bar.chpl";
}

where bar.chpl might be defined in the current directory or along a user-specified search path.

[bradcray]

I believe that PR #1354 took us a step toward having our parser/lexer be able to support this by switching files when we encounter such statements.

[BryantLam]

Ref #7847 (comment)

[BryantLam]

Ref #10796 (comment)

Also, this is relevant: "Is including headers inside a namespace always a bad idea?"

[mppf]

@BryantLam - is the thumbs-down on this issue intentional?

[BryantLam]

@mppf -- Yes. This feature isn't really appealing to me. It would work, but it would have some code-scaling issues:

• Refactoring is now harder because it relies on directory structure. You could argue that C/C++ #include have this problem today, and you wouldn't be wrong.
• I don't see how it differs from simply doing:

module MyModule {
  use Submodule1;
  use Submodule2;
}

• ... except in the case where individual files (= file-level modules) relied on each other beyond the scope of a file-level module, which now adds the complication that you would have to understand all of the code in every include to know whether you could make a modification to one of them.

I would prefer an approach like C++ namespaces where you could "define" a module multiple times and it would be treated as logically the same. Namespace collisions don't happen all that often when you are the owner of a namespace, so I don't see the necessity of file-level modules other than the notion that you can insert code at file-scope (which could simply be treated as concatenated all together and sequentially executed from arbitrarily ordered modules). .. Or get rid of file-scope code, but that's probably too drastic.

[bradcray]

I would prefer an approach like C++ namespaces where you could "define" a module multiple times and it would be treated as logically the same.

Does an issue for this proposal already exist? If not, would you consider creating one to illustrate what you're looking for?

Namespace collisions don't happen all that often when you are the owner of a namespace

It may be that I'm simply not understanding your proposal, but this statement seems odd to me: If the point of namespaces is to prevent two pieces of code that are unaware of each other from inadvertently having both define conflicting symbols with the same name, it seems weird to solve it using a solution in which two distinct pieces of code that happen to use the same name for a module get auto-merged into a single module (again, if I'm understanding properly). Maybe what I'm missing is: What does it mean to "own a namespace?"

(That said, I think these questions should be answered on an issue proposing C++-like namespaces in Chapel, though, rather than one proposing an include statement... I understand that the include directive is not your preferred solution for managing namespaces, but I think it still has value in terms of permitting disparate files from being stitched together to form one logical one (in addition to the potential for abuse).

I don't see how it differs from simply doing:

module MyModule {
  use Submodule1;
  use Submodule2;
}

This code says "Make the public symbols of Submodule1 and Submodule2 available when resolving references within this scope." It doesn't say anything about injecting the contents of specific files into this file.

[BryantLam]

Does an issue for this proposal already exist? If not, would you consider creating one to illustrate what you're looking for?

I don't think I want the proposed namespace approach anymore. A simpler solution exists: files are already modules. Break the files up along module boundaries and "package" it with a top-level module by use or import statements. The compiler's module search paths must only look down a directory tree for a used submodule, and it can only be in a few known locations so we're not stating all over the filesystem. In #12923, I've expanded more on this approach.

I understand that the include directive is not your preferred solution for managing namespaces, but I think it still has value in terms of permitting disparate files from being stitched together to form one logical one (in addition to the potential for abuse).

I disagree. The #include directive in C/C++ has led to modern pain with C++'s attempt to introduce modules. I recognize that the #include directive is exceptionally powerful (and simplistic) because it merely--wholesale--includes a file into another file. But that behavior alongside other preprocessor behaviors like #define makes modularizing an existing C++ codebase nontrivial when you start from its source files.

Chapel doesn't have a lot of these problems because we don't have macros (yet) or a preprocessor that can wholesale include files into other files. This feature, however, proposes that exact behavior that led languages like even C++ away from using filesystem paths towards a semantic model which would assist in incremental compilation.

I don't see how it differs from simply doing:

module MyModule {
  use Submodule1;
  use Submodule2;
}

This code says "Make the public symbols of Submodule1 and Submodule2 available when resolving references within this scope." It doesn't say anything about injecting the contents of specific files into this file.

Right. It shouldn't be injecting files. The use statement should simply be limiting the available paths to where it can find these submodules. My proposal is addressed in #12923.