fix nim-lang/RFCs#18; add reflection.moduleSymbols to list symbols in a module

[timotheecour]

• add reflections.moduleSymbols to get sequence of symbols in a module (for compile time reflection)
• fix Feature request: macros able to iterate over symbols RFCs#18

links

• Module Introspection - Nim forum : Module Introspection - Nim forum
• Transform procs at compile time? - Nim forum
• Get all procs in given module - Nim forum
• Feature request: macros able to iterate over symbols · Issue #18 · nim-lang/RFCs
• nimscripter https://discord.com/channels/371759389889003530/768367394547957761/894724802185687071

[Araq]

I don't like the feature (the implementation mostly follows the existing pattern). Previously I told people to use parseStmt on a full file to get the AST. I'd like to preserve the following invariant in Nim code: Given mymacro: x, only x can be affected by the transformation that mymacro does. Your macros.moduleSymbols breaks this invariant, local reasoning about Nim code gets harder.

[krux02]

I am closing as rejected. Adding API's like this needs careful thinking instead of exposing all internal structures.

[sighoya]

Question: Can we open up it again, I find this very useful.
Essentially, it replaces the need for custom builder/compiler plugins to do "reflection".
What you know in Java with annotations could then be realized in Nim with pragmas at compile time.
Examples:

• infer all children of a class like type
• first class dynamic dispatching by retrieving all overloads
• non-local dependency injection
• generating all c/c++ wrappers automatically by traversing all importcpp pragmas?

A disadvantage of this feature is the non-localness literally allowing any local code change to break your code, however at least at compile time.

@Araq wrote:

Given mymacro: x, only x can be affected by the transformation that mymacro does.

Understand, so assignments could be created for non local vars. But I think bindSym hackery already allows such things, okay dynamically scoped bindSyms are experimental yet.

[timotheecour]

Question: Can we open up it again, I find this very useful.

yes, it is very useful and an often requested feature (see some links i added on top), I've finally re-opened this PR and updated it.

@Araq PTAL

refs https://forum.nim-lang.org/t/1817#37522

It's been 4 years (EDIT: now 5), I cannot count how often I got this feature request. What does that mean? If a really suitable implementation comes along I won't fight it as hard anymore. Maybe.

update since original PR:

• rebased, fixed conflicts and updated implementation
• moved this from std/macros to std/reflection, where it belongs
• added more illustrative runnableExamples
• added enablePrivate = false optional param (follows import {.all.} PR)

why std/reflection

this is the most sensible place for both moduleSymbols (this PR) and the other reflection routines from #17641

#17641 (comment)
The name std / reflection is a recipe for bloat. Reflection is a wide field and many people expect there to exist RTTI related functionality. I propose std / sourceinfo instead.

nim is a static compiled language, so CT being implied by reflection will surprise no-one; in any case I've added a doc comment to make it clear that the scope of this module is CT reflection. For RT reflection / RTTI, we already have a module, https://nim-lang.github.io/Nim/typeinfo.html

std/sourceinfo doesn't reflect the intended scope as well as std/reflection.

why not parseStmt(readFile("module.nim"))

it suffers from the same pitfalls as include; you get different symbols from the ones actually used in the project. You can still use parseStmt(readFile("module.nim")) if you want, but the semantics are different and usually not what a user interested in reflection wants.

why not use compilerapi ?

compilerapi (which few people use anyway) is no replacement for a reflection module and moduleSymbols in particular; it doesn't have the ease of use of nim macros and is a meta-api that doesn't integrate easily with user code.

other languages have reflection too:

• D (at CT): __traits(allMembers, mymodule)
• go (at RT): NumMethod + friends in https://golang.org/pkg/reflect/
• java (at RT): http://tutorials.jenkov.com/java-reflection/methods.html
• python3 (at RT): dir([object]) which works with modules, classes etc; also, inspect
• js: var mod = require('./mod.js'); Object.keys(mod)
• C#: https://docs.microsoft.com/en-us/dotnet/api/system.reflection.module.getmethods?view=net-5.0 + firends
• ruby, etc...

honestly, it's a surprising omission in nim; std/macros doesn't replace the need for std/reflection which can be very useful for user code.

[timotheecour]

ping @Araq, see updates in #9560 (comment)

[Varriount]

Is there a general overview of what reflection capabilities Nim should have. and what are feasible? While I think this has merit, modules aiming to serve a functionality like "reflection" are best designed as a whole, rather than piecemeal. For example, see how https://blog.golang.org/laws-of-reflection and Go's reflection package work.

[Araq]

Is there a general overview of what reflection capabilities Nim should have. and what are feasible?

It's pretty simple: Nim should never have reflection like Java and C# because Dead-code elimination becomes fragile or impossible. In general, musing about what would harm an optimizer is a good design principle if "easy to reason about" is not just a buzzword phrase.

[timotheecour]

It's pretty simple: Nim should never have reflection like Java and C# because Dead-code elimination becomes fragile or impossible. In general, musing about what would harm an optimizer is a good design principle if "easy to reason about" is not just a buzzword phrase.

@Araq I fail to see how this PR impacts dead code elimination in any way. moduleSymbols only lists the symbols in a module; if they're not called from a top-level scope, dead-code elimination rules still apply exactly as before this PR. Listing properties (eg declaration line, doc comments, etc) derived from all symbols in a module also won't impact DCE.

Example

nim r -d:case4a main # no stdlib_strutils.nim.c generated
nim r -d:case4a main # stdlib_strutils.nim.c generated but only CmpIgnoreCase is codegen'ed

when true:
  import std/[macros, reflection, strutils]
  macro moduleSymbolsTest(mymod: typed, enablePrivate: static bool = false): untyped =
    for ai in mymod.moduleSymbols(enablePrivate):
      if $ai == "cmpIgnoreCase": return ai
  when defined case4a:
    const foo1 = moduleSymbolsTest(strutils, enablePrivate = true)
    echo foo1("ab", "Ab")
  when defined case4b:
    let foo2 = moduleSymbolsTest(strutils, enablePrivate = true)
    echo foo2("ab", "Ab")

The feature is useful, and has been requested many, many times over; can we please merge this?

Most other languages I've used (D, go, java, python3, js, C#, ruby ...) allow this kind of reflection APIs, those languages didn't stop being analyzable or optimizable because of this; it's a very useful thing to have when you need it.

[Araq]

@Araq I fail to see how this PR impacts dead code elimination in any way.

I didn't mean to imply that it does impact it. I merely replied to @Varriount and outlined some of the design principles that we follow.

[timotheecour]

ping @Araq this keeps coming up, eg https://discord.com/channels/371759389889003530/768367394547957761/894724802185687071 (for nimscripter)

[deech]

This would be very useful to me, I want to extern the exposed types/procs in a module compiled with the cpp backend so it can be used from other languages. Having a way to simply list the public contents of a module would make this a lot easier. We already have std/importutils so this seems like the next logical step.

[Araq]

In order to proceed

1. rename it to something else, reflection typically is used for runtime reflection.
2. Make it clear and ideally enforce it that the resulting sequence of NimNodes cannot be mutated. Or that mutations do not affect the compilation pipeline.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.