Roadmap for Nim

[Araq]

"The Cylons Were Created by Man. They Rebelled. They Evolved. They Look and Feel Human.
Some are programmed to think they are Human. There are many copies. And they have a Plan."

It seems to come up regularly that people don't know where Nim is heading, where our focus of development is and what's the PLAN. I'm sorry for not communicating it well but the plan is solid, doesn't change much and it "only" takes longer than I hoped it would take.

Incremental compilation (IC) / nimsuggest / IDE tooling

This is a single point as for me it's a single thing. If we had a compiler that only recompiles the module that was touched the "nimsuggest" tool can be a simple command line tool that answers a single question like "where is symbol X used?".

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR nim-lang/Nim#17814. Which is in the state "on hold" until version 1.6.2 is out.

IC is the most important thing for me personally. It's not scheduled for any particular release as it's not about the language nor about its library. Ideally IC will stabilize and be backported to the 1.6.x line.

Update: Good IC relies on refactorings of the compilation pipeline. We want to finish these refactorings, release version 2.0 and then finish IC in version 2.x.

Version 1.6.2

• We will release 1.6.2 near-term (ideally still in 2021) which is 1.6.0 with many bugs ironed out. It is expected that the 1.6.x line is a LTS version, much like 1.2.x is.

Version 2.0

Hard goals

For version 2.0 we will make --gc:orc, mimalloc and --threads:on the new defaults.

• --gc:orc koch boot compiler with orc Nim#20467 defaults to ORC Nim#19972
• --threads:on default threads on Nim#19368
• some way to make the allocator faster for shared memory. One option is mimalloc supports mimalloc for ARC/ORC (1.15x ~ 1.22x speedup booting the compiler) Nim#20359
• Refactorings of the compilation pipeline in anticipation of IC.

Furthermore, we consider the following experimental features to be stable enough for version 2.0:

• --experimental:strictEffects fixes #19162; enable strictEffects for v2 Nim#19380
• --experimental:unicodeOperators Unicode Operators are no longer experimental Nim#20444
• --experimental:overloadableEnums overloadable enums no longer experimental Nim#20298
• -d:nimPreviewFloatRoundtrip move formatfloat out of system Nim#20195
  -d:nimPreviewDotLikeOps enable nimPreviewDotLikeOps Nim#19598

Stretch goals

Language

• Enforce "Definite assignment analysis" for all types, this is easy to do, removes an error-prone aspect of the language, makes the language more consistent and mitigates the problem with nil pointers. Definite assignment analysis and out parameters #378. DAA and 'out' parameters Nim#20506

• Deprecate 'unsafeAddr' and make 'addr' available for all addressable locations, see Deprecate 'unsafeAddr' and make 'addr' available for all addressable locations #369 deprecate unsafeAddr; extend addr Nim#19373

• Support field = defaultValue inside object declarations. User-defined implicit initialization hooks #252
  add default field support for object in ARC/ORC Nim#20480

Cleanup of experimental language features

• "Void type" -- We should optimize away the "empty tuple" type in the backend and remove this feature.
• "Automatic dereferencing" -- We should remove the feature, IMO. It's not a problem worth solving. Automatic dereferencing is removed Nim#20531
• "AST based overloading" -- Should be removed.
• "Parallel & Spawn" -- Should be removed. Better done by external packages.
• "Guards and Locks" -- Remove lock levels. 'lock levels' are deprecated, now a noop Nim#20539

Stable experimental language features

These features are stable and should be part of the manual.

• "Named argument overloading". caseStmtMacros no longer experimental, experimental manual refactor Nim#19173
• "Do notation". caseStmtMacros no longer experimental, experimental manual refactor Nim#19173
• "Type bound operations". move type operation section and remove deepcopy document Nim#19389
• "Case statement macros". caseStmtMacros no longer experimental, experimental manual refactor Nim#19173
• "Guards and Locks" -- Move "guards" to the stable documentation. caseStmtMacros no longer experimental, experimental manual refactor Nim#19173
• "Noalias annotation" -- Stable. caseStmtMacros no longer experimental, experimental manual refactor Nim#19173

Library

1. Clean up the system module, move these submodules into their own modules:

   • IO move io out of system Nim#19442
   • assertion handling move assertions out of system Nim#19599
   • thread handling put std/threads under the umbrella of nimPreviewSlimSystem Nim#20711

2. Clean up the OS module, make it more type safe, split it in up in e.g.:

   • path handling (paths.nim) add typesafe std/paths, std/files, std/dirs, std/symlinks Nim#20582
   • directory handling (dirs.nim): creation, removal, iteration, existence checking add typesafe std/paths, std/files, std/dirs, std/symlinks Nim#20582
   • file handling (files.nim): creation, removal, existence checking add typesafe std/paths, std/files, std/dirs, std/symlinks Nim#20582
   • environment variables handling (envvars.nim) Added std/envvars for env vars handling Nim#19421
   • symlinks handling (symlinks.nim) add typesafe std/paths, std/files, std/dirs, std/symlinks Nim#20582
   • OS error reporting (oserrors.nim) Added std/oserrors for OS error reporting Nim#19390

3. The JSON module is also in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance. It could be split up into:

   • [-] jsonbuilder
   • [-] jsonparser
   • [-] jsonlexer
   • [-] jsontree
   • [-] jsonmapper: it maps Nim types (objects) to Json types

4. More of the standard library should be moved to external Nimble packages. How
   far we get with this is unclear.

Misc (in comments)

• register callback for marshal in VM register callback for marshal in VM Nim#19578; ref Roadmap for Nim #437 (comment)

Version 2.x

Every stretch goal of 2.0 that didn't make it into version 2.0, should be
considered a goal for version 2.x.

Hard goals

We should remove the requirement for forward declarations and allow for recursive module dependencies. Then we can remove these experimental features:

1. "Package level objects"
2. "code reordering"

see also nim-lang/Nim#18818

Stretch goals

1. Find a better design for "Term rewriting macros". Once done, deprecate the old feature.
2. Find a better design for "Special Operators". Once done, deprecate the old feature.
3. Implement "Aliasing restrictions in parameter passing". The compiler can always produce warnings first.
4. "Strict funcs" / "view types" should be made stable for Nim 2.x.
5. Concepts: Find a way of how to infer static values and stabilize the feature.
6. Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Nimble

We hope to be able to ship a new Nimble with the 2.0 release but we are happy to ship it whenever it's ready. See #398 for good ideas of how to further evolve Nimble. The most important missing feature of Nimble is "lock files" -- Nimble implements it, but not in a way that is compatible with the 1.6.0 Nim compiler.

How can you help?

The standard library cleanups should be rather easy to do.

[Vindaar]

"AST based overloading" -- Should be removed.

Not sure how I feel about this. Would this also imply removal of the same feature for NimNodes? I rather like that (i.e. specifying a CT proc is only for idents).

To be fair I didn't know this could be used with runtime procedures.

[metagn]

Find a way of allowing a syntax like of Value(let x, var y) and embrace an official pattern matching solution that makes use of this syntax.

Reminds me, default values for object fields was quite popular, is this planned as well? (just saw this higher up in the post) are any object variants improvements from many proposals going to be considered? I've also seen a lot of complaints about object construction, but I don't know which useful changes there are other than default values, if there was going to be a =init I'm guessing it'll probably be more coherent for it to be similar to =destroy instead of a complete override of initialization (after zeroing memory/initializing default field values).

The JSON module ... could be split up into:

Would json/builder, json/lexer be better (with a re-export of every submodule in a module named json)? I remember this being discussed for other modules but vaguely something about it not working or being inconvenient. I believe it was proposed for the xml modules and times, but deemed problematic for os.

[Araq]

Would json/builder, json/lexer be better

Yes, most people would like that better but I don't want to implement and support package.module.name just for a refactoring so I'd go with the name prefixes.

[konsumlamm]

"Void type" -- We should optimize away the "empty tuple" type in the backend and remove this feature.

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

[Araq]

So eventually, void would be replaced by the empty tuple, so that e.g. proc test() = discard would return () (like it is the case in Rust)?

No, for the return type and discard-checking there would still be a void.

[levovix0]

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

[Araq]

Update: Expanded the section on Nimble.

[juancarlospaco]

"Parallel & Spawn" -- Should be removed. Better done by external packages.

But stdlib keeps std/tasks ?,
I think a way to run multiple stuff at the same time in stdlib is useful !,
at least with a basic API, maybe think of removing Parallel and keep Spawn?

[Araq]

std/tasks is a "building block", so it should stay.

[Varriount]

+1 for moduleGroup/module instead of moduleGroupmodule

I don't want to implement and support package.module.name

why implementing and supporting package.module.name needed for package/module/name?

I believe that this is because import only considers the last path component as the module name. The code from json/lexer import nil allows accessing all exported routines/types in "./json/lexer.nim" through the lexer symbol, not the expression json.lexer.

If an expression like json.lexer won't be supported, then the symbol "jsonLexer" may be preferable to "lexer", since "jsonLexer" is less ambiguous.

[Araq]

Update:

• Nimble supports lockfiles and Nim 1.6.2 supports the new Nimble. (But it doesn't yet ship with it.)
• Nim version 1.6.2 has been released.

[juancarlospaco]

Would be nice to give the jsgen some love too!, at least improve jsgen a little,
the JS stdlib improved a lot and wont produce Showstopper complex bugs,
and the few JS bugs are actually jsgen related bugs actually so ... 🤔

[a-mr]

What is policy for breaking changes in 2.0?

I see 2 extreme options:

1. almost no breaking allowed as it is currently in Nim 1.x. The transition to gc:orc is an exception.
2. any breaking changes are potentially allowed if it is the right thing. Why would you call it version 2.0 otherwise...

Some concrete examples I know:

• slices are treated differently for different procs [], []=, delete and different types (e.g. Slice operations inconsistent between types, operations, bounds checks Nim#16492). It would be great to unify it, but it's a significant breaking change of course.
• os.walkDir silently ignores errors (see e.g. walkDir ignores errors reported by C Nim#10309 and references). It would be desirable to raise exception by default but we observed that it would break even some code inside Nim repo.
• anchors generated by docgen don't preserve information about e.g. var in parameter or type of optional parameter with default value (docgen: anchors for procs omit params if their default values are variable names Nim#12373). Fixing that would break previously made references to such anchors.

[Araq]

We should fix as much as reasonable, but we need to have a good upgrade path. The worst thing continues to be subtle changes in runtime behavior. That means:

• I don't know what to do with slices.
• Deprecate walkDir and introduce something that makes sense. Can also be in a module of its own.
• Anchors for v2 are not worth preserving, details like that are not worth

[zedeus]

Very cool, nice roadmap. What's the current status, what is the primary project being worked on?

The good news is that we have this tool already, it's nim check --navigate, the bad news is that its progress is tied to PR nim-lang/Nim#17814. Which is in the state "on hold" until version 1.6.2 is out.

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

[Araq]

1.6.2 is now out, but there hasn't been any updates to that PR. What's the status of it?

I never know how to update these RFCs. 1.6.2 is out, next week work on 2.0 will begin.

[Araq]

There is nim-lang/threading which is official plus a couple of external packages that do it better than we could hope to. There are currently no plans to bring this into the stdlib for these reasons:

• We lack the resources; our job shifted to compiler and tooling work which is demanding enough already.
• The stdlib should be trimmed down a bit, not expanded further.
• Most importantly: As separate packages they can evolve much more quickly, not hindered by a deprecation cycle that spans a decade. And not tied to our slow release cycle.

We merely offer the building blocks with destructors, atomics and std/tasks.

[metagn]

Note for the checklist: guards & locks are in both experimental features to clean up and stable experimental features. Since guards were moved to the stable documentation, it can be marked under the stable experimental features section.

Some RFCs covered by the items:

• RFC: move file I/O out of system.nim #67
• [RFC] switch threads to on by default #361

A few simple non-breaking features that could be done at any time, but would be nice to have soon, probably also under 1.6 line:

• overloadExists(foo(args)) to tell whether an overload exists without compiling/running body via compiles #164
• namespaced defines: -d:myproj.mykey to avoid clashes #181
• Redoing variable declaration macro pragmas #220

There are also tons of RFCs related to default object fields and pattern matching but they tend to be larger in scope and discuss other things.

I think "optimize away empty tuple" will take a while compared to the steps for removing the other experimental features. But since it's not tied to a super well known experimental feature it should be fine if it's not 2.0.

Does the var in of Value(let x, var y) mean var y: T or y: var T? I don't think it should be the first one given for loops and proc parameters presumably don't have it for a reason. Second one is something the language could do that macros couldn't.

"Not nil annotation" -- Should be planned for Nim 2.x but removed from the "experimental" manual.

Don't understand this. Is it supposed to be kept in as experimental but undocumented?

[beef331]

I just got reminded, is there any chance marshal will get removed from the stdlib for 2.0, or at least removing the VM support for it. Almost every json library made can be used at compile time without hard coded VM support, and as most know marshal is just a json serializer. https://github.com/nim-lang/Nim/blob/devel/compiler/vmgen.nim#L2057-L2065 can cause 10% slower compilation in compile time heavy code(macros included). Since every single call is done with two string searches to see if it's from marshal.

[ringabout]

Alternative way is to support vmmarshal properly with callback if it is worth.

[Araq]

@beef331 wow... that should be fixed indeed. It doesn't necessarily mean we need to disable marshal on the VM though, it should simply either use its own opcodes or the existing callback mechanism.

[ringabout]

A simple POC using callback mechanism: https://gist.github.com/xflywind/551d84567cb39d95d0ddcd71f8279e6a

[ringabout]

Path related RFCs：

#54 new path handling module that uses Nim's type system, using Path instead of string
#121 os module should have an "exists" function
#71 [compiler/pathutils] AbsoluteFile,AbsoluteDir,RelativeFile,RelativeDir is too complicated
#112 joinPath("foo", "/bar") should return "/bar"

[Menduist]

Could we consider #89 for nim V2?

A few options have been considered, eg the linked RFC, or deprecating naked excepts, all of them being better than the current status quo, imo

[juancarlospaco]

I would like to re-consider ideas proposed before this Nim 2.0 trend, like:

• Pattern Matching (Requested in many places multiple times).
• Add bigints to standard library Nim#14696
• Multiple Comparisons #242
• Add ZIP to stdlib #318
• Memoization stdlib #389
• Move choosenim under Nim org #374

[beef331]

Slightly related to arne's point. In my quest to make code more reusable I did attempt to make parseutils use openarray[char] but it's showed a few issues with openarray/string/cstring that will need to get resolved. Presently due to implicit conversion from string to cstring the following does not work.

proc doThing(oa: openarray[char]) = discard
proc doThing(str: cstring) = discard
doThing("hello")

Similarly the following perhaps should work. Where openArray[char] is considered equivalent to string in the case of converters.

type NotString = distinct string
converter toString(s: NotString): string = string(s)
proc doThing(oa: openarray[char]) = discard
doThing(NotString("hello"))

[juancarlospaco]

The Nim 2.0 can also be a good opportunity to make a decision about Nimble packages clean out,
packages that do not exist anymore and give 404 errors, but they are still in the packages list,
theres multiple issues open and closed in the repo about the problem,
and removing the packages from the JSON looks like it is not allowed (?),
but it is useless because the repo do not exist anymore,
and sometimes the user also do not exist anymore or is not coding anymore since years ago.

• Handling deleted packages packages#2075
• All the packages no longer availabe on the provided url packages#1438
• Issues with some packages packages#1807
• Removed repos packages#360
• Probaly more I am missing...

[CyberTailor]

"Parallel & Spawn" -- Should be removed. Better done by external packages

What about bootstrapping build systems/package managers that use threads? Will they need to supply their own implementation of threadpool?

[Araq]

No, package managers and bulid systems can use Nimble packages as dependencies if they do the "git clone" commands directly in the build script and/or can use git submodules. A package manager is not like a compiler, it doesn't have to "bootstrap" itself.

[CyberTailor]

This is not possible in network-sandboxed environments, and support for them matters.

[Araq]

The sandbox would prevent nimble install nimble just as it would prevent git clone depsThatNimbleRequires.

[CyberTailor]

I've successfully packaged lots of Nim libraries and programs for Gentoo, which package manager is network-sandboxed (this is the standard in *nix packaging). Neither git nor nimble are needed for it. Seems like this discussion is off-topic and should be over.

[Araq]

Fine with me, you started the subdiscussion.

[tomasky]

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

[Araq]

Just use a CI, cross-compilation won't let you run any tests anyway... I wish we would attract more users who understand software development.

[tomasky]

Ok, I spent some time cross-compiling using CI:

- name: build-x86
  image: dockcross/linux-x86
  commands:
  - nimble build --passL:-static -o:out-linux-x86 -d:release
 
- name: build-x86-64
  image: dockcross/linux-x64
  commands:
  - nimble build --passL:-static -o:out-linux-amd64 -d:release
    
- name: build-linux-arm64
  image: dockcross/linux-arm64
  commands:
  - nimble build --cc:env --os:linux --cpu:arm64 -o:out-linux-arm64 -d:release

- name: build-android-arm64
  image: ghcr.io/tomasky/drones/arm64:nim
  commands:
  - nimble build --cc:env --os:android --cpu:arm64 -o:out-android-arm64 -d:release

[Araq]

Postponed:

• Implement Extend the notion of a "type bound" operation #380, most probably by yet another opt-in switch like --experimental:typeBoundProcs. implements https://github.com/nim-lang/RFCs/issues/380; WIP Nim#20520

[Araq]

Postponed:

• The JSON module is also in a poor shape, the API exposes public fields, sometimes is not intuitive (the %* operator) and has poor performance.

[Araq]

I'm closing this since version 2.0 is effectively done, we'll only fix even more ORC bugs and release 2.0 in early 2023. The plan for 2023 will be published as an RFC here in the next couple of days.

[ShalokShalom]

I like nim, but cross-compiling is inconvenient. I wish I could cross-compile directly like Golang or ziglang, which would attract more users

https://github.com/enthus1ast/zigcc

Just use the Zig compiler 😉