Initial version

.gitignore

@@ -0,0 +1,5 @@
+nimcache
+*.exe
+*.swp
+test*
+toast

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Ganesh Viswanathan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,73 @@
+Nimterop is a [Nim](https://nim-lang.org/) package that aims to make C/C++ interop seamless
+
+Nim has one of the best FFI you can find - importing C/C++ is supported out of the box. All you need to provide is type and proc definitions for Nim to interop with C/C++ binaries. Generation of these wrappers is easy for simple libraries but quickly gets out of hand. [c2nim](https://github.com/nim-lang/c2nim) greatly helps here by parsing and converting C/C++ into Nim but is limited due to the complex and constantly evolving C/C++ grammar. [nimgen](https://github.com/genotrance/nimgen) mainly focuses on automating the wrapping process and fills some holes but is again limited to c2nim's capabilities.
+
+The goal of nimterop is to leverage the [tree-sitter](http://tree-sitter.github.io/tree-sitter/) engine to parse C/C++ code and then convert relevant portions of the AST into Nim definitions using compile-time macros. [tree-sitter](https://github.com/tree-sitter) is a Github sponsored project that can parse a variety of languages into an AST which is then leveraged by the [Atom](https://atom.io/) editor for syntax highlighting and code folding. The advantages of this approach are multifold:
+- Benefit from the tree-sitter community's investment into language parsing
+- Leverage Nim macros which are a user API and relatively stable
+- Avoid depending on Nim compiler API which is evolving constantly
+
+The nimterop feature set is still limited when compared with c2nim. Supported language constructs include:
+- `#define NAME VALUE` where `VALUE` is a number (int, float, hex)
+- `struct X`, `typedef struct`, `enum X`, `typedef enum`
+- Functions with primitive types, structs, enums and typedef structs/enums as params and return values
+
+Given the simplicity and success of this approach so far, it seems feasible to continue on for more complex code. The goal is to make interop seamless so nimterop will focus on wrapping headers and not the outright conversion of C/C++ implementation.
+
+C++ constructs are still TBD depending on the results of the C interop.
+
+__Installation__
+
+Nimterop can be installed via [Nimble](https://github.com/nim-lang/nimble):
+
+```
+> nimble install http://github.com/genotrance/nimtreesitter?subdir=treesitter
+> nimble install http://github.com/genotrance/nimtreesitter?subdir=treesitter_c
+> nimble install http://github.com/genotrance/nimtreesitter?subdir=treesitter_cpp
+
+> nimble install http://github.com/genotrance/nimterop
+```
+
+This will download and install nimterop in the standard Nimble package location, typically ~/.nimble. Once installed, it can be imported into any Nim program.
+
+__Usage__
+
+```nim
+import nimterop/cimport
+
+cDebug()
+cDefine("HAS_ABC")
+cDefine("HAS_ABC", "DEF")
+cIncludeDir("clib/include")
+cImport("clib.h")
+
+cCompile("clib/src/*.c")
+```
+
+__Documentation__
+
+Detailed documentation is still forthcoming.
+
+`cDebug()` - enable debug messages
+
+`cDefine()` - `#define` an identifer that is forwarded to the compiler using `{.passC: "-DXXX".}` as well as _eventually_ used in processing `#ifdef` statements
+
+`cIncludeDir()` - add an include directory that is forwarded to the compiler using `{.passC: "-IXXX".}` as well as searched for files included using `cImport()` statements and following `cIncludeDir()` statements
+
+`cImport()` - import all supported definitions from specific import header file
+
+__Implementation Details__
+
+In order to use the tree-sitter C library at compile-time, it has to be compiled into a separate binary called `toast` (to AST) since the Nim VM doesn't yet support FFI. `toast` takes a C/C++ file and runs it through the tree-sitter API which returns an AST data structure. This is then printed out to stdout in a Lisp S-Expression format.
+
+The `cImport()` proc runs `toast` on the specified header file and parses the resulting S-Expression back into an AST data structure at compile time. This AST is then processed to generate the relevant Nim definitions to interop with the code accordingly. A few other helper procs are provided to influence this process.
+
+The tree-sitter library is limited as well - it may fail on some advanced language constructs but is designed to handle them gracefully since it is expected to have bad code while actively typing in an editor. When an error is detected, tree-sitter includes an ERROR node at that location in the AST. At this time, `cImport()` will complain and continue if it encounters any errors. Depending on how severe the errors are, compilation may succeed or fail. Glaring issues will be communicated to the tree-sitter team but their goals may not always align with those of this project.
+
+__Credits__
+
+Nimterop depends on [tree-sitter](http://tree-sitter.github.io/tree-sitter/) and all licensing terms of [tree-sitter](https://github.com/tree-sitter/tree-sitter/blob/master/LICENSE) apply to the usage of this package. Interestingly, the tree-sitter functionality is [wrapped](https://github.com/genotrance/nimtreesitter) using c2nim and nimgen at this time. Depending on the success of this project, those could perhaps be bootstrapped using nimterop eventually.
+
+__Feedback__
+
+Nimterop is a work in progress and any feedback or suggestions are welcome. It is hosted on [GitHub](https://github.com/genotrance/nimterop) with an MIT license so issues, forks and PRs are most appreciated.

config.nims

@@ -0,0 +1 @@
+switch("gcc.linkerexe", "g++")

nimterop.nimble

@@ -0,0 +1,16 @@
+# Package
+
+version     = "0.1.0"
+author      = "genotrance"
+description = "C/C++ interop for Nim"
+license     = "MIT"
+
+bin = @["toast"]
+installDirs = @["nimterop"]
+
+# Dependencies
+
+requires "nim >= 0.19.0", "treesitter >= 0.1.0", "treesitter_c >= 0.1.0", "treesitter_cpp >= 0.1.0", "regex >= 0.10.0"
+
+task test, "Test":
+  exec "nim c -r tests/tnimterop"

nimterop/ast.nim

@@ -0,0 +1,218 @@
+import macros, os, strformat
+
+import regex
+
+import getters, globals
+
+proc addReorder*(): NimNode =
+  result = newNimNode(nnkStmtList)
+  if not gReorder:
+    gReorder = true
+    result.add parseStmt(
+      "{.experimental: \"codeReordering\".}"
+    )
+
+proc addHeader*(fullpath: string) =
+  gCurrentHeader = ("header" & fullpath.splitFile().name.replace(re"[-.]+", ""))
+  gConstStr &= &"  {gCurrentHeader} = \"{fullpath}\" # addHeader()\n"
+
+#
+# Preprocessor
+#
+
+proc preprocDef(node: Ast) =
+  if node.children.len() == 2:
+    let
+      name = getNodeValIf(node.children[0], "identifier")
+      val = getNodeValIf(node.children[1], "preproc_arg")
+
+    if name.nBl and val.nBl and name notin gConsts:
+      gConsts.add(name)
+      if val.getType().nBl:
+        # #define NAME VALUE
+        gConstStr &= &"  {name.getIdentifier()}* = {val} # preprocDef()\n"
+
+#
+# Types
+#
+
+proc typeScan(node: Ast, sym, identifier, offset: string): string =
+  if node.sym != sym or node.children.len() != 2:
+    return
+
+  let
+    pname = getNodeValIf(node.children[1], identifier)
+    ptyp = getNodeValIf(node.children[0], "primitive_type")
+    ttyp = getNodeValIf(node.children[0], "type_identifier")
+
+  if pname.len() == 0:
+    return
+  elif ptyp.nBl:
+    result = &"{offset}{pname.getIdentifier()}: {ptyp.getType()}"
+  elif ttyp.nBl:
+    result = &"{offset}{pname.getIdentifier()}: {ttyp}"
+  elif node.children[0].sym in ["struct_specifier", "enum_specifier"] and node.children[0].children.len() == 1:
+    let styp = getNodeValIf(node.children[0].children[0], "type_identifier")
+    if styp.nBl:
+      result = &"{offset}{pname.getIdentifier()}: {styp}"
+  else:
+    return
+
+proc structSpecifier(node: Ast, name = "") =
+  var stmt: string
+  if node.children.len() == 1 and name notin gTypes:
+    case node.children[0].sym:
+      of "type_identifier":
+        let typ = getNodeValIf(node.children[0], "type_identifier")
+        if typ.nBl:
+          # typedef struct X Y
+          gTypes.add(name)
+          gTypeStr &= &"  {name}* = {typ} #1 structSpecifier()\n"
+
+      of "field_declaration_list":
+        # typedef struct { fields } X
+        stmt = &"  {name}* {{.importc: \"{name}\", header: {gCurrentHeader}, bycopy.}} = object #2 structSpecifier()\n"
+
+        for field in node.children[0].children:
+          let ts = typeScan(field, "field_declaration", "field_identifier", "    ")
+          if ts.len() == 0:
+            return
+          stmt &= ts & "\n"
+
+        gTypes.add(name)
+        gTypeStr &= stmt
+  elif name.len() == 0 and node.children.len() == 2 and node.children[1].sym == "field_declaration_list":
+    let ename = getNodeValIf(node.children[0], "type_identifier")
+    if ename.nBl and ename notin gTypes:
+      # struct X { fields }
+      stmt &= &"  {ename}* {{.importc: \"struct {ename}\", header: {gCurrentHeader}, bycopy.}} = object #3 structSpecifier()\n"
+
+      for field in node.children[1].children:
+        let ts = typeScan(field, "field_declaration", "field_identifier", "    ")
+        if ts.len() == 0:
+          return
+        stmt &= ts & "\n"
+
+      gTypes.add(name)
+      gTypeStr &= stmt
+
+proc enumSpecifier(node: Ast, name = "") =
+  var
+    ename: string
+    elid: int
+    stmt: string
+
+  if node.children.len() == 1 and node.children[0].sym == "enumerator_list":
+    # typedef enum { fields } X
+    ename = name
+    elid = 0
+    stmt = &"  {name}* = enum #1 enumSpecifier()\n"
+  elif name.len() == 0 and node.children.len() == 2 and node.children[1].sym == "enumerator_list":
+    ename = getNodeValIf(node.children[0], "type_identifier")
+    elid = 1
+    if ename.nBl:
+      # enum X { fields }
+      stmt = &"  {ename}* = enum #2 enumSpecifier()\n"
+    else:
+      return
+
+  for field in node.children[elid].children:
+    if field.sym == "enumerator":
+      let fname = getNodeValIf(field.children[0], "identifier")
+      if field.children.len() == 1:
+        stmt &= &"    {fname}\n"
+      elif field.children.len() == 2 and field.children[1].sym == "number_literal":
+        let num = getNodeValIf(field.children[1], "number_literal")
+        stmt &= &"    {fname} = {num}\n"
+      else:
+        return
+
+  if ename notin gTypes:
+    gTypes.add(name)
+    gTypeStr &= stmt
+
+proc typeDefinition(node: Ast) =
+  if node.children.len() == 2:
+    let
+      name = getNodeValIf(node.children[1], "type_identifier")
+      ptyp = getNodeValIf(node.children[0], "primitive_type")
+      ttyp = getNodeValIf(node.children[0], "type_identifier")
+
+    if name.nBl and name notin gTypes:
+      if ptyp.nBl:
+        # typedef int X
+        gTypes.add(name)
+        gTypeStr &= &"  {name}* = {ptyp.getType()} #1 typeDefinition()\n"
+      elif ttyp.nBl:
+        # typedef X Y
+        gTypes.add(name)
+        gTypeStr &= &"  {name}* = {ttyp} #2 typeDefinition()\n"
+      else:
+        case node.children[0].sym:
+          of "struct_specifier":
+            structSpecifier(node.children[0], name)
+          of "enum_specifier":
+            enumSpecifier(node.children[0], name)
+
+proc functionDeclarator(node: Ast, typ: string) =
+  if node.children.len() == 2:
+    let
+      name = getNodeValIf(node.children[0], "identifier")
+
+    if name.nBl and name notin gProcs and node.children[1].sym == "parameter_list":
+      # typ function(typ param1, ...)
+      var stmt = &"# functionDeclarator()\nproc {name}*("
+
+      for i in 0 .. node.children[1].children.len()-1:
+        let ts = typeScan(node.children[1].children[i], "parameter_declaration", "identifier", "")
+        if ts.len() == 0:
+          return
+        stmt &= ts
+        if i != node.children[1].children.len()-1:
+          stmt &= ", "
+
+      if typ != "void":
+        stmt &= &"): {typ.getType()} "
+      else:
+        stmt &= ") "
+
+      stmt &= &"{{.importc: \"{name}\", header: {gCurrentHeader}.}}\n"
+
+      gProcs.add(name)
+      gProcStr &= stmt
+
+proc declaration*(node: Ast) =
+  if node.children.len() == 2 and node.children[1].sym == "function_declarator":
+    let
+      ptyp = getNodeValIf(node.children[0], "primitive_type")
+      ttyp = getNodeValIf(node.children[0], "type_identifier")
+
+    if ptyp.nBl:
+      functionDeclarator(node.children[1], ptyp.getType())
+    elif ttyp.nBl:
+      functionDeclarator(node.children[1], ttyp)
+    elif node.children[0].sym == "struct_specifier" and node.children[0].children.len() == 1:
+      let styp = getNodeValIf(node.children[0].children[0], "type_identifier")
+      if styp.nBl:
+        functionDeclarator(node.children[1], styp)
+
+proc genNimAst*(node: Ast) =
+  case node.sym:
+    of "ERROR":
+      let (line, col) = getLineCol(node)
+      echo &"Potentially invalid syntax at line {line} column {col}"
+    of "preproc_def":
+      preprocDef(node)
+    of "type_definition":
+      typeDefinition(node)
+    of "declaration":
+      declaration(node)
+    of "struct_specifier":
+      if node.parent.sym notin ["type_definition", "declaration"]:
+        structSpecifier(node)
+    of "enum_specifier":
+      if node.parent.sym notin ["type_definition", "declaration"]:
+        enumSpecifier(node)
+
+  for child in node.children:
+    genNimAst(child)