In the long term, we might need a different language

[poke1024]

This obviously is a bit crazy, and I won't have time to even investigate this in detail, but still I want to post the idea to port Mathics to Swift (https://swift.org/) or Rust(https://www.rust-lang.org/) here for reference, since I strongly believe it would carry Mathics to a place where it's highly competitive in terms of speed (esp. compared to packages like R, GNU Octave, and so on) and Swift or Rust would be the right languages for this (as compared to C++11, Go, and other typed languages).

Even though Python allows very quick growth and development of Mathics at the current point, and it's the ideal language for feature extension, the better choice for the core engine (everything that doesn't interface with libs) would be Swift or Rust. This is crazy as it's a complete rewrite concerning also the tool chain.

On the other hand, Swift and Rust are both very Python-like, which means the code would look very similar; yet using a statically typed, compiled language would takes Mathic to a different league in terms of speed and memory consumption:

http://www.marcinkliks.pl/2015/02/22/swift-vs-others/

[ghost]

But there's PyPy :(

[ghost]

I know PyPy is slower, but... yeah

[sn6uv]

It's certainly a crazy idea but it's one I've had before. I'm not familiar enough with either Rust or Swift to comment on their suitability specifically.

Off the top of my head, a few approaches/techniques I've considered:

Bootstrapping

Rewrite the core of Mathics in something low-level and as much extension stuff as possible in Wolfram code.

Pros:

• Less code to optimise.
• Large cross-over between those familiar with Wolfram syntax

Cons:

• Wolfram is not nice for writing compilers.
• Performance might mean shifting lots of things to the core.

We've been starting to do this already with the Autoload stuff but poor performance is limiting this at the moment.

Compilable Wolfram code

This is something that only occurred to me recently while working on #548 and builds on the 'Bootstrapping' approach above. Basically we find a subset of the Wolfram language compliable to LLVM IR and implement.

Pros:

• super fast Wolfram code

Cons:

• no nice features: bigints, python libs, etc.
• We're trying writing a compiler in a low-level, statically typed subset of the Wolfram language (Insane)

Embedding a Python intepreter

The idea is that we embed a python intepreter (like PyPy) within a compiled program (written something low-level) and re-use the existing Mathics codebase as much as possible.

Pros:

• Gradual porting to a rewritten core
• Performance should be no worse than it currently is
• Lots of extension code we'd want to run fits into the apply_something pattern which makes things simpler.

Cons:

• The codebase now consists of Wolfram, Python, and some additional low-level language.
• It's not clear how to structure this in a sane way.

Docs on embedding PyPy: http://doc.pypy.org/en/latest/embedding.html

Rewriting the core in RPython

RPython is a subset of the python language that implements PyPy. We could try porting the core to that subset and compiling it.

Pros:

• Performance, we can reuse the PyPy JIT
• RPython is not Python but it is similar
• Some success has been had with Ruby: http://docs.topazruby.com/en/latest/

Cons:

• RPython is not Python and can be frustrating to work with
• Difficult to re-use the existing Python codebase without embedding PyPy (we probably don't want to rewrite everything in RPython).
• requires quite a bit of study of the PyPy codebase to get the hang of things

This is something I started to look into but didn't get very far. It certainly looked promising though.

Rewriting the core as a C-extension / Using Cython more

Pros:

• Performance
• Well supported
• Gradual development from the current interpreter

Cons:

• no PyPy?
• having to write C code

Rewriting the core in Lisp

Might as well embrace Greenspun's tenth rule. An implementation with TCO would fix some issues with the tree-walking evaluator.

[wolfv]

I think the speed of the things that are really slow and happen in SymPy will become much faster once symengine is feature complete and used as SymPy backend (https://github.com/symengine/symengine).
Symengine is written in C++(11).

[sn6uv]

@wolfv absolutely, rewriting the mathics core in something lower-level than python might even allow us to call into symengine directly. The fact that sympy is currently implemented in python is a major hurdle for us moving to anything else right now.

[sn6uv]

I've got an experiment rewriting the Mathics core in C, see https://github.com/sn6uv/cmathics. I'm not convinced it's actually going to work out but I'm certainly getting a feel for how the core could be written in a statically typed language.

[isuruf]

@sn6uv, symengine's got all of the functionality in cmathics except for the pattern matching, but very less if you compared to SymPy. This should give a pretty good idea about what's implemented, https://github.com/symengine/symengine/blob/master/symengine/cwrapper.h
SymEngine (C++) has bindings to C, Python, Julia and Ruby.

[corywalker]

I've been working on a Golang version of something very similar to this. My intent was not to create an alternative to Mathics. I chose not to look at the source code here because I've been enjoying the journey of understanding how everything works from the outside. There are some very fun algorithms embedded in the workings of the language. The result of my work is located here. I thought that I would update the developers here. I haven't bothered to optimize the replacement engine too much (or find concurrency in the evaluation sequence), so it has the potential to be faster. This has been one of my favorite programming projects, and I hope some others will find it useful.

[ghost]

By the way, there's Nim language - it's very powerful, but unpopular :(
It can be compiled to C, C++, Objective C, and JS.
So if you compile to C (it's by default), you can easily create bindings to something.

Also it's very performant, thanks for years of hard work of GCC and CLang developers

[ghost]

@TiberiumN is there a reason it's unpopular, if it's so great?

[ghost]

@pavelbraginskiy
Because there's no a lot of sponsors.

[poke1024]

I like Nim, but I'm not so sure about the stability of its toolchain and its future support at this stage. Another language that looks very promising for building compilers and interpreters is terralang (http://terralang.org/). It's great stuff.

For my part, I'm still trying to push the C++ experiment Angus started, to see where it goes. The compile times are horrible due my (excessive?) use of templates and the code is complex, and I wonder whether this is fixable. On the plus side, it can handle roughly 500 test cases now and it's fast. You can take a look at the current state of things at https://github.com/poke1024/cmathics (just look at the embedded test cases to see what works).