Merge cpp

.gitignore

@@ -86,6 +86,10 @@ bilder.pyc
 bild.log
 
 bild_output.txt
+runtime/Cpp/demo/generated
+xcuserdata
+*.jar
+.vscode
 
 # VSCode Java plugin temporary files
 javac-services.0.log

.travis.yml

@@ -2,10 +2,12 @@ sudo: true
 language: java
 script:
   - mvn install
+
 jdk:
   - openjdk6
   - oraclejdk7
   - oraclejdk8
+
 before_install:
   - sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
   - sudo add-apt-repository ppa:fkrull/deadsnakes -y
@@ -19,3 +21,17 @@ before_install:
   - go version ; go env
   - python --version
   - python3 --version
+
+env:
+  - CXX=g++-5
+
+compiler:
+  - clang
+
+addons:
+  apt:
+    sources:
+      - ubuntu-toolchain-r-test
+    packages:
+      - g++-5
+      - uuid-dev

README.md

@@ -12,6 +12,9 @@ ANTLR project lead and supreme dictator for life
 * [Sam Harwell](http://tunnelvisionlabs.com/) (Tool co-author, Java and C# target)
 * Eric Vergnaud (Javascript, Python2, Python3 targets and significant work on C# target)
 * [Peter Boyer](https://github.com/pboyer) (Go target)
+* [Mike Lischke](http://www.soft-gems.net/) (C++ completed target)
+* Dan McLaughlin (C++ initial target)
+* David Sisson (C++ initial target and test)
 
 ## Useful information
 
@@ -20,7 +23,7 @@ ANTLR project lead and supreme dictator for life
 * [Official site](http://www.antlr.org/)
 * [Documentation](https://github.com/antlr/antlr4/blob/master/doc/index.md)
 * [FAQ](https://github.com/antlr/antlr4/blob/master/doc/faq/index.md)
-* [ANTLR code generation targets](https://github.com/antlr/antlr4/blob/master/doc/targets.md) (Currently: Java, C#, Python2|3, JavaScript, Go)
+* [ANTLR code generation targets](https://github.com/antlr/antlr4/blob/master/doc/targets.md)<br>(Currently: Java, C#, Python2|3, JavaScript, Go, C++)
 * [Java API](http://www.antlr.org/api/Java/index.html)
 * [ANTLR v3](http://www.antlr3.org/)
 * [v3 to v4 Migration, differences](https://github.com/antlr/antlr4/blob/master/doc/faq/general.md)

contributors.txt

@@ -1,4 +1,4 @@
-ANTLR Project Contributors Certification of Origin and Rights
+ANTLR Project Contributors Certification of Origin and Rights
 
 All contributors to ANTLR v4 must formally agree to abide by this
 certificate of origin by signing on the bottom with their github
@@ -58,6 +58,7 @@ YYYY/MM/DD, github id, Full name, email
 2014/03/18, aphyr, Kyle Kingsbury, aphyr@aphyr.com
 2014/06/07, ericvergnaud, Eric Vergnaud, eric.vergnaud@wanadoo.fr
 2014/07/04, jimidle, Jim Idle, jimi@Idle.ws
+2014/01/01, danmclaughlin, Dan McLaughlin, dan.mclaughlin@gmail.com
 2014/09/04. jeduden, Jan-Eric Duden, jeduden@gmail.com
 2014/09/27, petrbel, Petr Bělohlávek, antlr@petrbel.cz
 2014/10/18, sergiusignacius, Sérgio Silva, serge.a.silva@gmail.com
@@ -89,13 +90,21 @@ YYYY/MM/DD, github id, Full name, email
 2015/12/23, pboyer, Peter Boyer, peter.b.boyer@gmail.com
 2015/12/24, dtymon, David Tymon, david.tymon@gmail.com
 2016/02/18, reitzig, Raphael Reitzig, reitzig[at]cs.uni-kl.de
+2016/03/10, mike-lischke, Mike Lischke, mike@lischke-online.de
 2016/03/27, beardlybread, Bradley Steinbacher, bradley.j.steinbacher@gmail.com
 2016/03/29, msteiger, Martin Steiger, antlr@martin-steiger.de
 2016/03/28, gagern, Martin von Gagern, gagern@ma.tum.de
+2016/07/20, chrisheller, Chris Heller, chris.heller.greyheller@gmail.com
+2016/07/20, nburles, Nathan Burles, nburles@gmail.com
+2016/07/20, kosl90, Li Liqiang, kos1990l@gmail.com
+2016/07/27, timoc, Tim O'Callaghan, timo@linux.com
+2016/07/26, nic30, Michal Orsák, michal.o.socials@gmail.com
 2016/07/18, willfaught, Will Faught, will.faught@gmail.com
 2016/08/08, wjkohnen, Wolfgang Johannes Kohnen, wjkohnen-go-antlr@ko-sys.com
 2016/08/11, BurtHarris, Ralph "Burt" Harris, Burt_Harris_antlr4@azxs.33mail.com
 2016/08/19, andjo403, Andreas Jonson, andjo403@hotmail.com
 2016/10/13, cgudrian, Christian Gudrian, christian.gudrian@gmx.de
 2016/10/13, nielsbasjes, Niels Basjes, niels@basjes.nl
+2016/10/21, FloorGoddijn, Floor Goddijn, floor.goddijn[at]aimms.com
+2016/11/01, RYDB3RG, Kai Stammerjohann, RYDB3RG@users.noreply.github.com
 2016/11/05, runner-mei, meifakun, runner.mei@gmail.com

doc/building-antlr.md

@@ -112,7 +112,8 @@ antlr4-tool-testsuite/4.5.2-SNAPSHOT/antlr4-tool-testsuite-4.5.2-SNAPSHOT.jar
 
 Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.5.2-SNAPSHOT purposes.
 
-To build without running the tests (saves about 8 minutes), do this:
+## Building without testing
+To build without running the tests (saves a lot of time), do this:
 
 ```bash
 mvn -DskipTests install
@@ -122,4 +123,4 @@ mvn -DskipTests install
 
 After download ANTLR source, just "import project from existing sources" and click on the "Maven Projects" tab in right gutter of IDE. It should build stuff in the background automatically and look like:
 
-<img src=images/intellij-maven.png width=200>
+<img src=images/intellij-maven.png width=200>

doc/cpp-target.md

@@ -0,0 +1,137 @@
+# C++
+
+The C++ target supports all platforms that can either run MS Visual Studio 2013 (or newer), XCode 7 (or newer) or CMake (C++11 required). All build tools can either create static or dynamic libraries, both as 64bit or 32bit arch. Additionally, XCode can create an iOS library.
+
+## How to create a C++ lexer or parser?
+This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
+
+```
+$ antlr4 -Dlanguage=Cpp MyGrammar.g4
+```
+
+You will see that there are a whole bunch of files generated by this call. If visitor or listener are not suppressed (which is the default) you'll get:
+
+* MyGrammarLexer.h + MyGrammarLexer.cpp
+* MyGrammarParser.h + MyGrammarParser.cpp
+* MyGrammarVisitor.h + MyGrammarVisitor.cpp
+* MyGrammarBaseVisitor.h + MyGrammarBaseVisitor.cpp
+* MyGrammarListener.h + MyGrammarListener.cpp
+* MyGrammarBaseListener.h + MyGrammarBaseListener.cpp
+
+## Where can I get the runtime?
+
+Once you've generated the lexer and/or parser code, you need to download or build the runtime. Prebuilt C++ runtime binaries for Windows (VS 2013 runtime), OSX and iOS are available on the ANTLR web site:
+
+* http://www.antlr.org
+
+Use CMake to build a Linux library (works also on OSX, however not for the iOS library). Building your own library on OSX or Windows is trivial, however. Just open the VS or XCode project, select target + arch and build it. Should work out of the box without any additional dependency.
+
+
+## How do I run the generated lexer and/or parser?
+
+Putting it all together to get a working parser is really easy. Look in the [runtime/Cpp/demo](../runtime/Cpp/demo) folder for a simple example. The [README](../runtime/Cpp/demo/README.md) there describes shortly how to build and run the demo on OSX, Windows or Linux.
+
+## How do I create and run a custom listener?
+
+The generation step above created a listener and base listener class for you. The listener class is an abstract interface, which declares enter and exit methods for each of your parser rules. The base listener implements all those abstract methods with an empty body, so you don't have to do it yourself if you just want to implement a single function. Hence use this base listener as the base class for your custom listener:
+
+```c++
+#include <iostream>
+
+#include "antlr4-runtime.h"
+#include "MyGrammarLexer.h"
+#include "MyGrammarParser.h"
+#include "MyGrammarBaseListener.h"
+
+using namespace org::antlr::v4::runtime;
+
+class TreeShapeListener : public MyGrammarBaseListener {
+public:
+  void enterKey(Ref<ParserRuleContext> ctx) {
+	// Do something when entering the key rule.
+  }
+};
+
+
+int main(int argc, const char* argv[]) {
+  std::ifstream stream;
+  stream.open(argv[1]);
+  ANTLRInputStream input(stream);
+  MyGrammarLexer lexer(&input);
+  CommonTokenStream tokens(&lexer);
+  MyGrammarParser parser(&tokens);
+
+  Ref<tree::ParseTree> tree = parser.key();
+  Ref<TreeShapeListener> listener(new TreeShapeListener());
+  tree::ParseTreeWalker::DEFAULT->walk(listener, tree);
+
+  return 0;
+}
+
+```
+
+This example assumes your grammar contains a parser rule named `key` for which the enterKey function was generated. The `Ref<>` template is an alias for `std::shared_ptr<>` to simplify the runtime source code which often makes use of smart pointers.
+
+## Specialities of this ANTLR target
+
+There are a couple of things that only the C++ ANTLR target has to deal with. They are described here.
+
+### Memory Management
+Since C++ has no built-in memory management we need to take extra care. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable. Generally, when you see a raw pointer in code consider this as being managed elsewehere. You should never try to manage such a pointer (delete, assign to smart pointer etc.).
+
+### Unicode Support
+Encoding is mostly an input issue, i.e. when the lexer converts text input into lexer tokens. The parser is completely encoding unaware. However, lexer input in the grammar is defined by character ranges with either a single member (e.g. 'a' or [a] or [abc]), an explicit range (e.g. 'a'..'z' or [a-z]), the full Unicode range (for a wildcard) and the full Unicode range minus a sub range (for negated ranges, e.g. ~[a]). The explicit ranges (including single member ranges) are encoded in the serialized ATN by 16bit numbers, hence cannot reach beyond 0xFFFF (the Unicode BMP), while the implicit ranges can include any value (and hence support the full Unicode set, up to 0x10FFFF).
+
+> An interesting side note here is that the Java target fully supports Unicode as well, despite the inherent limitations from the serialized ATN. That's possible because the Java String class represents characters beyond the BMP as surrogate pairs (two 16bit values) and even reads them as 2 separate input characters. To make this work a character range for an identifier in a grammar must include the surrogate pairs area (for a Java parser).
+
+The C++ target however always expects UTF-8 input (either in a string or via a wide stream) which is then converted to UTF-32 (a char32_t array) and fed to the lexer. ANTLR, when parsing your grammar, limits character ranges explicitly to the BMP currently. So, in order to allow specifying the full Unicode set the C++ target uses a little trick: whenever an explicit character range includes the (unused) codepoint 0xFFFF in a grammar it is silently extended to the full Unicode range. It's clear that this is an all-or-nothing solution. You cannot define a subset of Unicode codepoints > 0xFFFF that way. This can only be solved if ANTLR supports larger character intervals.
+
+The differences in handling characters beyond the BMP leads to a difference between Java and C++ lexers: the character offsets may not concur. This is because Java reads two 16bit values per Unicode char (if that falls into the surrogate area) while a C++ parser only reads one 32bit value. That usually doesn't have practical consequences, but might confuse people when comparing token positions.
+
+### Named Actions
+In order to help customizing the generated files there are a number of additional socalled **named actions**. These actions are tight to specific areas in the generated code and allow to add custom (target specific) code. All targets support these actions 
+
+* @parser::header
+* @parser::members
+* @lexer::header
+* @lexer::members
+
+(and their scopeless alternatives `@header` and `@members`) where header doesn't mean a C/C++ header file, but the top of a code file. The content of the header action appears in all generated files at the first line. So it's good for things like license/copyright information.
+
+The content of a *members* action is placed in the public section of lexer or parser class declarations. Hence it can be used for public variables or predicate functions used in a grammar predicate. Since all targets support *header* + *members* they are the best place for stuff that should be available also in generated files for other languages.
+
+In addition to that the C++ target supports many more such named actions. Unfortunately, it's not possible to define new scopes (e.g. *listener* in addition to *parser*) so they had to be defined as part of the existing scopes (*lexer* or *parser*). The grammar in the demo application contains all of the named actions as well for reference. Here's the list:
+
+* **@lexer::preinclude** - Placed right before the first #include (e.g. good for headers that must appear first, for system headers etc.). Appears in both lexer h and cpp file.
+* **@lexer::postinclude** - Placed right after the last #include, but before any class code (e.g. for additional namespaces). Appears in both lexer h and cpp file.
+* **@lexer::context** - Placed right before the lexer class declaration. Use for e.g. additional types, aliases, forward declarations and the like. Appears in the lexer h file.
+* **@lexer::declarations** - Placed in the private section of the lexer declaration (generated sections in all classes strictly follow the pattern: public, protected, privat, from top to bottom). Use this for private vars etc.
+* **@lexer::definitions** - Placed before other implementations in the cpp file (but after *@postinclude*). Use this to implement e.g. private types.
+
+For the parser there are the same actions as shown above for the lexer. In addition to that there are even more actions for visitor and listener classes:
+
+* **@parser::listenerpreinclude**
+* **@parser::listenerpostinclude**
+* **@parser::listenerdeclarations**
+* **@parser::listenermembers**
+* **@parser::listenerdefinitions**
+* 
+* **@parser::baselistenerpreinclude**
+* **@parser::baselistenerpostinclude**
+* **@parser::baselistenerdeclarations**
+* **@parser::baselistenermembers**
+* **@parser::baselistenerdefinitions**
+* 
+* **@parser::visitorpreinclude**
+* **@parser::visitorpostinclude**
+* **@parser::visitordeclarations**
+* **@parser::visitormembers**
+* **@parser::visitordefinitions**
+* 
+* **@parser::basevisitorpreinclude**
+* **@parser::basevisitorpostinclude**
+* **@parser::basevisitordeclarations**
+* **@parser::basevisitormembers**
+* **@parser::basevisitordefinitions**
+
+and should be self explanatory now. Note: there is no *context* action for listeners or visitors, simply because they would be even less used than the other actions and there are so many already.

doc/creating-a-language-target.md

@@ -19,4 +19,4 @@ Creating a new target involves the following key elements:
 ```bash
 $ mvn compile
 ```
-That should proceed with success. See [Building ANTLR](building-antlr.md) for more details. (That link does not currently work as I have that documentation in a branch. see https://github.com/parrt/antlr4/blob/move-doc-to-repo/doc/building-antlr.md for now.)
+That should proceed with success. See [Building ANTLR](building-antlr.md) for more details.

doc/releasing-antlr.md

@@ -24,11 +24,14 @@ Edit the repository looking for 4.5 or whatever and update it. Bump version in t
  * runtime/CSharp/runtime/CSharp/Antlr4.Runtime/Properties/AssemblyInfo.cs
  * runtime/JavaScript/src/antlr4/package.json
  * runtime/JavaScript/src/antlr4/Recognizer.js
+ * tool/src/org/antlr/v4/codegen/target/CppTarget.java
  * tool/src/org/antlr/v4/codegen/target/CSharpTarget.java
  * tool/src/org/antlr/v4/codegen/target/JavaScriptTarget.java
  * tool/src/org/antlr/v4/codegen/target/Python2Target.java
  * tool/src/org/antlr/v4/codegen/target/Python3Target.java
-
+ * runtime/Cpp/VERSION
+ * runtime/Cpp/RuntimeMetaData.cpp
+
 Here is a simple script to display any line from the critical files with, say, `4.5` in it:
 
 ```bash
@@ -240,6 +243,52 @@ python setup.py sdist bdist_wininst upload -r pypi
 
 Add links to the artifacts from download.html
 
+### C++
+
+The C++ target is the most complex one, because it addresses multiple platforms, which require individual handling. We have 4 scenarios to cover:
+
+* **Windows**: static and dynamic libraries for the VC++ runtime 2013 or 2015 (corresponding to Visual Studio 2013 or 2015) + header files. All that in 32 and 64bit, debug + release.
+* **MacOS**: static and dynamic release libraries + header files.
+* **iOS**: no prebuilt binaries, but just a zip of the source, including the XCode project to build everything from source.
+* **Linux**: no prebuilt binaries, but just a zip of the source code, including the cmake file to build everything from source there.
+
+In theory we could also create a library for iOS, but that requires to sign it, which depends on an active iOS developer account. So we leave this up to the ANTLR user to build the iOS lib, like we do for Linux builds.
+
+For each platform there's a deployment script which generates zip archives and copies them to the target folder. The Windows deployment script must be run on a machine with VS 2013 + VS 2015 installed. The Mac script must be run on a machine with XCode 7+ installed. The source script can be executed on any Linux or Mac box.
+
+On a Mac (with XCode 7+ installed):
+
+```bash
+cd runtime/Cpp
+./deploy-macos.sh
+```
+
+On any Mac or Linux machine:
+
+```bash
+cd runtime/Cpp
+./deploy-source.sh
+```
+
+On a Windows machine the build scripts checks if VS 2013 and/or VS 2015 are installed and builds binaries for each, if found. This script requires 7z to be installed (http://7-zip.org).
+
+```bash
+cd runtime/Cpp
+deploy-windows.cmd
+```
+
+Move target to website (**_rename to a specific ANTLR version first if needed_**):
+
+```bash
+pushd ~/antlr/sites/website-antlr4/download
+git add antlr4cpp-runtime-macos.zip
+git add antlr4cpp-runtime-windows.zip
+git add antlr4cpp-runtime-source.zip
+git commit -a -m 'update C++ runtime'
+git push origin gh-pages
+popd
+```
+
 ## Update javadoc for runtime and tool
 
 First gen javadoc: