DARMA-tasking · lifflander · Jun 22, 2020 · Feb 5, 2020 · Feb 5, 2020 · Jun 15, 2020
diff --git a/.codacy.yml b/.codacy.yml
@@ -1,3 +1,4 @@
 exclude_paths:
   - 'tests/extern/**'
   - 'lib/**/*'
+  - 'docs/**/*'
diff --git a/README.md b/README.md
@@ -29,126 +29,15 @@ they can be automatically migrated and thereby executed on varying hardware
 resources without explicit programmer mapping, location, and communication
 management.
 
-## Building
-
-*vt* can be built with `cmake` or built inside a `docker` container.
-
-To build *vt*, one must obtain the following dependencies:
-
-### Optional:
-
-#### If threading is enabled:
-  - OpenMP       _or_
-  - `std::thread`s (default to from C++ compiler)
-
-#### Required:
-  - detector,   (*vt* ecosystem)
-  - checkpoint, (*vt* ecosystem)
-  - MPI         (mpich/openmpi/mvapich/IBM Spectrum MPI/Cray MPICH/etc.)
-
-### Automatically build dependencies
-
-Assuming MPI is installed and accessible via CC/CXX, the only other dependencies
-that are required are checkpoint and detector. The easiest way to get these
-built are to clone them inside `vt/lib`:
-
-```bash
-$ git clone git@github.com:DARMA-tasking/vt
-$ cd vt/lib
-$ git clone git@github.com:DARMA-tasking/checkpoint
-$ git clone git@github.com:DARMA-tasking/detector
-```
-
-With these in `vt/lib`, cmake will automatically build them and stitch them into
-*vt*'s linking process.
-
-Instead of running `cmake`, one may invoke the `vt/ci/build_cpp.sh` script which
-will run `cmake` for *vt* with environment variables for most configuration
-parameters.
-
-#### Environment Variables
-
-| Variable                    | Default Value   | Description |
-| ------------------          | --------------- | ----------- |
-| `CMAKE_BUILD_TYPE`          | Release         | The `cmake` build type |
-| `VT_LB_ENABLED`             | ON              | Enable compile-time load balancing support |
-| `VT_TRACE_ENABLED `         | OFF             | Enable compile-time tracing support |
-| `VT_TRACE_RUNTIME_ENABLED ` | OFF             | Force tracing on at runtime (used in CI for automatically testing tracing on all tests/examples) |
-| `VT_DOXYGEN_ENABLED `       | OFF             | Enable doxygen generation |
-| `VT_MIMALLOC_ENABLED `      | OFF             | Enable `mimalloc`, alternative allocator for debugging memory usage/frees/corruption |
-| `VT_ASAN_ENABLED `          | OFF             | Enable building with address sanitizer |
-| `VT_POOL_ENABLED `          | ON              | Use memory pool in *vt* for message allocation |
-| `VT_ZOLTAN_ENABLED `        | OFF             | Build with Zoltan enabled for `ZoltanLB` support |
-| `ZOLTAN_DIR `               | (empty)         | Directory pointing to Zoltan installation |
-| `VT_MPI_GUARD_ENABLED `     | OFF             | Enable compile-time load balancing support |
-
-With these set, invoke the script with two arguments: the path to the *vt* root directory and the build path. Here's an example assuming that *vt* is cloned into `/usr/src/vt` with trace enabled in debug mode.
-
-**Usage for building:**
+## Read the documentation
 
-```bash
-$ vt/ci/build_cpp.sh <full-path-to-vt-source> <full-path-to-build-dir>
-```
+To learn *vt*, read the [full
+documentation](https://darma-tasking.github.io/docs/html/index.html) that is
+automatically generated whenever a push occurs to "develop". It includes a
+walk-though of the tutorial and a overview of the components that make up a *vt*
+runtime.
 
-**Example:**
-
-```bash
-$ cd /usr/src
-$ git clone git@github.com:DARMA-tasking/vt
-$ VT_TRACE_ENABLED=1 CMAKE_BUILD_TYPE=Debug /usr/src/vt/ci/build_cpp.sh /usr/src/vt /usr/build/vt
-```
-
-### Building with `docker` containerization
-
-The easiest way to build *vt* is by using `docker` with the available containers that contain the proper compilers, MPI, and all other dependencies. First, install `docker` on the system. On some systems, `docker-compose` might also need to be installed.
-
-The `docker` builds are configured through `docker-compose` to use a shared, cached filesystem mount with the host for `ccache` to enable fast re-builds.
-
-For `docker-compose`, the following variables can be set to configure the build. One may configure the architecture, compiler type and version, Linux distro (ubuntu or alpine), and distro version.
-
-```
-# Variables:
-#   ARCH={amd64, arm64v8, ...}
-#   COMPILER_TYPE={gnu, clang}
-#   COMPILER={gcc-5, gcc-6, gcc-7, gcc-8, gcc-9, gcc-10,
-#             clang-3.9, clang-4.0, clang-5.0, clang-6.0, clang-7, clang-8,
-#             clang-9, clang-10}
-#   REPO=lifflander1/vt
-#   UBUNTU={18.04, 20.04}
-#   ULIMIT_CORE=0
-#
-# DARMA/vt Configuration Variables:
-#   VT_LB=1              # Enable load balancing
-#   VT_TRACE=0           # Enable tracing
-#   VT_MIMALLOC=0        # Enable mimalloc memory allocator
-#   VT_DOCS=0            # Enable doxygen build
-#   VT_TRACE_RT=0        # Enable tracing at runtime (for testing)
-#   VT_ASAN=0            # Enable address sanitizer
-#   BUILD_TYPE=release   # CMake build type
-```
-
-With these set, run the following for a non-interactive build:
-
-```bash
-$ cd vt
-$ docker-compose run -e BUILD_TYPE=debug -e VT_TRACE=1 ubuntu-cpp
-```
-
-For an interactive build, where one can build, debug, and run `valgrind`, etc:
-
-```bash
-$ cd vt
-$ docker-compose run -e BUILD_TYPE=debug -e VT_TRACE=1 ubuntu-cpp-interactive
-$ /vt/ci/build_cpp.sh /vt /build
-$ /vt/ci/test_cpp.sh /vt /build
-```
-
-For more detailed information on configuring the docker build, read the documentation in `vt/docker-compose.yml`.
-
-## Testing
-
-After *vt* is built succesfully, one may invoke the tests several ways. One may run `make test` or `ninja test` (depending on the generator used) or `ctest`, to run all the tests. Alternatively, the tests can be run automatically from the CI script:
+## Building
 
-```bash
-$ vt/ci/test_cpp.sh <full-path-to-vt-source> <full-path-to-build-dir>
-```
+[Learn how to build](https://darma-tasking.github.io/docs/html/vt-build.html)
+*vt* with `cmake` or `docker`.
diff --git a/cmake/load_doxygen.cmake b/cmake/load_doxygen.cmake
@@ -9,8 +9,16 @@ if (${vt_doxygen_enabled})
     set(doxygen_in ${CMAKE_CURRENT_SOURCE_DIR}/docs/Doxyfile.in)
     set(doxygen_out ${CMAKE_CURRENT_BINARY_DIR}/Doxyfile)
 
+    set(DOXYGEN_PROJECT_NAME "vt")
+    set(VERSION_MAJOR "1")
+    set(VERSION_MINOR "0")
+    set(VERSION_PATCH "0")
     set(DOXYGEN_INPUT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/src/")
+    set(DOXYGEN_DOCS_DIR "${CMAKE_CURRENT_SOURCE_DIR}/docs/")
+    set(DOXYGEN_EXAMPLE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/examples/")
+    set(DOXYGEN_TUTORIAL_DIR "${CMAKE_CURRENT_SOURCE_DIR}/tutorial/")
     set(DOXYGEN_OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}/docs/")
+    set(DOXYGEN_MAIN_PAGE "${CMAKE_CURRENT_SOURCE_DIR}/src/vt.md")
     set(DOXYGEN_INDEX_FILE ${DOXYGEN_OUTPUT_DIR}/xml/index.xml)
 
     configure_file(${doxygen_in} ${doxygen_out} @ONLY)

diff --git a/docs/Doxyfile.in b/docs/Doxyfile.in
@@ -32,7 +32,7 @@ DOXYFILE_ENCODING      = UTF-8
 # title of most generated pages and in a few other places.
 # The default value is: My Project.
 
-PROJECT_NAME           = "@CMAKE_PROJECT_NAME@"
+PROJECT_NAME           = "@DOXYGEN_PROJECT_NAME@"
 
 # The PROJECT_NUMBER tag can be used to enter a project or revision number. This
 # could be handy for archiving the generated documentation or if some version
@@ -44,7 +44,7 @@ PROJECT_NUMBER         = @VERSION_MAJOR@.@VERSION_MINOR@.@VERSION_PATCH@
 # for a project that appears at the top of each page and should give viewer a
 # quick idea about the purpose of the project. Keep the description short.
 
-PROJECT_BRIEF          =
+PROJECT_BRIEF          = "(Virtual Transport)"
 
 # With the PROJECT_LOGO tag one can specify a logo or an icon that is included
 # in the documentation. The maximum height of the logo should not exceed 55
@@ -464,7 +464,7 @@ EXTRACT_ALL            = NO
 # be included in the documentation.
 # The default value is: NO.
 
-EXTRACT_PRIVATE        = NO
+EXTRACT_PRIVATE        = YES
 
 # If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal
 # scope will be included in the documentation.
@@ -814,7 +814,8 @@ WARN_LOGFILE           =
 # spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
 # Note: If this tag is empty the current directory is searched.
 
-INPUT                  = "@DOXYGEN_INPUT_DIR@"
+INPUT                  = "@DOXYGEN_INPUT_DIR@" \
+                         "@DOXYGEN_DOCS_DIR@"
 
 # This tag can be used to specify the character encoding of the source files
 # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
@@ -931,21 +932,22 @@ EXCLUDE_SYMBOLS        =
 # that contain example code fragments that are included (see the \include
 # command).
 
-EXAMPLE_PATH           =
+EXAMPLE_PATH           = "@DOXYGEN_EXAMPLE_DIR@" \
+                         "@DOXYGEN_TUTORIAL_DIR@"
 
 # If the value of the EXAMPLE_PATH tag contains directories, you can use the
 # EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and
 # *.h) to filter out the source-files in the directories. If left blank all
 # files are included.
 
-EXAMPLE_PATTERNS       = *
+EXAMPLE_PATTERNS       =
 
 # If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be
 # searched for input files to be used with the \include or \dontinclude commands
 # irrespective of the value of the RECURSIVE tag.
 # The default value is: NO.
 
-EXAMPLE_RECURSIVE      = NO
+EXAMPLE_RECURSIVE      = YES
 
 # The IMAGE_PATH tag can be used to specify one or more files or directories
 # that contain images that are to be included in the documentation (see the
@@ -1007,7 +1009,7 @@ FILTER_SOURCE_PATTERNS =
 # (index.html). This can be useful if you have a project on for instance GitHub
 # and want to reuse the introduction page also for the doxygen output.
 
-USE_MDFILE_AS_MAINPAGE = @doxy_main_page@
+USE_MDFILE_AS_MAINPAGE = "@DOXYGEN_MAIN_PAGE@"
 
 #---------------------------------------------------------------------------
 # Configuration options related to source browsing

diff --git a/docs/Doxyfile.in-mcss b/docs/Doxyfile.in-mcss
@@ -1,6 +1,31 @@
 @INCLUDE                = Doxyfile
 GENERATE_HTML           = YES
 GENERATE_XML            = YES
-XML_PROGRAMLISTING      = NO
+XML_PROGRAMLISTING      = YES
 M_SHOW_UNDOCUMENTED     = YES
-EXTRACT_ALL             = YES
+EXTRACT_ALL             = YES
+
+M_LINKS_NAVBAR1 = \
+  "introduction context active-messenger collection collective group location objgroup pipe pool rdmahandle registry scheduler term trace" \
+  "learn vt-build tutorial examples" \
+  "pages" \
+  "namespaces namespacevt namespacevt_1_1collective namespacevt_1_1debug namespacevt_1_1group namespacevt_1_1location namespacevt_1_1messaging namespacevt_1_1objgroup namespacevt_1_1pipe namespacevt_1_1rdma namespacevt_1_1runtime namespacevt_1_1term namespacevt_1_1vrt"
+
+M_LINKS_NAVBAR2 = \
+  "annotated" \
+  "files" \
+  "<a href=\"https://github.com/DARMA-tasking/vt\">GitHub</a> <a href=\"https://github.com/DARMA-tasking/checkpoint\">Checkpoint</a> <a href=\"https://github.com/DARMA-tasking/detector\">Detector</a> <a href=\"https://github.com/DARMA-tasking/LB-analysis-framework\">LBAF</a> <a href=\"https://github.com/DARMA-tasking/checkpoint-member-analyzer\">Checkpoint Analyzer</a> <a href=\"https://github.com/DARMA-tasking/DARMA-tasking.github.io\">Documentation</a>"
+
+
+ALIASES += \
+    "vt=<b><i>vt</i></b>" \
+    "m_div{1}=@xmlonly<mcss:div xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:class=\"\1\">@endxmlonly" \
+    "m_enddiv=@xmlonly</mcss:div>@endxmlonly" \
+    "m_span{1}=@xmlonly<mcss:span xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:class=\"\1\">@endxmlonly" \
+    "m_endspan=@xmlonly</mcss:span>@endxmlonly" \
+    "m_class{1}=@xmlonly<mcss:class xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:class=\"\1\" />@endxmlonly" \
+    "m_footernavigation=@xmlonly<mcss:footernavigation xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" />@endxmlonly" \
+    "m_examplenavigation{2}=@xmlonly<mcss:examplenavigation xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:page=\"\1\" mcss:prefix=\"\2\" />@endxmlonly" \
+    "m_keywords{1}=@xmlonly<mcss:search xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:keywords=\"\1\" />@endxmlonly" \
+    "m_keyword{3}=@xmlonly<mcss:search xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:keyword=\"\1\" mcss:title=\"\2\" mcss:suffix-length=\"\3\" />@endxmlonly" \
+    "m_enum_values_as_keywords=@xmlonly<mcss:search xmlns:mcss=\"http://mcss.mosra.cz/doxygen/\" mcss:enum-values-as-keywords=\"true\" />@endxmlonly"
diff --git a/docs/md/active-messenger.md b/docs/md/active-messenger.md
@@ -0,0 +1,95 @@
+
+\page active-messenger Active Messenger
+\brief Asynchronous send/receive of messages
+
+The active messenger `vt::messaging::ActiveMessenger`, accessed via
+`vt::theMsg()`, asynchronously sends and receives messages across nodes using
+MPI internally. When sending a message, it uses the \vt registry to consistently
+dispatch messages and data to handlers (function pointers, functors, or methods)
+across nodes.
+
+Each message contains an envelope `vt::Envelope` to store meta-data associated
+with the message, such as the destination and handler to trigger when it
+arrives. Sending a message entails setting up the envelope, optionally
+serializing the message (depending on whether the serialize overload is
+present), and then using `MPI_Isend` to asynchronously transfer the bytes to the
+destination node. On the receive side, the active messenger is always probing
+for an incoming message and begins a transfer when it discovers one. The \vt
+\subpage{scheduler} polls the active messenger to make progress on any incoming
+messages.
+
+\section am-simple-example Sending a message
+
+\code{.cpp}
+#include <vt/transport.h>
+
+#include <vector>
+
+// Declare a serializable message
+struct MyMsg : vt::Message {
+  using MessageParentType = vt::Message;
+  vt_msg_serialize_required(); // for vector
+
+  MyMsg() = default; // default constructor for de-serialize
+  MyMsg(int in_val, std::vector<double> const& in_vec)
+    : val(in_val),
+      my_vec(in_vec)
+  { }
+
+  template <typename SerializerT>
+  void serialize(SerializerT& s) {
+    MessageParentType::serialize(s);
+    s | val;
+    s | my_vec;
+  }
+
+  int val = 0;
+  std::vector<double> my_vec;
+};
+
+// Active function pointer
+void myHandler(MyMsg* m) {
+  vt::NodeType this_node = vt::theContext()->getNode();
+  fmt::print("{}: val={}, vec size={}\n", this_node, m->val, m->my_vec.size());
+}
+
+// Active functor
+struct MyFunctor {
+  void operator()(MyMsg* m) {
+    vt::NodeType this_node = vt::theContext()->getNode();
+    fmt::print("{}: val={}, vec size={}\n", this_node, m->val, m->my_vec.size());
+  }
+};
+
+int main(int argc, char** argv) {
+  vt::initialize(argc, argv);
+
+  vt::NodeType this_node = vt::theContext()->getNode();
+
+  if (this_node == 0) {
+    // spins in scheduler until termination of the enclosed work
+    vt::runInEpochRooted([=]{
+      std::vector<double> vec_to_send;
+      vec_to_send.push_back(29.);
+      vec_to_send.push_back(54.);
+
+      auto msg = vt::makeMessage<MyMsg>(10, vec_to_send);
+      vt::theMsg()->sendMsg<MyMsg, myHandler>(1, msg.get()); // send to node 1
+
+      auto msg2 = vt::makeMessage<MyMsg>(11, vec_to_send);
+      vt::theMsg()->sendMsg<MyFunctor>(1, msg2.get());  // send to node 1
+    });
+  }
+
+  vt::finalize();
+  return 0;
+}
+
+\endcode
+
+Program output:
+
+\code{.shell-session}
+1: val=10, vec size=2
+1: val=11, vec size=2
+\endcode