ArrayBuilder refactoring (#977)

* make snapshot private to see where it is used

* dependent project fix

* remove dependency on arrays

* call python layout to construct the arrays

* move c++ calls to a separate file that can be safely deleted when the c++ tests are deleted and the json parser is refactored
Please enter the commit message for your changes. Lines starting

* make array builder snapshot public again - since all internal calls to it are done via pybind11

* remove c++ tests and RawArray

* remove dependent project and update documentation

* cleanup: remove snapshot methods

* remove config build for dependent project

* use shared pointers

* some more code moved to be handled via pybind11

* append moved to pybind11, numba tests expect to fail for now

* revert removing append - it's needed for numba for now

* final cleanup

.ci/azure-buildtest-awkward.yml

@@ -199,46 +199,6 @@ jobs:
           python -m pytest -vv -rs tests
         displayName: "Test"
 
-      - script: |
-          echo '===== include directory ============================='
-          ls -R `python -m awkward.config --incdir`
-          echo '===== library directory ============================='
-          ls `python -m awkward.config --libdir`
-          echo '===== compilation ==================================='
-          which g++
-          g++ minimal.cpp `python -m awkward.config --cflags --static-libs` -o minimal
-          otool -L minimal
-          ./minimal "[[1.1, 2.2, 3.3], [], [4.4, 5.5], [6.6]]" -2
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 1"
-        condition: "and(succeeded(), ne(variables['python.version'], '3.9'))"
-
-      - script: |
-          rm -f minimal
-          g++ minimal.cpp `python -m awkward.config --cflags --libs` -o minimal
-          export DYLD_LIBRARY_PATH=`python -m awkward.config --libdir`:$DYLD_LIBRARY_PATH
-          echo $DYLD_LIBRARY_PATH
-          otool -L minimal
-          ./minimal "[[1.1, 2.2, 3.3], [], [4.4, 5.5], [6.6]]" -2
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 2"
-        condition: "and(succeeded(), ne(variables['python.version'], '3.9'))"
-
-      - script: |
-          which python
-          python -c 'import sys; print(sys.version)'
-          cmake -S . -B build -DPYTHON_EXECUTABLE:FILEPATH=`which python`
-          cmake --build build
-          export DYLD_LIBRARY_PATH=`python -m awkward.config --libdir`:$DYLD_LIBRARY_PATH
-          echo $DYLD_LIBRARY_PATH
-          ls build/dependent*
-          otool -L build/dependent*
-          touch build/__init__.py
-          python -m pytest -vv test-python.py
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 3"
-        condition: "and(succeeded(), ne(variables['python.version'], '3.9'))"
-
   - job: Linux
 
     pool:
@@ -327,39 +287,3 @@ jobs:
       - script: |
           python -m pytest -vv -rs tests
         displayName: "Test"
-
-      - script: |
-          echo '===== include directory ============================='
-          ls -R `python -m awkward.config --incdir`
-          echo '===== library directory ============================='
-          ls `python -m awkward.config --libdir`
-          echo '===== compilation ==================================='
-          g++ minimal.cpp `python -m awkward.config --cflags --static-libs` -o minimal
-          ldd minimal
-          ./minimal "[[1.1, 2.2, 3.3], [], [4.4, 5.5], [6.6]]" -2
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 1"
-
-      - script: |
-          rm -f minimal
-          g++ minimal.cpp `python -m awkward.config --cflags --libs` -o minimal
-          export LD_LIBRARY_PATH=`python -m awkward.config --libdir`:$LD_LIBRARY_PATH
-          echo $LD_LIBRARY_PATH
-          ldd minimal
-          ./minimal "[[1.1, 2.2, 3.3], [], [4.4, 5.5], [6.6]]" -2
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 2"
-
-      - script: |
-          which python
-          python -c 'import sys; print(sys.version)'
-          cmake -S . -B build -DPYTHON_EXECUTABLE:FILEPATH=`which python`
-          cmake --build build
-          export LD_LIBRARY_PATH=`python -m awkward.config --libdir`:$LD_LIBRARY_PATH
-          echo $LD_LIBRARY_PATH
-          ls build/dependent*
-          ldd build/dependent*
-          touch build/__init__.py
-          python -m pytest -vv test-python.py
-        workingDirectory: dependent-project
-        displayName: "Test dependent project 3"

.gitignore

@@ -17,13 +17,6 @@ tests-cpu-kernels
 tests-cuda-kernels
 tests-spec
 
-dependent-project/CMakeFiles
-dependent-project/CMakeCache.txt
-dependent-project/Makefile
-dependent-project/cmake_install.cmake
-dependent-project/cmake_pybind11
-dependent-project/minimal
-
 # to use all-contributors-cli without adding it to the project
 node_modules
 package.json

README.md

@@ -191,13 +191,9 @@ python -m pytest -vv -rs tests-cuda
 
 Python projects can simply `import awkward`.
 
-C++ projects can link against the shared libraries `libawkward-cpu-kernels.so` and `libawkward.so` or their static library equivalents. These libraries are shipped, along with the include files, as part of pip's installation.
-
-   * See the [dependent-project](https://github.com/scikit-hep/awkward-1.0/tree/main/dependent-project) directory for examples.
-
 # Roadmap
 
-Since [version 0.4.0](https://github.com/scikit-hep/awkward-1.0/releases/tag/0.4.0), Awkward Array has been compiled with versions of pybind11 that have [ABI version 4](https://github.com/pybind/pybind11/blob/v2.6.2/include/pybind11/detail/internals.h#L152-L153). Any other Python extension built with pybind11 ABI version 4 can consume and produce Awkward Arrays in the way [described here](https://github.com/scikit-hep/awkward-1.0/tree/main/dependent-project).
+Since [version 0.4.0](https://github.com/scikit-hep/awkward-1.0/releases/tag/0.4.0), Awkward Array has been compiled with versions of pybind11 that have [ABI version 4](https://github.com/pybind/pybind11/blob/v2.6.2/include/pybind11/detail/internals.h#L152-L153).
 
 The table below indicates when interface-breaking changes are planned; each is discussed in pull requests and issues. It doesn't include new additions that don't interfere with old behavior or corrections to bugs (i.e. behaviors that were _never_ intended and fixed immediately). Each deprecated feature is announced by a `FutureWarning` that indicates the target removal version/date.
 

docs-doxygen/index.md

@@ -21,7 +21,7 @@ The high-level interface and Numba implementation are described in the [Python A
 
 This reference describes the
 
-   * **C++ classes:** in [namespace awkward](namespaceawkward.html) (often abbreviated as "ak"), which are compiled into **libawkward.so** (or dylib or lib). This library is fully usable from C++, without Python (see [dependent-project](https://github.com/scikit-hep/awkward-1.0/tree/main/dependent-project) for an example).
+   * **C++ classes:** in [namespace awkward](namespaceawkward.html) (often abbreviated as "ak"), which are compiled into **libawkward.so** (or dylib or lib).
    * **pybind11 interface:** no namespace, but contained entirely within the [python directory](dir_91f33a3f1dd6262845ebd1570075970c.html), which are compiled into **awkward._ext** for use in Python.
    * **CPU kernels:** no namespace, but contained entirely within the [kernels directory](dir_6225843069e7cc68401bbec110a1667f.html), which are compiled into **libawkward-cpu-kernels.so** (or dylib or lib). This library is fully usable from any language that can call functions through [FFI](https://en.wikipedia.org/wiki/Foreign_function_interface).
    * **GPU kernels:** FIXME! (not implemented yet)

docs-src/quickstart.md

@@ -85,7 +85,6 @@ For developers
 See Awkward Array's [GitHub page](https://github.com/scikit-hep/awkward-1.0) for more on the following.
 
    * [Installation for developers](https://github.com/scikit-hep/awkward-1.0#installation-for-developers).
-   * [Using Awkward as a dependency](https://github.com/scikit-hep/awkward-1.0/tree/main/dependent-project) (example projects).
    * [Continuous integration](https://dev.azure.com/jpivarski/Scikit-HEP/_build?definitionId=3&_a=summary) and [continuous deployment](https://dev.azure.com/jpivarski/Scikit-HEP/_build?definitionId=4&_a=summary) are hosted by [Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/).
    * [Release history](https://awkward-array.readthedocs.io/en/latest/_auto/changelog.html) (changelog) is hosted by [ReadTheDocs](https://readthedocs.org).
    * [Roadmap](https://github.com/scikit-hep/awkward-1.0#roadmap) of future releases.