[ENH] add PermutationForests and FeatureImportanceForests to sktree

.codespellignore

@@ -1,4 +1,5 @@
 raison
 nd
 parth
-ot
+ot
+fpr

.github/workflows/circle_artifacts.yml

@@ -4,20 +4,21 @@ on: [status]
 # Restrict the permissions granted to the use of secrets.GITHUB_TOKEN in this
 # github actions workflow:
 # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
-permissions:
-  statuses: write
+permissions: read-all
 
 jobs:
   circleci_artifacts_redirector_job:
     runs-on: ubuntu-20.04
     if: "github.repository == 'neurodata/scikit-tree' && github.event.context == 'ci/circleci: build_docs'"
+    permissions:
+      statuses: write
     name: Run CircleCI artifacts redirector
     steps:
       - name: GitHub Action step
         uses: larsoner/circleci-artifacts-redirector-action@master
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
-          api-token: ${{ secrets.CIRCLE_TOKEN }}
+          api-token: ${{ secrets.CIRCLECI_TOKEN }}
           artifact-path: 0/dev/index.html
           circleci-jobs: build_docs
           job-title: Check the rendered docs here!

.github/workflows/main.yml

@@ -116,7 +116,115 @@ jobs:
           ./spin --help
           ./spin coverage --help
           ./spin test --help
-          ./spin coverage
+          ./spin test
+
+      - name: debug
+        run: |
+          ls $PWD/build-install/usr/lib/python${{matrix.python-version}}/site-packages/
+          echo "Okay..."
+          ls $PWD/build
+          ls ./
+
+      - name: Save build
+        uses: actions/upload-artifact@v3
+        with:
+          name: sktree-build
+          path: $PWD/build
+
+  build_and_test_slow:
+    name: Slow Meson build ${{ matrix.os }} - py${{ matrix.python-version }}
+    timeout-minutes: 20
+    needs: [build_and_test]
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-22.04]
+        python-version: ["3.11"]
+        poetry-version: [1.5.0]
+    runs-on: ${{ matrix.os }}
+    defaults:
+      run:
+        shell: bash
+    env:
+      # to make sure coverage/test command builds cleanly
+      FORCE_SUBMODULE: True
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4.6.1
+        with:
+          python-version: ${{ matrix.python-version }}
+          architecture: "x64"
+          cache: "pip"
+          cache-dependency-path: "requirements.txt"
+
+      - name: show-gcc
+        run: |
+          gcc --version
+
+      - name: Install Ccache for MacOSX
+        if: ${{ matrix.os == 'macos-latest'}}
+        run: |
+          brew install ccache
+
+      - name: Install packages for Ubuntu
+        if: ${{ matrix.os == 'ubuntu-22.04'}}
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libopenblas-dev libatlas-base-dev liblapack-dev gfortran libgmp-dev libmpfr-dev libsuitesparse-dev ccache libmpc-dev
+
+      - name: Install Python packages
+        run: |
+          python -m pip install -r build_requirements.txt
+          python -m pip install spin
+          python -m pip install -r test_requirements.txt
+
+      - name: Prepare compiler cache
+        id: prep-ccache
+        shell: bash
+        run: |
+          mkdir -p "${CCACHE_DIR}"
+          echo "dir=$CCACHE_DIR" >> $GITHUB_OUTPUT
+          NOW=$(date -u +"%F-%T")
+          echo "timestamp=${NOW}" >> $GITHUB_OUTPUT
+
+      - name: Setup compiler cache
+        uses: actions/cache@v3
+        id: cache-ccachev1
+        # Reference: https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
+        # NOTE: The caching strategy is modeled in a way that it will always have a unique cache key for each workflow run
+        # (even if the same workflow is run multiple times). The restore keys are not unique and for a partial match, they will
+        # return the most recently created cache entry, according to the GitHub Action Docs.
+        with:
+          path: ${{ steps.prep-ccache.outputs.dir }}
+          # Restores ccache from either a previous build on this branch or on main
+          key: ${{ github.workflow }}-${{ matrix.python-version }}-ccache-linux-${{ steps.prep-ccache.outputs.timestamp }}
+          # This evaluates to `Linux Tests-3.9-ccache-linux-` which is not unique. As the CI matrix is expanded, this will
+          # need to be updated to be unique so that the cache is not restored from a different job altogether.
+          restore-keys: |
+            ${{ github.workflow }}-${{ matrix.python-version }}-ccache-linux-
+
+      - name: Setup build and install scikit-tree
+        run: |
+          ./spin build -j 2 --forcesubmodule
+
+      - name: Ccache performance
+        shell: bash -l {0}
+        run: ccache -s
+
+      - name: build-path
+        run: |
+          echo "$PWD/build-install/"
+          export INSTALLED_PATH=$PWD/build-install/usr/lib/python${{matrix.python-version}}/site-packages
+
+      - name: Run unit tests and coverage
+        run: |
+          ./spin --help
+          ./spin coverage --help
+          ./spin test --help
+          ./spin coverage -k "slowtest"
           cp $PWD/build-install/usr/lib/python${{matrix.python-version}}/site-packages/coverage.xml ./coverage.xml
 
       - name: debug
@@ -127,7 +235,6 @@ jobs:
           ls ./
 
       - name: Upload coverage stats to codecov
-        if: ${{ matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.10'}}
         uses: codecov/codecov-action@v3
         with:
           # python spin goes into the INSTALLED path in order to run pytest
@@ -146,7 +253,7 @@ jobs:
   release:
     name: Release
     runs-on: ubuntu-latest
-    needs: [build_and_test]
+    needs: [build_and_test_slow]
     if: startsWith(github.ref, 'refs/tags/')
     steps:
       - name: Checkout repository

.spin/cmds.py

@@ -1,5 +1,4 @@
 import os
-import shutil
 import subprocess
 import sys
 
@@ -13,39 +12,27 @@ def get_git_revision_hash(submodule) -> str:
 
 
 @click.command()
-@click.option("--build-dir", default="build", help="Build directory; default is `$PWD/build`")
-@click.option("--clean", is_flag=True, help="Clean previously built docs before building")
-@click.option("--noplot", is_flag=True, help="Build docs without plots")
+@click.argument("slowtest", default=True)
 @click.pass_context
-def docs(ctx, build_dir, clean=False, noplot=False):
-    """📖 Build documentation"""
-    if clean:
-        doc_dir = "./docs/_build"
-        if os.path.isdir(doc_dir):
-            print(f"Removing `{doc_dir}`")
-            shutil.rmtree(doc_dir)
-
-    site_path = meson._get_site_packages()
-    if site_path is None:
-        print("No built scikit-tree found; run `./spin build` first.")
-        sys.exit(1)
-
-    util.run(["pip", "install", "-q", "-r", "doc_requirements.txt"])
-
-    ctx.invoke(meson.docs)
-    # os.environ["SPHINXOPTS"] = "-W"
-    # os.environ["PYTHONPATH"] = f'{site_path}{os.sep}:{os.environ.get("PYTHONPATH", "")}'
-    # if noplot:
-    #     util.run(["make", "-C", "docs", "clean", "html-noplot"], replace=True)
-    # else:
-    #     util.run(["make", "-C", "docs", "clean", "html"], replace=True)
-
-
-@click.command()
-@click.pass_context
-def coverage(ctx):
+def coverage(ctx, slowtest):
     """📊 Generate coverage report"""
-    pytest_args = ("-o", "python_functions=test_*", "sktree", "--cov=sktree", "--cov-report=xml")
+    if slowtest:
+        pytest_args = (
+            "-o",
+            "python_functions=test_*",
+            "sktree",
+            "--cov=sktree",
+            "--cov-report=xml",
+            "-k slowtest",
+        )
+    else:
+        pytest_args = (
+            "-o",
+            "python_functions=test_*",
+            "sktree",
+            "--cov=sktree",
+            "--cov-report=xml",
+        )
     ctx.invoke(meson.test, pytest_args=pytest_args)
 
 

README.md

@@ -13,19 +13,14 @@ scikit-tree is a scikit-learn compatible API for building state-of-the-art decis
 
 Tree-models have withstood the test of time, and are consistently used for modern-day data science and machine learning applications. They especially perform well when there are limited samples for a problem and are flexible learners that can be applied to a wide variety of different settings, such as tabular, images, time-series, genomics, EEG data and more.
 
-We welcome contributions for modern tree-based algorithms. We use Cython to achieve fast C/C++ speeds, while abiding by a scikit-learn compatible (tested) API. Moreover, our Cython internals are easily extensible because they follow the internal Cython API of scikit-learn as well.
-
-Due to the current state of scikit-learn's internal Cython code for trees, we have to instead leverage a fork of scikit-learn at https://github.com/neurodata/scikit-learn when
-extending the decision tree model API of scikit-learn. Specifically, we extend the Python and Cython API of the tree submodule in scikit-learn in our submodule, so we can introduce the tree models housed in this package. Thus these extend the functionality of decision-tree based models in a way that is not possible yet in scikit-learn itself. As one example, we introduce an abstract API to allow users to implement their own oblique splits. Our plan in the future is to benchmark these functionalities and introduce them upstream to scikit-learn where applicable and inclusion criterion are met.
-
 Documentation
 =============
 
 See here for the documentation for our dev version: https://docs.neurodata.io/scikit-tree/dev/index.html
 
 Why oblique trees and why trees beyond those in scikit-learn?
 =============================================================
-In 2001, Leo Breiman proposed two types of Random Forests. One was known as ``Forest-RI``, which is the axis-aligned traditional random forest. One was known as ``Forest-RC``, which is the random oblique linear combinations random forest. This leveraged random combinations of features to perform splits. [MORF](1) builds upon ``Forest-RC`` by proposing additional functions to combine features. Other modern tree variants such as Canonical Correlation Forests (CCF), or unsupervised random forests are also important at solving real-world problems using robust decision tree models.
+In 2001, Leo Breiman proposed two types of Random Forests. One was known as ``Forest-RI``, which is the axis-aligned traditional random forest. One was known as ``Forest-RC``, which is the random oblique linear combinations random forest. This leveraged random combinations of features to perform splits. [MORF](1) builds upon ``Forest-RC`` by proposing additional functions to combine features. Other modern tree variants such as Canonical Correlation Forests (CCF), Extended Isolation Forests, Quantile Forests, or unsupervised random forests are also important at solving real-world problems using robust decision tree models.
 
 Installation
 ============
@@ -105,6 +100,13 @@ Alternatively, you can use editable installs
 
     pip install --no-build-isolation --editable .
 
+Development
+===========
+We welcome contributions for modern tree-based algorithms. We use Cython to achieve fast C/C++ speeds, while abiding by a scikit-learn compatible (tested) API. Moreover, our Cython internals are easily extensible because they follow the internal Cython API of scikit-learn as well.
+
+Due to the current state of scikit-learn's internal Cython code for trees, we have to instead leverage a fork of scikit-learn at https://github.com/neurodata/scikit-learn when
+extending the decision tree model API of scikit-learn. Specifically, we extend the Python and Cython API of the tree submodule in scikit-learn in our submodule, so we can introduce the tree models housed in this package. Thus these extend the functionality of decision-tree based models in a way that is not possible yet in scikit-learn itself. As one example, we introduce an abstract API to allow users to implement their own oblique splits. Our plan in the future is to benchmark these functionalities and introduce them upstream to scikit-learn where applicable and inclusion criterion are met.
+
 References
 ==========
 [1]: [`Li, Adam, et al. "Manifold Oblique Random Forests: Towards Closing the Gap on Convolutional Deep Networks" SIAM Journal on Mathematics of Data Science, 5(1), 77-96, 2023`](https://doi.org/10.1137/21M1449117)