Ark Restructuring, Adding More Development Tools and Overhauls

[srivarra]

Motivation

Currently both Ark, and us developers can benefit from improved organization, and taking advantage of more development tools.

Lets use this issue to gather general organization ideas, handy development tools for us to use, overhauls and more.

[srivarra]

Poetry and PyProject.toml to Organize, Build, Package, Publish and more

Poetry is a very easy to use package for dependency management and packaging in Python.

We can add packages for different steps in the development process, for example build, test, docs are some that are commonly used.

These are split up based on which group the developer wants to add them. Poetry also takes advantage of the pyproject.toml PyProject file which is an easy to use structured document somewhat similar to yaml.

We can move the setup.py to the PyProject structure. The file would start with:

[tool.poetry]
name = "ark-analysis"
version = "0.4.1"
description = "Toolbox for analysis on segmented images from MIBI"
authors = [
    "Noah Frey Greenwald <nfgreen@stanford.edu>",
    "Adam Kagel <ackagel@stanford.edu>",
    "Alex Kong <alkong@stanford.edu>",
    "Candace Liu <cliu@stanford.edu>",
    "Cami Laura Sowers <csowers@stanford.edu>",
    "Sricharan Reddy Varra <srivarra@stanford.edu>",
]
classifiers = [
    'License :: OSI Approved :: Apache Software License',
    'Development Status :: 4 - Beta',
    'Programming Language :: Python :: 3',
    'Programming Language :: Python :: 3.8',
]
license = "Modified Apache License 2.0"
readme = "README.md"
packages = [{ include = "ark", from = "src" }]
homepage = "https://github.com/angelolab/ark-analysis"
repository = "https://github.com/angelolab/ark-analysis"
documentation = "https://ark-analysis.readthedocs.io/en/latest/"

As an example, say we want to add pytest to our poetry project, (after initializing it with poetry init) we would run: `poetry add pytest -G test). This will add pytest to the test group. Below is a real example of modifying the current ark-analysis to use Poetry, adding relevant packages to their groups. It will also solve version mismatches automatically for the majority of the time.

## BUILD DEPENDENCIES
[tool.poetry.dependencies]
cryptography = "^38.0.1"
Cython = "^0.29.32"
datasets = "^2.5.1"
decorator = "^5.1.1"
feather-format = "^0.4.1"
importlib-metadata = "^4.12.0"
jedi = "^0.18.1"
jupyter = "^1.0.0"
jupyter-contrib-nbextensions = "^0.5.1"
jupyterlab = "^3.4.7"
matplotlib = "^3.6.0"
natsort = "^8.2.0"
numpy = "^1.23.3"
palettable = "^3.3.0"
pandas = "^1.5.0"
python = ">=3.8,<3.12"
scikit-image = "^0.19.3"
scikit-learn = "^1.1.2"
scipy = "^1.9.1"
seaborn = "^0.12.0"
spatial-lda = "^0.1.3"
statsmodels = "^0.13.2"
tifffile = "^2022.8.12"
tqdm = "^4.64.1"
umap-learn = "^0.5.3"
xarray = "^2022.9.0"

## TEST DEPENDENCIES ##
[tool.poetry.group.test.dependencies]
attrs = "^22.1.0"
coveralls = "^3.3.1"
pytest = "^7.1.3"
pytest-asyncio = "^0.19.0"
pytest-cases = "^3.6.13"
pytest-cov = "^4.0.0"
pytest-mock = "^3.9.0"
pytest-order = "^1.0.1"
pytest-pycodestyle = "^2.3.0"
testbook = "^0.4.2"

## DOCUMENTATION DEPENDENCIES ##
[tool.poetry.group.docs.dependencies]
commonmark = "^0.9.1"
docutils = "^0.19"
ipython = "^8.5.0"
ipywidgets = "^8.0.2"
m2r2 = "^0.3.3"
mistune = "~0.8.4"
mock = "^4.0.3"
nbsphinx = "^0.8.9"
nbsphinx-link = "^1.3.0"
recommonmark = "^0.7.1"
Sphinx = "^5.2.3"
sphinx-rtd-theme = "0.5.0"

In addition it comes with it's own build system, but we can add our own in the case of building C++ or Cython files

[tool.poetry.build]
generate-setup-file = false
script = "build.py"

[build-system]
requires = ["poetry-core", "Cython", "setuptools", "numpy"]
build-backend = "poetry.core.masonry.api"

When poetry builds the project (or when you install it locally for local testing, and usage with poetry install it'll run the build.py script.

General Advantages of PyProject.toml

Many Python applications support the toml format, such as Coverage and Pytest

## COVERAGE ##
[tool.coverage.paths]
source = ["src", "*/site-packages"]

[tool.coverage.run]
branch = true
source = ["ark"]

[tool.coverage.report]
exclude_lines = [
    "except ImportError",
    "raise AssertionError",
    "raise NotImplementedError",
]
show_missing = true
ignore_errors = false
fail_under = 90
omit = ["*/**/*_test.py"]

## TESTING ##
[tool.pytest.ini_options]
addopts = ["-vv", "-s", "--cov", "--pycodestyle"]
filterwarnings = [
    "ignore::DeprecationWarning",
    "ignore::PendingDeprecationWarning",
]

To run pytest we use poetry run pytest. It searches for the section labeled [tool.pytest.ini_options] and runs pytest with any arguments given in the PyProject file.

Poetry creates a localized virtual environment for all the python packages installed, and most importantly keeps track of each, with their versions in a file called poetry.lock. If you've used JavaScript, it's a similar idea to a Nodejs.lock.

Keeping in line with the pytest example, here is it's index in the poetry.lock file:

[[package]]
name = "pytest"
version = "7.1.3"
description = "pytest: simple powerful testing with Python"
category = "dev"
optional = false
python-versions = ">=3.7"

[package.dependencies]
attrs = ">=19.2.0"
colorama = {version = "*", markers = "sys_platform == \"win32\""}
iniconfig = "*"
packaging = "*"
pluggy = ">=0.12,<2.0"
py = ">=1.8.2"
tomli = ">=1.0.0"

[package.extras]
testing = ["argcomplete", "hypothesis (>=3.56)", "mock", "nose", "pygments (>=2.7.2)", "requests", "xmlschema"]

[srivarra]

Build.py for Cython, C++ Scripts

If we decide to have more C++ / Cython included in Ark, it would be beneficial to have a file which compiles all extensions and installs them with respect to the user's architecture, CPU, and operating System.
Each extension can be appended to make a list of them, and compiled together.

import os
from setuptools import Extension, Distribution
import multiprocessing
import platform
import numpy as np
from Cython.Build import cythonize
from Cython.Distutils.build_ext import new_build_ext as cython_build_ext


CYTHON_DEBUG = False

if CYTHON_DEBUG:
    from Cython.Compiler.Options import get_directive_defaults
    directive_defaults = get_directive_defaults()

    directive_defaults['linetrace'] = True
    directive_defaults['binding'] = True

INCLUDE_DIR_ARGS = [np.get_include()]
DEFINE_MACROS_ARGS = [('CYTHON_TRACE', '1')] if CYTHON_DEBUG else None
LIBRARIES_ARGS = []
COMPILER_ARGS = []

# UNIX system args
if os.name == "posix": 
    LIBRARIES_ARGS = ["m", "pthread", "dl"]
    COMPILER_ARGS = ["-g1", "-O3", "-ffast-math"]

# Apple Silicon args
if platform.system() == "Darwin" and platform.machine() == "arm64":
    COMPILER_ARGS += ["-arch=aarch_64"]
else:
    COMPILER_ARGS += ["-mtune=native", "-march=native"]

def build():
    extensions = [
        Extension(
            name="src.ark.utils._bootstrapping",
            sources=["src/ark/utils/_bootstrapping.pyx"],
            extra_compile_args=COMPILER_ARGS,
            include_dirs=INCLUDE_DIR_ARGS,
            libraries=LIBRARIES_ARGS,
            define_macros=DEFINE_MACROS_ARGS,
            ),
    ]
    extension_modules = cythonize(
        module_list = extensions,
        nthreads=multiprocessing.cpu_count(),
        compiler_directives={"binding": True, "language_level": "3"},
        force=True,
    )

    distribution = Distribution({
        "ext_modules": extension_modules,
        "cmdclass": {
            "build_ext": cython_build_ext,
        },
    })
    
    # Grab the build_ext command and copy all files back to source dir.
    # Done so Poetry grabs the files during the next step in its build.
    distribution.run_command("build_ext")
    build_ext_cmd = distribution.get_command_obj("build_ext")
    build_ext_cmd.copy_extensions_to_source()



if __name__ == "__main__":
    build()

[srivarra]

Move Ark code to src/ark instead of housing it in ark

I highly suggest giving a quick read to this section in the Py-Pkgs book. The rest of the online book is a great resource for packaging Python projects.

tldr:

• A “src” layout forces you to install your package before it can be tested.
• A “src” layout leads to cleaner editable installs of your package.

[srivarra]

Use a dedicated Autoformatter

Black is super opionated, but it just works out of the box. No hassle, and has very little customization. It takes advantage of PyProject.toml as well.

Here is an example from the official docs.

[tool.black]
line-length = 100
target-version = ['py37']
include = '\.pyi?$'
# 'extend-exclude' excludes files or directories in addition to the defaults
extend-exclude = '''
# A regex preceded with ^/ will apply only to files and directories
# in the root of the project.
(
  ^/foo.py    # exclude a file named foo.py in the root of the project
  | .*_pb2.py  # exclude autogenerated Protocol Buffer files anywhere in the project
)
'''

[srivarra]

Using Git-hooks and Pre-commit

Pre-commit runs a series of checks before each commit. We can add things like pytest, and formatting to it. So before a developer pushes to a branch, it's already tested locally, and formatted properly.