feat(back): fluidattacks#1155 support batch pipelines

- implement compute-on-batch with python
- add support for batch pipelines
- create and adjust python docs
- add makePythonVscodeSettings util

Signed-off-by: Daniel F. Murcia Rivera <danmur97@outlook.com>

.gitignore

@@ -0,0 +1,2 @@
+__pycache__
+.vscode

docs/src/api/builtins/deploy.md

@@ -2,70 +2,80 @@
 
 Submit a job to a [AWS BATCH](https://aws.amazon.com/batch/) queue.
 
+When used as a Makes declaration (at makes.nix attrs):
+
+- computeOnAwsBatch: `AttrsOf[JobType]` (Optional Attr)
+
 Types:
 
-- computeOnAwsBatch (`attrsOf jobType`): Optional.
+- computeOnAwsBatch: `JobType -> SourceAble`
     Job groups to submit.
     Defaults to `{ }`.
-- jobType (`submodule`):
-    - allowDuplicates (`bool`): Optional.
+
+- `JobType` = `Attrs`
+    - allowDuplicates: `bool` (Optional Attr)
         Set to `false` in order to prevent submitting the job
         if there is already a job in the queue with the same name.
         Defaults to `false`.
-    - attempts (`ints.positive`): Optional.
+    - attempts: `PositiveInt` (Optional Attr)
         If the value of attempts is greater than one,
         the job is retried on failure the same number of attempts as the value.
         Defaults to `1`.
-    - attemptDurationSeconds (`ints.positive`): Optional.
+    - attemptDurationSeconds: `PositiveInt`
         The time duration in seconds
         (measured from the job attempt's startedAt timestamp)
         after which Batch terminates your jobs
         if they have not finished.
-    - command (`listOf str`):
+    - command: `List[str]`
         The command to send to the container.
         It overrides the one specified
         in the Batch job definition.
         Additional arguments can be propagated when running this module output.
-    - definition (`str`):
+    - definition: `str`
         Name of the Batch job definition
         that we will use as base for submitting the job.
         In general an Batch job definition is required
         in order to specify which container image
         our job is going to run on.
-    - environment (`listOf str`): Optional.
+    - environment: `List[str]` (Optional Attr)
         Name of the environment variables
         whose names and values should be copied from the machine running Makes
         to the machine on Batch running the job.
         Defaults to `[ ]`.
-    - includePositionalArgsInName (`bool`): Optional.
+    - includePositionalArgsInName: `bool` (Optional Attr).
         Enable to make positional arguments part of the job name.
         This is useful for identifying jobs
         in the Batch console
         more easily.
         Defaults to `true`.
-    - memory (`ints.positive`):
+    - nextJob: `Attrs` (Optional Attr)
+        The next job that will be executed after its parent finish.
+        You must provide a `name` attribute and all the required
+        attrs of `JobType`.
+        Defaults to `{ }`.
+    - memory: `PositiveInt`
         Amount of memory, in MiB that is reserved for the job.
-    - parallel (`ints.positive`): Optional.
+    - parallel: `PositiveInt` (Optional Attr)
         Number of parallel jobs to trigger using
         [Batch Array Jobs](https://docs.aws.amazon.com/batch/latest/userguide/array_jobs.html).
-    - propagateTags (`bool`): Optional.
+        Defaults to `1`.
+    - propagateTags: `bool` (Optional Attr)
         Enable tags to be propagated into the ECS tasks.
         Defaults to `true`.
-    - queue (`nullOr str`):
+    - queue: `str | null`
         Name of the Batch queue we should submit the job to.
-        It can be set to `null`,
-        causing Makes to read
+        If `null` then queue is fetch from
         the `MAKES_COMPUTE_ON_AWS_BATCH_QUEUE` environment variable at runtime.
-    - setup (`listOf package`):
+    - setup: `List[Source-able]`
         [Makes Environment][makes_environment]
         or [Makes Secrets][makes_secrets]
         to `source` (as in Bash's `source`)
         before anything else.
         Defaults to `[ ]`.
-    - tags (`attrsOf str`): Optional.
+    - tags: `AttrsOf[str]` (Optional Attr).
         Tags to apply to the batch job.
         Defaults to `{ }`.
-    - vcpus (`ints.positive`):
+    - vcpus: `PositiveInt`
         Amount of virtual CPUs that is reserved for the job.
 
 Example:

docs/src/api/extensions/python.md

@@ -4,9 +4,9 @@ Get a specific Python interpreter.
 
 Types:
 
-- makePythonVersion (`function str -> package`):
-    - (`enum ["3.8" "3.9" "3.10" "3.11"]`):
-        Python version of the interpreter to return.
+- makePythonVersion: `SupportedVersions -> Derivation`
+- `SupportedVersions` = `"3.8" | "3.9" | "3.10" | "3.11"`
+    Supported Python versions.
 
 Example:
 
@@ -50,38 +50,39 @@ Pre-requisites:
 
 Types:
 
-- makePythonPypiEnvironment (`function { ... } -> package`):
-    - name (`str`):
+- makePythonPypiEnvironment: `Input -> SourceAble`
+- `Input` = `Attrs`
+    - name: `Str`
         Custom name to assign to the build step, be creative, it helps in debugging.
-    - searchPathsBuild (`asIn makeSearchPaths`): Optional.
+    - searchPathsBuild: `makeSearchPaths` (Optional Attr)
         Arguments here will be passed as-is to `makeSearchPaths`
         and used while installing the Python dependencies.
         Defaults to `makeSearchPaths`'s defaults.
-    - searchPathsRuntime (`asIn makeSearchPaths`): Optional.
+    - searchPathsRuntime: `makeSearchPaths` (Optional Attr)
         Arguments here will be passed as-is to `makeSearchPaths`
         and propagated to the runtime environment.
         Defaults to `makeSearchPaths`'s defaults.
-    - sourcesYaml (`package`):
+    - sourcesYaml: `NixPath`
         `sources.yaml` file
         computed as explained in the pre-requisites section.
 
   For building a few special packages you may need to boostrap
   dependencies in the build environment.
   The following flags are available for convenience:
 
-    - withCython_0_29_24 (`bool`): Optional.
+    - withCython_0_29_24: `Bool` (Optional Attr)
         Bootstrap cython 0.29.24 to the environment
         Defaults to `false`.
-    - withNumpy_1_24_0 (`bool`): Optional.
+    - withNumpy_1_24_0: `Bool` (Optional Attr)
         Bootstrap numpy 1.24.0 to the environment
         Defaults to `false`.
-    - withSetuptools_67_7_2 (`bool`): Optional.
+    - withSetuptools_67_7_2: `Bool` (Optional Attr)
         Bootstrap setuptools 67.7.2 to the environment
         Defaults to `false`.
-    - withSetuptoolsScm_7_1_0 (`bool`) Optional.
+    - withSetuptoolsScm_7_1_0: `Bool` (Optional Attr)
         Bootstrap setuptools-scm 7.1.0 to the environment
         Defaults to `false`.
-    - withWheel_0_40_0 (`bool`): Optional.
+    - withWheel_0_40_0: `Bool` (Optional Attr)
         Bootstrap wheel 0.40.0 to the environment
         Defaults to `false`.
 
@@ -131,41 +132,42 @@ over the tests and its python environments.
 
 Types:
 
-- makePythonPypiEnvironment: (`function Input -> Bundle`):
-  - Input: `AttrsOf`
-    - buildEnv: `function {...} -> Derivation`
-      The nixpkgs buildEnv.override function.
-      Commonly found at `nixpkgs."${python_version}".buildEnv.override`
-    - buildPythonPackage: `function {...} -> Derivation`
-      The nixpkgs buildPythonPackage function.
-      Commonly found at `nixpkgs."${python_version}".pkgs.buildPythonPackage`
-    - pkgDeps: `AttrsOf`
-      The package dependencies.
-      Usually other python packages build with nix,
-      but can be also a nix derivation of a binary.
-      - runtime_deps: `List[Derivation]`
-      - build_deps: `List[Derivation]`
-      - test_deps: `List[Derivation]`
+- makePythonPyprojectPackage: `Input -> Bundle`
+- Input: `Attrs`
+    - buildEnv: `Attrs -> PythonEnvDerivation`
+        The nixpkgs buildEnv.override function.
+        Commonly found at `nixpkgs."${python_version}".buildEnv.override`
+    - buildPythonPackage: `Attrs -> PythonPkgDerivation`
+        The nixpkgs buildPythonPackage function.
+        Commonly found at `nixpkgs."${python_version}".pkgs.buildPythonPackage`
+    - pkgDeps: `Attrs`
+        The package dependencies.
+        Usually other python packages build with nix,
+        but can be also a nix derivation of a binary.
+
+        - runtime_deps: `List[Derivation]`
+        - build_deps: `List[Derivation]`
+        - test_deps: `List[Derivation]`
     - src: `NixPath`
-      The nix path to the source code of the python package.
-      i.e. not only be the package itself, it should also contain
-      a tests folder/module, the pyproject conf and any other meta-package
-      data that the build or tests requires (e.g. custom mypy conf).
-  - Bundle: `AttrsOf`
-    - check: `AttrsOf`
-      Builds of the package only including one test.
-      - tests:`Derivation`
-      - types: `Derivation`
-    - env: `AttrsOf`
-      - dev: `Derivation`
-        The python environment containing only
-        runtime_deps and test_deps
-      - runtime: `Derivation`
-        The python environment containing only
-        the package itself and its runtime_deps.
-    - pkg: `Derivation`
-      The output of the nixpkgs buildPythonPackage function
-      i.e. the python package
+        The nix path to the source code of the python package.
+        i.e. not only the package itself, it should also contain
+        a tests folder/module, the pyproject conf and any other meta-package
+        data that the build or tests requires (e.g. custom mypy conf).
+- Bundle: `Attrs`
+    - check: `Attrs`
+        Builds of the package only including one test.
+        - tests: `Derivation`
+        - types: `Derivation`
+    - env: `Attrs`
+        - dev: `PythonEnvDerivation`
+          The python environment containing only
+          runtime_deps and test_deps
+        - runtime: `PythonEnvDerivation`
+          The python environment containing only
+          the package itself and its runtime_deps.
+    - pkg: `PythonPkgDerivation`
+        The output of the nixpkgs buildPythonPackage function
+        i.e. the python package
 
 ???+ tip
 
@@ -233,3 +235,96 @@ Example:
     executed only once (or never).
     This can also help on performance over heavy
     compilation/build processes.
+
+## makePythonVscodeSettings
+
+Generate visual studio code configuration for python development.
+
+Types:
+
+- makePythonVscodeSettings: `Input -> SourceAble`
+- `Input` = `Attrs`
+    - name: `Str`
+    - env: `PythonEnvDerivation`
+        A python environment derivation. e.g. can be builded from nixpkgs
+        standard builders or from some `env` of the outputs of `makePythonPyprojectPackage`
+    - bins: `List[Derivation]`
+        Derivations to include on the `searchPaths.bins` input
+
+Example:
+
+=== "my-env/makes.nix"
+
+    ```nix
+    {
+      inputs,
+      makePythonPyprojectPackage,
+      makePythonVscodeSettings,
+      projectPath,
+      ...
+    }: let
+      root = projectPath "/my_package";
+      bundle = makePythonPyprojectPackage {
+        inherit (inputs) buildEnv buildPythonPackage;
+        pkgDeps = {
+          runtime_deps = [];
+          build_deps = [];
+          test_deps = with inputs.python_pkgs; [
+            mypy
+            pytest
+          ];
+        };
+        src = root;
+      };
+    in
+      makePythonVscodeSettings {
+        env = bundle.env.dev;
+        bins = [];
+        name = "my-package-env-dev";
+      }
+    ```
+
+=== "makes.nix"
+
+    ```nix
+    {outputs, ...}: {
+      dev = {
+        myPackage = {
+          source = [outputs."/my-env"];
+        };
+      };
+    }
+    ```
+
+=== ".envrc"
+
+    ```bash
+    source "$(m . /dev/myPackage)/template"
+    ```
+
+## pythonOverrideUtils
+
+Integrating python packages built with nix can create conflicts when
+integrating various into one environment. This utils helps unifying
+the dependencies into one and only one version per package.
+
+Types:
+
+- `PythonOverride` = `PythonPkgDerivation -> PythonPkgDerivation`
+    A functions that creates a new modified `PythonPkgDerivation` from the original.
+
+- pythonOverrideUtils: `Attrs`
+    - compose: `List[_T -> _T] -> _T -> _T`
+        Function composition
+    - no_check_override: `PythonOverride`
+        Skips the python package tests that are triggered on the build process.
+        This override is defined through `recursive_python_pkg_override`.
+    - recursive_python_pkg_override: `(Derivation -> Bool) -> PythonOverride -> PythonOverride`
+        Search over all the tree of sub-dependencies the derivation
+        that evaluates to true as defined by the supplied first argument
+        filter `Derivation -> Bool`.
+        If match, the supplied `PythonOverride` (second arg) is applied.
+    - replace_pkg: `List[Str] -> PythonPkgDerivation -> PythonOverride`
+        Replace all python packages that match the supplied list of names,
+        with the supplied python package.
+        The returned override is defined through `recursive_python_pkg_override`

makes.nix

@@ -91,6 +91,9 @@
       ];
       source = [outputs."/cli/env/runtime"];
     };
+    batchClient = {
+      source = [outputs."/envs/compute-on-aws-batch"];
+    };
   };
   envVars = {
     example = {

makes/envs/compute-on-aws-batch/main.nix

@@ -0,0 +1,18 @@
+{
+  __nixpkgs__,
+  makePythonPyprojectPackage,
+  makePythonVscodeSettings,
+  projectPath,
+  ...
+}: let
+  root = projectPath "/src/args/compute-on-aws-batch/batch-client";
+  pkg = import "${root}/entrypoint.nix" {
+    inherit makePythonPyprojectPackage;
+    nixpkgs = __nixpkgs__;
+  };
+in
+  makePythonVscodeSettings {
+    env = pkg.env.dev;
+    bins = [];
+    name = "makes-batch-client-env-dev";
+  }

src/args/agnostic.nix

@@ -83,6 +83,7 @@
     makePythonPypiEnvironment = import ./make-python-pypi-environment/default.nix self;
     makePythonPyprojectPackage = import ./make-python-pyproject-package/default.nix;
     makePythonVersion = import ./make-python-version/default.nix self;
+    makePythonVscodeSettings = import ./make-python-vscode-settings/default.nix self;
     makeRubyGemsEnvironment = import ./make-ruby-gems-environment/default.nix self;
     makeRubyGemsInstall = import ./make-ruby-gems-install/default.nix self;
     makeRubyVersion = import ./make-ruby-version/default.nix self;

src/args/compute-on-aws-batch/batch-client/.envrc

@@ -0,0 +1 @@
+source "$(m . /dev/batchClient)/template"

src/args/compute-on-aws-batch/batch-client/batch_client/__init__.py

@@ -0,0 +1,9 @@
+from ._logger import (
+    setup_logger,
+)
+from fa_purity.cmd import (
+    unsafe_unwrap,
+)
+
+__version__ = "1.0.0"
+unsafe_unwrap(setup_logger(__name__))