Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expressions, Parameters and Renderers #55

Merged
merged 108 commits into from
Sep 30, 2024
Merged
Show file tree
Hide file tree
Changes from 98 commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
2e7e6e7
wip(renderers): can pick renderers dynamically
philtweir Jul 20, 2024
aad87a5
fix(renderers): make the renderer semantics less janky
philtweir Jul 20, 2024
c58c17e
feat(renderers): add the ability to pass renderer arguments via CLI/YAML
philtweir Jul 20, 2024
d792f76
feat(renderers): add way to output subworkflows to files
philtweir Jul 20, 2024
b2c10e2
feat: support list/tuple return values
philtweir Jul 24, 2024
f9e6c9b
feat(annotations): add ability to annotate variables
philtweir Aug 3, 2024
d5b8c65
fix: correct tests
philtweir Aug 3, 2024
470de20
wip: add configuration
philtweir Aug 3, 2024
be9b862
fix: scope thread pools to a single render call and add context copying
philtweir Aug 3, 2024
7c34dbe
fix: make sure the globals are updated when a subworkflow is running
philtweir Aug 4, 2024
c6dc355
fix: tests
philtweir Aug 6, 2024
a7c1fe9
fix: integrate flatten_all_nested configuration
philtweir Aug 6, 2024
dee6815
fix: correctly handle flattened nested tasks
philtweir Aug 6, 2024
363be81
fix: confirm tuples can be returned
philtweir Aug 6, 2024
7aeccd9
fix: step reference works as tuple
philtweir Aug 6, 2024
03c8c8b
fix: tidy up type parsing
philtweir Aug 6, 2024
b50b150
fix: nested raw is parseable
philtweir Aug 6, 2024
109dbf9
fix(annotations): correct name to AtRender
philtweir Aug 7, 2024
6cb2ca1
feat(sympy): handle arithmetic/structure manipulation of references
philtweir Aug 10, 2024
0f846e7
feat(sympy): more flexible parameter support
philtweir Aug 10, 2024
ff81ad4
fix: fix render load order
philtweir Aug 10, 2024
e3b9b28
feat(iterated): allow spreading
philtweir Aug 11, 2024
38825fa
feat(fields): make it possible to use a plain dict if configured for …
philtweir Aug 11, 2024
0f047fb
fix: is_raw_type should handle union types
philtweir Aug 11, 2024
9468595
feat(positional): allow positional arguments if configuration overrid…
philtweir Aug 11, 2024
333967f
fix(sympy): support none in dicts
philtweir Aug 11, 2024
079c00b
fix(sympy): support none in dicts
philtweir Aug 11, 2024
213c514
fix: expressions should consider Raw as a raw type
philtweir Aug 11, 2024
08db69f
fix: sorted should not break for two identical steps
philtweir Aug 11, 2024
3dd35d1
fix: efficiency improvements
philtweir Aug 12, 2024
7e8ab1a
fix: efficiency improvements
philtweir Aug 12, 2024
2aba6e6
fix: efficiency improvements
philtweir Aug 12, 2024
36c74f1
feat: add construct args to CLI
philtweir Aug 13, 2024
0517594
fix: make field separator configurable
philtweir Aug 14, 2024
9b5d03a
fix: construct kwargs used to set_configuration
philtweir Aug 14, 2024
77c46bf
fix: construct kwargs used to set_configuration
philtweir Aug 14, 2024
19f1a38
fix: ignore annotations when looking at fieldability
philtweir Aug 14, 2024
ca47bb8
feat(imports): retrieve annotations from imported modules
philtweir Aug 17, 2024
4bdcd8b
feat(fixed): add loopable parameters
philtweir Aug 17, 2024
5773fae
feat(fixed): add loopable parameters
philtweir Aug 17, 2024
0c2cbdc
fix: better error message for iterating a non-iterable
philtweir Aug 17, 2024
00036d0
fix: add is_firm
philtweir Aug 17, 2024
5b83266
fix: move indexing to find_field
philtweir Aug 18, 2024
fc5309e
fix: resolve the parameter deduplication
philtweir Aug 18, 2024
36d15c6
fix: resolve the parameter deduplication
philtweir Aug 18, 2024
f6de37f
fix: fixups in iteration
philtweir Aug 18, 2024
3dbe777
fix(typing): make types consistent in core
philtweir Aug 19, 2024
bd7ee4a
fix(typing): move @subworkflow to @workflow
philtweir Aug 19, 2024
71a35c3
fix: allow workflows to come from packages
philtweir Aug 19, 2024
03e5209
fix: allow workflows to come from packages
philtweir Aug 19, 2024
7d880c0
fix: allow workflows to come from packages
philtweir Aug 19, 2024
aee1ce1
fix: allow workflows to come from packages
philtweir Aug 19, 2024
fc7495e
feat(iterable): provide len for Fixed
philtweir Aug 19, 2024
093b1d6
fix: allow fields of parameters
philtweir Aug 19, 2024
362af32
fix: reduce number of workflow merges
philtweir Aug 19, 2024
828d58c
fix: render configuration
philtweir Aug 20, 2024
8988037
fix: render configuration
philtweir Aug 20, 2024
f03b58c
fix: address field and hashing consistency
philtweir Aug 20, 2024
8c00247
fix: correct logic for remapping before a workflow has a name
philtweir Aug 20, 2024
0be1fea
fix: remove duplicate __hash__ method
philtweir Aug 20, 2024
0bae23c
fix: tidy up 0.9.2 merge, mostly label corrections
philtweir Aug 24, 2024
8e03137
fix: tests not working
philtweir Aug 24, 2024
173b2c7
fix: make sure we can identify an iterated ref
philtweir Aug 24, 2024
0d8b3d1
docs: tidyup docstrings and structures
philtweir Aug 24, 2024
f1dd479
fix: throw import exceptions
philtweir Aug 24, 2024
5c2d7a2
fix: add configuration for simplify_ids
philtweir Aug 24, 2024
4aef9c4
docs: tidyup docstrings and structures
philtweir Aug 24, 2024
8c84ab2
chore: fix up typehints and disable mypy sympy checking, as sympy doe…
philtweir Aug 24, 2024
b4a4cbc
fix: unbreak the doublequotes
philtweir Aug 24, 2024
394e5a6
fix: unbreak the doublequotes
philtweir Aug 24, 2024
88472e4
fix: unbreak the doublequotes
philtweir Aug 24, 2024
861420b
fix: unbreak the doublequotes
philtweir Aug 24, 2024
99f26fd
wip: try 3.12 in test
philtweir Aug 24, 2024
3274ea9
fix: sort examples - TODO: fix explanatory text
philtweir Aug 24, 2024
1dc5692
chore: fix test linting - TODO: add docstrings
philtweir Aug 25, 2024
3871ef4
fix: make result ordering consistent
philtweir Aug 25, 2024
fc88fb4
fix: ensure that fieldability doesn't mean dask matches references as…
philtweir Aug 25, 2024
780a7d6
fix: restore 3.11 in test
philtweir Aug 25, 2024
3e589ee
ci: add 3.12 to the build matrix
philtweir Aug 25, 2024
0c45e51
fix: restore commented test
philtweir Aug 25, 2024
269e591
fix: mypy issue in tests on 3.11
philtweir Aug 25, 2024
4dfa8bd
chore: fix tests and types for mypy and ruff on 3.11 and 3.12
philtweir Aug 25, 2024
33874a3
ci: allow subclassing any for mypy to enable subclassing Symbol
philtweir Aug 25, 2024
3ec7e39
fix: make __make_reference__ pass workflow if none
philtweir Aug 25, 2024
d0e0d37
fix: remove variable re-use
philtweir Aug 25, 2024
e27db97
feat: can go upwards in fields
philtweir Aug 25, 2024
d890bd3
fix: do not lose the field structure when iterating
philtweir Aug 25, 2024
351d344
fix: add a catch for a bug in the first import
philtweir Aug 29, 2024
3bd55e4
fix: remove unnecessary abstractmethod annotation
philtweir Aug 30, 2024
a7c0d8d
fix: remove unnecessary abstractmethod annotation
philtweir Aug 30, 2024
c32469e
chore: Replace all Unions with | for consistency
KamenDimitrov97 Sep 2, 2024
f6f478b
chore: Refactored render.py:get_render_method for better readability,…
KamenDimitrov97 Sep 2, 2024
e76a0a9
chore: Formated methods
KamenDimitrov97 Sep 2, 2024
a0655c0
docs: Added docstrings to annotation tests
KamenDimitrov97 Sep 2, 2024
aa7d6e4
docs: Added docstrings for tests
KamenDimitrov97 Sep 2, 2024
3bff5e5
fix: Formating
KamenDimitrov97 Sep 2, 2024
7ae5a7a
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
2a2d790
fix: check that importing a non-compliant render module throws an error
philtweir Sep 9, 2024
241f338
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
fe5253c
fix: double-check importing correctly errors if render neither behave…
philtweir Sep 9, 2024
c28a31e
fix: ensure pytest ignores tests._lib
philtweir Sep 9, 2024
713882a
fix: correct name of default_config
philtweir Sep 10, 2024
6714786
fix: remove Graph import from dask.typing in backend_dask as it seems…
philtweir Sep 10, 2024
dd27801
ops: enable example tests in CI
KamenDimitrov97 Sep 17, 2024
b6677c0
feat: Added a test for default renderer
KamenDimitrov97 Sep 17, 2024
3b9ccfe
feat: Added a test for default renderer
KamenDimitrov97 Sep 17, 2024
f519c47
feat: Abstracted write_rendered_output from the main dewret CLI
KamenDimitrov97 Sep 30, 2024
a25d3c2
docs: Updated renderer tutorial with the protocols implementation nee…
KamenDimitrov97 Sep 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/python-static-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
- uses: python/mypy@v1.8.0
with:
paths: "./src"
- run: pip install .[test] && python -m mypy --strict --install-types --non-interactive ./src ./tests ./example
- run: pip install .[test] && python -m mypy --strict --install-types --allow-subclassing-any --non-interactive ./src ./tests ./example

audit:
runs-on: ubuntu-latest
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/python-test-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@ on: [push, pull_request]
jobs:
unit-pip:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12"]
KamenDimitrov97 marked this conversation as resolved.
Show resolved Hide resolved
steps:
- uses: actions/checkout@v4
- name: Set up Python

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: '3.11'
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip "hatchling < 1.22"
Expand Down
4 changes: 2 additions & 2 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,14 @@ and backends, as well as bespoke serialization or formatting.
>>>
>>> result = increment(num=3)
>>> workflow = construct(result, simplify_ids=True)
>>> cwl = render(workflow)
>>> cwl = render(workflow)["__root__"]
philtweir marked this conversation as resolved.
Show resolved Hide resolved
>>> yaml.dump(cwl, sys.stdout, indent=2)
class: Workflow
cwlVersion: 1.2
inputs:
increment-1-num:
default: 3
label: increment-1-num
label: num
type: int
outputs:
out:
Expand Down
9 changes: 3 additions & 6 deletions docs/renderer_tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,11 +401,11 @@ import inspect
import typing

from attrs import define
from dewret.utils import Raw, BasicType
from dewret.workflow import Lazy
from dewret.workflow import Reference, Raw, Workflow, Step, Task
from dewret.utils import BasicType
from dewret.workflow import Reference, Workflow, Step, Task

RawType = typing.Union[BasicType, list[str], list["RawType"], dict[str, "RawType"]]
RawType = BasicType | list[str] | list["RawType"] | dict[str, "RawType"]
```

## To run this example:
Expand All @@ -416,6 +416,3 @@ RawType = typing.Union[BasicType, list[str], list["RawType"], dict[str, "RawType
```shell
python snakemake_tasks.py
```

### Q: Should I add a brief description of dewret in step 1? Should link dewret types/docs etc here?
### A: Get details on how that happens and probably yes.
KamenDimitrov97 marked this conversation as resolved.
Show resolved Hide resolved
Loading
Loading