Allow complex `Filters` #2145

ychiucco · 2024-12-16T09:28:42Z

Checklist before merging

I added an appropriate entry to CHANGELOG.md
I added logging to new code - if appropriate.
I merged main into the current branch.

github-actions · 2024-12-16T14:07:45Z

Benchmark comparison

GET /api/alive/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`
vanilla	0.8	1.0	0.80
power	0.8	0.9	0.89
dataset	0.8	0.9	0.89
project	0.8	0.9	0.89
job	0.8	0.9	0.89

GET /api/v2/dataset/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	10.3	9.4	1.10	0.7	0.6
power	434.9	410.8	1.06	404.6	366.2
dataset	162.9	149.7	1.09	147.4	134.0
project	51.3	43.2	1.19	52.0	47.2
job	30.0	29.3	1.02	35.6	32.4

GET /api/v2/job/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.6	6.4	1.03	2.2	2.2
power	149.9	141.4	1.06	665.0	652.2
dataset	151.6	146.8	1.03	918.0	905.2
project	49.0	49.0	1.00	287.8	283.8
job	40.3	42.3	0.95	228.1	224.9

GET /api/v2/project/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	7.8	7.4	1.05	0.1	0.1
power	7.5	7.1	1.06	0.1	0.1
dataset	7.7	7.3	1.05	0.1	0.1
project	9.5	9.5	1.00	2.3	2.3
job	7.6	7.4	1.03	0.1	0.1

GET /api/v2/task/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.4	5.8	1.10	0.8	0.8
power	6.3	5.7	1.11	0.8	0.8
dataset	7.0	6.1	1.15	0.8	0.8
project	6.1	5.9	1.03	0.8	0.8
job	5.6	6.2	0.90	0.8	0.8

GET /api/v2/task-group/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	6.9	7.6	0.91	1.6	1.6
power	6.7	7.5	0.89	1.6	1.6
dataset	6.8	7.4	0.92	1.6	1.6
project	6.8	7.3	0.93	1.6	1.6
job	6.9	7.3	0.95	1.6	1.6

GET /api/v2/workflow/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	14.2	10.8	1.31	0.9	0.8
power	25.6	24.8	1.03	30.4	29.1
dataset	19.7	20.0	0.98	17.4	16.7
project	23.1	22.2	1.04	21.8	21.0
job	10.9	12.0	0.91	0.9	0.8

POST /api/v2/project/3/dataset/417/images/query/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	155.9	152.4	1.02	275.1	275.1

GET /auth/current-user/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
vanilla	4.7	4.8	0.98	0.2	0.2
power	4.4	4.8	0.92	0.2	0.2
dataset	4.5	4.8	0.94	0.2	0.2
project	4.5	4.8	0.94	0.2	0.2
job	4.4	4.8	0.92	0.1	0.1

POST /auth/token/login/

User	Time current (ms)	Time `main` (ms)	Ratio current/`main`	Size current (Kb)	Size `main` (Kb)
dataset	274.3	273.8	1.00	0.2	0.2

github-actions · 2024-12-17T10:09:24Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)
fractal_server
__init__.py	$This PR doesn't change the number of statements in fractal_server/__init__.py, which is 1.$	$This PR doesn't change the number of statements missing coverage in fractal_server/__init__.py, which is 0.$	$This PR doesn't change the coverage rate in fractal_server/__init__.py, which is 100% (1/1).$	$In this PR, 1 new statements are added to fractal_server/__init__.py, 1 of which are covered (100%).$
fractal_server/app/models/v2
dataset.py	$This PR adds 1 to the number of statements in fractal_server/app/models/v2/dataset.py, taking it from 26 to 27.$	$This PR doesn't change the number of statements missing coverage in fractal_server/app/models/v2/dataset.py, which is 0.$	$This PR doesn't change the coverage rate in fractal_server/app/models/v2/dataset.py, which is 100% (27/27).$	$In this PR, 2 new statements are added to fractal_server/app/models/v2/dataset.py, 2 of which are covered (100%).$
workflowtask.py	$This PR adds 1 to the number of statements in fractal_server/app/models/v2/workflowtask.py, taking it from 24 to 25.$	$This PR doesn't change the number of statements missing coverage in fractal_server/app/models/v2/workflowtask.py, which is 0.$	$This PR doesn't change the coverage rate in fractal_server/app/models/v2/workflowtask.py, which is 100% (25/25).$	$In this PR, 1 new statements are added to fractal_server/app/models/v2/workflowtask.py, 1 of which are covered (100%).$
fractal_server/app/routes/api/v2
images.py	$This PR doesn't change the number of statements in fractal_server/app/routes/api/v2/images.py, which is 107.$	$This PR doesn't change the number of statements missing coverage in fractal_server/app/routes/api/v2/images.py, which is 3.$	$This PR doesn't change the coverage rate in fractal_server/app/routes/api/v2/images.py, which is 95.86% (104/107).$	$In this PR, 1 new statements are added to fractal_server/app/routes/api/v2/images.py, 1 of which are covered (100%).$
fractal_server/app/runner/executors/slurm/sudo
executor.py	$This PR doesn't change the number of statements in fractal_server/app/runner/executors/slurm/sudo/executor.py, which is 437.$	$This PR adds 7 to the number of statements missing coverage in fractal_server/app/runner/executors/slurm/sudo/executor.py, taking it from 53 to 60.$	$This PR removes 1.48 percentage points from the coverage rate in fractal_server/app/runner/executors/slurm/sudo/executor.py, taking it from 85.76% (384/437) to 84.28% (377/437).$	$This PR does not seem to add statements to fractal_server/app/runner/executors/slurm/sudo/executor.py.$
fractal_server/app/runner/v2
runner.py	$This PR adds 2 to the number of statements in fractal_server/app/runner/v2/runner.py, taking it from 123 to 125.$	$This PR doesn't change the number of statements missing coverage in fractal_server/app/runner/v2/runner.py, which is 4.$	$This PR adds 0.07 percentage points to the coverage rate in fractal_server/app/runner/v2/runner.py, taking it from 93.93% (119/123) to 94.01% (121/125).$	$In this PR, 4 new statements are added to fractal_server/app/runner/v2/runner.py, 4 of which are covered (100%).$
fractal_server/app/runner/v2/_local_experimental
executor.py	$This PR doesn't change the number of statements in fractal_server/app/runner/v2/_local_experimental/executor.py, which is 74.$	$This PR adds 1 to the number of statements missing coverage in fractal_server/app/runner/v2/_local_experimental/executor.py, taking it from 0 to 1.$	$This PR removes 2.22 percentage points from the coverage rate in fractal_server/app/runner/v2/_local_experimental/executor.py, taking it from 97.77% (74/74) to 95.55% (73/74).$	$This PR does not seem to add statements to fractal_server/app/runner/v2/_local_experimental/executor.py.$
fractal_server/images
models.py	$This PR adds 14 to the number of statements in fractal_server/images/models.py, taking it from 64 to 78.$	$This PR doesn't change the number of statements missing coverage in fractal_server/images/models.py, which is 1.$	$This PR adds 0.46 percentage points to the coverage rate in fractal_server/images/models.py, taking it from 97.61% (63/64) to 98.07% (77/78).$	$In this PR, 20 new statements are added to fractal_server/images/models.py, 20 of which are covered (100%).$
tools.py	$This PR adds 3 to the number of statements in fractal_server/images/tools.py, taking it from 29 to 32.$	$This PR adds 1 to the number of statements missing coverage in fractal_server/images/tools.py, taking it from 0 to 1.$	$This PR removes 4.17 percentage points from the coverage rate in fractal_server/images/tools.py, taking it from 100% (29/29) to 95.83% (31/32).$	$In this PR, 7 new statements are added to fractal_server/images/tools.py, 7 of which are covered (100%).$
Project Total

_{This report was generated by python-coverage-comment-action}

tcompa · 2024-12-17T11:03:30Z

benchmarks/runner/mocks.py

+class FiltersMock(TypedDict):
+    types: dict[str, bool]
+    attributes_include: dict[str, list[Any]]
+    attributes_exclude: dict[str, list[Any]]
+
+    @classmethod
+    def get_default(cls):
+        return cls(types={}, attributes_include={}, attributes_exclude={})


Let's re-use something like for sqlmodel models, and move it to fractal_server/images/models.py next to the actual Filters model.

tcompa · 2024-12-17T11:05:32Z

fractal_server/app/models/v2/workflowtask.py

    ] = Field(
        sa_column=Column(
            JSON,
            nullable=False,
-            server_default='{"attributes": {}, "types": {}}',
+            server_default=json.dumps(


Let's use a FILTERS_DEFAULT_JSON variable, defined in fractal_server/images/models.py. It would be something equivalent to

FILTERS_DEFAULT_JSON = json.dumps(FiltersMock.get_defaults()) # replace FiltersMock with name

change Filters class

cdb783b

ychiucco linked an issue Dec 16, 2024 that may be closed by this pull request

Allow complex filters being set #1401

Open

ychiucco added 4 commits December 16, 2024 12:03

fix validation

90ea2fe

fix test_schemas_dataset

9d7aab2

fix default

c7a273a

fix test

832e0e2

ychiucco added 13 commits December 16, 2024 15:33

fix type hint

48f6eb6

fix input_filters

6bed22b

special case [None]

26c4c8a

fix test_filters

56610c3

migration

232ebde

fix runner

63ca24d

mock filter class

94359e1

fix migration

39fc994

add FiltersMock to benchmarks

72c0028

fix test images

a8af1bb

fix migrations

51b1103

Merge branch 'main' into 1401-allow-complex-filters-being-set

39f2d0d

fix test

f66a091

tcompa reviewed Dec 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow complex `Filters` #2145

Allow complex `Filters` #2145

ychiucco commented Dec 16, 2024

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024

tcompa Dec 17, 2024

tcompa Dec 17, 2024

Allow complex Filters #2145

Are you sure you want to change the base?

Allow complex Filters #2145

Conversation

ychiucco commented Dec 16, 2024

Checklist before merging

github-actions bot commented Dec 16, 2024 • edited Loading

Benchmark comparison

GET /api/alive/

GET /api/v2/dataset/

GET /api/v2/job/

GET /api/v2/project/

GET /api/v2/task/

GET /api/v2/task-group/

GET /api/v2/workflow/

POST /api/v2/project/3/dataset/417/images/query/

GET /auth/current-user/

POST /auth/token/login/

github-actions bot commented Dec 17, 2024

Coverage report

tcompa Dec 17, 2024

Choose a reason for hiding this comment

tcompa Dec 17, 2024

Choose a reason for hiding this comment

Allow complex `Filters` #2145

Allow complex `Filters` #2145

github-actions bot commented Dec 16, 2024 •

edited

Loading