Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Convert post-processing to Transform classes that can be specified in config #537

Closed
Tracked by #614
NickleDave opened this issue Jul 8, 2022 · 1 comment
Closed
Tracked by #614

Comments

@NickleDave
Copy link
Collaborator

e.g. MajorityVote

[vak.model.TweetyNet]

[vak.model.TweetyNet.output-transform.MajorityVote]
[vak.model.TweetyNet.output_transform]
MinSegmentDur = {min_dur = 2}

🤔 not sure if this is valid TOML, but ... something like this

@NickleDave
Copy link
Collaborator Author

I think something like this will be necessary to achieve #472 without adding a ton of very specific logic to the _eval function

Basically the _eval function should not have to know anything about the inputs and outputs of the model, and how metrics get computed from those. It should just be something like:

def _eval():
    self.metrics.reinit()
    for batch in loader:
        output = self.model(input)
        self.metrics.add(output)
    return self.metrics.vals

One way to achieve this would be to have Networks return an Output class with attributes, and to have metrics look for those attributes. But that starts to also feel very constrained, i.e. we might regret what we chose for the expected attributes down the road

NickleDave added a commit that referenced this issue Feb 5, 2023
- Factor all functions out of labeled_timebins except has_unlabeled
- Only test has_unlabeled in test_labeled_timebins,
  parametrize the unit test
- Add 'multi_char_labels_to_single_char' to vak.labels
  - and add type annotations and clean up docstrings
- Clean up / add tests to tests/test_labels.py
- Add src/vak/transforms/labeled_timebins/functional.py
- Add src/vak/transforms/labeled_timebins/transforms.py
- Fix imports in vak/transforms/ __init__.py files
  - Import labeled_timebins transforms in transforms/__init__.py
  - Fix other imports
  - Import classes from transforms in
    src/vak/transforms/labeled_timebins/__init__.py
- TST: Move paths outside fixtures in fixtures/annot.py
  to be able to import directly in test modules
  to parametrize there.
- TST: Move path to constant outside fixture in fixtures/config.py
- Add tests/test_transforms/__init__.py
  so we can do relative import from tests/fixtures
- Add tests/test_transforms/test_labeled_timebins/test_functional.py
- Add tests/test_transforms/test_labeled_timebins/test_transforms.py
NickleDave added a commit that referenced this issue Feb 6, 2023
- Factor all functions out of labeled_timebins except has_unlabeled
- Only test has_unlabeled in test_labeled_timebins,
  parametrize the unit test
- Add 'multi_char_labels_to_single_char' to vak.labels
  - and add type annotations and clean up docstrings
- Clean up / add tests to tests/test_labels.py
- Add src/vak/transforms/labeled_timebins/functional.py
- Add src/vak/transforms/labeled_timebins/transforms.py
- Fix imports in vak/transforms/ __init__.py files
  - Import labeled_timebins transforms in transforms/__init__.py
  - Fix other imports
  - Import classes from transforms in
    src/vak/transforms/labeled_timebins/__init__.py
- TST: Move paths outside fixtures in fixtures/annot.py
  to be able to import directly in test modules
  to parametrize there.
- TST: Move path to constant outside fixture in fixtures/config.py
- Add tests/test_transforms/__init__.py
  so we can do relative import from tests/fixtures
- Add tests/test_transforms/test_labeled_timebins/test_functional.py
- Add tests/test_transforms/test_labeled_timebins/test_transforms.py
- Use `vak.transforms.labeled_timebins.from_segments`
  in vocal_dataset.py
- Use vak.transforms.labeled_timebins in engine/model.py
- Use transforms.labeled_timebins.from_segments
  in datasets/window_dataset.py
- Use transforms.labeled_timebins.to_segments in core/predict.py
- Use labeled_timebins transforms 'postprocess' and 'to_segments'
  in core/predict.py
NickleDave added a commit that referenced this issue Feb 7, 2023
- Factor all functions out of labeled_timebins except has_unlabeled
- Only test has_unlabeled in test_labeled_timebins,
  parametrize the unit test
- Add 'multi_char_labels_to_single_char' to vak.labels
  - and add type annotations and clean up docstrings
- Clean up / add tests to tests/test_labels.py
- Add src/vak/transforms/labeled_timebins/functional.py
- Add src/vak/transforms/labeled_timebins/transforms.py
- Fix imports in vak/transforms/ __init__.py files
  - Import labeled_timebins transforms in transforms/__init__.py
  - Fix other imports
  - Import classes from transforms in
    src/vak/transforms/labeled_timebins/__init__.py
- TST: Move paths outside fixtures in fixtures/annot.py
  to be able to import directly in test modules
  to parametrize there.
- TST: Move path to constant outside fixture in fixtures/config.py
- Add tests/test_transforms/__init__.py
  so we can do relative import from tests/fixtures
- Add tests/test_transforms/test_labeled_timebins/test_functional.py
- Add tests/test_transforms/test_labeled_timebins/test_transforms.py
- Use `vak.transforms.labeled_timebins.from_segments`
  in vocal_dataset.py
- Use vak.transforms.labeled_timebins in engine/model.py
- Use transforms.labeled_timebins.from_segments
  in datasets/window_dataset.py
- Use transforms.labeled_timebins.to_segments in core/predict.py
- Use labeled_timebins transforms 'postprocess' and 'to_segments'
  in core/predict.py
NickleDave added a commit that referenced this issue Feb 7, 2023
- Factor all functions out of labeled_timebins except has_unlabeled
- Only test has_unlabeled in test_labeled_timebins,
  parametrize the unit test
- Add 'multi_char_labels_to_single_char' to vak.labels
  - and add type annotations and clean up docstrings
- Clean up / add tests to tests/test_labels.py
- Add src/vak/transforms/labeled_timebins/functional.py
- Add src/vak/transforms/labeled_timebins/transforms.py
- Fix imports in vak/transforms/ __init__.py files
  - Import labeled_timebins transforms in transforms/__init__.py
  - Fix other imports
  - Import classes from transforms in
    src/vak/transforms/labeled_timebins/__init__.py
- TST: Move paths outside fixtures in fixtures/annot.py
  to be able to import directly in test modules
  to parametrize there.
- TST: Move path to constant outside fixture in fixtures/config.py
- Add tests/test_transforms/__init__.py
  so we can do relative import from tests/fixtures
- Add tests/test_transforms/test_labeled_timebins/test_functional.py
- Add tests/test_transforms/test_labeled_timebins/test_transforms.py
- Use `vak.transforms.labeled_timebins.from_segments`
  in vocal_dataset.py
- Use vak.transforms.labeled_timebins in engine/model.py
- Use transforms.labeled_timebins.from_segments
  in datasets/window_dataset.py
- Use transforms.labeled_timebins.to_segments in core/predict.py
- Use labeled_timebins transforms 'postprocess' and 'to_segments'
  in core/predict.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant