Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spans question in Feedback datasets #1868

Closed
11 tasks done
Tracked by #19
davidsbatista opened this issue Apr 7, 2022 · 8 comments · Fixed by #4622
Closed
11 tasks done
Tracked by #19

Spans question in Feedback datasets #1868

davidsbatista opened this issue Apr 7, 2022 · 8 comments · Fixed by #4622
Assignees
@dvsrepo
Labels
type: community request Indicates a feature requested by someone outside of the Argilla organization type: enhancement Indicates new feature requests type: popular request Indicates that several people outside of the Argilla organization are interested in this feature
Milestone
v1.26.0

Comments

@davidsbatista
Copy link

davidsbatista commented Apr 7, 2022 •
edited by jfcalvo
Loading

I need to annotate data where for each document I need to annotate named-entities (i.e., token level classification) and also a set of labels (doc-level classification)

How can one achieve this with rubrix, as far as I understood you need to have two different collections, for the two different tasks, and then, after annotating and exporting them, you need to merge the annotations.

Is there any way to have this done inside rubrix - also on the interface?

Tasks
Preview Give feedback
  1. [TASK] Allow to create responses for span questions argilla-server#45
    jfcalvo
  2. [TASK] Allow to update span questions to reorder labels argilla-server#49
    jfcalvo
  3. [TASK] Configure span questions from SDK #4616
    area: python sdk language: python severity: major team: backend type: enhancement
    frascuchon
  4. [TASK] Allow to provide suggestions and responses for span question from SDK #4618
    area: python sdk language: python severity: minor team: backend type: enhancement
    frascuchon
  5. [TASK] Impor/export dataset with span questions from/to HF hub #4635
    area: python sdk language: python severity: minor team: ml type: integration
    frascuchon
  6. [TASK] Allow score fields for suggestion values associated to a span question argilla-server#55
    jfcalvo
  7. [TASK] Improve how response and suggestion values are parsed by schemas argilla-server#61
    jfcalvo
  8. [TASK] Add new validator classes for Response and Suggestions argilla-server#64
    jfcalvo
  9. [TASK] Add new validator classes for Questions argilla-server#73
    jfcalvo
  10. [TASK] Add questions update visible_options validation argilla-server#75
    jfcalvo
  11. [TASK] Update docs for the new Span Question #4651
    area: ui severity: minor team: ml type: documentation
    nataliaElv
@davidsbatista davidsbatista added the type: enhancement Indicates new feature requests label Apr 7, 2022
@frascuchon frascuchon transferred this issue from argilla-io/argilla May 26, 2022
@frascuchon frascuchon added this to the 2022 Q4 milestone May 27, 2022
@frascuchon frascuchon added Manual Labeling type: community request Indicates a feature requested by someone outside of the Argilla organization labels Jul 12, 2022
@frascuchon frascuchon modified the milestones: 2022 Q4, 2023 Q1 Nov 3, 2022
@frascuchon frascuchon transferred this issue from another repository Nov 7, 2022
@frascuchon frascuchon moved this to 2023 Q1 in 🚲 Argilla Roadmap Nov 11, 2022
@frascuchon frascuchon modified the milestones: 2023 Q1, 2023 Q3 Nov 15, 2022
@frascuchon frascuchon moved this from 2023 Q1 to 2023 Q3 in 🚲 Argilla Roadmap Nov 15, 2022
@frascuchon frascuchon modified the milestones: 2023 Q3, 2023 Q2 Nov 15, 2022
@frascuchon frascuchon moved this from 2023 Q3 to 2023 Q2 in 🚲 Argilla Roadmap Nov 15, 2022
@davidberenstein1957
Copy link
Member

https://github.com/argilla-io/argilla/discussions/1969

@davidsbatista
Copy link
Author

so we are now 2, the number of users who already suggested this task, we're growing 😄

@nataliaElv nataliaElv moved this from Q2 2023 to Q3 2023 in 🚲 Argilla Roadmap May 4, 2023
@davidberenstein1957 davidberenstein1957 mentioned this issue Jun 6, 2023
Closed
@davidberenstein1957
Copy link
Member

Keep this issue updated

@davidberenstein1957
Copy link
Member

@davidsbatista We added this to Q3 2023 in our Roadmap

@davidsbatista
Copy link
Author

nice, thanks @davidberenstein1957 for keeping this as a feature 👍🏽 I think it will pay off

@nataliaElv nataliaElv mentioned this issue Sep 28, 2023
Closed
@nataliaElv nataliaElv removed this from the 2023 Q2 milestone Jul 4, 2023
This was referenced Nov 16, 2023
Closed
Closed
@nataliaElv nataliaElv added type: popular request Indicates that several people outside of the Argilla organization are interested in this feature and removed Manual Labeling labels Nov 23, 2023
@davidberenstein1957
Copy link
Member

@davidsbatista we are hard at work at tackling this issue at the moment. Would any of you be interested in providing some feedback and pointers w.r.t. what you would expect from the implementation? If so, could you ping me on Slack or send me an email at david@argilla.io?

@nataliaElv nataliaElv added this to the v1.25.0 milestone Feb 1, 2024
@nataliaElv nataliaElv mentioned this issue Feb 9, 2024
Closed
@davidsbatista
Copy link
Author

@davidberenstein1957 I currently don't have much time since I start a new project, but I can in between pull a docker image with the new features and try to run it and test it

@frascuchon frascuchon mentioned this issue Feb 12, 2024
Merged
13 tasks
@nataliaElv nataliaElv changed the title TokenClassification and TextClassification on the same interface Spans question in Feedback datasets Feb 13, 2024
frascuchon added a commit to argilla-io/argilla-server that referenced this issue Feb 15, 2024
@frascuchon @pre-commit-ci @jfcalvo
feat: Create span questions (#24)
b052c4c 
<!-- Thanks for your contribution! As part of our Community Growers
initiative 🌱, we're donating Justdiggit bunds in your name to reforest
sub-Saharan Africa. To claim your Community Growers certificate, please
contact David Berenstein in our Slack community or fill in this form
https://tally.so/r/n9XrxK once your PR has been merged. -->

# Description

This PR adds support for questions with `SpansQuestionSettings` to
endpoints `POST /api/v1/datasets/:dataset_id/questions` and `GET
/api/v1/datasets/:dataset_id/questions`

Also, dataset questions-related routes have been moved into a new module
called `datasets/questions.py`

Refs: argilla-io/argilla#1868

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [X] New feature (non-breaking change which adds functionality)
- [X] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

Changes have been tested locally

- [ ] Test A
- [ ] Test B

**Checklist**

- [ ] I added relevant documentation
- [X] I followed the style guidelines of this project
- [X] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [X] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] I have added relevant notes to the `CHANGELOG.md` file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: José Francisco Calvo <jose@argilla.io>
@nataliaElv nataliaElv modified the milestones: v1.25.0, v.1.26.0 Feb 20, 2024
@jfcalvo jfcalvo mentioned this issue Mar 6, 2024
Merged
15 tasks
@frascuchon frascuchon mentioned this issue Mar 11, 2024
Merged
13 tasks
@damianpumar damianpumar linked a pull request Mar 13, 2024 that will close this issue
🚀 feat/span questions #4622
Merged
@nataliaElv
Copy link
Member

@davidsbatista As you can probably see, we're working to deliver this feature in the next release. We'd love to know what you think of how the feature looks 😄 You can try it out following the instructions here: https://rubrixworkspace.slack.com/archives/C05DHJ3LGQM/p1710504136136119

jfcalvo added a commit to argilla-io/argilla-server that referenced this issue Mar 18, 2024
@jfcalvo @frascuchon @pre-commit-ci
[FEATURE-BRANCH] feat: support for span questions (#54)
4c8c259 
# Description

This is the feature branch supporting all the changes related with the
first iteration for span questions feature.

Refs argilla-io/argilla#1868

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)
- [ ] Documentation update

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

- [x] Adding required tests.

**Checklist**

- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [ ] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: Francisco Aranda <francis@argilla.io>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
frascuchon added a commit that referenced this issue Mar 18, 2024
@frascuchon @pre-commit-ci
feat: span questions support for SDK (#4643)
5d2bfd2 
<!-- Thanks for your contribution! As part of our Community Growers
initiative 🌱, we're donating Justdiggit bunds in your name to reforest
sub-Saharan Africa. To claim your Community Growers certificate, please
contact David Berenstein in our Slack community or fill in this form
https://tally.so/r/n9XrxK once your PR has been merged. -->

# Description

This PR contains the feature branch for span question support from SDK,
allowing to create span questions and providing suggestions and
responses from SDK.

Ref #1868

Closes #4618 
Closes #4635

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [X] New feature (non-breaking change which adds functionality)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [X] Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

Tested locally

- [ ] Test A
- [ ] Test B

**Checklist**

- [ ] I added relevant documentation
- [ ] I followed the style guidelines of this project
- [X] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [X] I have added relevant notes to the `CHANGELOG.md` file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Francisco Aranda <frascuchon@MacBook-Pro-de-Francisco.local>
frascuchon added a commit that referenced this issue Mar 18, 2024
@frascuchon @pre-commit-ci
feat: span questions support for SDK (#4643)
7585c8d 
<!-- Thanks for your contribution! As part of our Community Growers
initiative 🌱, we're donating Justdiggit bunds in your name to reforest
sub-Saharan Africa. To claim your Community Growers certificate, please
contact David Berenstein in our Slack community or fill in this form
https://tally.so/r/n9XrxK once your PR has been merged. -->

# Description

This PR contains the feature branch for span question support from SDK,
allowing to create span questions and providing suggestions and
responses from SDK.

Ref #1868

Closes #4618 
Closes #4635

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [X] New feature (non-breaking change which adds functionality)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [X] Improvement (change adding some improvement to an existing
functionality)

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

Tested locally

- [ ] Test A
- [ ] Test B

**Checklist**

- [ ] I added relevant documentation
- [ ] I followed the style guidelines of this project
- [X] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [X] I have added relevant notes to the `CHANGELOG.md` file (See
https://keepachangelog.com/)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Francisco Aranda <frascuchon@MacBook-Pro-de-Francisco.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dvsrepo dvsrepo

Labels
type: community request Indicates a feature requested by someone outside of the Argilla organization type: enhancement Indicates new feature requests type: popular request Indicates that several people outside of the Argilla organization are interested in this feature
Projects
None yet
Milestone
v1.26.0
Development

Successfully merging a pull request may close this issue.

🚀 feat/span questions argilla-io/argilla
5 participants
@dvsrepo @frascuchon @davidsbatista @davidberenstein1957 @nataliaElv