queue remove: can remove successed,failed tasks msgs

[karajan1001]

Followed from #7592

1. Seperate remove method to a new file.
2. Add queued, failed, processed flags to remove.
3. Add new unit tests for queue status --queue/fail/success
4. Implement methods to remove done tasks.
5. add two new unit test for celery_queue.remove
6. bump into dvc-task version 0.0.13

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

A demo for this PR.

[karajan1001]

@pmrowla
The only problem remained is that the queued experiment I prepared will be executed wrongly by some celery worker in the background, making the tests really unstable. Any ideas about it?

[pmrowla]

@pmrowla The only problem remained is that the queued experiment I prepared will be executed wrongly by some celery worker in the background, making the tests really unstable. Any ideas about it?

@karajan1001 it's likely due to using the celery pytest worker inside multiple fixtures at once.

In general, we should try to move away from adding complex func tests in favor of unit testing whenever possible. In this case, for celery we do need a couple of func tests to make sure that execution and the UI work properly (in both failed and successful cases), but for testing specific backend or API calls (like remove/status) it's better to use unit tests + mocked celery behavior

[pmrowla]

One other thing to note here regarding commits, ideally we want to keep changes in separate commits where possible. In this PR it would be nice to have the separation with:

• 1 commit for the dependency version bump. If there are specific changes required to make the new dep version work with DVC then those changes can also go into the version bump commit (like when a dependency has a backwards incompatible API change)
• 1 commit for the queue remove related changes
• 1 commit for the queue status related changes

Basically for small "review fix" type changes it makes sense to squash them into the larger commit, but for larger independent changes that affect different parts of the code base or different parts of the CLI/API, having separate commits makes it easier to keep track of what changed and when. So follow-up changes/fixes to remove or status can get squashed into each relevant commit before merging the PR, but ideally the final PR would not contain commits that mix changes to both remove and status.

---

This isn't really a requirement for this PR, but it's something to be aware of in the future

[karajan1001]

One other thing to note here regarding commits, ideally we want to keep changes in separate commits where possible. In this PR it would be nice to have the separation with:

• 1 commit for the dependency version bump. If there are specific changes required to make the new dep version work with DVC then those changes can also go into the version bump commit (like when a dependency has a backwards incompatible API change)
• 1 commit for the queue remove related changes
• 1 commit for the queue status related changes

After removing the functional test of queue status. It only contains queue remove related changes and the dependency bump. I can separate them later.

[pmrowla]

Do we need still need this test? With unit tests for remove() we should not need any additional func tests here

[karajan1001]

I think we can simplify them. We tested the remove method case by case as well as the cli test. But we don't have a place to test the whole process. This is why I added a functional test here, But I think we can simplify it, We don't need to test all of the cases, but only one of them.