Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Collect Plugin re-use #569

Closed
rickycao-qy opened this issue Sep 8, 2020 · 0 comments · Fixed by #607
Closed

Data Collect Plugin re-use #569

rickycao-qy opened this issue Sep 8, 2020 · 0 comments · Fixed by #607
Assignees
Labels
pipeline Idea and discussion to create a pipeline for something

Comments

@rickycao-qy
Copy link
Collaborator

Pipeline works as following now:

  1. provide a plugin-specific working dir for data-collect plugin. One data-collect plugin will download and maintain data in this working dir.

  2. provide the same working dir to data-access plugin. Data-access plugin can also modify contents in this working directory.

This mechanism has the following problems that we hope data-collect plugin can make full use of this working directory. For example, they can reuse existing data, do some data cache, and optimize storage space. But since data-access plugin will also modify the same directory, the data-collect plugin will get lost

To solve this problem, I suggest we keep the current style for data-collect. Meanwhile, we provide a temp directory (job-specific) directory for data-access. This can be removed after the whole job is finished.

@rickycao-qy rickycao-qy added the pipeline Idea and discussion to create a pipeline for something label Sep 8, 2020
@rickycao-qy rickycao-qy self-assigned this Sep 8, 2020
@FeelyChau FeelyChau mentioned this issue Sep 8, 2020
16 tasks
@WenheLI WenheLI changed the title Dataset downloaded by data-collect plugins should be reused and not affected by other plugins Data Collect Plugin re-use Sep 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pipeline Idea and discussion to create a pipeline for something
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant