You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
provide a plugin-specific working dir for data-collect plugin. One data-collect plugin will download and maintain data in this working dir.
provide the same working dir to data-access plugin. Data-access plugin can also modify contents in this working directory.
This mechanism has the following problems that we hope data-collect plugin can make full use of this working directory. For example, they can reuse existing data, do some data cache, and optimize storage space. But since data-access plugin will also modify the same directory, the data-collect plugin will get lost
To solve this problem, I suggest we keep the current style for data-collect. Meanwhile, we provide a temp directory (job-specific) directory for data-access. This can be removed after the whole job is finished.
The text was updated successfully, but these errors were encountered:
WenheLI
changed the title
Dataset downloaded by data-collect plugins should be reused and not affected by other plugins
Data Collect Plugin re-use
Sep 8, 2020
Pipeline works as following now:
provide a plugin-specific working dir for data-collect plugin. One data-collect plugin will download and maintain data in this working dir.
provide the same working dir to data-access plugin. Data-access plugin can also modify contents in this working directory.
This mechanism has the following problems that we hope data-collect plugin can make full use of this working directory. For example, they can reuse existing data, do some data cache, and optimize storage space. But since data-access plugin will also modify the same directory, the data-collect plugin will get lost
To solve this problem, I suggest we keep the current style for data-collect. Meanwhile, we provide a temp directory (job-specific) directory for data-access. This can be removed after the whole job is finished.
The text was updated successfully, but these errors were encountered: