Allow overriding raw data files with processed derived data files in load function #42
Assignees
Labels
feature
New planned feature
Comments
glopesdev
added a commit
that referenced
this issue
Oct 13, 2021
glopesdev
added a commit
that referenced
this issue
Nov 4, 2021
|
PR merged, closing |
iamamutt
pushed a commit
that referenced
this issue
Feb 2, 2022
* Allow load to resolve priority list of paths Fixes #42 * Add support for parsing multianimal position data Fixes #55 * add plot for experiment-level average time spent at different locations of the arena * update docker stuff * Merge remote-tracking branch 'origin/datajoint_pipeline' * python package changes for docker install * docker updates * fix context path * fix dockerfile path * docker ghcr.io PAT instructions for private repo * use datajoint_pipeline for actions branch * pull latest conda/mamba image * Docker updates (#7) * new branch avoid build * changes to docker dev yml * user group settings * minor updates * update docker image * docker container paths permissions fixes * minor rename * refactor docker for aeon_db worker * incorporate thinhs changes * get env for linux amd64 * image updates * add script for sudo * process root dirs separately then concat * some keys not in json file * docker env updates * update sciviz deployment * confirm API works, revert back the temporary changes in `chunkdata` * gh actions github context * change cache type * remove old cache * devcontainer updates * dockerfile updates * docker updates * docker updates * Docker updates (#61) Updates to docker and trigger gh actions * test action * env instructions * fix repo owner * add locked envs * Docker updates Updates to docker and trigger gh actions * docker updates * docker updates * rm tmp dir gh actions * rm tmp dir gh actions * add back api fix * gh action * none path * docker updates * Docker ingestion tested (#64) * docker updates * try without cache * debug buildx * ingestion updates Co-authored-by: glopesdev <glopesdev@users.noreply.github.com> Co-authored-by: Thinh Nguyen <thinh@vathes.com> Co-authored-by: Jai Bhagat <jkbhagatio@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We would like to allow for curation and cleaning of low-level data, while keeping the original raw data immutable. Ideally downstream analysis scripts would not need to be changed in any major way to consume the cleaned up data.
It would be ideal if this can be transparently supported by the aeon file system by allowing the
loadmethod to receive a list of dataset paths which are used for file resolution in priority order (paths in the list are searched in FIFO order).We need to consider whether / how to make this process traceable, either by adding an attribute identifying the source of the dataset chunk, or some other means of logging.
The text was updated successfully, but these errors were encountered: