-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposed changes to framework #106
Comments
Merged
Martin, This sounds reasonable. We also need to think about specialized runners for the tasks we are thinking about. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The current workflow to run algorithm X on dataset Y is something like this:
python install.py
builds docker containerpython create_dataset.py
sets up datasetspython run.py --dataset Y --algorithm X
mountsdata/
,results/
,benchmark/
into the container for Xpython plot.py / data_export.py / ...
to evaluate the resultsGiven @harsha-simhadri's and @sourcesync's frustrations and some directions discussed in other meetings, I think we should relax step 3 a bit and allow more flexibility in the container setup. One direction could look like this:
python install.py
builds docker container, participants are expected to overwrite the entry point to point to their own implementation (filealgorithms/X/Dockerfile
)python create_dataset.py
sets up datasetsalgorithms/X/run.{py,sh}
)task
,dataset
, where the results should be written, and some additional parametersdata/
,results/
, and the config file that is used by the implementation (algorithms/X/config.yaml
, maybe task specific)a. file I/O in the container, loading/building index
b. running the experiment and providing timings
c. writing results in a standard format (as before
results/Y/X/run_identifier.hdf5
)python plot.py / data_export.py / ...
to evaluate the resultsWe provide a default run script for inspiration, which would be pretty close to the current setup. Putting all the logic into the container could mean a lot of code duplication, but isolated containers will allow for a much easier orchestration.
I can provide a proof-of-concept if this sounds promising.
The text was updated successfully, but these errors were encountered: