create a table with all tool settings used for the paper?

[alicemchardy]

We do have all tools fully reproducible in docker containers, however - for the making it easier for the reader, we could include a table with all tool parameter settings based on these dockers for the supplement. Any thoughts on this?

[sjanssen2]

If we go into this direction, we should also include used inputs like databases. Furthermore, it is not always clear what type of input sequences could be used for some of the profiling tools. Besides raw reads some offer the option to input contigs (I can't see how one would end up with the same abundances as with reads) but it is a situation where we could guide potential uses.

[sjanssen2]

I do know which parameters I used for those containers I compiled, but I am not sure if it is strait forward to investigate the parameter settings in partitcipant submitted containers - if we have those at all.
In what depth do we want to explain the underlying algorithms of the tools in our competition? If we don't, it might be hard to grasp what the parameter settings are good for.

[sjanssen2]

to make results comparable, I often had to add an additional conversion step to the actual profiling tool to e.g. make it speak NCBI tax IDs instead of GreenGenes strings. Thus, the containers are not pure wrapper to the tools. We should clarify that to the users.

[alicemchardy]

Good points. How about a table with the following columns

tool with version and reference / parameters used (where deviating from default) /modifications included by us / reference databases used (link to our collection)/ keywords to its approach

[p-hofmann]

Some tools are part of a pipeline, using blast, diamond, bowtie2, kallisto as a preparation step.
Either we need to include a column for like 'additional info' or we could see it as part of the parameter column?

[sjanssen2]

I started to compile this table: https://docs.google.com/spreadsheets/d/12flNC5JlhVl-qY_O9CSUs6Zauj6HWNU5NlC3kUn7WQc/edit?usp=sharing
It is hard to find out settings for the submitted dockerized tools. Do you think we could reach out to the submitters and ask them to fill out the missing information?

[alicemchardy]

Good idea. I made a copy of the document the CAMI evaluation workshop folder and called it now CAMI_tool_settings - once the draft is circulated, developers should add this