Skip to content

Redbiom improving listing #2132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 7, 2017
Merged

Redbiom improving listing #2132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
aca72c3
init commit for redbiom-improving-listing
antgonza May 11, 2017
5532e10
improving listing and grouping of artifacts
antgonza May 12, 2017
58edff2
show_loading needs to be in sitebase.html cause o the import
antgonza May 12, 2017
8a2a167
order matters for DataTables
antgonza May 12, 2017
9fa79e2
adding drop-down
antgonza May 20, 2017
bfb7efc
fixing categories
antgonza May 22, 2017
95c80c2
addressing @wasade comments
antgonza May 22, 2017
8abab71
addressing @josenavas comments
antgonza May 23, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 89 additions & 24 deletions qiita_pet/handlers/qiita_redbiom.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
# The full license is in the file LICENSE, distributed with this software.
# -----------------------------------------------------------------------------

from future.utils import viewitems
from requests import ConnectionError
import redbiom.summarize
import redbiom.search
Expand Down Expand Up @@ -71,7 +72,22 @@ def _redbiom_search(self, query, search_on, callback):
import qiita_db.sql_connection as qdbsc
if features:
if search_on in ('metadata', 'observations'):
# This query basically selects all studies/prep-infos
# that contain the given sample ids (main_query); with
# those we select the artifacts that are only BIOM and
# their parent (artifact_query); then we can select
# the processing parameters of the parents. The
# structure might seem excessive but it's the only way
# to work with the json elements of the database and
# make it run in an OK manner (see initial comments)
# in https://github.com/biocore/qiita/pull/2132. Also
# this query could be simplified if we used a newer
# version of postgres, which doesn't mean faster
# performance
sql = """
-- main_query will retrieve all basic information
-- about the study based on which samples that exist
-- in the prep info file, and their artifact_ids
WITH main_query AS (
SELECT study_title, study_id, artifact_id,
array_agg(DISTINCT sample_id) AS samples,
Expand All @@ -83,42 +99,91 @@ def _redbiom_search(self, query, search_on, callback):
(prep_template_id)
JOIN qiita.study USING (study_id)
WHERE sample_id IN %s
GROUP BY study_title, study_id, artifact_id)
SELECT study_title, study_id, samples,
name, command_id,
(main_query.children).artifact_id AS artifact_id
FROM main_query
JOIN qiita.artifact a ON
(main_query.children).artifact_id = a.artifact_id
JOIN qiita.artifact_type at ON (
at.artifact_type_id = a.artifact_type_id
AND artifact_type = 'BIOM')
ORDER BY artifact_id
GROUP BY study_title, study_id, artifact_id),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be possible to get an english description of what this query does? inline comments within the query to describe the subqueries would also be helpful

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added some information.

-- now, we can take all the artifacts and just select
-- the BIOMs, while selecting the parent of the
-- artifacts, note that we are selecting the main
-- columns (discardig children) from the main_query +
-- the children artifact_id
artifact_query AS (
SELECT study_title, study_id, samples,
name, command_id,
(main_query.children).artifact_id AS
artifact_id
FROM main_query
JOIN qiita.artifact a ON
(main_query.children).artifact_id =
a.artifact_id
JOIN qiita.artifact_type at ON (
at.artifact_type_id = a.artifact_type_id
AND artifact_type = 'BIOM')),
-- now, we can select the parent processing parameters
-- of the children, note that we are selecting all
-- columns returned from artifact_query and the
-- parent processing parameters
parent_query AS (
SELECT artifact_query.*,
array_agg(parent_params) as parent_parameters
FROM artifact_query
LEFT JOIN qiita.parent_artifact pa ON (
artifact_query.artifact_id = pa.artifact_id)
LEFT JOIN qiita.artifact a ON (
pa.parent_id = a.artifact_id),
json_each_text(command_parameters)
parent_params
GROUP BY artifact_query.study_title,
artifact_query.study_id,
artifact_query.samples, artifact_query.name,
artifact_query.command_id,
artifact_query.artifact_id)
-- just select everything that is the parent_query
SELECT * FROM parent_query
ORDER BY parent_parameters, artifact_id
"""

sql_params = """
SELECT parameter_set_name, array_agg(ps) AS param_set
FROM qiita.default_parameter_set,
json_each_text(parameter_set) ps
GROUP BY parameter_set_name"""

with qdbsc.TRN:
qdbsc.TRN.add(sql, [tuple(features)])
results = []
commands = {}
# obtaining all existing parameters, note that
# they are not that many (~40) and we don't expect
# to have a huge growth in the near future
qdbsc.TRN.add(sql_params)
params = {pname: eval(params) for pname, params
in qdbsc.TRN.execute_fetchindex()}

# now let's get the actual artifacts
qdbsc.TRN.add(sql, [tuple(features)])
for row in qdbsc.TRN.execute_fetchindex():
title, sid, samples, name, cid, aid = row
title, sid, samples, name, cid, aid, pp = row
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont understand how the select on main_query did not change but somehow row has an additional element now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the with is basically a select after another select, this allows us to make faster queries and "work" with json ... hopefully the explanation added helps.

nr = {'study_title': title, 'study_id': sid,
'artifact_id': aid, 'aname': name,
'samples': samples}
if cid is not None:
if cid not in commands:
c = qdb.software.Command(cid)
commands[cid] = {
'sfwn': c.software.name,
'sfv': c.software.version,
'cmdn': c.name
}
nr['command'] = commands[cid]['cmdn']
nr['software'] = commands[cid]['sfwn']
nr['version'] = commands[cid]['sfv']
commands[cid] = '%s - %s v%s' % (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should trim length be represented?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has it, in the next block the processing parameters of the parent are added.

c.name, c.software.name,
c.software.version)

# [-1] taking the last cause it's sorted by
# the number of overlapping parameters
# [0] then taking the first element that is
# the name of the parameter set
ppc = sorted(
[[k, len(eval(pp) & v)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is being evald?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psql return a set as a string so we are eval that ..

for k, v in viewitems(params)],
key=lambda x: x[1])[-1][0]

nr['command'] = '%s @ %s' % (
commands[cid], ppc)
else:
nr['command'] = None
nr['software'] = None
nr['version'] = None
nr['command'] = ''
results.append(nr)
else:
sql = """
Expand Down
14 changes: 0 additions & 14 deletions qiita_pet/static/js/qiita.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,17 +301,3 @@ function display_number_of_samples_added(num_samples) {
bootstrapAlert(num_samples + ' samples selected.', "success", 10000);
$('#dflt-sel-info').css('color', 'rgb(0, 160, 0)');
}

/**
*
* Function to show the loading gif in a given div
*
* @param div_name string with the name of the div to populate with the loading gif
*
* This function replaces the content of the given div with the
* gif to show that the section of page is loading
*
*/
function show_loading(div_name) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this function removed from here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previous PR, you requested to be moved here and I did. However, while working on this version I realized that it was a bad decision because it uses {% raw qiita_config.portal_dir %}, which it's not executed correctly as it's in the qiita.js file. Moved it sitebase.html so it works correctly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

$("#" + div_name).html("<img src='{% raw qiita_config.portal_dir %}/static/img/waiting.gif' style='display:block;margin-left: auto;margin-right: auto'/>");
}
154 changes: 99 additions & 55 deletions qiita_pet/templates/redbiom.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,11 @@
moi.init(null, window.location.host + '{% raw qiita_config.portal_dir %}/study/list/socket/', function(){}, error, error);
moi.add_callback('sel', display_number_of_samples_added);
$("#redbiom-table").dataTable({
"dom": 'lfrt<"footer">ip',
"dom": '<"ps">lfr<t><"footer">ip',
'footerCallback': function ( row, data, start, end, display ) {

// adding total features found

var api = this.api(), data;
var total_features = api
.column(4)
Expand All @@ -30,23 +33,52 @@
}
}
$('.footer').html(text)

// adding selectors for filtering

this.api().columns().every( function (iterator) {
var column = this;
if (iterator == 3 && column.data().length != 0) {
var placer = $(".ps");
placer.empty();
placer.append("<b>Filter by Processing paramters</b>: ")
var selector = $('<select><option value=""></option></select>')
.appendTo( placer )
.on('change', function () {
var val = $.fn.dataTable.util.escapeRegex( $(this).val() );
column
.search( val ? '^'+val+'$' : '', true, false )
.draw();
});

var added = [];
column.data().each(function (d){
var cmd = d.command;
if (added.indexOf(cmd) == -1){
added.push(cmd);
selector.append('<option value="' + cmd + '">' + cmd + '</option>');
}
});
}
});
},
"columns": [
{ "data": null, "orderable": false, "width": "10%" },
{ "data": null, "width": "40%" },
{ "data": "aname", "width": "20%" },
{ "data": null, "width": "20%" },
{ "data": null, "width": "10%" },
{ "data": null, "width": "30%" },
{ "data": null, "width": "10%" }
],
columnDefs: [
{type:'natural', targets:[1,2,3,4]},
{ type:'natural', targets: [1,2,3,4] },
// render Add to Analysis button
{"render": function ( data, type, row, meta ) {
var text = '';
{% if current_user is not None %}
var text = '<input type="button" class="btn btn-sm" value="Add to analysis" onclick="redbiom_send_to_moi(' +
row.artifact_id + ', ' + meta.row + ')">';
{% else %}
var text = '';
if ($("#search_on").val() != 'categories'){
var text = '<input type="button" class="btn btn-sm" value="Add to analysis" onclick="redbiom_send_to_moi(' +
row.artifact_id + ', ' + meta.row + ')">';
}
{% end %}
return text;
}, targets: [0]},
Expand All @@ -60,71 +92,83 @@
{% end %}
return text;
}, targets: [1]},
// render Artifact Processing
// render Artifact Name
{"render": function ( data, type, row, meta ) {
var text = '';
if (!(!row.command)) {
text += row.command;
}
if (!(!row.software)) {
text += ' ' + row.software;
}
if (!(!row.version)) {
text += ' v' + row.version;
}
var text = row.aname;
if ($("#search_on").val() == 'categories'){
text = '';
}
return text;
}, targets: [2]},
// render Processing Parameters or Metadata Categories
{"render": function ( data, type, row, meta ) {
var text = row.command;
if ($("#search_on").val() == 'categories'){
text = row.samples.join(', ');
}
return text;
}, targets: [3]},
// render Artifact Processing
// render # of samples/categories
{"render": function ( data, type, row, meta ) {
return row.samples.length;
}, targets: [4]}
],
"language": {
"search": "Filter results by column data (Study Name, Artifact Name, Software, etc):",
"search": "Filter results by column data:",
"loadingRecords": "Please wait - loading information ...",
"zeroRecords": " "
},
});

$("#submitForm").submit(function(event){
event.preventDefault();
$("#submitForm").submit(function(event){
event.preventDefault();

show_loading("redbiom-info");
show_loading("redbiom-info");

var search = $("#search").val();
var search_on = $("#search_on").val();
var search = $("#search").val();
var search_on = $("#search_on").val();

$.post("{% raw qiita_config.portal_dir %}/redbiom/", {'search': search, 'search_on': search_on})
.done(function ( data ){
var redbiom_table = $('#redbiom-table').DataTable();
var redbiom_info = $('#redbiom-info');
redbiom_table.clear().draw();
if(data.status == "error") {
bootstrapAlert(data.message.replace("\n", "<br/>"), "danger");
} else {
if(data.message != ''){
redbiom_info.html(
`<div class="alert alert-warning alert-dismissible" role="alert">
<button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">&times;</span></button>
<strong>Warning!</strong> ` + data.message + `</div><br/>`)
} else if(data.data){
$('#redbiom-table tr:eq(0) th:eq(1)').text("Study Title");
if (search_on != 'categories') {
{% if current_user is not None %}
$('#redbiom-table tr:eq(0) th:eq(0)').text("Add to Analysis");
{% end %}
$('#redbiom-table tr:eq(0) th:eq(2)').text("Artifact Name");
$('#redbiom-table tr:eq(0) th:eq(3)').text("Artifact Processing");
$('#redbiom-table tr:eq(0) th:eq(4)').text("# of Samples Found");
} else {
$('#redbiom-table tr:eq(0) th:eq(3)').text("Metadata Categories");
$('#redbiom-table tr:eq(0) th:eq(4)').text("# of Categories Found");
}
redbiom_table.rows.add(data.data).draw();
redbiom_info.html('');
$.post("{% raw qiita_config.portal_dir %}/redbiom/", {'search': search, 'search_on': search_on})
.done(function ( data ){
var redbiom_table = $('#redbiom-table').DataTable();
var redbiom_info = $('#redbiom-info');
// the next 4 lines reset the column filtering
var placer = $(".ps");
redbiom_table.column(3).search("").draw();
redbiom_table.clear().draw();
placer.empty();
if(data.status == "error") {
bootstrapAlert(data.message.replace("\n", "<br/>"), "danger");
} else {
if(data.message != ''){
redbiom_info.html(
`<div class="alert alert-warning alert-dismissible" role="alert">
<button type="button" class="close" data-dismiss="alert" aria-label="Close"><span aria-hidden="true">&times;</span></button>
<strong>Warning!</strong> ` + data.message + `</div><br/>`)
} else if(data.data){
{% if current_user is not None %}
var header_0 = "Add to Analysis";
{% else %}
var header_0 = "";
{% end %}
var header_2 = "Artifact Name";
var header_3 = "Processing Parameters";
var header_4 = "# of Samples";
if (search_on == 'categories') {
header_2 = "";
header_3 = "Metadata Categories";
header_4 = "# of Categories";
}
$('#redbiom-table tr:eq(0) th:eq(0)').text(header_0);
$('#redbiom-table tr:eq(0) th:eq(1)').text("Study Title");
$('#redbiom-table tr:eq(0) th:eq(2)').text(header_2);
$('#redbiom-table tr:eq(0) th:eq(3)').text(header_3);
$('#redbiom-table tr:eq(0) th:eq(4)').text(header_4);
redbiom_table.rows.add(data.data).draw();
redbiom_info.html('');
}
});
}
});
});
});
</script>
Expand Down
14 changes: 14 additions & 0 deletions qiita_pet/templates/sitebase.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@
<link rel="shortcut icon" href="{% raw qiita_config.portal_dir %}/static/img/favicon.ico">

<script>
/**
*
* Function to show the loading gif in a given div
*
* @param div_name string with the name of the div to populate with the loading gif
*
* This function replaces the content of the given div with the
* gif to show that the section of page is loading
*
*/
function show_loading(div_name) {
$("#" + div_name).html("<img src='{% raw qiita_config.portal_dir %}/static/img/waiting.gif' style='display:block;margin-left: auto;margin-right: auto'/>");
}

function overlay_check() {
// from http://stackoverflow.com/q/5916900/379593
var ua=navigator.userAgent,tem,M=ua.match(/(opera|chrome|safari|firefox|msie|trident(?=\/))\/?\s*(\d+)/i) || [];
Expand Down
Loading