minor improvements

[antgonza]

Two minor improvements:

1. Improve performance of _common_to_dataframe_steps ~17 seconds for AGP:

In [50]: %timeit SampleTemplate(10317).to_dataframe()                                                                                                                                                                                                                                       
1min 8s ± 224 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [51]: %timeit ST(10317).to_dataframe()                                                                                                                                                                                                                                                   
51.8 s ± 259 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

2. Turns out that jobs actually append the node where the job was submitted so that list comprehension will miss jobs submitted in the workers.

[codecov-io]

Codecov Report

Merging #3067 (a5b9be5) into dev (36faa57) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              dev    #3067   +/-   ##
=======================================
  Coverage   95.02%   95.03%           
=======================================
  Files          74       74           
  Lines       14279    14295   +16     
=======================================
+ Hits        13569    13585   +16     
  Misses        710      710           

Impacted Files	Coverage Δ
qiita_db/metadata_template/constants.py	100.00% <ø> (ø)
qiita_db/metadata_template/prep_template.py	99.11% <ø> (+0.04%)	⬆️
qiita_db/__init__.py	100.00% <100.00%> (ø)
qiita_db/meta_util.py	88.62% <100.00%> (ø)
...ita_db/metadata_template/base_metadata_template.py	96.25% <100.00%> (-0.08%)	⬇️
qiita_db/metadata_template/sample_template.py	100.00% <100.00%> (ø)
...ta_db/metadata_template/test/test_prep_template.py	99.55% <100.00%> (+<0.01%)	⬆️
..._db/metadata_template/test/test_sample_template.py	99.86% <100.00%> (+<0.01%)	⬆️
qiita_db/processing_job.py	73.80% <100.00%> (ø)
qiita_db/study.py	94.11% <100.00%> (ø)
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6e5d80...a5b9be5. Read the comment docs.

[wasade]

Would it be possible to do pd.concat([df.drop(1, axis=1), df[1]])? It not really obvious why the new DataFrame construction is needed or the tolist. Oh wait, is this supposed to be a reorder operation moving the first column to the end of the frame? If so, then i think this does it:

cols = [df.columns[0]] + df.columns[2:].tolist() + [df.columns[1]]
df = df[cols]

[antgonza]

Good question. The metadata tables are basically 2 columns: sample_id, metadata (json, which is returned as a dict). In other words, the SQL will return [[sample_id, {'col_a': 'a', 'col_b': 'b'}, ...], [sample_id, {'col_a': 'a', 'col_b': 'b'}, ...], ...] so that line basically expands the dict into actual columns in the dataframe, like this:

In [61]: data = ['x', {'a': 1, 'b': 2}], ['y', {'a': 1, 'c': 2}]                                                                                                                                                                                                                            

In [62]: df = pd.DataFrame(data)                                                                                                                                                                                                                                                            

In [63]: df                                                                                                                                                                                                                                                                                 
Out[63]: 
   0                 1
0  x  {'a': 1, 'b': 2}
1  y  {'a': 1, 'c': 2}

In [64]: pd.concat([df.drop(1, axis=1), pd.DataFrame(df[1].tolist())], axis=1)                                                                                                                                                                                                              
Out[64]: 
   0  a    b    c
0  x  1  2.0  NaN
1  y  1  NaN  2.0

[wasade]

Okay, thanks. What about this? It's a little less juggling and you get the index "for free"?

>>> import pandas as pd
>>> data = ['x', {'a': 1, 'b': 2}], ['y', {'a': 1, 'c': 2}]
>>> pd.DataFrame([d for _, d in data], index=[i for i, _ in data])
   a    b    c
x  1  2.0  NaN
y  1  NaN  2.0

[antgonza]

Thanks, this made the code a bit faster too. If you agree with the changes, and the tests pass, could you merge?

[ElDeveloper]

These changes look good. @antgonza I'm not familiar with the pgsql json side of things but, this seems like the kind of thing that pgsql could probably do much faster than Pandas/Python. Did you happen to look into querying the data directly into a table form? Rather than querying a JSON string, and then converting it to a table?

I'm looking into it now, but just want to make sure I shared this observation.

[ElDeveloper]

Following up on this @antgonza shared some benchmarks showing that the pgsql implementation was in fact slower than the changes in this PR.

[ElDeveloper]

Thanks @wasade and @antgonza