Skip to content

Commit 76cf8e3

Browse files
authored
Merge branch 'qiita-spots:dev' into dev
2 parents 1b55c47 + 1b8b0c7 commit 76cf8e3

File tree

28 files changed

+7013
-1135
lines changed

28 files changed

+7013
-1135
lines changed

.github/workflows/qiita-ci.yml

Lines changed: 25 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
on:
44
push:
5-
branches: [ dev ]
5+
branches: [dev]
66
pull_request:
77

88
jobs:
@@ -45,7 +45,7 @@ jobs:
4545
uses: conda-incubator/setup-miniconda@v2
4646
with:
4747
auto-update-conda: true
48-
python-version: '3.9'
48+
python-version: "3.9"
4949

5050
- name: Basic dependencies install
5151
env:
@@ -148,6 +148,7 @@ jobs:
148148
149149
echo "4. Setting up nginx"
150150
mkdir -p /usr/share/miniconda/envs/qiita/var/run/nginx/
151+
sed -i "s|alias /Users/username|alias /home/runner/work/qiita|" ${PWD}/qiita_pet/nginx_example.conf
151152
nginx -c ${PWD}/qiita_pet/nginx_example.conf
152153
153154
echo "5. Setting up qiita"
@@ -169,17 +170,20 @@ jobs:
169170
conda deactivate
170171
171172
echo "8. Setting up SSH"
172-
ssh-keygen -t rsa -b 4096 -N '' -f $PWD/qiita_ware/test/test_data/test_key
173+
ssh-keygen -t ed25519 -a 200 -N '' -f $PWD/qiita_ware/test/test_data/test_key
173174
mkdir ~/.ssh/
174175
cp $PWD/qiita_ware/test/test_data/test_key* ~/.ssh/
175176
cat ~/.ssh/test_key.pub > ~/.ssh/authorized_keys
177+
cat ~/.ssh/test_key.pub > ~/.ssh/authorized_keys2
176178
chmod 600 $PWD/qiita_ware/test/test_data/test_key*
177179
chmod 600 ~/.ssh/*
178180
chmod 700 ~/.ssh/
179181
echo "Connecting as $USER@localhost"
180182
# this line (and the -o StrictHostKeyChecking=no) is so the server
181183
# is added to the list of known servers
182-
scp -O -o StrictHostKeyChecking=no -i $PWD/qiita_ware/test/test_data/test_key $USER@localhost:/home/runner/work/qiita/qiita/qiita_ware/test/test_data/random_key /home/runner/work/qiita/qiita/qiita_ware/test/test_data/random_key_copy_1
184+
185+
# 05/22/25: commenting this line out as github actions is not allowing this step
186+
# scp -O -o StrictHostKeyChecking=no -i $PWD/qiita_ware/test/test_data/test_key $USER@localhost:/home/runner/work/qiita/qiita/qiita_ware/test/test_data/random_key /home/runner/work/qiita/qiita/qiita_ware/test/test_data/random_key_copy_1
183187
184188
- name: Main tests
185189
shell: bash -l {0}
@@ -239,24 +243,24 @@ jobs:
239243
needs: main
240244
runs-on: ubuntu-latest
241245
steps:
242-
- name: Coveralls Finished
243-
uses: AndreMiras/coveralls-python-action@develop
244-
with:
245-
github-token: ${{ secrets.github_token }}
246-
parallel-finished: true
246+
- name: Coveralls Finished
247+
uses: AndreMiras/coveralls-python-action@develop
248+
with:
249+
github-token: ${{ secrets.github_token }}
250+
parallel-finished: true
247251

248252
lint:
249253
runs-on: ubuntu-latest
250254
steps:
251-
- name: flake8
252-
uses: actions/setup-python@v2
253-
with:
254-
python-version: '3.9'
255-
- name: install dependencies
256-
run: python -m pip install --upgrade pip
257-
- name: Check out repository code
258-
uses: actions/checkout@v2
259-
- name: lint
260-
run: |
261-
pip install -q flake8
262-
flake8 qiita_* setup.py scripts/qiita* notebooks/*/*.py
255+
- name: flake8
256+
uses: actions/setup-python@v2
257+
with:
258+
python-version: "3.9"
259+
- name: install dependencies
260+
run: python -m pip install --upgrade pip
261+
- name: Check out repository code
262+
uses: actions/checkout@v2
263+
- name: lint
264+
run: |
265+
pip install -q flake8
266+
flake8 qiita_* setup.py scripts/qiita* notebooks/*/*.py

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,3 +75,6 @@ notebooks/*/*.tsv.gz
7575

7676
# jupyter notebooks input data
7777
notebooks/resource-allocation/data
78+
79+
# ignore plugin SQL injection file
80+
qiita_db/support_files/patches/test_db_sql/91.sql

CHANGELOG.md

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,33 @@
11
# Qiita changelog
22

3+
Version 2025.07
4+
---------------
5+
6+
Deployed on July 15th, 2025
7+
8+
* During EBI-ENA submissions now we automatically renmae `country` to `geographic location (country and/or sea)` and `collection_date` to `collection date`; as the ENA requirements changed.
9+
* Added platform `DNBSEQ1` and instruments: `DNBSEQ-G400`, `DNBSEQ-T7`, `DNBSEQ-G800` for EBI-ENA submissions.
10+
* Other general fixes [#3474](https://github.com/qiita-spots/qiita/pull/3474), [#3475](https://github.com/qiita-spots/qiita/pull/3475).
11+
* SPP: merged and deprecated [mg-scripts](https://github.com/qiita-spots/mg-scripts) to [qp-knight-lab-processing](https://github.com/qiita-spots/qp-knight-lab-processing).
12+
* SPP: Added integration tests via a couple of PRs: [#129](https://github.com/qiita-spots/qp-knight-lab-processing/pull/129) & [#131](https://github.com/qiita-spots/qp-knight-lab-processing/pull/131).
13+
* SPP: Added new command `Human Filter & QC existing Prep` to facilitate human-filtering existing preparations.
14+
* SPP: Cleaned and centralized sequencers information to [kl-metapool](https://github.com/biocore/kl-metapool) & added `MiSeq i100`, thank you @AmandaBirmingham.
15+
* SPP: Added a new [subsample method](https://github.com/qiita-spots/qp-knight-lab-processing/pull/138) to randomly select sequences from paired files when they are larger than expected: 720,000,000.
16+
* SPP: Storing `TellReadJob/QC_Analysis_TellReadJob.html` in the results folder.
17+
18+
19+
20+
Version 2025.04
21+
---------------
22+
23+
Deployed on April 11th, 2025
24+
25+
* General improvements for automatic environment generation by @sjanssen2 (thank you!): [#3462](https://github.com/qiita-spots/qiita/pull/3462), [#3463](https://github.com/qiita-spots/qiita/pull/3463), [3464](https://github.com/qiita-spots/qiita/pull/3464), [#3465](https://github.com/qiita-spots/qiita/pull/3465).
26+
* When ProcessingJob.resource_allocation_info fails, it will now be [set as the error for the job](https://github.com/qiita-spots/qiita/pull/3466).
27+
* SPP: General updates and clean up: [#169](https://github.com/biocore/mg-scripts/pull/169), [#101](https://github.com/qiita-spots/qp-knight-lab-processing/pull/101).
28+
* `Remove SynDNA plasmid, insert, & CP026085 reads` superseded `Remove SynDNA inserts & plasmid reads`; which now removes SynDNA plasmids, inserts, and CP026085 reads, in this order.
29+
30+
331
Version 2025.02
432
---------------
533

@@ -8,7 +36,7 @@ Deployed on February 24th, 2025
836
* Replaced os.rename for shutil.move in the code to fix [#3455](https://github.com/qiita-spots/qiita/issues/3455).
937
* Via qp-spades, replaced the legacy `spades` command for `cloudSPAdes` for TellSeq.
1038
* `FASTA_preprocessed` within qtp-sequencing now allows for results to be named using their sample-name, extra from run-prefix.
11-
* `Remove SynDNA inserts & plasmid reads` superseded `Remove SynDNA reads`, which now removes SynDna inserts and plasmids.
39+
* `Remove SynDNA inserts & plasmid reads` superseded `Remove SynDNA reads`, which now removes SynDNA inserts and plasmids.
1240
* `update_resource_allocation_redis` now relies on using equations stored in the database vs. hardcoded; thank you @Gossty!
1341
* SPP: Updated prep-info file generation to identify and report filtered fastq files that could not be matched to a sample-id instead of silently ignoring them.
1442
* SPP: Removed legacy test code and example files for amplicon processing. Some other tests updated and repurposed.

qiita_core/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@
66
# The full license is in the file LICENSE, distributed with this software.
77
# -----------------------------------------------------------------------------
88

9-
__version__ = "2025.02"
9+
__version__ = "2025.07"

qiita_core/support_files/config_test.cfg

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,6 @@ REQUIRE_APPROVAL = True
2525
# Base URL: DO NOT ADD TRAILING SLASH
2626
BASE_URL = https://localhost:8383
2727

28-
# Download path files
29-
UPLOAD_DATA_DIR = /home/runner/work/qiita/qiita/qiita_db/support_files/test_data/uploads/
30-
3128
# Working directory path
3229
WORKING_DIR = /home/runner/work/qiita/qiita/qiita_db/support_files/test_data/working_dir/
3330

qiita_core/tests/test_configuration_manager.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -306,9 +306,6 @@ def test_get_portal_latlong(self):
306306
# Base URL: DO NOT ADD TRAILING SLASH
307307
BASE_URL = https://localhost
308308
309-
# Download path files
310-
UPLOAD_DATA_DIR = /tmp/
311-
312309
# Working directory path
313310
WORKING_DIR = /tmp/
314311

qiita_db/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
from . import user
2828
from . import processing_job
2929

30-
__version__ = "2025.02"
30+
__version__ = "2025.07"
3131

3232
__all__ = ["analysis", "artifact", "archive", "base", "commands",
3333
"environment_manager", "exceptions", "investigation", "logger",

qiita_db/environment_manager.py

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# -----------------------------------------------------------------------------
88

99
from os.path import abspath, dirname, join, exists, basename, splitext
10+
from shutil import copytree
1011
from functools import partial
1112
from os import mkdir
1213
import gzip
@@ -127,6 +128,36 @@ def _download_reference_files():
127128
_insert_processed_params(ref)
128129

129130

131+
def create_mountpoints():
132+
r"""In a fresh qiita setup, sub-directories under
133+
qiita_config.base_data_dir might not yet exist. To avoid failing in
134+
later steps, they are created here.
135+
"""
136+
with qdb.sql_connection.TRN:
137+
sql = """SELECT DISTINCT mountpoint FROM qiita.data_directory
138+
WHERE active = TRUE"""
139+
qdb.sql_connection.TRN.add(sql)
140+
created_subdirs = []
141+
for mountpoint in qdb.sql_connection.TRN.execute_fetchflatten():
142+
for (ddid, subdir) in qdb.util.get_mountpoint(mountpoint,
143+
retrieve_all=True):
144+
if not exists(join(qiita_config.base_data_dir, subdir)):
145+
if qiita_config.test_environment:
146+
# if in test mode, we want to potentially fill the
147+
# new directory with according test data
148+
copytree(get_support_file('test_data', mountpoint),
149+
join(qiita_config.base_data_dir, subdir))
150+
else:
151+
# in production mode, an empty directory is created
152+
mkdir(join(qiita_config.base_data_dir, subdir))
153+
created_subdirs.append(subdir)
154+
155+
if len(created_subdirs) > 0:
156+
print("Created %i sub-directories as 'mount points':\n%s"
157+
% (len(created_subdirs),
158+
''.join(map(lambda x: ' - %s\n' % x, created_subdirs))))
159+
160+
130161
def make_environment(load_ontologies, download_reference, add_demo_user):
131162
r"""Creates the new environment specified in the configuration
132163
@@ -397,6 +428,9 @@ def patch(patches_dir=PATCHES_DIR, verbose=False, test=False):
397428
with qdb.sql_connection.TRN:
398429
_populate_test_db()
399430

431+
# create mountpoints as subdirectories in BASE_DATA_DIR
432+
create_mountpoints()
433+
400434
patch_update_sql = "UPDATE settings SET current_patch = %s"
401435
for sql_patch_fp in sql_patch_files[next_patch_index:]:
402436
sql_patch_filename = basename(sql_patch_fp)

qiita_db/handlers/tests/oauthbase.py

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,10 @@
66
# The full license is in the file LICENSE, distributed with this software.
77
# -----------------------------------------------------------------------------
88

9-
from qiita_core.qiita_settings import r_client
9+
import requests
10+
import os
11+
import sys
12+
from qiita_core.qiita_settings import r_client, qiita_config
1013

1114
from qiita_pet.test.tornado_test_base import TestHandlerBase
1215

@@ -19,3 +22,49 @@ def setUp(self):
1922
r_client.hset(self.token, 'grant_type', 'client')
2023
r_client.expire(self.token, 20)
2124
super(OauthTestingBase, self).setUp()
25+
self._session = requests.Session()
26+
# should point to client certificat file:
27+
# /qiita/qiita_core/support_files/ci_rootca.crt
28+
self._verify = os.environ['QIITA_ROOTCA_CERT']
29+
self._fetch_token()
30+
31+
self._files_to_remove = []
32+
33+
def tearDown(self):
34+
for fp in self._files_to_remove:
35+
if os.path.exists(fp):
36+
os.remove(fp)
37+
38+
def _fetch_token(self):
39+
data = {
40+
'client_id': '4MOBzUBHBtUmwhaC258H7PS0rBBLyGQrVxGPgc9g305bvVhf6h',
41+
'client_secret':
42+
('rFb7jwAb3UmSUN57Bjlsi4DTl2owLwRpwCc0SggRN'
43+
'EVb2Ebae2p5Umnq20rNMhmqN'),
44+
'grant_type': 'client'}
45+
resp = self._session.post(
46+
"%s/qiita_db/authenticate/" % qiita_config.base_url,
47+
verify=self._verify, data=data, timeout=80)
48+
if resp.status_code != 200:
49+
raise ValueError("_fetchToken() POST request failed")
50+
self._token = resp.json()['access_token']
51+
print('obtained access_token = %s' % self._token, file=sys.stderr)
52+
53+
def post_authed(self, url, **kwargs):
54+
if 'headers' not in kwargs:
55+
kwargs['headers'] = {}
56+
if 'Authorization' not in kwargs['headers']:
57+
kwargs['headers']['Authorization'] = 'Bearer %s' % self._token
58+
59+
r = self._session.post(
60+
qiita_config.base_url + url, verify=self._verify, **kwargs)
61+
r.close()
62+
63+
return r
64+
65+
def get_authed(self, url):
66+
r = self._session.get(qiita_config.base_url + url, verify=self._verify,
67+
headers={'Authorization': 'Bearer %s' %
68+
self._token})
69+
r.close()
70+
return r

qiita_db/metadata_template/test/test_base_metadata_template.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,12 +125,13 @@ def test_restrictions(self):
125125
obs = MT.prep_template.PrepTemplate(1).restrictions
126126
exp = {
127127
'target_gene': ['16S rRNA', '18S rRNA', 'ITS1/2', 'LSU'],
128-
'platform': ['FASTA', 'Illumina', 'Ion_Torrent', 'LS454',
128+
'platform': ['DNBSEQ', 'FASTA', 'Illumina', 'Ion_Torrent', 'LS454',
129129
'Oxford Nanopore'],
130130
'target_subfragment': ['V3', 'V4', 'V6', 'V9', 'ITS1/2'],
131131
'instrument_model': [
132132
'454 GS', '454 GS 20', '454 GS FLX', '454 GS FLX+',
133-
'454 GS FLX Titanium', '454 GS Junior',
133+
'454 GS FLX Titanium', '454 GS Junior', 'DNBSEQ-G400',
134+
'DNBSEQ-T7', 'DNBSEQ-G800',
134135
'Illumina Genome Analyzer', 'Illumina Genome Analyzer II',
135136
'Illumina Genome Analyzer IIx', 'Illumina HiScanSQ',
136137
'Illumina HiSeq 1000', 'Illumina HiSeq 1500',

0 commit comments

Comments
 (0)