Skip to content

Commit 8f37b8a

Browse files
antgonzaElDeveloper
antgonza
and
ElDeveloper
authored
2021.03 (#3086)
* adding creation and modification timestamps to prep_templates (#3066) * adding creation and modification timestamps to prep_templates * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * fix #3068 (#3069) * minor improvements (#3067) * minor improvements * fix test * adding @wasade suggestion * minor fixes (#3073) * fix #3072 (#3074) * fix #3072 * flake8 * fix #3076 (#3077) * fix #3076 * addressing @ElDeveloper comments * fix #3070 (#3075) * fix #3070 * Apply suggestions from code review Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * flake8 Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * fix-3064 (#3080) * Create qiita-ci.yml * add passwd to psql * nginx + redis * redis via conda * redis-server - -> redis-server -- * daemonize -> daemonize * conda libevent * apt-get libevent-dev * adding multiple sections * rm travis_retry * add missing activagte * onfig_test.cfg * rm indent * add 1 indent * pgport * add drop * change db name * redis version * redis -> redbiom version * split tests * source .profile * adding ssh command * mkdir ~/.ssh/ * only ssh * pushing to retrigger * add run * split ssh conf/run * 60367309 * add bash * fix ident * rm ssh cmd * strict=no * localhost * mod permissions * user ssh * rm keys * echos * start ssh service * ssh -vvv * split setup/test * localhost -> 127 * localhost -> 127 * rm ssh setup * -o StrictHostKeyChecking=no * change key creation * echo sshd_config * readd ssh config * cp sshd_config and then move * rm echos * extra qiita folder * ls test_data * rm qiita folder * use key in .ssh * more ssh params * tee -> > * chmod * fullpath * ls key * rm localhost * params * rsa -> rsa-sha2-256 * rm root login * readd -o StrictHostKeyChecking=no * minimizing params * permissions * cleaning up * comments => echo * -o StrictHostKeyChecking=no * export REDBIOM_HOST=http://127.0.0.1:7379 * localhost -> 127.0.0.1 * add runner * cover package * cleaning env vars * renaming jobs * redbiom host * add env vars via export * changing echo * add env COVER_PACKAGE * rm user sftp * just qiita_db * open_sftp -> SFTPClient * +1 worker * only SCPClient * ssh.SCPClient -> SCPClient * return sftp/ssh back * just qiita_db * rm sftp * add all tests * BASE_URL = https://127.0.0.1:8383 * bash -c * start_biom * start_biom 127 * populate QIITA_SERVER_CERT * nginx 127 * adding ports to conf * lowering ports * service qiita * 8383 postgres * curl test * mkdir nginx * moving tests around * moving curl up * echos for curl * sleep 10 * 127.0.0.1 -> localhost * returning some localhost to 127... * qiita_pet! * final clean up * kill qiita first * addreesing @ElDeveloper comments * checkout repo for flake8 * split cover_package * Workflows GUI (#3078) * fix #3076 * addressing @ElDeveloper comments * adding workflows GUI * improve HTML * db changes and improvements * comment out new db changes * rm default_workflows from PluginHandler.get * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * fix #3081 (#3082) * fix #3081 * generate_analysis_list_per_study to speed up retrival * minor modifications * split sql * addressing @ElDeveloper comments * add Command.processing_jobs (#3085) * add Command.processing_jobs * Update qiita_db/software.py [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * Improve workflows (#3083) * fix #3076 * addressing @ElDeveloper comments * adding workflows GUI * improve HTML * db changes and improvements * comment out new db changes * rm default_workflows from PluginHandler.get * improving workflows after deployment * fix error * clean up * minor fixes * improving html * comments from @justinshaffer * addressing @ElDeveloper comments * 2021.03 (#3084) * 2021.03 * add Command.processing_jobs to changelog * codecov * codecov required * codecov token Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>
1 parent e6e5d80 commit 8f37b8a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+2997
-1952
lines changed

‎.github/workflows/qiita-ci.yml

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# name: Qiita CI
2+
3+
on:
4+
push:
5+
branches: [ dev ]
6+
pull_request:
7+
8+
jobs:
9+
# derived from https://github.com/actions/example-services/blob/master/.github/workflows/postgres-service.yml
10+
main:
11+
runs-on: ubuntu-latest
12+
13+
strategy:
14+
matrix:
15+
include:
16+
- cover_package: "qiita_db"
17+
- cover_package: "qiita_pet qiita_core qiita_ware"
18+
19+
services:
20+
postgres:
21+
# Docker Hub image
22+
image: postgres:9.5
23+
env:
24+
POSTGRES_DB: postgres
25+
POSTGRES_USER: postgres
26+
POSTGRES_PASSWORD: postgres
27+
COVER_PACKAGE: ${{ matrix.cover_package }}
28+
29+
# Set health checks to wait until postgres has started
30+
options: >-
31+
--health-cmd pg_isready
32+
--health-interval 10s
33+
--health-timeout 5s
34+
--health-retries 5
35+
ports:
36+
# based on https://github.com/actions/example-services/blob/master/.github/workflows/postgres-service.yml#L44-L72
37+
- 5432/tcp
38+
39+
steps:
40+
# Downloads a copy of the code in your repository before running CI tests
41+
- name: Check out repository code
42+
uses: actions/checkout@v2
43+
44+
- name: Setup for conda
45+
uses: conda-incubator/setup-miniconda@v2
46+
with:
47+
auto-update-conda: true
48+
python-version: 3.6
49+
50+
- name: Basic dependencies install
51+
env:
52+
COVER_PACKAGE: ${{ matrix.cover_package }}
53+
shell: bash -l {0}
54+
run: |
55+
echo "Testing: " $COVER_PACKAGE
56+
57+
# pull out the port so we can modify the configuration file easily
58+
pgport=${{ job.services.postgres.ports[5432] }}
59+
sed -i "s/PORT = 5432/PORT = $pgport/" qiita_core/support_files/config_test.cfg
60+
61+
# PGPASSWORD is read by pg_restore, which is called by the build_db process.
62+
export PGPASSWORD=postgres
63+
64+
# Setting up main qiita conda environment
65+
conda config --add channels conda-forge
66+
conda create -q --yes -n qiita python=3.6 pip libgfortran numpy nginx cython redis
67+
conda activate qiita
68+
pip install sphinx sphinx-bootstrap-theme nose-timer codecov Click
69+
70+
# Configuring SSH
71+
cp /etc/ssh/sshd_config sshd_config
72+
echo "RSAAuthentication yes" > sshd_config
73+
echo "PubkeyAuthentication yes" > sshd_config
74+
echo "StrictModes no" > sshd_config
75+
sudo mv sshd_config /etc/ssh/sshd_config
76+
sudo systemctl restart ssh
77+
78+
- name: Webdis install
79+
shell: bash -l {0}
80+
run: |
81+
sudo apt-get -y install libevent-dev
82+
git clone https://github.com/nicolasff/webdis
83+
cd webdis
84+
make
85+
86+
- name: Main install
87+
shell: bash -l {0}
88+
run: |
89+
conda activate qiita
90+
export QIITA_SERVER_CERT=`pwd`/qiita_core/support_files/server.crt
91+
export QIITA_CONFIG_FP=`pwd`/qiita_core/support_files/config_test.cfg
92+
export REDBIOM_HOST="http://localhost:7379"
93+
pip install . --no-binary redbiom
94+
pwd
95+
mkdir ~/.qiita_plugins
96+
97+
- name: Install plugins
98+
shell: bash -l {0}
99+
run: |
100+
wget https://data.qiime2.org/distro/core/qiime2-2019.4-py36-linux-conda.yml
101+
conda env create -q -n qtp-biom --file qiime2-2019.4-py36-linux-conda.yml
102+
rm qiime2-2019.4-py36-linux-conda.yml
103+
export QIITA_SERVER_CERT=`pwd`/qiita_core/support_files/server.crt
104+
export QIITA_CONFIG_FP=`pwd`/qiita_core/support_files/config_test.cfg
105+
export REDBIOM_HOST="http://localhost:7379"
106+
conda activate qtp-biom
107+
pip install -U pip
108+
pip install https://github.com/qiita-spots/qiita_client/archive/master.zip
109+
pip install https://github.com/qiita-spots/qtp-biom/archive/master.zip
110+
configure_biom --env-script "source /home/runner/.profile; conda activate qtp-biom" --server-cert $QIITA_SERVER_CERT
111+
112+
- name: Starting services
113+
shell: bash -l {0}
114+
run: |
115+
conda activate qiita
116+
export QIITA_SERVER_CERT=`pwd`/qiita_core/support_files/server.crt
117+
export QIITA_CONFIG_FP=`pwd`/qiita_core/support_files/config_test.cfg
118+
export REDBIOM_HOST="http://localhost:7379"
119+
120+
echo "1. Setting up redis"
121+
redis-server --daemonize yes --port 7777
122+
redis-server --daemonize yes --port 6379
123+
124+
echo "2. Starting webdis"
125+
pushd webdis
126+
./webdis &
127+
popd
128+
129+
echo "3. Starting redbiom: " $REDBIOM_HOST
130+
curl -s http://localhost:7379/FLUSHALL > /dev/null
131+
redbiom --version
132+
redbiom admin scripts-writable
133+
redbiom admin create-context --name "qiita-test" --description "qiita-test context"
134+
redbiom admin load-sample-metadata --metadata `pwd`/qiita_db/support_files/test_data/templates/1_19700101-000000.txt
135+
redbiom admin load-sample-metadata-search --metadata `pwd`/qiita_db/support_files/test_data/templates/1_19700101-000000.txt
136+
redbiom admin load-sample-data --table `pwd`/qiita_db/support_files/test_data/processed_data/1_study_1001_closed_reference_otu_table.biom --context qiita-test --tag 4
137+
redbiom admin load-sample-data --table `pwd`/qiita_db/support_files/test_data/processed_data/1_study_1001_closed_reference_otu_table-for_redbiom_tests.biom --context qiita-test --tag 5
138+
139+
echo "4. Setting up nginx"
140+
mkdir -p /usr/share/miniconda/envs/qiita/var/run/nginx/
141+
nginx -c ${PWD}/qiita_pet/nginx_example.conf
142+
143+
echo "5. Setting up qiita"
144+
conda activate qiita
145+
qiita-env make --no-load-ontologies
146+
qiita-test-install
147+
qiita plugins update
148+
149+
echo "6. Starting supervisord => multiple qiita instances"
150+
supervisord -c ${PWD}/qiita_pet/supervisor_example.conf
151+
sleep 10
152+
cat /tmp/supervisord.log
153+
154+
echo "7. Starting plugins"
155+
conda deactivate
156+
conda activate qtp-biom
157+
export QIITA_CLIENT_DEBUG_LEVEL=DEBUG
158+
start_biom https://localhost:8383 register ignored
159+
conda deactivate
160+
161+
echo "8. Setting up SSH"
162+
ssh-keygen -t rsa-sha2-256 -b 4096 -N '' -f $PWD/qiita_ware/test/test_data/test_key
163+
mkdir ~/.ssh/
164+
cp $PWD/qiita_ware/test/test_data/test_key* ~/.ssh/
165+
cat ~/.ssh/test_key.pub > ~/.ssh/authorized_keys
166+
chmod 600 $PWD/qiita_ware/test/test_data/test_key*
167+
chmod 600 ~/.ssh/*
168+
chmod 700 ~/.ssh/
169+
scp -o StrictHostKeyChecking=no -i $PWD/qiita_ware/test/test_data/test_key $USER@localhost:$PWD/qiita_ware/test/test_data/random_key $PWD/qiita_ware/test/test_data/random_key_copy
170+
171+
- name: Main tests
172+
shell: bash -l {0}
173+
env:
174+
COVER_PACKAGE: ${{ matrix.cover_package }}
175+
run: |
176+
conda activate qiita
177+
export QIITA_SERVER_CERT=`pwd`/qiita_core/support_files/server.crt
178+
export QIITA_CONFIG_FP=`pwd`/qiita_core/support_files/config_test.cfg
179+
export REDBIOM_HOST="http://localhost:7379"
180+
nosetests $COVER_PACKAGE --with-doctest --with-coverage --with-timer -v --cover-package=$COVER_PACKAGE -e 'test_submit_EBI_parse_EBI_reply_failure' -e 'test_full_submission'
181+
182+
# killing the qiita server to run the next commands
183+
QIITA_PID=`cat /tmp/supervisord.pid`
184+
kill $QIITA_PID
185+
sleep 10
186+
if [[ "$COVER_PACKAGE" == *"qiita_db"* ]]; then test_data_studies/commands.sh; all-qiita-cron-job; fi
187+
188+
- uses: codecov/codecov-action@v1
189+
with:
190+
token: ${{ secrets.CODECOV_TOKEN }}
191+
file: codecov.yml
192+
193+
lint:
194+
runs-on: ubuntu-latest
195+
steps:
196+
- name: flake8
197+
uses: actions/setup-python@v2
198+
with:
199+
python-version: 3.6
200+
- name: install dependencies
201+
run: python -m pip install --upgrade pip
202+
- name: Check out repository code
203+
uses: actions/checkout@v2
204+
- name: lint
205+
run: |
206+
pip install -q flake8
207+
flake8 qiita_* setup.py scripts/qiita*

‎.travis.yml

Lines changed: 0 additions & 108 deletions
This file was deleted.

‎CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,19 @@
11
# Qiita changelog
22

33

4+
Version 2021.03
5+
---------------
6+
7+
* Fixed [issue](https://github.com/qiita-spots/qtp-target-gene/issues/32) that left behind non gz per sample FASTQ files.
8+
* [Recommended Workflows](https://qiita-rc.ucsd.edu/workflows/) are now stored in the database.
9+
* Added a new button only for owners and admins within the Study page to display a list of all the Analyses generated with that study; helpful to clean up Studies and for general information.
10+
* The Qiita CI now runs as [GitHub Actions](https://github.com/qiita-spots/qiita/actions); moving away from Travis CI.
11+
* Prep information file object now stores its creation and modification timestamps.
12+
* Improved creation time for all information files via the to_dataframe() method.
13+
* Split the "other" category of the storage stats plot (https://qiita.ucsd.edu/stats/) into "other" and "biom" so biom can be its own category.
14+
* Added a processing_jobs property to qiita_db.software.Command to easily retrieve all jobs in the system that have ran the given command.
15+
* Fixed the following issues: [#3068](https://github.com/qiita-spots/qiita/issues/3068), [#3072](https://github.com/qiita-spots/qiita/issues/3072), [#3076](https://github.com/qiita-spots/qiita/issues/3076), and [#3070](https://github.com/qiita-spots/qiita/issues/3070).
16+
417
Version 2021.01
518
---------------
619

‎README.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ future.
7171
existing studies by submitting a fix proposals to the authors of the study.
7272

7373

74-
.. |Build Status| image:: https://travis-ci.org/qiita-spots/qiita.png?branch=master
75-
:target: https://travis-ci.org/qiita-spots/qiita
74+
.. |Build Status| image:: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml/badge.svg
75+
:target: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml
7676
.. |Coverage Status| image:: https://codecov.io/gh/qiita-spots/qiita/branch/master/graph/badge.svg
7777
:target: https://codecov.io/gh/qiita-spots/qiita

‎codecov.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,6 @@
11
ignore:
2-
- "qiita_db/support_files/**/*"
2+
- "qiita_db/support_files/**/*"
3+
4+
codecov:
5+
max_report_age: off
6+
require_ci_to_pass: yes

‎qiita_core/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@
66
# The full license is in the file LICENSE, distributed with this software.
77
# -----------------------------------------------------------------------------
88

9-
__version__ = "2021.01"
9+
__version__ = "2021.03"

‎qiita_core/support_files/config_test.cfg

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,16 +26,16 @@ REQUIRE_APPROVAL = True
2626
BASE_URL = https://localhost:8383
2727

2828
# Download path files
29-
UPLOAD_DATA_DIR = /home/travis/miniconda3/envs/qiita/lib/python3.6/site-packages/qiita_db/support_files/test_data/uploads/
29+
UPLOAD_DATA_DIR = /home/runner/work/qiita/qiita/qiita_db/support_files/test_data/uploads/
3030

3131
# Working directory path
32-
WORKING_DIR = /home/travis/miniconda3/envs/qiita/lib/python3.6/site-packages/qiita_db/support_files/test_data/working_dir/
32+
WORKING_DIR = /home/runner/work/qiita/qiita/qiita_db/support_files/test_data/working_dir/
3333

3434
# Maximum upload size (in Gb)
3535
MAX_UPLOAD_SIZE = 100
3636

3737
# Path to the base directory where the data files are going to be stored
38-
BASE_DATA_DIR = /home/travis/miniconda3/envs/qiita/lib/python3.6/site-packages/qiita_db/support_files/test_data/
38+
BASE_DATA_DIR = /home/runner/work/qiita/qiita/qiita_db/support_files/test_data/
3939

4040
# Valid upload extension, comma separated. Empty for no uploads
4141
VALID_UPLOAD_EXTENSION = fastq,fastq.gz,txt,tsv,sff,fna,qual
@@ -118,10 +118,10 @@ HOST = localhost
118118
PORT = 5432
119119

120120
# The password to use to connect to the database
121-
PASSWORD =
121+
PASSWORD = postgres
122122

123123
# The postgres password for the admin_user
124-
ADMIN_PASSWORD =
124+
ADMIN_PASSWORD = postgres
125125

126126
# ----------------------------- Torque settings -----------------------------
127127
[torque]

0 commit comments

Comments
 (0)