Merge pull request espnet#4372 from kamo-naoyuki/isort

Add isort checking to the CI tests

CONTRIBUTING.md

@@ -64,21 +64,21 @@ To port models from zenodo using Hugging Face hub,
 1. Create a Hugging Face account - https://huggingface.co/
 2. Request to be added to espnet organisation - https://huggingface.co/espnet
 3. Go to `egs2/RECIPE/*/scripts/utils` and run `./upload_models_to_hub.sh "ZENODO_MODEL_NAME"`
-   
+
 To upload models using Huggingface-cli follow the following steps:
 You can also refer to https://huggingface.co/docs/transformers/model_sharing
 1. Create a Hugging Face account - https://huggingface.co/
-2. Request to be added to espnet organisation - https://huggingface.co/espnet 
-3. Run huggingface-cli login (You can get the token request at this step under setting > Access Tokens > espnet token  
+2. Request to be added to espnet organisation - https://huggingface.co/espnet
+3. Run huggingface-cli login (You can get the token request at this step under setting > Access Tokens > espnet token
 4. `huggingface-cli repo create your-model-name --organization espnet`
 5. `git clone https://huggingface.co/username/your-model-name` (clone this outside ESPNet to avoid issues as this a git repo)
 6. `cd your-model-name`
 7. `git lfs install`
-8. copy contents from exp diretory of your recipe into this directory (Check other models of similar task under ESPNet to confirm your directory structure) 
+8. copy contents from exp diretory of your recipe into this directory (Check other models of similar task under ESPNet to confirm your directory structure)
 9. `git add . `
 10. `git commit -m "Add model files"`
 11. `git push`
-12. Check if the inference demo on HF is running successfully to verify the upload      
+12. Check if the inference demo on HF is running successfully to verify the upload
 
 #### 1.3.3 Additional requirements for new recipe
 
@@ -91,12 +91,13 @@ to its differences.
 - If a recipe for a new corpus is proposed, you should add its name and information to:
 https://github.com/espnet/espnet/blob/master/egs/README.md if it's a ESPnet1 recipe,
 or https://github.com/espnet/espnet/blob/master/egs2/README.md + `db.sh` if it's a ESPnet2 recipe.
-   
+
 #### 1.3.4 Checklist before you submit the recipe-based PR
 
 - [ ] be careful about the name for the recipe. It is recommended to follow naming conventions of the other recipes
 - [ ] common/shared files are linked with **soft link** (see Section 1.3.3)
-- [ ] modified or new python scripts should be passed through **latest** black formating (by using python package black). The command to be executed could be `black espnet espnet2 test utils setup.py egs*/*/*/local egs2/TEMPLATE/asr1/pyscripts`
+- [ ] modified or new python scripts should be passed through **latest** black formating (by using python package black). The command to be executed could be `black espnet espnet2 test utils setup.py egs*/*/*/local egs2/TEMPLATE/*/pyscripts tools/*.py ci/*.py`
+- [ ] modified or new python scripts should be passed through **latest** isort formating (by using python package isort). The command to be executed could be `isort espnet espnet2 test utils setup.py egs*/*/*/local egs2/TEMPLATE/*/pyscripts tools/*.py ci/*.py`
 - [ ] cluster settings should be set as **default** (e.g., cmd.sh conf/slurm.conf conf/queue.conf conf/pbs.conf)
 - [ ] update `egs/README.md` or `egs2/README.md` with corresponding recipes
 - [ ] add corresponding entry in `egs2/TEMPLATE/db.sh` for a new corpus
@@ -135,7 +136,7 @@ $ pip install -e ".[test]"
 
 ### 4.1 Python
 
-Then you can run the entire test suite using [flake8](http://flake8.pycqa.org/en/latest/), [autopep8](https://github.com/hhatto/autopep8), [black](https://github.com/psf/black) and [pytest](https://docs.pytest.org/en/latest/) with [coverage](https://pytest-cov.readthedocs.io/en/latest/reporting.html) by
+Then you can run the entire test suite using [flake8](http://flake8.pycqa.org/en/latest/), [autopep8](https://github.com/hhatto/autopep8), [black](https://github.com/psf/black), [isort](https://github.com/PyCQA/isort) and [pytest](https://docs.pytest.org/en/latest/) with [coverage](https://pytest-cov.readthedocs.io/en/latest/reporting.html) by
 ``` console
 ./ci/test_python.sh
 ```

README.md

@@ -18,7 +18,8 @@
 [![GitHub license](https://img.shields.io/github/license/espnet/espnet.svg)](https://github.com/espnet/espnet)
 [![codecov](https://codecov.io/gh/espnet/espnet/branch/master/graph/badge.svg)](https://codecov.io/gh/espnet/espnet)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![Mergify Status](https://img.shields.io/endpoint.svg?url=https://gh.mergify.io/badges/espnet/espnet&style=flat)](https://mergify.io)
+[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![Mergify Status](https://img.shields.io/endpoint.svg?url=https://api.mergify.com/v1/badges/espnet/espnet&style=flat)](https://mergify.com)
 [![Gitter](https://badges.gitter.im/espnet-en/community.svg)](https://gitter.im/espnet-en/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 
 [**Docs**](https://espnet.github.io/espnet/)
@@ -78,11 +79,11 @@ ESPnet uses [pytorch](http://pytorch.org/) as a deep learning engine and also fo
 - Self-supervised learning representations as features, using upstream models in [S3PRL](https://github.com/s3prl/s3prl) in frontend.
   - Set `frontend` to be `s3prl`
   - Select any upstream model by setting the `frontend_conf` to the corresponding name.
-- Transfer Learning : 
+- Transfer Learning :
   - easy usage and transfers from models previously trained by your group, or models from [ESPnet huggingface repository](https://huggingface.co/espnet).
   - [Documentation](https://github.com/espnet/espnet/tree/master/egs2/mini_an4/asr1/transfer_learning.md) and [toy example runnable on colab](https://github.com/espnet/notebook/blob/master/espnet2_asr_transfer_learning_demo.ipynb).
 - Streaming Transformer/Conformer ASR with blockwise synchronous beam search.
-- Restricted Self-Attention based on [Longformer](https://arxiv.org/abs/2004.05150) as an encoder for long sequences 
+- Restricted Self-Attention based on [Longformer](https://arxiv.org/abs/2004.05150) as an encoder for long sequences
 
 Demonstration
 - Real-time ASR demo with ESPnet2  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_asr_realtime_demo.ipynb)
@@ -579,10 +580,10 @@ We list the performance on various SLU tasks and dataset using the metric report
 | Dialogue  Act Classification                                                 |     Switchboard     |       Acc       |       67.5       |                [link](https://github.com/espnet/espnet/tree/master/egs2/swbd_da/asr1/README.md)                |
 | Dialogue  Act Classification                                                 |     Jdcinal (Jp)    |       Acc       |       67.4       |                [link](https://github.com/espnet/espnet/tree/master/egs2/jdcinal/asr1/README.md)                |
 | Emotion Recognition                                                  |     IEMOCAP     |       Acc       |       69.4       |                [link](https://github.com/espnet/espnet/tree/master/egs2/iemocap/asr1/README.md)                |
-| Emotion Recognition                                                  |     swbd_sentiment     |       Macro F1       |       61.4       |                [link](https://github.com/espnet/espnet/tree/master/egs2/swbd_sentiment/asr1/README.md)                | 
-| Emotion Recognition                                                  |     slue_voxceleb     |       Macro F1       |       44.0       |                [link](https://github.com/espnet/espnet/tree/master/egs2/slue-voxceleb/asr1/README.md)                | 
+| Emotion Recognition                                                  |     swbd_sentiment     |       Macro F1       |       61.4       |                [link](https://github.com/espnet/espnet/tree/master/egs2/swbd_sentiment/asr1/README.md)                |
+| Emotion Recognition                                                  |     slue_voxceleb     |       Macro F1       |       44.0       |                [link](https://github.com/espnet/espnet/tree/master/egs2/slue-voxceleb/asr1/README.md)                |
+
 
-
 If you want to check the results of the other recipes, please check `egs2/<name_of_recipe>/asr1/RESULTS.md`.
 
 
@@ -737,7 +738,7 @@ See the module documentation for more information.
 It is recommended to use models with RNN-based encoders (such as BLSTMP) for aligning large audio files;
 rather than using Transformer models that have a high memory consumption on longer audio data.
 The sample rate of the audio must be consistent with that of the data used in training; adjust with `sox` if needed.
-  
+
 Also, we can use this tool to provide token-level segmentation information if we prepare a list of tokens instead of that of utterances in the `text` file. See the discussion in https://github.com/espnet/espnet/issues/4278#issuecomment-1100756463.
 
 </div></details>

ci/test_python.sh

@@ -5,11 +5,16 @@
 
 set -euo pipefail
 
-modules="espnet espnet2 test utils setup.py egs*/*/*/local egs2/TEMPLATE/asr1/pyscripts"
+modules="espnet espnet2 test utils setup.py egs*/*/*/local egs2/TEMPLATE/*/pyscripts tools/*.py ci/*.py"
 
 # black
 if ! black --check ${modules}; then
-    printf 'Please apply:\n    $ black %s\n' "${modules}"
+    printf '[INFO] Please apply black:\n    $ black %s\n' "${modules}"
+    exit 1
+fi
+# isort
+if ! isort -c -v ${modules}; then
+    printf '[INFO] Please apply isort:\n    $ isort %s\n' "${modules}"
     exit 1
 fi
 

egs/arctic/tts1/local/clean_text.py

@@ -8,7 +8,6 @@
 
 from tacotron_cleaner.cleaners import custom_english_cleaners
 
-
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("text", type=str, help="text to be cleaned")

egs/chime6/asr1/local/extract_noises.py

@@ -6,11 +6,12 @@
 import argparse
 import json
 import math
-import numpy as np
 import os
-import scipy.io.wavfile as siw
 import sys
 
+import numpy as np
+import scipy.io.wavfile as siw
+
 
 def get_args():
     parser = argparse.ArgumentParser(

egs/chime6/asr1/local/make_noise_list.py

@@ -7,7 +7,6 @@
 import os
 import sys
 
-
 if len(sys.argv) != 2:
     print("Usage: {} <noises-dir>".format(sys.argv[0]))
     raise SystemExit(1)

egs/cmu_indic/tts1/local/clean_text.py

@@ -8,7 +8,6 @@
 
 from tacotron_cleaner.cleaners import custom_english_cleaners
 
-
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("text", type=str, help="text to be cleaned")

egs/covost2/st1/local/process_tsv.py

@@ -5,8 +5,8 @@
 
 import argparse
 import codecs
-import pandas as pd
 
+import pandas as pd
 
 parser = argparse.ArgumentParser(description="extract translation from tsv file")
 parser.add_argument("tsv_path", type=str, default=None, help="input tsv path")

egs/csj/asr1/local/csj_rm_tag.py

@@ -6,9 +6,8 @@
 
 import argparse
 import codecs
-from io import open
 import sys
-
+from io import open
 
 PY2 = sys.version_info[0] == 2
 sys.stdin = codecs.getreader("utf-8")(sys.stdin if PY2 else sys.stdin.buffer)

egs/iwslt16/mt1/local/extract_recog_text.py

@@ -4,9 +4,9 @@
 """
 import argparse
 import glob
-from itertools import takewhile
 import json
 import os
+from itertools import takewhile
 
 
 def get_args():

egs/iwslt16/mt1/local/generate_json.py

@@ -5,11 +5,9 @@
 """
 import argparse
 import json
-from logging import getLogger
 import os
-from typing import Dict
-from typing import List
-
+from logging import getLogger
+from typing import Dict, List
 
 logger = getLogger(__name__)
 

egs/iwslt16/mt1/local/generate_vocab.py

@@ -6,8 +6,8 @@
 format: token + whitespace + index
 """
 import argparse
-from collections import defaultdict
 import fileinput
+from collections import defaultdict
 
 
 def get_args():

egs/iwslt18/st1/local/parse_xml.py

@@ -6,10 +6,10 @@
 
 import argparse
 import codecs
-from collections import OrderedDict
 import os
 import re
 import xml.etree.ElementTree as etree
+from collections import OrderedDict
 
 
 def main():

egs/iwslt21/asr1/local/filter_parentheses.py

@@ -7,6 +7,7 @@
 import argparse
 import codecs
 import re
+
 import regex
 
 parser = argparse.ArgumentParser()

egs/iwslt21_low_resource/st1/local/data_prep.py

@@ -1,7 +1,6 @@
 import argparse
 import os
 
-
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Convert data into kaldi format")
     parser.add_argument("data_dir", type=str)

egs/jnas/asr1/local/filter_text.py

@@ -6,9 +6,8 @@
 
 import argparse
 import codecs
-from io import open
 import sys
-
+from io import open
 
 PY2 = sys.version_info[0] == 2
 sys.stdin = codecs.getreader("utf-8")(sys.stdin if PY2 else sys.stdin.buffer)

egs/ksponspeech/asr1/local/get_space_normalized_hyps.py

@@ -4,11 +4,11 @@
 # Copyright 2020 Electronics and Telecommunications Research Institute (Jeong-Uk, Bang)
 #  Apache 2.0  (http://www.apache.org/licenses/LICENSE-2.0)
 
-import configargparse
 import logging
 import os
 import sys
 
+import configargparse
 from numpy import zeros
 
 space_sym = "▁"

egs/ksponspeech/asr1/local/get_transcriptions.py

@@ -5,13 +5,14 @@
 #  Apache 2.0  (http://www.apache.org/licenses/LICENSE-2.0)
 
 import codecs
-import configargparse
 import logging
 import os
 import re
 import shutil
 import sys
 
+import configargparse
+
 
 def get_parser():
     """Get default arguments."""

egs/libri_css/asr1/local/best_wer_matching.py

@@ -5,9 +5,10 @@
 import io
 import itertools
 import math
+import sys
+
 import numpy as np
 from scipy.optimize import linear_sum_assignment
-import sys
 
 
 # Helper function to group the list by ref/hyp ids

egs/libri_css/asr1/local/get_perspeaker_output.py

@@ -5,9 +5,9 @@
   into per_speaker output (text) file"""
 
 import argparse
-from collections import defaultdict
 import itertools
 import os
+from collections import defaultdict
 
 
 def get_args():

egs/libri_css/asr1/local/prepare_data.py

@@ -7,6 +7,7 @@
 import argparse
 import glob
 import os
+
 import soundfile as sf
 import tqdm
 

egs/libri_css/asr1/local/segmentation/apply_webrtcvad.py

@@ -12,6 +12,7 @@
 import os
 import sys
 import wave
+
 import webrtcvad
 
 

egs/ljspeech/tts1/local/clean_text.py

@@ -5,8 +5,8 @@
 
 import argparse
 import codecs
-import nltk
 
+import nltk
 from tacotron_cleaner.cleaners import custom_english_cleaners
 
 try:

egs/lrs/avsr1/local/se_batch.py

@@ -5,11 +5,12 @@
 License, v. 2.0. If a copy of the MPL was not distributed with this
 file, You can obtain one at http://mozilla.org/MPL/2.0/."""
 
-from deepxi.utils import read_wav
 import glob
-import numpy as np
 import os
 
+import numpy as np
+from deepxi.utils import read_wav
+
 
 def Batch(fdir, snr_l=[]):
     """REQUIRES REWRITING. WILL BE MOVED TO deepxi/utils.py

egs/mgb2/asr1/local/process_xml.py

@@ -1,9 +1,10 @@
 #!/usr/bin/env python3
 
 import argparse
-from bs4 import BeautifulSoup
 import sys
 
+from bs4 import BeautifulSoup
+
 
 def get_args():
     parser = argparse.ArgumentParser(description="""This script process xml file.""")

egs/mgb2/asr1/local/text_segmenting.py

@@ -4,6 +4,7 @@
 # Apache 2.0  (http://www.apache.org/licenses/LICENSE-2.0)
 
 import argparse
+
 import pandas as pd
 
 

egs/polyphone_swiss_french/asr1/local/data_prep.py

@@ -1,11 +1,11 @@
 #!/usr/bin/env python3
-from collections import defaultdict
 import os
 import pathlib
-from random import shuffle
 import re
 import subprocess
 import sys
+from collections import defaultdict
+from random import shuffle
 
 
 class FrPolyphonePrepper:
@@ -570,6 +570,7 @@ def _generate_random(self, corpus, splits):
 
 if __name__ == "__main__":
     import argparse
+
     import yaml
 
     example = "{0} --config conf/dataprep.yml".format(sys.argv[0])

egs/puebla_nahuatl/asr1/local/construct_dataset.py

@@ -1,5 +1,4 @@
 import os
-
 from argparse import ArgumentParser