Skip to content

Commit

Permalink
Run pre-commit hooks on all files
Browse files Browse the repository at this point in the history
Also:

- Update pre-commit hooks via autoupdate
- Tweak config of various hooks
- Add pip-compile hook
  • Loading branch information
tchamberlin committed Jul 26, 2023
1 parent d2a0c0f commit c7bf1b4
Show file tree
Hide file tree
Showing 54 changed files with 2,412 additions and 1,846 deletions.
14 changes: 14 additions & 0 deletions .github/workflows/pre-commit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: pre-commit

on:
pull_request:
push:
branches: [release-0.1.0-precommit]

jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v3
- uses: pre-commit/action@v3.0.0
4 changes: 2 additions & 2 deletions .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
jobs:
build:
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
Expand All @@ -32,4 +32,4 @@ jobs:
pip install -e .
- name: Test with pytest
run: |
pytest
pytest
2 changes: 1 addition & 1 deletion .github/workflows/release-test.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
on:
testrelease:
types:
types:
- published

name: testrelease
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
on:
release:
types:
types:
- published

name: release
Expand Down
21 changes: 15 additions & 6 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
# See https://pre-commit.com for more information
default_language_version:
python: python3.8
python: python3.11

# See https://pre-commit.com/hooks.html for more hooks
repos:
Expand All @@ -10,10 +10,7 @@ repos:
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
exclude: LICENSE
- id: check-yaml
# Might make sense once everything is at Python 3
# - id: check-ast
- id: check-added-large-files
args:
- '--maxkb=1024'
Expand All @@ -27,12 +24,24 @@ repos:
hooks:
- id: isort
- repo: 'https://github.com/psf/black'
rev: 23.1.0
rev: 23.7.0
hooks:
- id: black
- repo: 'https://github.com/PyCQA/autoflake'
rev: v2.2.0
hooks:
- id: autoflake
- repo: 'https://github.com/PyCQA/docformatter'
rev: v1.7.5
hooks:
- id: docformatter
args:
- '--in-place'
- '--config=./pyproject.toml'
- repo: https://github.com/jazzband/pip-tools
rev: 6.12.3
rev: 7.1.0
hooks:
- id: pip-compile
files: pyproject.toml
args:
- '--resolver=backtracking'
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
## Getting Started
### Installation

Dysh requires Python 3.8+ and recent versions of [astropy]( https://astropy.org), [numpy](https://numpy.org), [scipy](https://scipy.org), [pandas](https://pandas.pydata.org), [specutils](https://specutils.readthedocs.io/en/stable/), and [matplotlib](https://matplotlib.org).
Dysh requires Python 3.8+ and recent versions of [astropy]( https://astropy.org), [numpy](https://numpy.org), [scipy](https://scipy.org), [pandas](https://pandas.pydata.org), [specutils](https://specutils.readthedocs.io/en/stable/), and [matplotlib](https://matplotlib.org).

#### With pip from PyPi
Dysh is most easily installed with *pip*, which will take care of any dependencies. The packaged code is hosted at the [Python Packaging Index](https://pypi.org/project/dysh).
Expand All @@ -15,7 +15,7 @@ Dysh is most easily installed with *pip*, which will take care of any dependenci
```

#### From github
To install from github without creating a separate virtual environment:
To install from github without creating a separate virtual environment:

```bash
$ git clone git@github.com:GreenBankObservatory/dysh.git
Expand Down Expand Up @@ -64,7 +64,7 @@ The virtual environment will be created the first time; subsequent invoking ``ha

```bash
(dysh) $ exit
$
$
```

4. Each time when you come back in this directory without being in this virtual environment, you'll need to load the virtual environment
Expand All @@ -80,4 +80,3 @@ The virtual environment will be created the first time; subsequent invoking ``ha
```bash
$ pytest
```

15 changes: 7 additions & 8 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Notes for benchmarking Dysh

The purpose of benchmarking is to test performance of
The purpose of benchmarking is to test performance of

- opening/loading an SDFITS file/hdu
- opening/loading an SDFITS file/hdu
- optionally creating an index of some sort (GBTIDL or pandas)
- creating spectra from each DATA row of the SDFITS table.
- remove baselines of order 1, 2, and 3 from each spectrum.

Timing is 'wall clock time', i.e., it includes and kernel/sleep operations. In python, we are using *time.perfcounter_ns()*

## Avoiding File caching
Expand All @@ -22,7 +22,7 @@ The OMP_NUM_THREADS environment variable sets the number of threads to use for
and
setenv OMP_NUM_THREADS 1

## SDFITS Files
## SDFITS Files

The standard set of SDFITS files to run the benchmark on are in /lma1/mpound/GBT/examples/ on LMA machines and /home/scratch/mpound/examples in GBO machines. The files are (in ascending order of size):

Expand All @@ -33,15 +33,15 @@ The standard set of SDFITS files to run the benchmark on are in /lma1/mpound/GBT
- onoff-L/data/TGBT21A_501_11.raw.vegas.fits
- mapping-L/data/TGBT17A_506_11.raw.vegas/
TGBT17A_506_11.raw.vegas.A.fits
- nod-KFPA/data/TGBT22A_503_02.raw.vegas/TGBT22A_503_02.raw.vegas.F.fits
- nod-KFPA/data/TGBT22A_503_02.raw.vegas/TGBT22A_503_02.raw.vegas.F.fits
- mixed-fs-ps/data/AGBT16B_225_05/AGBT16B_225_05.raw.vegas/AGBT16B_225_05.raw.vegas.B.fits

## Code details
1. Benchmark code is in the subdirectory *benchmark*
2. The python benchmark code is *revisedstructure.py* and is invoked with *benchmark_py.csh*
- To run it, you have to have dysh installed. This is done with hatch and I haven't written up the notes for that yet!
- It has various options to turn on/off pieces of the benchmark, see *revisedstructure.py -h*

## Output
Output should be into a (IPAC-formatted) table with the following columns:

Expand All @@ -50,7 +50,7 @@ Output should be into a (IPAC-formatted) table with the following columns:
- N_hdu - number of HDU in the file
- HDU - number of the HDU for which this row contains the benchmark
- N_rows - number of rows in this HDU
- N_chan - number of chan per spectum in this HDU
- N_chan - number of chan per spectum in this HDU
- Load - time to load/read the SDFITs file, in ms . e.g. astropy.io.fits.open()
- Index - time to create index (e.g. pandas, GBTIDL) in ms
- Create_Obslocks - time to create Spectrum object for every row of HDU, in ms
Expand All @@ -62,4 +62,3 @@ The python script *makeplots.py* will make some standard plots given the IPAC f

./makeplots.py -f tab_omp=1.out -t "Timing for Load/Obsblocks/Baseline (OMP=1)" -o t1.png
./makeplots.py -f tab_omp_unset.out -t "Timing for Load/Obsblocks/Baseline (OMP unset)" -o t2.png

2 changes: 1 addition & 1 deletion benchmark/benchmark_py.csh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#set path = ( . $path )
#
# BE sure to echo 1 > /proc/sys/vm/drop_caches before running this.
unsetenv OMP_NUM_THREADS
unsetenv OMP_NUM_THREADS
set root = "/data/gbt/examples/"
#set root = "/lma1/mpound/GBT/examples/"
#set root = "/home/scratch/mpound/examples/"
Expand Down
Loading

0 comments on commit c7bf1b4

Please sign in to comment.