Skip to content

Commit a5343b1

Browse files
committed
update Readme.md
1 parent 453e1e1 commit a5343b1

File tree

1 file changed

+13
-11
lines changed

1 file changed

+13
-11
lines changed

Readme.md

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
# batch tool
2+
python script tool for parallel processing of files
3+
24
```
35
usage: pybatch.py [-h] [-p [PARAMETER]] [-v] [-r] -s SCRIPT [-m [MAX_WORKERS]]
46
[-n] [-x [SYS_PATH]] [--no-shadow] -i INFILES [INFILES ...]
@@ -184,59 +186,59 @@ files per batch: [2, 1, 1, 2, 4]
184186

185187
simple example
186188
```bash
187-
pybatch.py --max_workers 4 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
189+
pybatch.py --max_workers 4 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
188190
```
189191

190192
uses a shared dict which can be accessed by each batch
191193
```bash
192-
pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script3.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
194+
pybatch.py --max_workers 10 --script samples/test_script3.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
193195
```
194196

195197
count python files and folders which have files (example needs numpy)
196198
```bash
197-
pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script4.py --infiles ~/python/batch_tool/**/*.py --verbose
199+
pybatch.py --conda-env pytorch2 --script samples/test_script4.py --infiles **/*.py --verbose
198200
```
199201

200202
it is possible to use multiple processes in a batch process
201203
```bash
202-
pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script5.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
204+
pybatch.py --max_workers 10 --script samples/test_script5.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
203205
```
204206

205207
it is possible to use multiple threads in a batch process
206208
```bash
207-
pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script6.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
209+
pybatch.py --max_workers 10 --script samples/test_script6.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
208210
```
209211

210212
read from file
211213
```bash
212214
pybatch.py @samples/cmdline.txt --dry-run
213215
```
214216

215-
Generate INFILES count python files and folders which have files (example needs numpy)
217+
count python files and folders which contain python files (example needs numpy)
216218
```bash
217-
pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script7.py --infiles ~/python/batch_tool/**/*.py --verbose
219+
pybatch.py --conda-env pytorch2 --script samples/test_script7.py --infiles /path/to/batch_tool/**/*.py --verbose
218220
```
219221
## conda
220222

221223
run in a conda env
222224
```bash
223-
pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
225+
pybatch.py --conda-env pytorch2 --script samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
224226
```
225227

226228
note: when using conda run signal handlers will not work
227229
```bash
228-
conda run -n pytorch pybatch.py --script ~/python/batch_tool/samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
230+
conda run -n pytorch pybatch.py --script samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
229231
```
230232

231233
## slurm
232234
use slurm to run scripts
233235

234236
```--max_workers``` and ```-c, --cpus-per-task``` should make sense
235237
```bash
236-
srun --pty --mem 16G -c 10 -t 240 pybatch.py --conda-env pytorch2 --max_workers 10 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
238+
srun --pty --mem 16G -c 10 -t 240 pybatch.py --conda-env pytorch2 --max_workers 10 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
237239
```
238240
```bash
239-
srun --nodelist=rtxa5000-01 --gres=gpu:1 --pty --mem 16G -c 2 -t 240 pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
241+
srun --nodelist=rtxa5000-01 --gres=gpu:1 --pty --mem 16G -c 2 -t 240 pybatch.py --conda-env pytorch2 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose
240242
```
241243

242244

0 commit comments

Comments
 (0)