|
1 | 1 | # batch tool
|
| 2 | +python script tool for parallel processing of files |
| 3 | + |
2 | 4 | ```
|
3 | 5 | usage: pybatch.py [-h] [-p [PARAMETER]] [-v] [-r] -s SCRIPT [-m [MAX_WORKERS]]
|
4 | 6 | [-n] [-x [SYS_PATH]] [--no-shadow] -i INFILES [INFILES ...]
|
@@ -184,59 +186,59 @@ files per batch: [2, 1, 1, 2, 4]
|
184 | 186 |
|
185 | 187 | simple example
|
186 | 188 | ```bash
|
187 |
| -pybatch.py --max_workers 4 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 189 | +pybatch.py --max_workers 4 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
188 | 190 | ```
|
189 | 191 |
|
190 | 192 | uses a shared dict which can be accessed by each batch
|
191 | 193 | ```bash
|
192 |
| -pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script3.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 194 | +pybatch.py --max_workers 10 --script samples/test_script3.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
193 | 195 | ```
|
194 | 196 |
|
195 | 197 | count python files and folders which have files (example needs numpy)
|
196 | 198 | ```bash
|
197 |
| -pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script4.py --infiles ~/python/batch_tool/**/*.py --verbose |
| 199 | +pybatch.py --conda-env pytorch2 --script samples/test_script4.py --infiles **/*.py --verbose |
198 | 200 | ```
|
199 | 201 |
|
200 | 202 | it is possible to use multiple processes in a batch process
|
201 | 203 | ```bash
|
202 |
| -pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script5.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 204 | +pybatch.py --max_workers 10 --script samples/test_script5.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
203 | 205 | ```
|
204 | 206 |
|
205 | 207 | it is possible to use multiple threads in a batch process
|
206 | 208 | ```bash
|
207 |
| -pybatch.py --max_workers 10 --script ~/python/batch_tool/samples/test_script6.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 209 | +pybatch.py --max_workers 10 --script samples/test_script6.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
208 | 210 | ```
|
209 | 211 |
|
210 | 212 | read from file
|
211 | 213 | ```bash
|
212 | 214 | pybatch.py @samples/cmdline.txt --dry-run
|
213 | 215 | ```
|
214 | 216 |
|
215 |
| -Generate INFILES count python files and folders which have files (example needs numpy) |
| 217 | +count python files and folders which contain python files (example needs numpy) |
216 | 218 | ```bash
|
217 |
| -pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script7.py --infiles ~/python/batch_tool/**/*.py --verbose |
| 219 | +pybatch.py --conda-env pytorch2 --script samples/test_script7.py --infiles /path/to/batch_tool/**/*.py --verbose |
218 | 220 | ```
|
219 | 221 | ## conda
|
220 | 222 |
|
221 | 223 | run in a conda env
|
222 | 224 | ```bash
|
223 |
| -pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 225 | +pybatch.py --conda-env pytorch2 --script samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
224 | 226 | ```
|
225 | 227 |
|
226 | 228 | note: when using conda run signal handlers will not work
|
227 | 229 | ```bash
|
228 |
| -conda run -n pytorch pybatch.py --script ~/python/batch_tool/samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 230 | +conda run -n pytorch pybatch.py --script samples/test_script.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
229 | 231 | ```
|
230 | 232 |
|
231 | 233 | ## slurm
|
232 | 234 | use slurm to run scripts
|
233 | 235 |
|
234 | 236 | ```--max_workers``` and ```-c, --cpus-per-task``` should make sense
|
235 | 237 | ```bash
|
236 |
| -srun --pty --mem 16G -c 10 -t 240 pybatch.py --conda-env pytorch2 --max_workers 10 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 238 | +srun --pty --mem 16G -c 10 -t 240 pybatch.py --conda-env pytorch2 --max_workers 10 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
237 | 239 | ```
|
238 | 240 | ```bash
|
239 |
| -srun --nodelist=rtxa5000-01 --gres=gpu:1 --pty --mem 16G -c 2 -t 240 pybatch.py --conda-env pytorch2 --script ~/python/batch_tool/samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
| 241 | +srun --nodelist=rtxa5000-01 --gres=gpu:1 --pty --mem 16G -c 2 -t 240 pybatch.py --conda-env pytorch2 --script samples/test_script2.py --infiles 1 2 3 4 5 6 7 8 9 10 --verbose |
240 | 242 | ```
|
241 | 243 |
|
242 | 244 |
|
|
0 commit comments