Failed to run the training example

[zhong-yy]

Hi, thank you for providing these useful benchmark examples.This package provides very useful encapsulation of training and evaluation processes and detailed examples.

I have gone through the examples given in the seisbench documentation. One of the seisbench examples refers to this package for more elaborate training and evaluation experiments.

Problem

After I installed the package from source pip install -r requirements.txt in the root directory, I tried to run an example with the following command

python benchmark/train.py --config=configs/stead_phasenet.json

but I got the following ImportError

Traceback (most recent call last):
  File "/home/zhongyiyuan/pick-benchmark/benchmark/train.py", line 8, in <module>
    import pytorch_lightning as pl
  File "/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning import metrics  # noqa: E402
  File "/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/pytorch_lightning/metrics/__init__.py", line 15, in <module>
    from pytorch_lightning.metrics.classification import (  # noqa: F401
  File "/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/pytorch_lightning/metrics/classification/__init__.py", line 14, in <module>
    from pytorch_lightning.metrics.classification.accuracy import Accuracy  # noqa: F401
  File "/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/pytorch_lightning/metrics/classification/accuracy.py", line 18, in <module>
    from pytorch_lightning.metrics.utils import deprecated_metrics
  File "/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/pytorch_lightning/metrics/utils.py", line 22, in <module>
    from torchmetrics.utilities.data import get_num_classes as _get_num_classes
ImportError: cannot import name 'get_num_classes' from 'torchmetrics.utilities.data' (/home/zhongyiyuan/miniconda3/envs/pickbenchmark/lib/python3.9/site-packages/torchmetrics/utilities/data.py)

The versions of related packages are

# Name                    Version                   Build  Channel
seisbench                 0.1.16                   pypi_0    pypi
pytorch-lightning         1.3.4                    pypi_0    pypi
torch                     1.8.1                    pypi_0    pypi
torchmetrics              0.11.0                   pypi_0    pypi
scikit-learn              0.24.2                   pypi_0    pypi
seaborn                   0.11.2                   pypi_0    pypi
bayesian-optimization     1.2.0                    pypi_0    pypi

My try

I updated the packages

# Name                    Version                   Build  Channel
seisbench                 0.2.8                    pypi_0    pypi
pytorch-lightning         1.8.4.post0              pypi_0    pypi
torch                     1.13.0                   pypi_0    pypi
torchmetrics              0.11.0                   pypi_0    pypi
scikit-learn              1.2.0                    pypi_0    pypi
seaborn                   0.12.1                   pypi_0    pypi
bayesian-optimization     1.4.2                    pypi_0    pypi

and tried agaiin, but got the following error

Traceback (most recent call last):
  File "/home/zhongyiyuan/pick-benchmark/benchmark/train.py", line 10, in <module>
    from pytorch_lightning.callbacks import GPUStatsMonitor
ImportError: cannot import name 'GPUStatsMonitor' from 'pytorch_lightning.callbacks' (/home/zhongyiyuan/miniconda3/envs/seisbench/lib/python3.9/site-packages/pytorch_lightning/callbacks/__init__.py)

This may be caused by Lightning-AI/pytorch-lightning#12554

[zhong-yy]

My current solution: replace GPUstatsMonitor with DeviceStatsMonitor.
Lightning-AI/pytorch-lightning#9032

[yetinam]

Hi, thanks for figuring this out. I'll update the repository to reflect this but it will take a few days until I get around to this.

[maihao14]

Hi, @yetinam,
Thanks for developing this amazing project. However, after a few tests of changing package dependencies, I still can't successfully install this repo (it will raise the error at line from benchmark import models ). My question is: Instead of trying to fix the dependencies, is it possible to write some scripts to load these pre-trained model weights (e.g., cross-domain training results) to SeisBench (to test or train on a new dataset) and use the evaluation modules provided in this repo to evaluate the new results? In another word, if the dependencies problem cannot be fixed shortly, do you think it's possible to merge part of the codes to SeisBench? Any suggestions are welcome!

[yetinam]

Hi @maihao14 ,

sorry for not following up on this before. The issue above should now be fixed. It was a problem with the version of torchmetrics. Please rerun pip install -r requirements.txt and check if this solves your issue too. Otherwise, please reopen the issue and provide the full error trace.

Regarding the suggestion to include this in SeisBench, all the model weights are already part of SeisBench. These are the ones that are loaded through from_pretrained. However, we only use the ones with the optimal learning rate, so you won't get access to the others. Regarding evaluation, I'm very hesitant to include it into SeisBench. SeisBench is more a library for the tools but not for the actual script for training an evaluation. So I'd rather fix this repository.