nohup: ignoring input
proline.gs.washington.edu
Sun Jul  2 12:48:01 PDT 2023
PID=89285
+ casanovo --mode=train --config config.yaml --peak_path=target.train.100k.mgf --peak_path_val=target.train.100k.mgf
Global seed set to 454
2023-07-02 12:48:09,732 INFO [casanovo/MainProcess] casanovo.main : Casanovo version 3.4.0
2023-07-02 12:48:09,732 DEBUG [casanovo/MainProcess] casanovo.main : mode = train
2023-07-02 12:48:09,732 DEBUG [casanovo/MainProcess] casanovo.main : model = None
2023-07-02 12:48:09,732 DEBUG [casanovo/MainProcess] casanovo.main : peak_path = target.train.100k.mgf
2023-07-02 12:48:09,732 DEBUG [casanovo/MainProcess] casanovo.main : peak_path_val = target.train.100k.mgf
2023-07-02 12:48:09,732 DEBUG [casanovo/MainProcess] casanovo.main : config = config.yaml
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : output = /net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/casanovo_20230702124809
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : random_seed = 454
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : n_peaks = 150
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : min_mz = 50.0
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : max_mz = 2500.0
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : min_intensity = 0.01
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : remove_precursor_tol = 2.0
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : max_charge = 10
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : precursor_mass_tol = 50.0
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : isotope_error_range = (0, 1)
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : min_peptide_len = 6
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : dim_model = 512
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : n_head = 8
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : dim_feedforward = 1024
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : n_layers = 9
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : dropout = 0.0
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : dim_intensity = None
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : custom_encoder = None
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : max_length = 100
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : residues = {'G': 57.021464, 'A': 71.037114, 'S': 87.032028, 'P': 97.052764, 'V': 99.068414, 'T': 101.04767, 'C+57.021': 160.030649, 'L': 113.084064, 'I': 113.084064, 'N': 114.042927, 'D': 115.026943, 'Q': 128.058578, 'K': 128.094963, 'E': 129.042593, 'M': 131.040485, 'H': 137.058912, 'F': 147.068414, 'R': 156.101111, 'Y': 163.063329, 'W': 186.079313, 'M+15.995': 147.0354, 'N+0.984': 115.026943, 'Q+0.984': 129.042594, '+42.011': 42.010565, '+43.006': 43.005814, '-17.027': -17.026549, '+43.006-17.027': 25.980265}
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : n_log = 1
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : tb_summarywriter = tb_logs
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : warmup_iters = 100000
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : max_iters = 600000
2023-07-02 12:48:09,733 DEBUG [casanovo/MainProcess] casanovo.main : learning_rate = 0.0005
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : weight_decay = 1e-05
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : train_batch_size = 32
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : predict_batch_size = 1024
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : n_beams = 1
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : top_match = 1
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : logger = None
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : max_epochs = 30
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : num_sanity_val_steps = 0
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : train_from_scratch = True
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : save_model = True
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : model_save_folder_path = 
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : save_weights_only = True
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : every_n_train_steps = 30
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : no_gpu = False
2023-07-02 12:48:09,734 DEBUG [casanovo/MainProcess] casanovo.main : n_workers = 36
2023-07-02 12:48:09,734 INFO [casanovo/MainProcess] casanovo.main : Train the Casanovo model.
2023-07-02 12:48:09,737 INFO [depthcharge.data.hdf5/MainProcess] hdf5.__init__ : Reading 1 files...

/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 0spectra [00:00, ?spectra/s]
/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 628spectra [00:00, 6274.19spectra/s]
/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 1187spectra [00:00, 6515.93spectra/s]
2023-07-02 12:48:09,929 INFO [depthcharge.data.hdf5/MainProcess] hdf5.__init__ : Reading 1 files...

/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 0spectra [00:00, ?spectra/s]
/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 775spectra [00:00, 7742.07spectra/s]
/net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/target.train.100k.mgf: 1187spectra [00:00, 7425.05spectra/s]
2023-07-02 12:48:10,166 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:589: LightningDeprecationWarning: The Trainer argument `auto_select_gpus` has been deprecated in v1.9.0 and will be removed in v2.0.0. Please use the function `pytorch_lightning.accelerators.find_usable_cuda_devices` instead.
  rank_zero_deprecation(

2023-07-02 12:48:10,769 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:613: UserWarning: Checkpoint directory  exists and is not empty.
  rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")


Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/76 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/76 [00:00<?, ?it/s] 2023-07-02 12:48:34,858 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/torch/nn/functional.py:4999: UserWarning: Support for mismatched key_padding_mask and attn_mask is deprecated. Use same type for both instead.
  warnings.warn(

2023-07-02 12:48:34,906 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/utilities/data.py:83: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 32. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
  warning_cache.warn(


Epoch 0:   1%|▏         | 1/76 [00:24<30:11, 24.16s/it]
Epoch 0:   1%|▏         | 1/76 [00:24<30:18, 24.24s/it, loss=3.49]
Epoch 0:   3%|▎         | 2/76 [00:24<15:00, 12.18s/it, loss=3.49]
Epoch 0:   3%|▎         | 2/76 [00:24<15:04, 12.22s/it, loss=3.44]
Epoch 0:   4%|▍         | 3/76 [00:24<09:57,  8.18s/it, loss=3.44]
Epoch 0:   4%|▍         | 3/76 [00:24<09:59,  8.21s/it, loss=3.44]
Epoch 0:   5%|▌         | 4/76 [00:24<07:24,  6.18s/it, loss=3.44]
Epoch 0:   5%|▌         | 4/76 [00:24<07:25,  6.19s/it, loss=3.45]
Epoch 0:   7%|▋         | 5/76 [00:24<05:53,  4.98s/it, loss=3.45]
Epoch 0:   7%|▋         | 5/76 [00:24<05:54,  4.99s/it, loss=3.47]
Epoch 0:   8%|▊         | 6/76 [00:25<04:52,  4.17s/it, loss=3.47]
Epoch 0:   8%|▊         | 6/76 [00:25<04:52,  4.18s/it, loss=3.46]
Epoch 0:   9%|▉         | 7/76 [00:25<04:08,  3.60s/it, loss=3.46]
Epoch 0:   9%|▉         | 7/76 [00:25<04:09,  3.61s/it, loss=3.46]
Epoch 0:  11%|█         | 8/76 [00:25<03:35,  3.17s/it, loss=3.46]
Epoch 0:  11%|█         | 8/76 [00:25<03:36,  3.18s/it, loss=3.47]
Epoch 0:  12%|█▏        | 9/76 [00:25<03:10,  2.84s/it, loss=3.47]
Epoch 0:  12%|█▏        | 9/76 [00:25<03:10,  2.85s/it, loss=3.47]
Epoch 0:  13%|█▎        | 10/76 [00:25<02:49,  2.57s/it, loss=3.47]
Epoch 0:  13%|█▎        | 10/76 [00:25<02:50,  2.58s/it, loss=3.47]
Epoch 0:  14%|█▍        | 11/76 [00:25<02:32,  2.35s/it, loss=3.47]
Epoch 0:  14%|█▍        | 11/76 [00:25<02:33,  2.36s/it, loss=3.47]
Epoch 0:  16%|█▌        | 12/76 [00:26<02:18,  2.17s/it, loss=3.47]
Epoch 0:  16%|█▌        | 12/76 [00:26<02:19,  2.18s/it, loss=3.47]
Epoch 0:  17%|█▋        | 13/76 [00:26<02:07,  2.02s/it, loss=3.47]
Epoch 0:  17%|█▋        | 13/76 [00:26<02:07,  2.03s/it, loss=3.48]
Epoch 0:  18%|█▊        | 14/76 [00:26<01:57,  1.89s/it, loss=3.48]
Epoch 0:  18%|█▊        | 14/76 [00:26<01:57,  1.90s/it, loss=3.48]
Epoch 0:  20%|█▉        | 15/76 [00:26<01:48,  1.78s/it, loss=3.48]
Epoch 0:  20%|█▉        | 15/76 [00:26<01:48,  1.78s/it, loss=3.48]
Epoch 0:  21%|██        | 16/76 [00:26<01:40,  1.68s/it, loss=3.48]
Epoch 0:  21%|██        | 16/76 [00:26<01:41,  1.68s/it, loss=3.47]
Epoch 0:  22%|██▏       | 17/76 [00:27<01:33,  1.59s/it, loss=3.47]
Epoch 0:  22%|██▏       | 17/76 [00:27<01:34,  1.60s/it, loss=3.47]
Epoch 0:  24%|██▎       | 18/76 [00:27<01:27,  1.51s/it, loss=3.47]
Epoch 0:  24%|██▎       | 18/76 [00:27<01:27,  1.52s/it, loss=3.47]
Epoch 0:  25%|██▌       | 19/76 [00:27<01:22,  1.44s/it, loss=3.47]
Epoch 0:  25%|██▌       | 19/76 [00:27<01:22,  1.45s/it, loss=3.48]
Epoch 0:  26%|██▋       | 20/76 [00:27<01:17,  1.38s/it, loss=3.48]
Epoch 0:  26%|██▋       | 20/76 [00:27<01:17,  1.39s/it, loss=3.48]
Epoch 0:  28%|██▊       | 21/76 [00:27<01:12,  1.32s/it, loss=3.48]
Epoch 0:  28%|██▊       | 21/76 [00:27<01:12,  1.33s/it, loss=3.48]
Epoch 0:  29%|██▉       | 22/76 [00:27<01:08,  1.27s/it, loss=3.48]
Epoch 0:  29%|██▉       | 22/76 [00:28<01:08,  1.28s/it, loss=3.48]
Epoch 0:  30%|███       | 23/76 [00:28<01:04,  1.22s/it, loss=3.48]
Epoch 0:  30%|███       | 23/76 [00:28<01:05,  1.23s/it, loss=3.48]
Epoch 0:  32%|███▏      | 24/76 [00:28<01:01,  1.18s/it, loss=3.48]
Epoch 0:  32%|███▏      | 24/76 [00:28<01:01,  1.19s/it, loss=3.48]
Epoch 0:  33%|███▎      | 25/76 [00:28<00:58,  1.14s/it, loss=3.48]
Epoch 0:  33%|███▎      | 25/76 [00:28<00:58,  1.15s/it, loss=3.47]
Epoch 0:  34%|███▍      | 26/76 [00:28<00:55,  1.11s/it, loss=3.47]
Epoch 0:  34%|███▍      | 26/76 [00:28<00:55,  1.11s/it, loss=3.47]
Epoch 0:  36%|███▌      | 27/76 [00:28<00:52,  1.07s/it, loss=3.47]
Epoch 0:  36%|███▌      | 27/76 [00:29<00:52,  1.08s/it, loss=3.47]
Epoch 0:  37%|███▋      | 28/76 [00:29<00:49,  1.04s/it, loss=3.47]
Epoch 0:  37%|███▋      | 28/76 [00:29<00:50,  1.05s/it, loss=3.47]
Epoch 0:  38%|███▊      | 29/76 [00:29<00:47,  1.01s/it, loss=3.47]
Epoch 0:  38%|███▊      | 29/76 [00:29<00:47,  1.02s/it, loss=3.47]
Epoch 0:  39%|███▉      | 30/76 [00:29<00:45,  1.01it/s, loss=3.47]
Epoch 0:  39%|███▉      | 30/76 [00:29<00:45,  1.01it/s, loss=3.47]2023-07-02 12:48:40,656 DEBUG [fsspec.local/MainProcess] local.__init__ : open file: /net/noble/vol1/home/noble/proj/2023_wnoble_denovo-fdr/results/wnoble/2023-07-01bug/epoch=0-step=30.ckpt


Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/38 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/38 [00:00<?, ?it/s]2023-07-02 12:49:02,483 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/torch/nn/modules/transformer.py:296: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ../aten/src/ATen/NestedTensorImpl.cpp:177.)
  output = torch._nested_tensor_from_mask(output, src_key_padding_mask.logical_not(), mask_check=False)

2023-07-02 12:49:02,529 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/torch/nn/modules/activation.py:1160: UserWarning: Converting mask without torch.bool dtype to bool; this will negatively affect performance. Prefer to use a boolean mask directly. (Triggered internally at ../aten/src/ATen/native/transformers/attention.cpp:150.)
  return torch._native_multi_head_attention(



Validation DataLoader 0:   3%|▎         | 1/38 [00:00<00:07,  5.04it/s]
Epoch 0:  41%|████      | 31/76 [00:51<01:15,  1.67s/it, loss=3.47]

Validation DataLoader 0:   5%|▌         | 2/38 [00:00<00:05,  6.23it/s]
Epoch 0:  42%|████▏     | 32/76 [00:51<01:11,  1.62s/it, loss=3.47]

Validation DataLoader 0:   8%|▊         | 3/38 [00:00<00:04,  7.22it/s]
Epoch 0:  43%|████▎     | 33/76 [00:52<01:07,  1.58s/it, loss=3.47]

Validation DataLoader 0:  11%|█         | 4/38 [00:00<00:04,  8.01it/s]
Epoch 0:  45%|████▍     | 34/76 [00:52<01:04,  1.53s/it, loss=3.47]

Validation DataLoader 0:  13%|█▎        | 5/38 [00:00<00:03,  8.46it/s]
Epoch 0:  46%|████▌     | 35/76 [00:52<01:01,  1.49s/it, loss=3.47]

Validation DataLoader 0:  16%|█▌        | 6/38 [00:00<00:03,  8.92it/s]
Epoch 0:  47%|████▋     | 36/76 [00:52<00:58,  1.45s/it, loss=3.47]

Validation DataLoader 0:  18%|█▊        | 7/38 [00:00<00:03,  9.13it/s]
Epoch 0:  49%|████▊     | 37/76 [00:52<00:55,  1.42s/it, loss=3.47]

Validation DataLoader 0:  21%|██        | 8/38 [00:00<00:03,  9.31it/s]
Epoch 0:  50%|█████     | 38/76 [00:52<00:52,  1.38s/it, loss=3.47]

Validation DataLoader 0:  24%|██▎       | 9/38 [00:00<00:03,  9.50it/s]
Epoch 0:  51%|█████▏    | 39/76 [00:52<00:49,  1.35s/it, loss=3.47]

Validation DataLoader 0:  26%|██▋       | 10/38 [00:01<00:02,  9.64it/s]
Epoch 0:  53%|█████▎    | 40/76 [00:52<00:47,  1.32s/it, loss=3.47]

Validation DataLoader 0:  29%|██▉       | 11/38 [00:01<00:02,  9.78it/s]
Epoch 0:  54%|█████▍    | 41/76 [00:52<00:45,  1.29s/it, loss=3.47]

Validation DataLoader 0:  32%|███▏      | 12/38 [00:01<00:02,  9.82it/s]
Epoch 0:  55%|█████▌    | 42/76 [00:52<00:42,  1.26s/it, loss=3.47]

Validation DataLoader 0:  34%|███▍      | 13/38 [00:01<00:02,  9.86it/s]
Epoch 0:  57%|█████▋    | 43/76 [00:52<00:40,  1.23s/it, loss=3.47]

Validation DataLoader 0:  37%|███▋      | 14/38 [00:01<00:02,  9.90it/s]
Epoch 0:  58%|█████▊    | 44/76 [00:53<00:38,  1.21s/it, loss=3.47]

Validation DataLoader 0:  39%|███▉      | 15/38 [00:01<00:02,  9.93it/s]
Epoch 0:  59%|█████▉    | 45/76 [00:53<00:36,  1.18s/it, loss=3.47]

Validation DataLoader 0:  42%|████▏     | 16/38 [00:01<00:02,  9.94it/s]
Epoch 0:  61%|██████    | 46/76 [00:53<00:34,  1.16s/it, loss=3.47]

Validation DataLoader 0:  45%|████▍     | 17/38 [00:01<00:02,  9.93it/s]
Epoch 0:  62%|██████▏   | 47/76 [00:53<00:32,  1.13s/it, loss=3.47]

Validation DataLoader 0:  47%|████▋     | 18/38 [00:01<00:01, 10.01it/s]
Epoch 0:  63%|██████▎   | 48/76 [00:53<00:31,  1.11s/it, loss=3.47]

Validation DataLoader 0:  50%|█████     | 19/38 [00:01<00:01, 10.03it/s]
Epoch 0:  64%|██████▍   | 49/76 [00:53<00:29,  1.09s/it, loss=3.47]

Validation DataLoader 0:  53%|█████▎    | 20/38 [00:01<00:01, 10.04it/s]
Epoch 0:  66%|██████▌   | 50/76 [00:53<00:27,  1.07s/it, loss=3.47]

Validation DataLoader 0:  55%|█████▌    | 21/38 [00:02<00:01, 10.10it/s]
Epoch 0:  67%|██████▋   | 51/76 [00:53<00:26,  1.05s/it, loss=3.47]

Validation DataLoader 0:  58%|█████▊    | 22/38 [00:02<00:01, 10.09it/s]
Epoch 0:  68%|██████▊   | 52/76 [00:53<00:24,  1.03s/it, loss=3.47]

Validation DataLoader 0:  61%|██████    | 23/38 [00:02<00:01, 10.10it/s]
Epoch 0:  70%|██████▉   | 53/76 [00:53<00:23,  1.02s/it, loss=3.47]

Validation DataLoader 0:  63%|██████▎   | 24/38 [00:02<00:01, 10.12it/s]
Epoch 0:  71%|███████   | 54/76 [00:53<00:21,  1.00it/s, loss=3.47]

Validation DataLoader 0:  66%|██████▌   | 25/38 [00:02<00:01, 10.09it/s]
Epoch 0:  72%|███████▏  | 55/76 [00:54<00:20,  1.02it/s, loss=3.47]

Validation DataLoader 0:  68%|██████▊   | 26/38 [00:02<00:01, 10.01it/s]
Epoch 0:  74%|███████▎  | 56/76 [00:54<00:19,  1.03it/s, loss=3.47]

Validation DataLoader 0:  71%|███████   | 27/38 [00:02<00:01,  9.98it/s]
Epoch 0:  75%|███████▌  | 57/76 [00:54<00:18,  1.05it/s, loss=3.47]

Validation DataLoader 0:  74%|███████▎  | 28/38 [00:02<00:01,  9.97it/s]
Epoch 0:  76%|███████▋  | 58/76 [00:54<00:16,  1.07it/s, loss=3.47]

Validation DataLoader 0:  76%|███████▋  | 29/38 [00:02<00:00,  9.93it/s]
Epoch 0:  78%|███████▊  | 59/76 [00:54<00:15,  1.08it/s, loss=3.47]

Validation DataLoader 0:  79%|███████▉  | 30/38 [00:03<00:00,  9.90it/s]
Epoch 0:  79%|███████▉  | 60/76 [00:54<00:14,  1.10it/s, loss=3.47]

Validation DataLoader 0:  82%|████████▏ | 31/38 [00:03<00:00,  9.89it/s]
Epoch 0:  80%|████████  | 61/76 [00:54<00:13,  1.11it/s, loss=3.47]

Validation DataLoader 0:  84%|████████▍ | 32/38 [00:03<00:00,  9.87it/s]
Epoch 0:  82%|████████▏ | 62/76 [00:54<00:12,  1.13it/s, loss=3.47]

Validation DataLoader 0:  87%|████████▋ | 33/38 [00:03<00:00,  9.87it/s]
Epoch 0:  83%|████████▎ | 63/76 [00:54<00:11,  1.15it/s, loss=3.47]

Validation DataLoader 0:  89%|████████▉ | 34/38 [00:03<00:00,  9.85it/s]
Epoch 0:  84%|████████▍ | 64/76 [00:55<00:10,  1.16it/s, loss=3.47]

Validation DataLoader 0:  92%|█████████▏| 35/38 [00:03<00:00,  9.87it/s]
Epoch 0:  86%|████████▌ | 65/76 [00:55<00:09,  1.18it/s, loss=3.47]

Validation DataLoader 0:  95%|█████████▍| 36/38 [00:03<00:00,  9.84it/s]
Epoch 0:  87%|████████▋ | 66/76 [00:55<00:08,  1.19it/s, loss=3.47]

Validation DataLoader 0:  97%|█████████▋| 37/38 [00:03<00:00,  9.83it/s]
Epoch 0:  88%|████████▊ | 67/76 [00:55<00:07,  1.21it/s, loss=3.47]2023-07-02 12:49:06,235 WARNING [py.warnings/MainProcess] warnings._showwarnmsg : /net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/utilities/data.py:83: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 3. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.
  warning_cache.warn(



Validation DataLoader 0: 100%|██████████| 38/38 [00:03<00:00,  9.99it/s]
Epoch 0:  89%|████████▉ | 68/76 [00:55<00:06,  1.23it/s, loss=3.47]2023-07-02 12:49:06,261 INFO [casanovo/MainProcess] model._log_history : Step	Train loss	Valid loss	Peptide precision	AA precision
2023-07-02 12:49:06,261 INFO [casanovo/MainProcess] model._log_history : 30	nan	3.451766	0.000000	0.000000
Traceback (most recent call last):
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/bin/casanovo", line 8, in <module>
    sys.exit(main())
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/casanovo/casanovo.py", line 173, in main
    model_runner.train(peak_path, peak_path_val, model, config)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model_runner.py", line 318, in train
    trainer.fit(
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
    self.fit_loop.run()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.on_advance_end()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
    self._run_validation()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
    self.val_loop.run()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 206, in run
    output = self.on_run_end()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 184, in on_run_end
    self._on_evaluation_epoch_end()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 294, in _on_evaluation_epoch_end
    self.trainer._call_lightning_module_hook(hook_name)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1356, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model.py", line 851, in on_validation_epoch_end
    self._log_history()
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/casanovo/denovo/model.py", line 916, in _log_history
    if not np.isnan(metric_value):
  File "/net/noble/vol1/home/noble/miniconda3/envs/casanovo/lib/python3.10/site-packages/torch/_tensor.py", line 970, in __array__
    return self.numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Epoch 0:  89%|████████▉ | 68/76 [00:56<00:06,  1.20it/s, loss=3.47]