Skip to content
This repository has been archived by the owner on Dec 16, 2022. It is now read-only.

CUDA Out of memory error after training has stopped #5511

Closed
10 tasks done
vikigenius opened this issue Dec 13, 2021 · 3 comments · Fixed by #5518
Closed
10 tasks done

CUDA Out of memory error after training has stopped #5511

vikigenius opened this issue Dec 13, 2021 · 3 comments · Fixed by #5518
Assignees
@epwalsh
Labels
bug

Comments

@vikigenius
Copy link
Contributor

vikigenius commented Dec 13, 2021 •
edited
Loading

Checklist

  • I have verified that the issue exists against the main branch of AllenNLP.
  • I have read the relevant section in the contribution guide on reporting bugs.
  • I have checked the issues list for similar or identical bug reports.
  • I have checked the pull requests list for existing proposed fixes.
  • I have checked the CHANGELOG and the commit log to find out if the bug was already fixed in the main branch.
  • I have included in the "Description" section below a traceback from any exceptions related to this bug.
  • I have included in the "Related issues or possible duplicates" section beloew all related issues and possible duplicate issues (If there are none, check this box anyway).
  • I have included in the "Environment" section below the name of the operating system and Python version that I was using when I discovered this bug.
  • I have included in the "Environment" section below the output of pip freeze.
  • I have included in the "Steps to reproduce" section below a minimally reproducible example.

Description

Additional GPU memory is used even after training stops. This issue does not happen when the last epoch is also the best model so far.

Python traceback: 

-- Process 3 terminated with the following error:
Traceback (most recent call last):
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/allennlp/commands/train.py", line 504, in _train_worker
    metrics = train_loop.run()
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/allennlp/commands/train.py", line 577, in run
    return self.trainer.train()
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/allennlp/training/gradient_descent_trainer.py", line 769, in train
    metrics, epoch = self._try_train()
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/allennlp/training/gradient_descent_trainer.py", line 956, in _try_train
    self._load_model_state(self._best_model_filename)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/allennlp/training/gradient_descent_trainer.py", line 968, in _load_model_state
    self._ddp_wrapped_model.load_state_dict(torch.load(path))
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 882, in _load
    result = unpickler.load()
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 857, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 846, in load_tensor
    loaded_storages[key] = restore_location(storage, location)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/serialization.py", line 157, in _cuda_deserialize
    return obj.cuda(device)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/_utils.py", line 79, in _cuda
    return new_type(self.size()).copy_(self, non_blocking)
  File "/home/void/miniconda3/envs/lexsiamese/lib/python3.9/site-packages/torch/cuda/__init__.py", line 606, in _lazy_new
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Related issues or possible duplicates

  • None

Environment

OS:

Python version: 3.9.7

Output of pip freeze: 

aiohttp @ file:///home/void/.cache/pypoetry/artifacts/c3/d1/97/85daa5493c9bf3eb25c509bc3eb629985f82b89336bfcb5adcb76b78fe/aiohttp-3.8.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
aiosignal @ file:///home/void/.cache/pypoetry/artifacts/d3/d7/92/f7aa28b137b4b00e62270f58330cedde299917a71a3c00ee64913aa3c2/aiosignal-1.2.0-py3-none-any.whl
alabaster @ file:///home/void/.cache/pypoetry/artifacts/48/57/b7/0e44080d4c6f80ece83e427f90368b376fdec42759b60c2c37a04da23b/alabaster-0.7.12-py2.py3-none-any.whl
alembic @ file:///home/void/.cache/pypoetry/artifacts/a9/7f/76/cc692e51f952a683bbb802478dbcc64da55ae1365d161446cca6ce0809/alembic-1.7.5-py3-none-any.whl
allennlp @ file:///home/void/.cache/pypoetry/artifacts/5f/bc/2c/31dfa2e9194c9081a89d19148cac449d570f5f104d71adce4b92505e29/allennlp-2.8.0-py3-none-any.whl
allennlp-optuna @ file:///home/void/.cache/pypoetry/artifacts/5b/51/0e/8776f8115f6e5dd56c3b1a005c81c38696b961a98e5db932ee4ef8cb6f/allennlp_optuna-0.1.7-py3-none-any.whl
arger @ file:///home/void/.cache/pypoetry/artifacts/be/a9/97/92eedbef45f7661c903a6a82517d2d19b7b808717bee4ca85c941531b0/arger-1.4.2-py3-none-any.whl
argon2-cffi @ file:///home/void/.cache/pypoetry/artifacts/e5/36/34/bf219434f7144f39ec847b03dc7626b6d79dd2ce22f0dd27be6a6f2656/argon2_cffi-21.3.0-py3-none-any.whl
argon2-cffi-bindings @ file:///home/void/.cache/pypoetry/artifacts/dc/7d/af/6367bb3a6693241419d236ecccbc664f4efa6b8bdebb6a5594e1508633/argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
astor @ file:///home/void/.cache/pypoetry/artifacts/1f/03/36/982d1222edac5e8cb8ac6e0464249747fa800d4fb04728a99153ecfe4d/astor-0.8.1-py2.py3-none-any.whl
async-timeout @ file:///home/void/.cache/pypoetry/artifacts/4a/a8/7d/ffb2ac78e2c6ac4cbf6bbf5d29a0afb3cf7a1b3f18ffbb3a0adfe89d5d/async_timeout-4.0.1-py3-none-any.whl
attrs @ file:///home/void/.cache/pypoetry/artifacts/6f/a9/ee/569c37f69a8c365ee41d2340aeac0214ee8c0086b8d8db43a21545204b/attrs-21.2.0-py2.py3-none-any.whl
autopage @ file:///home/void/.cache/pypoetry/artifacts/73/11/fb/97c1c7b5e56a1b4a358f24e359119a21f7e5a2165ad9fc37738091d3ba/autopage-0.4.0-py3-none-any.whl
autorepr @ file:///home/void/.cache/pypoetry/artifacts/79/b0/2c/559adfad5f45b74ef658fdab7bff52f955339a62d1aeae798f390f88d9/autorepr-0.3.0-py2.py3-none-any.whl
Babel @ file:///home/void/.cache/pypoetry/artifacts/1e/28/b6/110e8fc8ccc0dafe21bbc521139d0d0674574f7daa6f6f90137bdb4759/Babel-2.9.1-py2.py3-none-any.whl
backcall @ file:///home/void/.cache/pypoetry/artifacts/43/8e/e8/4e598704edf6fb4a53d552ea511c04e9958dcf850897760e5387878b99/backcall-0.2.0-py2.py3-none-any.whl
backports.csv @ file:///home/void/.cache/pypoetry/artifacts/30/84/1a/81a42cff31ce7f0b7a86ab54e2cbcb610d96fa8b735d63bdb7251e91cb/backports.csv-1.0.7-py2.py3-none-any.whl
bandit @ file:///home/void/.cache/pypoetry/artifacts/5a/52/5e/30a779fa19fdd8ac741622674d12f76ad48c8f76074fe83dfc2ea2d0af/bandit-1.7.1-py3-none-any.whl
base58 @ file:///home/void/.cache/pypoetry/artifacts/2e/53/33/a15f42f485704caf86395e9701bb3440206ec9bffc062d7a597f43518e/base58-2.1.1-py3-none-any.whl
beautifulsoup4 @ file:///home/void/.cache/pypoetry/artifacts/d4/3e/b2/b4df19cca3023c21eb4a289b88c132ac71a9517bddef2055fd38eadd39/beautifulsoup4-4.10.0-py3-none-any.whl
bleach @ file:///home/void/.cache/pypoetry/artifacts/db/42/1e/1062d618eece64f515d716a5cb98eace54768d5af54a9273b1f554eaec/bleach-4.1.0-py2.py3-none-any.whl
blis @ file:///home/void/.cache/pypoetry/artifacts/ed/19/db/9c4d35a6ae4d8bafd131c6e29638af4528680f853c54cbd074ec874162/blis-0.7.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
boto3 @ file:///home/void/.cache/pypoetry/artifacts/0a/57/02/42339b65f77c54b1c0d1f35c3d3c6c7a084c961afce3c50c56e080ab14/boto3-1.20.23-py3-none-any.whl
botocore @ file:///home/void/.cache/pypoetry/artifacts/b3/3d/aa/30d9a04e5a70bef8c817f173dc388ed4116621b0611b24df8026ffe0f8/botocore-1.23.23-py3-none-any.whl
cached-path @ file:///home/void/.cache/pypoetry/artifacts/94/57/de/0f18078c0fcdba822b1236767fb6b51db6aac0d047b39fe790dfe94540/cached_path-0.3.2-py3-none-any.whl
cachetools @ file:///home/void/.cache/pypoetry/artifacts/ab/92/05/de1deda089a4f18230cef3edfa7da0739da7e4482a5727422e9a5db078/cachetools-4.2.4-py3-none-any.whl
cachy @ file:///home/void/.cache/pypoetry/artifacts/c0/ce/2b/be65c61ed593659749cff39804b346369e358f14f09151abe591f0c5ab/cachy-0.3.0-py2.py3-none-any.whl
catalogue @ file:///home/void/.cache/pypoetry/artifacts/7c/6a/8a/f666a0e711d9153f80a844329970fcf567da43df07fef3127356219777/catalogue-2.0.6-py3-none-any.whl
certifi==2021.10.8
cffi @ file:///home/void/.cache/pypoetry/artifacts/0b/b7/48/30ff76ec327f0041e809dd9c35b741a68d6d8566a123e6dabda722536a/cffi-1.15.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
chardet @ file:///home/void/.cache/pypoetry/artifacts/11/63/f9/797eda27963177a6b75a340f62aa194d462ea69e6b0dbb77a651fa2b62/chardet-4.0.0-py2.py3-none-any.whl
charset-normalizer @ file:///home/void/.cache/pypoetry/artifacts/9e/1a/c0/f7635922644ab43cdae873172296c99b00796f2e281669e0fb7516bf88/charset_normalizer-2.0.9-py3-none-any.whl
checklist @ file:///home/void/.cache/pypoetry/artifacts/04/84/f3/1324eec13577715f52121b3073ab37792c08483aae7faa7d22b7dd5e1d/checklist-0.0.11.tar.gz
cheroot @ file:///home/void/.cache/pypoetry/artifacts/3c/cf/87/c9bb0e3b0d4c43affeb8f9714d5791b61c97750bc3a2ee35d276d425b5/cheroot-8.5.2-py2.py3-none-any.whl
CherryPy @ file:///home/void/.cache/pypoetry/artifacts/dd/d5/0c/5289a45f52e9aa001f7b8c2b9c792377e79ad572bc759f1a692d160818/CherryPy-18.6.1-py2.py3-none-any.whl
click @ file:///home/void/.cache/pypoetry/artifacts/34/4d/5b/2e936191a48c61e37319c157af7e3c0dbdde9b80ccdd04b7940f99c5db/click-8.0.3-py3-none-any.whl
cliff @ file:///home/void/.cache/pypoetry/artifacts/b8/19/16/acf36d1f3ec0fb46cde9e3700a44f14796366bd39542c9adedeefebbca/cliff-3.10.0-py3-none-any.whl
cmaes @ file:///home/void/.cache/pypoetry/artifacts/25/75/d0/dd27383149a81e2f7be6fc9ac708fa8b06ea14d96b18d645f8f3cf85c5/cmaes-0.8.2-py3-none-any.whl
cmd2 @ file:///home/void/.cache/pypoetry/artifacts/7a/79/21/3ecc24e0d01c3e69c8d67262be1acf2db7425452f0f84972f9044802e8/cmd2-2.3.3-py3-none-any.whl
colorlog @ file:///home/void/.cache/pypoetry/artifacts/44/b1/86/5279793ca939562b56cd91947a520f212e585ad0c4252cb01c0ec59dd8/colorlog-6.6.0-py2.py3-none-any.whl
configparser @ file:///home/void/.cache/pypoetry/artifacts/5d/f4/e0/46a13abe32ddb24eefbec544131a7050f0951e8023028eefa49aa1e17b/configparser-5.2.0-py3-none-any.whl
ConfigUpdater @ file:///home/void/.cache/pypoetry/artifacts/ac/04/8d/bed8325152ace634ebb44caa22d0eece05748e6a11d69c5ca90bd23bc4/ConfigUpdater-3.0.1-py2.py3-none-any.whl
coverage @ file:///home/void/.cache/pypoetry/artifacts/80/36/13/bcbeede3f084e1a60abc76b394f326ef569436d4ed68aa0f8ccf5505b2/coverage-6.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
cryptography @ file:///home/void/.cache/pypoetry/artifacts/3c/18/cb/9631c1afaf817699720296a54b598cba8e50ddba8b42e89c91f593f416/cryptography-36.0.0-cp36-abi3-manylinux_2_24_x86_64.whl
cymem @ file:///home/void/.cache/pypoetry/artifacts/f1/e5/0d/418ca076950a2a7841dd4199fe239ce71934f6eed6448b117560c8b594/cymem-2.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
darglint @ file:///home/void/.cache/pypoetry/artifacts/2d/47/f9/e4df12fe8185cdb64f2f77c23d349693941e09d8d5374c68454fb12e4f/darglint-1.8.1-py3-none-any.whl
datasets @ file:///home/void/.cache/pypoetry/artifacts/34/03/25/6bfe1bccb74e348f99c4ec6ecd4c7d26bc49685ccf029f9f57a32e8367/datasets-1.16.1-py3-none-any.whl
debugpy @ file:///home/void/.cache/pypoetry/artifacts/7b/52/41/857f16c34fdc6e0393b19a8ceed45255a606010a8753e9c594132e53f2/debugpy-1.5.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
decorator @ file:///home/void/.cache/pypoetry/artifacts/c5/6b/f9/35190d429d462647b42d2847ab554759b98334d6479bc354c719947ce6/decorator-5.1.0-py3-none-any.whl
defusedxml @ file:///home/void/.cache/pypoetry/artifacts/2b/69/07/7b13f7eaf3a4d7af737dcebe24d3d17b1c2a2f457fbddf746f5642bc43/defusedxml-0.7.1-py2.py3-none-any.whl
dictdiffer @ file:///home/void/.cache/pypoetry/artifacts/bc/85/a4/4043bd24ee0498bfa674ee88c3b151fb29f0946a2b17b0f3e5f02c906b/dictdiffer-0.9.0-py2.py3-none-any.whl
dill @ file:///home/void/.cache/pypoetry/artifacts/26/56/9e/73963d2285e6c700801f185e8c1d28f1f971c09aaa411cec9b799a5fca/dill-0.3.4-py2.py3-none-any.whl
doc8 @ file:///home/void/.cache/pypoetry/artifacts/02/8c/ea/ee72e46c18bb92524f64422cb67e575cca213c43a4c48411b1bac48dd3/doc8-0.10.1-py3-none-any.whl
docker-pycreds @ file:///home/void/.cache/pypoetry/artifacts/9b/0b/be/891931da9caf5e55102337a635d3a7eeeb92c93b4bd39c24d0810f1f25/docker_pycreds-0.4.0-py2.py3-none-any.whl
docutils @ file:///home/void/.cache/pypoetry/artifacts/43/28/92/79000933ad30371dc938d9b368a9000e20ac0bb467a716c19ef1fbd3c7/docutils-0.17.1-py2.py3-none-any.whl
dparse @ file:///home/void/.cache/pypoetry/artifacts/0e/d3/3f/27b08502c8c8da888ca1b2701c4f1d91b6777be03197cc2566b65eb3dd/dparse-0.5.1-py3-none-any.whl
entrypoints @ file:///home/void/.cache/pypoetry/artifacts/63/c1/af/bbfdd91bcb544e62ac8f1567ef23c243cb188d1a9cb933532999c9bbb0/entrypoints-0.3-py2.py3-none-any.whl
eradicate @ file:///home/void/.cache/pypoetry/artifacts/12/ce/ac/197035fe6d51568abb7ea160f5ad416d2164a2010005e8356b8229e550/eradicate-2.0.0.tar.gz
fairscale @ file:///home/void/.cache/pypoetry/artifacts/af/4f/57/87d33c2f59c3d0bbc847cb9880c3b1504f03ced8353fd70924a1ebb6c4/fairscale-0.4.0.tar.gz
feedparser @ file:///home/void/.cache/pypoetry/artifacts/d1/81/87/0f3c1c0b02176b2bf05af85261d8ce7522e4d241e5d9f7b3f0ec4f2a10/feedparser-6.0.8-py3-none-any.whl
filelock @ file:///home/void/.cache/pypoetry/artifacts/2b/5e/92/0d4f081b9ca0f6e1a3a88db44eb2e55c0c3d723288fe74e32c41235525/filelock-3.3.2-py3-none-any.whl
flake8 @ file:///home/void/.cache/pypoetry/artifacts/3f/57/d5/11722093c13092cc3bfc3dd7c88aef6f8e4d5ac97cfe5fd054d5aba412/flake8-3.9.2-py2.py3-none-any.whl
flake8-bandit @ file:///home/void/.cache/pypoetry/artifacts/7e/e4/46/e15782d941f9cde39b64ca5b636180f47573f2b2c9315be56b55152f17/flake8_bandit-2.1.2.tar.gz
flake8-broken-line @ file:///home/void/.cache/pypoetry/artifacts/51/ea/87/37348b281b73d7df44fc46b09c0430e2984e991df11998e2e9bb459fce/flake8_broken_line-0.3.0-py3-none-any.whl
flake8-bugbear @ file:///home/void/.cache/pypoetry/artifacts/93/54/65/c5a4be2d8c60c8f88ee3122e7798d259db9f00c76b0ae20626b2fc8388/flake8_bugbear-21.11.29-py36.py37.py38-none-any.whl
flake8-commas @ file:///home/void/.cache/pypoetry/artifacts/32/06/3d/62f07a797bd4a6d9f15ead0b6306a47bd0650ca9b3e20aa08d4ee2c23d/flake8_commas-2.1.0-py2.py3-none-any.whl
flake8-comprehensions @ file:///home/void/.cache/pypoetry/artifacts/69/82/2b/b41b89274a557fa065257c074e1bff9ddb1bf7e97ca7831345247be00a/flake8_comprehensions-3.7.0-py3-none-any.whl
flake8-debugger @ file:///home/void/.cache/pypoetry/artifacts/66/04/47/7bef98a8d237eb17cbfbcb803343be1c79e2c0674ceba163717b6c8e1b/flake8_debugger-4.0.0-py3-none-any.whl
flake8-docstrings @ file:///home/void/.cache/pypoetry/artifacts/e0/85/e9/6b482a11d48cf26e1170d9f5bf0b044a5a6c9b816ffe70945e90fc3e56/flake8_docstrings-1.6.0-py2.py3-none-any.whl
flake8-eradicate @ file:///home/void/.cache/pypoetry/artifacts/94/a4/1b/56cd39944d5d4d2639e80929837e7fc3b2ff432e095fd7a4d528c56090/flake8_eradicate-1.2.0-py3-none-any.whl
flake8-isort @ file:///home/void/.cache/pypoetry/artifacts/73/53/0f/7e71fed3a8631dfabd048e16a84f9fb8e15c8c9a381bda2a444bf42961/flake8_isort-4.1.1-py3-none-any.whl
flake8-plugin-utils @ file:///home/void/.cache/pypoetry/artifacts/d1/c5/cd/002f81a0e6fe87b5dc718a0f7381e43f0814e72acccd04d84899cb342b/flake8_plugin_utils-1.3.2-py3-none-any.whl
flake8-polyfill @ file:///home/void/.cache/pypoetry/artifacts/28/17/cc/952c11cd5ffb2608137557f928dc4f9365b4dbe1e2a6015eeea78583ac/flake8_polyfill-1.0.2-py2.py3-none-any.whl
flake8-pytest-style @ file:///home/void/.cache/pypoetry/artifacts/06/f5/ba/9e9e06bdcc5972b1eb2a190e39d1607b314ef79bae22e26e243725f37b/flake8_pytest_style-1.5.1-py3-none-any.whl
flake8-quotes @ file:///home/void/.cache/pypoetry/artifacts/76/04/0d/e3326c63986618bbd2e54c3211274f66a0bfae21c700e3e67ed14640cd/flake8-quotes-3.3.1.tar.gz
flake8-rst-docstrings @ file:///home/void/.cache/pypoetry/artifacts/07/75/20/16ffb132e46114df2fbe48fe2f50254b3f8ea6016371f04addc6738f41/flake8_rst_docstrings-0.2.5-py3-none-any.whl
flake8-string-format @ file:///home/void/.cache/pypoetry/artifacts/24/89/bb/7ce8e216f8c7289aa8a2ad4c44f30f87af6c7cdaf5d510110d566d66ec/flake8_string_format-0.3.0-py2.py3-none-any.whl
frozenlist @ file:///home/void/.cache/pypoetry/artifacts/16/82/59/e9de178960fa722dabaa376316661aeedaca1bce820657f3c841fc51ae/frozenlist-1.2.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
fsspec @ file:///home/void/.cache/pypoetry/artifacts/21/c5/18/c49ba5010477a09fddd7a039e5d6496310652a5637ced473b5b1e7331d/fsspec-2021.11.1-py3-none-any.whl
future @ file:///home/void/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz
gitdb @ file:///home/void/.cache/pypoetry/artifacts/90/22/b2/13e35fcefe45f6779d7507377070865acf263775bda5c08fff422c1a4d/gitdb-4.0.9-py3-none-any.whl
GitPython @ file:///home/void/.cache/pypoetry/artifacts/a5/0c/54/252d7dfe30f738045d07a47072bf784c68b68cce6d4600869cf335c1b2/GitPython-3.1.24-py3-none-any.whl
google-api-core @ file:///home/void/.cache/pypoetry/artifacts/2c/84/57/bd52ce51ad2bacdcef1c0308c0b64c38cc33219fa8f5342003c9594e40/google_api_core-2.3.0-py2.py3-none-any.whl
google-auth @ file:///home/void/.cache/pypoetry/artifacts/b3/ff/ed/2b0fd60d6a2ca3cd399676776f0c09264d0e94f1667269ecd96d136ff8/google_auth-2.3.3-py2.py3-none-any.whl
google-cloud-core @ file:///home/void/.cache/pypoetry/artifacts/77/e4/3d/ce8ea415961c50b94280469610815b8d70c8ba17f6bbacb65dbf84ebee/google_cloud_core-2.2.1-py2.py3-none-any.whl
google-cloud-storage @ file:///home/void/.cache/pypoetry/artifacts/b8/b9/58/eafb345088c65da05e0224b0186ad9a2a543023de0ab7e775b901ffe04/google_cloud_storage-1.43.0-py2.py3-none-any.whl
google-crc32c @ file:///home/void/.cache/pypoetry/artifacts/dc/73/5d/3e90ed7e68039176e9833fa7936e473a5542d1c38f19a3d63e42ef3e0a/google_crc32c-1.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
google-resumable-media @ file:///home/void/.cache/pypoetry/artifacts/ee/56/cd/4225643d5f5931b0f867ee44e7dccdd99315ce4c372ba315f3387c51a6/google_resumable_media-2.1.0-py2.py3-none-any.whl
googleapis-common-protos @ file:///home/void/.cache/pypoetry/artifacts/77/7b/91/7cb1de6c5653a728a51c1d6c251dc38468c5b2e597067a35eed2285043/googleapis_common_protos-1.54.0-py2.py3-none-any.whl
greenlet @ file:///home/void/.cache/pypoetry/artifacts/50/37/3a/8b9fdf0fe03f0f52d1bb7c3505cdec8e2b0d72b9fedf093b84332bf97c/greenlet-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
h5py @ file:///home/void/.cache/pypoetry/artifacts/f1/de/3b/b6b056d985b3b06bfcbb42eaa59871c2c47aae1687e406be20e01d1541/h5py-3.6.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
huggingface-hub @ file:///home/void/.cache/pypoetry/artifacts/9e/bf/cf/655508ef1a6ac197efc182642122d99d0bbd6b68f63278cfff0bda75f7/huggingface_hub-0.1.2-py3-none-any.whl
identify @ file:///home/void/.cache/pypoetry/artifacts/ae/f6/e7/902cc86fc3e112d585e0fecc9cf81d49590a3d0cfaa0f46f382f8ad93b/identify-2.4.0-py2.py3-none-any.whl
idna @ file:///home/void/.cache/pypoetry/artifacts/d5/04/80/9c17fd3240a37d12ff2ef042b0306aeb1abd2d8b95f150fd60be938352/idna-3.3-py3-none-any.whl
imagesize @ file:///home/void/.cache/pypoetry/artifacts/96/5c/68/c505397f41dd445fe3305ed42a0c203fe0f36fee2cbb522ef564d90d71/imagesize-1.3.0-py2.py3-none-any.whl
importlib-metadata @ file:///home/void/.cache/pypoetry/artifacts/53/7a/9a/4d94d104305a079e224206038434d261e1d9cc1beb771d02a0715d5e65/importlib_metadata-4.8.2-py3-none-any.whl
iniconfig @ file:///home/void/.cache/pypoetry/artifacts/fa/b0/c6/10cfac68c9e6de9d2a1678366ca89fd9292b362c1760dbe758e41691cb/iniconfig-1.1.1-py2.py3-none-any.whl
ipykernel @ file:///home/void/.cache/pypoetry/artifacts/89/23/5d/c45e6bbf8fe53de475787afcf084ce795f90b515764c40736e0f7ad352/ipykernel-6.6.0-py3-none-any.whl
ipython @ file:///home/void/.cache/pypoetry/artifacts/e0/59/72/4d9bd2e1728ce74ef79b7a4d42048fa75076bb9e276ff9f0378a2d4e3c/ipython-7.30.1-py3-none-any.whl
ipython-genutils @ file:///home/void/.cache/pypoetry/artifacts/b4/31/01/6f96480580d1674cab0b5e26dc9fca7bbdf7a2fd5811a7807a92436268/ipython_genutils-0.2.0-py2.py3-none-any.whl
ipywidgets @ file:///home/void/.cache/pypoetry/artifacts/fc/29/4b/f750c0f70ce7f9529dc7dcae670c7a69248e56cec4a62e16cdf4febd70/ipywidgets-7.6.5-py2.py3-none-any.whl
iso-639 @ file:///home/void/.cache/pypoetry/artifacts/f0/a6/d6/17ede193e09cf4ac45d787cbef2e1c78ff7dff5a58775728212204bbd0/iso-639-0.4.5.tar.gz
isort @ file:///home/void/.cache/pypoetry/artifacts/9d/c0/d5/617ee6ec6b065b9a3c8c3f7a0b06604f1a47738904535923554e135a7d/isort-5.10.1-py3-none-any.whl
jaraco.classes @ file:///home/void/.cache/pypoetry/artifacts/d0/3c/d2/f157bfb781b294c3d68a29e898ab39327bc2397eea1b42cf8afdfda14b/jaraco.classes-3.2.1-py3-none-any.whl
jaraco.collections @ file:///home/void/.cache/pypoetry/artifacts/b5/37/dc/9c36b359f5ee40235f16be580710ca07f20d5f03be3d28a10df354e707/jaraco.collections-3.4.0-py3-none-any.whl
jaraco.functools @ file:///home/void/.cache/pypoetry/artifacts/77/50/ae/8893dee5640d369f64a95c45a9e7d2b70892e34051958e9bb44a0cd881/jaraco.functools-3.4.0-py3-none-any.whl
jaraco.text @ file:///home/void/.cache/pypoetry/artifacts/3b/b4/df/e40175a7b285d6abc4be0e5301cbcd2f73a40decd507c3b2f0d8a236c7/jaraco.text-3.6.0-py3-none-any.whl
jedi @ file:///home/void/.cache/pypoetry/artifacts/c0/16/7b/b208472f00204d5aaeb0895fcde8e681c56c250bdf8d106fa76cdf7b30/jedi-0.18.1-py2.py3-none-any.whl
Jinja2 @ file:///home/void/.cache/pypoetry/artifacts/3b/12/80/ed05a8559d6ad1e85324dbcebfe790670d339c674fb15c419b29aae138/Jinja2-3.0.3-py3-none-any.whl
jmespath @ file:///home/void/.cache/pypoetry/artifacts/2c/f0/52/b0ba93d941bd49c8719dee7ca27d2096bf96e17948667388c3ee2ac8f8/jmespath-0.10.0-py2.py3-none-any.whl
joblib @ file:///home/void/.cache/pypoetry/artifacts/a6/ca/34/f3a58b34616787c399095242ae8633fc32e061f8f76debd987dcecb325/joblib-1.1.0-py2.py3-none-any.whl
jsonnet @ file:///home/void/.cache/pypoetry/artifacts/60/4e/2d/acde747a02049d38e6dbda9dc3fbf64f03bf2e14c8e9ad04f07edcc66b/jsonnet-0.17.0.tar.gz
jsonschema @ file:///home/void/.cache/pypoetry/artifacts/ac/85/79/ccb562dde487c36a54f9c9e0872b5991960e317d7286c714723b7d399d/jsonschema-4.2.1-py3-none-any.whl
jupyter @ file:///home/void/.cache/pypoetry/artifacts/bb/e8/12/09df1332820a1126a780ab09cec78d2f50457f79bcd0cb2fbb07b19ef4/jupyter-1.0.0-py2.py3-none-any.whl
jupyter-client @ file:///home/void/.cache/pypoetry/artifacts/27/6c/bc/333a7cf743ca6eaba129fbc32cbd844b16a60d1a00d4837ea6a83d953c/jupyter_client-7.1.0-py3-none-any.whl
jupyter-console @ file:///home/void/.cache/pypoetry/artifacts/4b/2f/b8/119f975fb811a5e911beacc7a1bb8f8e1154254fb33204e753131e7aca/jupyter_console-6.4.0-py3-none-any.whl
jupyter-core @ file:///home/void/.cache/pypoetry/artifacts/94/16/ab/109e77a494c40a8da71a914e4c4fced4f49ccdc3ba8dc603c10c12d01b/jupyter_core-4.9.1-py3-none-any.whl
jupyterlab-pygments @ file:///home/void/.cache/pypoetry/artifacts/de/ef/fc/5883436de4b7865f082f7cba0e0e0ff5fbf229fe55d6e7d5431a6080f4/jupyterlab_pygments-0.1.2-py2.py3-none-any.whl
jupyterlab-widgets @ file:///home/void/.cache/pypoetry/artifacts/b0/2b/0f/f7f65b7f0839db0577f58ba1135a0b2592c80c8c44cfe0feade7ffea95/jupyterlab_widgets-1.0.2-py3-none-any.whl
lmdb @ file:///home/void/.cache/pypoetry/artifacts/8c/76/54/b721202dccac74078c06fb8bb746bc85c679ef6ed6488ff62e97f597c5/lmdb-1.2.1-cp39-cp39-manylinux2010_x86_64.whl
loguru @ file:///home/void/.cache/pypoetry/artifacts/86/46/fb/57abc2a88da92cb957a6f2c96bd3e58662eede7ebc1adce938a5ea46c4/loguru-0.5.3-py3-none-any.whl
lxml @ file:///home/void/.cache/pypoetry/artifacts/71/9f/af/bb6db64668ef6754d242c72616cbd050d11016729520a8de21ecf6b12b/lxml-4.6.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
m2r2 @ file:///home/void/.cache/pypoetry/artifacts/13/22/0c/6840c0a42d0edbca824d90ebfdf23e3514d686ab1df51c393580ff1ca2/m2r2-0.3.2-py3-none-any.whl
Mako @ file:///home/void/.cache/pypoetry/artifacts/f5/35/12/a1a9bd3bdefed47c1b61fb976364f50a2fb431b0d8cc52001a44920780/Mako-1.1.6-py2.py3-none-any.whl
MarkupSafe @ file:///home/void/.cache/pypoetry/artifacts/8b/03/f1/977f12728ea53158567728db9f624a06196c1eadef033ecd621df81b47/MarkupSafe-2.0.1-cp39-cp39-manylinux2010_x86_64.whl
marshmallow @ file:///home/void/.cache/pypoetry/artifacts/7e/41/f1/81d9a89f25ac2e4c2bf3c19eb031d7f3294c006480818b5d32bd3e9c17/marshmallow-3.14.1-py3-none-any.whl
marshmallow-polyfield @ file:///home/void/.cache/pypoetry/artifacts/15/54/1a/bf7b323a7e9bef7044335503b044f792374a95673e74fcf13d80690a10/marshmallow_polyfield-5.10-py3-none-any.whl
matplotlib-inline @ file:///home/void/.cache/pypoetry/artifacts/b5/de/c6/6384d0999287fcdc9e88d38f7951106dcb7ffac7c4e8c0d0c665e12cac/matplotlib_inline-0.1.3-py3-none-any.whl
mccabe @ file:///home/void/.cache/pypoetry/artifacts/96/5e/5f/21ae5296697ca7f94de4da6e21d4936d74029c352a35202e4c339a4253/mccabe-0.6.1-py2.py3-none-any.whl
mistune @ file:///home/void/.cache/pypoetry/artifacts/33/31/4c/2d69dc65d06d1c8f8b00b8e995e24bae97fce2e1f8ec5d8d2d98e852da/mistune-0.8.4-py2.py3-none-any.whl
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work
mkl-service==2.4.0
more-itertools @ file:///home/void/.cache/pypoetry/artifacts/a6/67/cb/fbf4fb4bbfedae39080e351b12ca64287d47c839769eeaf91434f3274c/more_itertools-8.12.0-py3-none-any.whl
multidict @ file:///home/void/.cache/pypoetry/artifacts/22/0a/79/83ddc41690ddab080e4124552f4a06bef7689df7f71ff8d022fd183a3c/multidict-5.2.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
multiprocess @ file:///home/void/.cache/pypoetry/artifacts/5d/fb/ce/029cafd015834319024c43f708d975b006d115de466309cb8f4d8c1353/multiprocess-0.70.12.2-py39-none-any.whl
munch @ file:///home/void/.cache/pypoetry/artifacts/c3/f9/98/c46b861b1fe10f4d4fecd0ed8752a968be33d2c7e698b70589015aa0b2/munch-2.5.0-py2.py3-none-any.whl
murmurhash @ file:///home/void/.cache/pypoetry/artifacts/c5/62/3e/e9b9045eda9a62bac496552aaab7dca42fa7ed5eb8ad1e2e47334f2e7b/murmurhash-1.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
mypy @ file:///home/void/.cache/pypoetry/artifacts/36/49/09/f6ff69bf72a43d56fd882450780491cf84523aa3cb39b1d0e3f2c6c9a4/mypy-0.910-cp39-cp39-manylinux2010_x86_64.whl
mypy-extensions @ file:///home/void/.cache/pypoetry/artifacts/92/45/bf/1807ce854ff668d92602207a37bfa9316def2a3f257bd03c4c5be4bc9b/mypy_extensions-0.4.3-py2.py3-none-any.whl
nbclient @ file:///home/void/.cache/pypoetry/artifacts/4f/7f/d9/a2258a0ec621b96205a9c5d1522d2c1a16f1e08360af4938d6af875ccb/nbclient-0.5.9-py3-none-any.whl
nbconvert @ file:///home/void/.cache/pypoetry/artifacts/0c/6a/ef/958991527ceb81a24b5211a099566d3ff16370b77496246e48525635a4/nbconvert-6.3.0-py3-none-any.whl
nbformat @ file:///home/void/.cache/pypoetry/artifacts/36/a7/e7/e1a0c1c54f6151e23afd51bc71e3f6e0b24a96dd1e693b92dd9a4e4ab3/nbformat-5.1.3-py3-none-any.whl
nest-asyncio @ file:///home/void/.cache/pypoetry/artifacts/b3/29/3c/b5aaddd006ebba37d2ae62d54ebf0e5a536b927e00b2e3a7b034b82e4a/nest_asyncio-1.5.4-py3-none-any.whl
nitpick @ file:///home/void/.cache/pypoetry/artifacts/d1/78/f1/f1b50c86f3a30423aa445ff3b36353427817a99ce9c4c005f96ab3e2e7/nitpick-0.29.0-py3-none-any.whl
nltk @ file:///home/void/.cache/pypoetry/artifacts/da/4b/7c/7cdcd55f26e3f858a1fa5f755b799911525ed98ca5cae6385001dfa07c/nltk-3.6.5-py3-none-any.whl
notebook @ file:///home/void/.cache/pypoetry/artifacts/1a/6a/22/3e21fed164be61b5708342540da0b106f4e490683849fe458a5632d50d/notebook-6.4.6-py3-none-any.whl
numpy @ file:///home/void/.cache/pypoetry/artifacts/35/1c/d2/841d67d0d5cb5184003bb5c34e40a0bcf166dda67c86bef5528cfa003a/numpy-1.21.4-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
optuna @ file:///home/void/.cache/pypoetry/artifacts/a5/9a/c9/62c0937c6a185daa56f35d881db7ee99d097ee07c462a4b536d689759b/optuna-2.10.0-py3-none-any.whl
overrides @ file:///home/void/.cache/pypoetry/artifacts/24/45/16/62e842b5cdff34f5106ee676232cbcc7d7a1333e4900d111bca737b13a/overrides-3.1.0.tar.gz
packaging @ file:///home/void/.cache/pypoetry/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl
pandas @ file:///home/void/.cache/pypoetry/artifacts/09/8b/ad/dea1bcb23c34aa56f16ddbf322155f5e0a2cb73a12e4ab0a51cdc35711/pandas-1.3.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pandocfilters @ file:///home/void/.cache/pypoetry/artifacts/31/93/b3/c3f289a2e857d460f34581e351cfe3ac509300d47c487f135c96e39349/pandocfilters-1.5.0-py2.py3-none-any.whl
parso @ file:///home/void/.cache/pypoetry/artifacts/4e/d3/c5/d91e200388f1ce44b74484eb39fbfd0797eb236741d3672a7d728668b2/parso-0.8.3-py2.py3-none-any.whl
pastel @ file:///home/void/.cache/pypoetry/artifacts/da/84/f3/3e4d8b15eabeba62960ed9d3ccc1e30b7ae5f1b93e6c28d291c67eaf93/pastel-0.2.1-py2.py3-none-any.whl
pathtools @ file:///home/void/.cache/pypoetry/artifacts/ce/ff/c7/31da76336d55d51d979a50868616c867c7b2ea6f2d2084b8c744726ae7/pathtools-0.1.2.tar.gz
pathy @ file:///home/void/.cache/pypoetry/artifacts/50/12/78/e6a2a43271455193874ba977034f2eb9ce0c4993ce86418a8903962049/pathy-0.6.1-py3-none-any.whl
patternfork-nosql @ file:///home/void/.cache/pypoetry/artifacts/fa/53/d7/a04c2b1cd20312460da3f82ca634ac259fc581089956ed73763c0757cc/patternfork_nosql-3.6.tar.gz
pbr @ file:///home/void/.cache/pypoetry/artifacts/ae/21/53/3fee9e8edcd01b131a623fffcfef07c401beb748a2bd52a8e3fd7d12eb/pbr-5.8.0-py2.py3-none-any.whl
pdfminer.six @ file:///home/void/.cache/pypoetry/artifacts/cb/48/9a/333cadec1385a8846110abec74e0e0d3a75e5fc706ec6c9746bc1cbf0a/pdfminer.six-20211012-py3-none-any.whl
pep8-naming @ file:///home/void/.cache/pypoetry/artifacts/a4/93/c7/a3b9b8b4aef682b4caa67015d897aff3d064860a460124ad8a23b6f45f/pep8_naming-0.11.1-py2.py3-none-any.whl
pexpect @ file:///home/void/.cache/pypoetry/artifacts/5c/c2/43/b54fe59cab7e831df35401c8e6840162bf4a2ae5862604e7bc22db3000/pexpect-4.8.0-py2.py3-none-any.whl
pickleshare @ file:///home/void/.cache/pypoetry/artifacts/b5/48/a1/d2b823337003d531d87cf0d503ef28bb579703a74d14ad24a88863d616/pickleshare-0.7.5-py2.py3-none-any.whl
Pillow==8.4.0
pluggy @ file:///home/void/.cache/pypoetry/artifacts/81/78/ca/13f743a3628faf5a0b7f021efb45f2193acba3a13663d498f6b34bf02e/pluggy-1.0.0-py2.py3-none-any.whl
poethepoet @ file:///home/void/.cache/pypoetry/artifacts/26/05/ba/93c09ab34cd65fa679517a054858388ea3d6872a2d2c935968ac69d41a/poethepoet-0.11.0-py3-none-any.whl
portend @ file:///home/void/.cache/pypoetry/artifacts/b3/f1/c9/2742382df341b37e926f70577d5e6e1941593daa76f445b011b5760ba3/portend-3.1.0-py3-none-any.whl
preshed @ file:///home/void/.cache/pypoetry/artifacts/32/18/f6/d3ef3856138e839656c9506ee150937257b0be46dcefb42b33572b9a01/preshed-3.0.6-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
prettytable @ file:///home/void/.cache/pypoetry/artifacts/41/92/33/a7b29c9b27ef82b89fcc84a93098e457b165dc0afa43255c01ddb8c35a/prettytable-2.4.0-py3-none-any.whl
prometheus-client @ file:///home/void/.cache/pypoetry/artifacts/e5/94/14/4c38534d91e7278ccdf17c9e95312bb7dc0616281412eb11d863b4f6b4/prometheus_client-0.12.0-py2.py3-none-any.whl
promise @ file:///home/void/.cache/pypoetry/artifacts/d6/c6/43/95f1e737b1dd79d3a5ac6cfb264a889716bab4cd9d28a9bc8c69591d53/promise-2.3.tar.gz
prompt-toolkit @ file:///home/void/.cache/pypoetry/artifacts/ec/2a/77/5024fe40d375fe25a5d638cb593ec62b68f66f18670ea6eb8be5e0d414/prompt_toolkit-3.0.24-py3-none-any.whl
protobuf @ file:///home/void/.cache/pypoetry/artifacts/67/3c/11/7ad8ef245e4143fa80e62004142d61b5f9bd8179e65185b45479a09fee/protobuf-3.19.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
psutil @ file:///home/void/.cache/pypoetry/artifacts/06/3a/4a/5e88d8f14e18b10c222f81667132086849930f068d68d064ceb8a87834/psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl
ptyprocess @ file:///home/void/.cache/pypoetry/artifacts/2a/29/5d/0cdc5ec916431d60f03d2f725c54edbfa9fe53700b75fdfee209a3291e/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///home/void/.cache/pypoetry/artifacts/b3/5c/47/ba5a596e01a2b61fa2daa6a438252483ad8c04e6c99e5dc22eaf8a489a/py-1.11.0-py2.py3-none-any.whl
pyarrow @ file:///home/void/.cache/pypoetry/artifacts/28/3c/3e/fe875ac36b574da1e98180fa8d5230599b1af79bf6a6c6004470215308/pyarrow-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pyasn1 @ file:///home/void/.cache/pypoetry/artifacts/7b/3a/54/42ce43b579bda01b9d79022fb733811594441e7a32e9f9a5a98f672bdc/pyasn1-0.4.8-py2.py3-none-any.whl
pyasn1-modules @ file:///home/void/.cache/pypoetry/artifacts/dd/b8/4f/b56433e0354274a31074995e01b8671751e9f0ed0001f5254e5b03a54f/pyasn1_modules-0.2.8-py2.py3-none-any.whl
pycodestyle @ file:///home/void/.cache/pypoetry/artifacts/4c/30/97/026c283ef67eb248e5b7e6fad1f8ffb99dae50c11fd93eb939fd7c1f46/pycodestyle-2.7.0-py2.py3-none-any.whl
pycparser @ file:///home/void/.cache/pypoetry/artifacts/fb/06/dd/b5671b47dd0597663bc05d60d324bb315a8cef56f3179b8f9067f88e50/pycparser-2.21-py2.py3-none-any.whl
pydantic @ file:///home/void/.cache/pypoetry/artifacts/e1/bf/c1/7cdd43a1b6ea4e9cfd1e0e9e5b3276ad83c6b57194b4449da3e64fe6cf/pydantic-1.8.2-cp39-cp39-manylinux2014_x86_64.whl
pydocstyle @ file:///home/void/.cache/pypoetry/artifacts/75/e7/e5/1acad15a51efd39cf39259c7888c205fd787a92efea28f7afc5a9e315c/pydocstyle-6.1.1-py3-none-any.whl
pyflakes @ file:///home/void/.cache/pypoetry/artifacts/eb/c4/2c/47fcc1b3f387b1f7033e85b3ac6ee7772338461a8de8ac3977c6a7dcc1/pyflakes-2.3.1-py2.py3-none-any.whl
Pygments @ file:///home/void/.cache/pypoetry/artifacts/d0/e9/f0/fe2df77eaa1b942fb9b6c52c8c433da9ce654580696e74423c9a9ad814/Pygments-2.10.0-py3-none-any.whl
pyparsing @ file:///home/void/.cache/pypoetry/artifacts/61/f1/da/6428e68d48dbbc9e9b942cf2dfb6b5a7faa052423078aae46465c3c8e0/pyparsing-3.0.6-py3-none-any.whl
pyperclip @ file:///home/void/.cache/pypoetry/artifacts/35/fc/1a/52ce52d3f2894189c10f68a583aa30936be8b3f2ae63ed12a58c2ab370/pyperclip-1.8.2.tar.gz
pyrsistent @ file:///home/void/.cache/pypoetry/artifacts/78/42/af/83d41626a30713ec48d568b27fc65ea9a098dd76da0e034fa6fa37c127/pyrsistent-0.18.0-cp39-cp39-manylinux1_x86_64.whl
pytest @ file:///home/void/.cache/pypoetry/artifacts/28/be/f1/ce8fa22c989106fc7c1ee7fa31d55d7f31aee1c58c63b2a43670601b79/pytest-6.2.5-py3-none-any.whl
pytest-cov @ file:///home/void/.cache/pypoetry/artifacts/50/df/c5/4bd35027c6247daac4ba547aff90d97ccdc7daf4aa62e110d69cfd39de/pytest_cov-3.0.0-py3-none-any.whl
pytest-randomly @ file:///home/void/.cache/pypoetry/artifacts/d0/be/d5/ab32188862bfab68e4fe043037f5c29f4b1657fa9b11ee6215bbbce29e/pytest_randomly-3.10.3-py3-none-any.whl
python-dateutil @ file:///home/void/.cache/pypoetry/artifacts/53/f8/2a/7d63ce15df7386e9536e83413453f8aa845b47fb425f05c4ca2fb231c3/python_dateutil-2.8.2-py2.py3-none-any.whl
python-docx @ file:///home/void/.cache/pypoetry/artifacts/7f/3f/b0/ca05b61dd6a8beb8bc8317700154416271ddda4db5425c92e9d780cba7/python-docx-0.8.11.tar.gz
python-lsp-jsonrpc @ file:///home/void/.cache/pypoetry/artifacts/e6/0e/80/b725147ba93341249a4ba1659a62c1e5fdff33d830389c548e06f75065/python_lsp_jsonrpc-1.0.0-py3-none-any.whl
python-lsp-server @ file:///home/void/.cache/pypoetry/artifacts/5d/f5/2f/a6a8117edeecd8d8bfc62f5737a05bb40fe86cd4e4647510868bffd40a/python_lsp_server-1.3.2-py3-none-any.whl
python-slugify @ file:///home/void/.cache/pypoetry/artifacts/55/30/64/5d47f177ea315b707abbd2c99158590fbcb2e16d64954fd6866264b6b1/python_slugify-5.0.2-py2.py3-none-any.whl
pytz @ file:///home/void/.cache/pypoetry/artifacts/3e/f3/dd/6629bf5ed0a970a128e0a015f2af3b67280e749a56fdaabdaa06d5fc6b/pytz-2021.3-py2.py3-none-any.whl
PyYAML @ file:///home/void/.cache/pypoetry/artifacts/4e/12/e6/32a4a77023f06c3061d2fc6d4692aa531b8530e211b24ff1f77a39e6ee/PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
pyzmq @ file:///home/void/.cache/pypoetry/artifacts/55/34/c8/5202023ce699a94ec8e3208751b3199ff36dda321909e46071ac56ff99/pyzmq-22.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
qtconsole @ file:///home/void/.cache/pypoetry/artifacts/c6/8e/42/23f9ca96f920c5a6d025e320f260cd6cfa562ad4ac8c8630fdfd4719e9/qtconsole-5.2.1-py3-none-any.whl
QtPy @ file:///home/void/.cache/pypoetry/artifacts/bd/e8/2d/4e139a4203207e267a82367a23f300bb4ea3aa19cc84b79e08dcee10ee/QtPy-1.11.3-py2.py3-none-any.whl
regex @ file:///home/void/.cache/pypoetry/artifacts/69/3f/17/3188c9bca1e4732b00fb9bfcb2517add6231b3912ff2fc88e163377c06/regex-2021.11.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
requests @ file:///home/void/.cache/pypoetry/artifacts/13/00/3d/5574756561eed6b522f5b13d5a2c05c2920860b05b1e687c3792495ba0/requests-2.26.0-py2.py3-none-any.whl
restructuredtext-lint @ file:///home/void/.cache/pypoetry/artifacts/b6/09/80/91d176f17ba9a28291203e41600b294aa26214e185082bcb0cc3543588/restructuredtext_lint-1.3.2.tar.gz
rsa @ file:///home/void/.cache/pypoetry/artifacts/29/ba/41/0ee0fcca877c94f32799d12775b513c1314a9712a5c2833dc5bacff2ab/rsa-4.8-py3-none-any.whl
ruamel.yaml @ file:///home/void/.cache/pypoetry/artifacts/74/59/e5/d0f5b8a5babb30e50b50ebbc49f00d4567db12f4f93dee4d256ad2045a/ruamel.yaml-0.17.17-py3-none-any.whl
ruamel.yaml.clib @ file:///home/void/.cache/pypoetry/artifacts/52/95/75/d1a001b469ea6ed66dd6f4194c9f7eb370223816cf6292be2a8213e9a4/ruamel.yaml.clib-0.2.6-cp39-cp39-manylinux1_x86_64.whl
s3transfer @ file:///home/void/.cache/pypoetry/artifacts/24/03/95/68c447d2eda2cc1924350cd0fdc25702e449ec27e9ac07b9106d57ee0b/s3transfer-0.5.0-py3-none-any.whl
sacremoses @ file:///home/void/.cache/pypoetry/artifacts/65/e4/32/31727975bdcbad897dad7897d4dd2f2cc417567cb8cdb062a33ff203dd/sacremoses-0.0.46-py3-none-any.whl
safety @ file:///home/void/.cache/pypoetry/artifacts/8f/cc/5a/1dc0196c65cdb3f015bad143c000d159aca6f838a21d7d10872b092b83/safety-1.10.3-py2.py3-none-any.whl
scikit-learn @ file:///home/void/.cache/pypoetry/artifacts/7e/53/6a/d82bb17e553455e6e77c903e8f9073d80cb89be3001094e1bc6b61c2da/scikit_learn-1.0.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
scipy @ file:///home/void/.cache/pypoetry/artifacts/f6/a9/24/5707b4b4771bb5870386d5f9a054a34c61d719c835eb5411d6f9c2172d/scipy-1.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Send2Trash @ file:///home/void/.cache/pypoetry/artifacts/63/2c/10/d48818ab0722d2d6ac42f70c8b9b6567ff3a7936bdc06dedddda4398b8/Send2Trash-1.8.0-py3-none-any.whl
sentencepiece @ file:///home/void/.cache/pypoetry/artifacts/82/c3/b5/7dcf4928f5159deaba8f451549bd8b063b13e500873857524fc8f893c9/sentencepiece-0.1.96-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sentry-sdk @ file:///home/void/.cache/pypoetry/artifacts/30/32/20/e1a1ec2d14fe92277fe6de796bd2788aa637b4453fae25dccfb91cf181/sentry_sdk-1.5.0-py2.py3-none-any.whl
sgmllib3k @ file:///home/void/.cache/pypoetry/artifacts/48/41/c1/47c574e94f31057312eab350c2a7e7b75d1105eb5b673a14efe485c128/sgmllib3k-1.0.0.tar.gz
shortuuid @ file:///home/void/.cache/pypoetry/artifacts/1b/c2/26/0a03ab3637895180121fd749a44e5005e71b8300cdd863c20ddbf0d318/shortuuid-1.0.8-py3-none-any.whl
siamenc==0.1.0
six @ file:///tmp/build/80754af9/six_1623709665295/work
smart-open @ file:///home/void/.cache/pypoetry/artifacts/90/9d/8f/b3121f6940407c06e50f8f81646a84f4551ec214bb239013488c3492e8/smart_open-5.2.1-py3-none-any.whl
smmap @ file:///home/void/.cache/pypoetry/artifacts/76/3f/89/377e56f6e08e5b1fa88da762382b4c9c817c6dd24eae2e0e190898511d/smmap-5.0.0-py3-none-any.whl
snowballstemmer @ file:///home/void/.cache/pypoetry/artifacts/a4/95/b0/c0f70d4b9bb0bac123e716da53ba9b012071cedf7c99bcf030757530f4/snowballstemmer-2.2.0-py2.py3-none-any.whl
sortedcontainers @ file:///home/void/.cache/pypoetry/artifacts/b9/80/e1/4bdfa349488797fd308ecbe48f4fad57a3245777fb47c8741730583262/sortedcontainers-2.4.0-py2.py3-none-any.whl
soupsieve @ file:///home/void/.cache/pypoetry/artifacts/d6/e3/46/df82bc73fb10e31f7b04a84f47a5779830a07eeb6a6fa547e6d39ba88f/soupsieve-2.3.1-py3-none-any.whl
spacy @ file:///home/void/.cache/pypoetry/artifacts/d7/0d/95/78417ba0b0e2e9fad4c3a79d8ffa424452dda63eda7e7928268b3f9e6e/spacy-3.1.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
spacy-legacy @ file:///home/void/.cache/pypoetry/artifacts/3a/2e/7c/ea1cd14cc8de699a5021f6aeb2d53e5951af637177cc9cbab44e17b4b2/spacy_legacy-3.0.8-py2.py3-none-any.whl
Sphinx @ file:///home/void/.cache/pypoetry/artifacts/8c/bb/4c/0ee38eab2be2a2d2f4623c9931920764ce590096adac2d693793344e12/Sphinx-4.3.1-py3-none-any.whl
sphinx-autodoc-typehints @ file:///home/void/.cache/pypoetry/artifacts/e1/00/00/fde2ece43111d9fa6d415ed6fc4b0ef02b562fd106f3b6d636c3290d69/sphinx_autodoc_typehints-1.12.0-py3-none-any.whl
sphinxcontrib-applehelp @ file:///home/void/.cache/pypoetry/artifacts/8d/eb/86/eec708bb3ff50c9780e78f36a9cb82cd9ff8030a90bd23b9a6f20aecca/sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl
sphinxcontrib-devhelp @ file:///home/void/.cache/pypoetry/artifacts/56/a5/74/11ccaa7737f06a10422027e0595b24d243af7a7a1dc4982dec22044c28/sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl
sphinxcontrib-htmlhelp @ file:///home/void/.cache/pypoetry/artifacts/66/d0/cb/7228297c74d9280e7246b52187704724b0b0881e2762cdef34e04be778/sphinxcontrib_htmlhelp-2.0.0-py2.py3-none-any.whl
sphinxcontrib-jsmath @ file:///home/void/.cache/pypoetry/artifacts/d2/22/96/2076357e64b369910aa24a20d5b719beb24a1487146e4742476ee1e2d8/sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl
sphinxcontrib-qthelp @ file:///home/void/.cache/pypoetry/artifacts/32/fc/a9/112a82396d53ec629c1450253a6ded4d94d7ffffd63acd49879543ece9/sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl
sphinxcontrib-serializinghtml @ file:///home/void/.cache/pypoetry/artifacts/e6/9a/17/830e357f3aee36549c613a2d660b5cf38d70c27ecb7c218d15c7bfffe1/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl
SQLAlchemy @ file:///home/void/.cache/pypoetry/artifacts/25/53/55/c917b4ef4af377fda67fa5112c8e9f7d76d6a01ac079e6117731103750/SQLAlchemy-1.4.28-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sqlitedict @ file:///home/void/.cache/pypoetry/artifacts/30/da/52/503d8c476c0ad677cb6bcf1714381aa53b2500752bacc499aa0a5f4283/sqlitedict-1.7.0.tar.gz
srsly @ file:///home/void/.cache/pypoetry/artifacts/89/a3/39/8c42cc259a1c485abc0d750eab45ade4628e27862bbc13e1b55e08fdcc/srsly-2.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
stevedore @ file:///home/void/.cache/pypoetry/artifacts/42/17/61/45c6b9d8b0d45dd19994d1f96aa784b4e9792c968eb24e21b0816a258b/stevedore-3.5.0-py3-none-any.whl
subprocess32 @ file:///home/void/.cache/pypoetry/artifacts/b9/91/2e/cc8d3ccbf05fa27ee73859de9d02ef1a7eba84ed701970db1063a1848d/subprocess32-3.5.4.tar.gz
tempora @ file:///home/void/.cache/pypoetry/artifacts/cb/55/38/93541362ca01d0ad882d583b2ede1e85283b618443fe69c3c3ed766f42/tempora-4.1.2-py3-none-any.whl
tensorboardX @ file:///home/void/.cache/pypoetry/artifacts/94/c5/ae/2f09a78d1accfca59ec6e4a2abba0672b6d5d4186a88aae7c52abcb8ae/tensorboardX-2.4.1-py2.py3-none-any.whl
termcolor @ file:///home/void/.cache/pypoetry/artifacts/a2/5d/c7/e4ccb3b3bb8d3e3aff995fb6ffb12cfc78bbc8affa283907ee5eb5a5a5/termcolor-1.1.0.tar.gz
terminado @ file:///home/void/.cache/pypoetry/artifacts/02/75/b5/4e735b6c2cb848bfe48f9fdbdff83761cb24fb4d7efabc73f87129bfc7/terminado-0.12.1-py3-none-any.whl
testfixtures @ file:///home/void/.cache/pypoetry/artifacts/f1/d5/a7/c4faba6a07aaf55807b3ccd90e164452bdf933dba1d25f66e8f71c451f/testfixtures-6.18.3-py2.py3-none-any.whl
testpath @ file:///home/void/.cache/pypoetry/artifacts/1e/2d/08/76691a9e7e429930fb378dd96f760de96f2686841c47da2b35a04c5aad/testpath-0.5.0-py3-none-any.whl
text-unidecode @ file:///home/void/.cache/pypoetry/artifacts/34/f9/c2/484c44b08bab89d472229bbd257fcc1d1c6273ee027f01cb08c4e3c309/text_unidecode-1.3-py2.py3-none-any.whl
thinc @ file:///home/void/.cache/pypoetry/artifacts/0b/d9/75/057cbfbcba5d08f7a45fc2007dc0bf6960c2deaabf14dc9a84d94ca162/thinc-8.0.13-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
threadpoolctl @ file:///home/void/.cache/pypoetry/artifacts/c6/32/f8/3fa7f1449c7563f03c74b00ea430b78691e4fec87a7b0098c82d4e65ab/threadpoolctl-3.0.0-py3-none-any.whl
tokenizers @ file:///home/void/.cache/pypoetry/artifacts/b8/c9/5c/4f70bb6fca31833a346653e969bb4f5d7261302f6b828e918709a81ec3/tokenizers-0.10.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
toml @ file:///home/void/.cache/pypoetry/artifacts/6b/6a/c9/53b19f7870a77d855e8b05ecdc98193944e5d246dafe11bbcad850ecba/toml-0.10.2-py2.py3-none-any.whl
tomli @ file:///home/void/.cache/pypoetry/artifacts/04/b1/42/66b120badb2e2c588323f2066e27e85314a70d09e9bd0d571ea28b888b/tomli-1.2.2-py3-none-any.whl
tomlkit @ file:///home/void/.cache/pypoetry/artifacts/fd/06/32/b79e75623225a9b5af79899482b9c2933c2fa2c6fb0eff80fcec10ae48/tomlkit-0.7.2-py2.py3-none-any.whl
torch==1.10.0
torchvision==0.11.1
tornado @ file:///home/void/.cache/pypoetry/artifacts/32/ff/76/0fc7d26eab2180c92fa57552212f38defaf212ceb7ddc9432bd84d646c/tornado-6.1-cp39-cp39-manylinux2010_x86_64.whl
tqdm @ file:///home/void/.cache/pypoetry/artifacts/c1/d2/95/984649137d2c9fac6d0f7a883c2007c07e7bbac3b692cad1004d495455/tqdm-4.62.3-py2.py3-none-any.whl
traitlets @ file:///home/void/.cache/pypoetry/artifacts/ee/01/57/93229f3393af5b0814aff282450fcac279f4d7f32188b49db882ebf459/traitlets-5.1.1-py3-none-any.whl
transformers @ file:///home/void/.cache/pypoetry/artifacts/2f/44/2e/3d86a39afc1169ee45cd28d73172f9d258f429a3993931ca6103e3a6ac/transformers-4.12.5-py3-none-any.whl
typer @ file:///home/void/.cache/pypoetry/artifacts/45/11/e9/0fd368a9452c62570ff1775b64e1fee58dc51e24327652495876625be0/typer-0.4.0-py3-none-any.whl
types-toml @ file:///home/void/.cache/pypoetry/artifacts/e1/31/59/1539e56caa1c8887484812d0d9a0384ba535b41664b6ccfbb6670a25fd/types_toml-0.10.1-py3-none-any.whl
typing-extensions @ file:///tmp/build/80754af9/typing_extensions_1631814937681/work
ujson @ file:///home/void/.cache/pypoetry/artifacts/e1/c6/4f/d6e2db8bf28a838a8458e3099d839390ef47f0f913b111365f4de539d0/ujson-4.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
urllib3 @ file:///home/void/.cache/pypoetry/artifacts/55/46/d1/1acba0600879bfaada54191116fbaea89a464f437b3f949bc9c2a220e7/urllib3-1.26.7-py2.py3-none-any.whl
wandb @ file:///home/void/.cache/pypoetry/artifacts/4a/87/ae/59e6cb93ad9fa66e2a38a2ab1823ce7acbd4f407416559707b9adc56f8/wandb-0.12.7-py2.py3-none-any.whl
wasabi @ file:///home/void/.cache/pypoetry/artifacts/38/0b/ad/70a82d637fbe35102583fb03600f76fcbaf3928d6e074bd295e50bbd5c/wasabi-0.9.0-py3-none-any.whl
wcwidth @ file:///home/void/.cache/pypoetry/artifacts/7d/f4/60/0737157bb9711fec72c70dff523aa54491eef317e0d586cf5388ff0908/wcwidth-0.2.5-py2.py3-none-any.whl
webencodings @ file:///home/void/.cache/pypoetry/artifacts/ed/d4/da/61384706cfac042ba3bd148746d66e50695463993be117c7c8dadeef7a/webencodings-0.5.1-py2.py3-none-any.whl
wemake-python-styleguide @ file:///home/void/.cache/pypoetry/artifacts/b2/30/6d/aff16dd6cc6e7169bd34e8f2e8feff71a3cc593d6ebb617acc9cf7e927/wemake_python_styleguide-0.15.3-py3-none-any.whl
widgetsnbextension @ file:///home/void/.cache/pypoetry/artifacts/1e/8c/72/6bea2521eb3293d1d676d0bd992472ac2749ffc9159df93c30cb550027/widgetsnbextension-3.5.2-py2.py3-none-any.whl
xxhash @ file:///home/void/.cache/pypoetry/artifacts/10/96/f5/ba104c6099c04fc1a07a01edd26d0c96519e713dadecd057566e8ce1f7/xxhash-2.0.2-cp39-cp39-manylinux2010_x86_64.whl
yarl @ file:///home/void/.cache/pypoetry/artifacts/a3/6b/bb/af40f3ac1dfb8650848efd34d43176c01039c7203c8f8f615543baf9d2/yarl-1.7.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
yaspin @ file:///home/void/.cache/pypoetry/artifacts/08/96/72/a995df36daf20ca713ef9c496ef7cc2595253341a2ed64056cb8ddb82c/yaspin-2.1.0-py3-none-any.whl
zc.lockfile @ file:///home/void/.cache/pypoetry/artifacts/b5/a8/c8/e94e98335e585be92e35e5d07dd8a75e5c2e7774c8bd24410160f9cfe0/zc.lockfile-2.0-py2.py3-none-any.whl
zipp @ file:///home/void/.cache/pypoetry/artifacts/84/c4/6a/f8c4d0deac7fc967e69e2e3b99eead4735005380e0c47b335bebf943f4/zipp-3.6.0-py3-none-any.whl

Steps to reproduce

This is hard to reproduce in a minimal example because it depends on GPU memory. But I can reproduce it by training my model with batch size large enough to take up all the GPU space.

  1. In first run tweak number of epochs so that the last epoch also ends up being the best epoch. In this scenario the error does not happen.

  2. In 2nd run increase the number of epochs or change metric such that the last epoch is no longer the best epoch, can reliably do this by making the model run out of patience. This will produce an out of memory error after the training has finished.

From looking I think this is the potential problem:

allennlp/allennlp/training/gradient_descent_trainer.py

Lines 953 to 957 in 38436d8

if self._best_model_filename is None or self._metric_tracker.is_best_so_far():
self._finalize_model()
else:
# The model we're loading here has already been finalized.
self._load_model_state(self._best_model_filename)

The else condition where the model state dict is loaded again is causing the issue maybe?

Edit

Here is the worker 3 logs that confirm it has finished it's entire training and validation and has ran out of patience.
out_worker3.log
@vikigenius vikigenius added the bug label Dec 13, 2021
@epwalsh
Copy link
Member

epwalsh commented Dec 17, 2021

Hey @vikigenius, I think this is a quick fix. The issue is that when self._load_model_state(self._best_model_filename) is called, we keep the model on GPU, and so have to load the state dictionary onto the same GPU as well. But there's really no reason why we need to keep the model on GPU at that point. So what we should do is move the model to CPU and then load the state dict to CPU.

If that makes sense, would you like to make a PR to fix? I think all of that could happen in the _load_model_state() method.

@vikigenius
Copy link
Contributor Author

Ah that makes sense, I thought the same thing, I will make a quick PR to fix this.

@epwalsh
Copy link
Member

epwalsh commented Dec 18, 2021

Awesome! Please ping me when the PR is ready 😄

vikigenius added a commit to vikigenius/allennlp that referenced this issue Dec 18, 2021
@vikigenius
Fixes allenai#5511: Load model on CPU post training to save GPU memory
2992ac9
@vikigenius vikigenius mentioned this issue Dec 18, 2021
Merged
6 tasks
epwalsh pushed a commit that referenced this issue Dec 20, 2021
@vikigenius
Fix model loading on GPU post training (#5518)
ab4f7b5 
* Fixes #5511: Load model on CPU post training to save GPU memory

* Edited Changelog
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@epwalsh epwalsh

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix model loading on GPU post training vikigenius/allennlp
2 participants
@epwalsh @vikigenius