Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible memory leaks can lead to server crash on dataset export #2241

Closed
2 tasks
zhiltsov-max opened this issue Sep 29, 2020 · 3 comments
Closed
2 tasks

Possible memory leaks can lead to server crash on dataset export #2241

zhiltsov-max opened this issue Sep 29, 2020 · 3 comments
Labels
bug Something isn't working

Comments

@zhiltsov-max
Copy link
Contributor

My actions before raising this issue

Expected Behaviour

Current Behaviour

Test on a large video with chunks and cache enabled.

  1. Run the code snippet with python manage.py shell
  2. Hit ctrl+C at some point
  3. check top output

There should not be any extra memory allocated, but after few runs over a video task with 3k frames I have 5-7 gb of allocated memory. Gc tells it knows nothing about these allocations.

import gc
import tracemalloc
tracemalloc.start(10)
try:    
    from cvat.apps.engine.models import Task
    from cvat.apps.engine.frame_provider import FrameProvider
    fp = FrameProvider(Task.objects.get(pk=39).data) # a large video task

    while True:
        for i, f in enumerate(fp.get_frames(out_type=fp.Type.NUMPY_ARRAY)):
            pass

except:    
    del fp
    del f
    snapshot = tracemalloc.take_snapshot()    
    gc.collect()

Env:

Python 3.6.5

absl-py==0.8.1
alabaster==0.7.12
appdirs==1.4.4
asgiref==3.2.10
asn1crypto==0.24.0
astor==0.8.0
astroid==2.4.2
astunparse==1.6.3
atpublic==2.0
attrs==19.3.0
autopep8==1.5.3
av==8.0.2
Babel==2.8.0
bandit==1.6.2
bcrypt==3.1.7
cached-property==1.5.1
cachetools==4.0.0
certifi==2020.6.20
cffi==1.12.3
chardet==3.0.4
click==7.1.2
colorama==0.4.3
commonmark==0.9.1
configobj==5.0.6
coreapi==2.3.3
coreschema==0.0.4
coverage==5.2.1
coveralls==2.1.1
croniter==0.3.30
cryptography==2.7
cycler==0.10.0
dataclasses==0.7
-e git+https://github.com/openvinotoolkit/datumaro.git@3157e749f304635775f10a46b7b15ef09d7b4374#egg=datumaro
ddt==1.2.1
decorator==4.4.0
defusedxml==0.6.0
dictdiffer==0.8.1
diskcache==4.1.0
distro==1.5.0
dj-pagination==2.5.0
Django==3.1.1
django-allauth==0.42.0
django-appconf==1.0.4
django-auth-ldap==2.2.0
django-cacheops==5.0.1
django-compressor==2.4
django-cors-headers==3.5.0
django-extensions==3.0.6
django-filter==2.3.0
django-rest-auth==0.9.5
django-revproxy==0.10.0
django-rq==2.3.2
django-sendfile==0.3.11
django-silk==4.0.1
djangorestframework==3.11.1
docker==3.7.3
docker-compose==1.24.1
docker-pycreds==0.4.0
dockerpty==0.4.1
docopt==0.6.2
docutils==0.16
dpath==2.0.1
drf-yasg==1.17.1
dvc==1.7.2
EasyProcess==0.3
editdistance==0.5.3
entrypoint2==0.2.1
fakeredis==1.4.1
ffmpy==0.2.2
flatten-dict==0.3.0
flufl.lock==3.2
funcy==1.14
furl==2.1.0
future==0.18.2
gast==0.3.3
gitdb==4.0.5
gitdb2==3.0.2
GitPython==3.1.3
google-auth==1.11.0
google-auth-oauthlib==0.4.1
google-pasta==0.1.8
gprof2dot==2019.11.30
grandalf==0.6
grpcio==1.24.1
h5py==2.10.0
idna==2.7
imageio==2.6.1
imagesize==1.2.0
imgaug==0.4.0
importlib-metadata==1.7.0
inflection==0.3.1
isort==4.3.21
itypes==1.1.0
Jinja2==2.10.3
joblib==0.16.0
jsonpath-ng==1.5.2
jsonschema==2.6.0
Keras==2.4.2
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lazy-object-proxy==1.5.1
lxml==4.4.1
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.3.1
mccabe==0.6.1
mod-wsgi==4.7.1
nanotime==0.5.2
networkx==2.3
nibabel==3.1.1
nose==1.3.7
numpy==1.17.5
oauthlib==3.1.0
opencv-python==4.4.0.42
opencv-python-headless==4.2.0.34
opt-einsum==3.1.0
orderedmultidict==1.0.1
packaging==20.3
paramiko==2.6.0
pascal-voc-writer==0.1.4
pathlib2==2.3.5
pathspec==0.8.0
patool==1.12
pbr==5.4.5
pdf2image==1.14.0
Pillow==7.2.0
pipdeptree==0.13.2
piprot==0.9.11
ply==3.11
protobuf==3.10.0
psycopg2-binary==2.8.5
py-cpuinfo==4.0.0
pyasn1==0.4.7
pyasn1-modules==0.2.7
pycocotools==2.0.0
pycodestyle==2.6.0
pycparser==2.19
pydicom==2.0.0
pydot==1.4.1
Pygments==2.6.1
pygtrie==2.3.2
pyhash==0.9.3
pylint==2.6.0
pylint-django==2.3.0
pylint-plugin-utils==0.6
PyNaCl==1.3.0
pyparsing==2.4.2
pypi-kenlm==0.1.20190403
python-dateutil==2.8.0
python-ldap==3.3.1
python-logstash==0.4.6
python3-openid==3.1.0
pytz==2020.1
pyunpack==0.2.1
PyWavelets==1.0.3
PyYAML==5.3.1
rcssmin==1.0.6
redis==3.5.3
requests==2.24.0
requests-futures==1.0.0
requests-oauthlib==1.2.0
rich==6.2.0
rjsmin==1.1.0
rope==0.17.0
rq==1.5.1
rq-scheduler==0.10.0
rsa==4.0
rstcheck==3.3.1
ruamel.yaml==0.16.5
ruamel.yaml.clib==0.2.0
rules==2.2
scikit-image==0.15.0
scikit-learn==0.23.2
scipy==1.4.1
sentencepiece==0.1.91
Shapely==1.7.1
shortuuid==1.0.1
shtab==1.3.1
six==1.15.0
smmap==3.0.4
smmap2==2.0.5
snakeviz==2.1.0
snowballstemmer==2.0.0
sortedcontainers==2.1.0
Sphinx==3.2.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sqlparse==0.3.1
stevedore==3.2.0
tabulate==0.8.7
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorboardX==1.8
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
texttable==0.9.1
threadpoolctl==2.1.0
tokenizers==0.8.1
toml==0.10.1
torch==1.5.1
torchvision==0.6.1
tornado==6.0.3
tqdm==4.49.0
typed-ast==1.4.0
typing-extensions==3.7.4.3
uritemplate==3.0.0
urllib3==1.24.3
voluptuous==0.11.7
websocket-client==0.56.0
Werkzeug==1.0.1
wrapt==1.12.1
yamlloader==0.5.5
zc.lockfile==2.0
zipp==3.1.0

Your Environment

  • Git hash commit (git log -1): f2c84a2653fc601c90bcb84154f0fe7a6be937df
  • Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 18.04

Next steps

You may join our Gitter channel for community support.

@zhiltsov-max zhiltsov-max added the bug Something isn't working label Sep 29, 2020
@zhiltsov-max
Copy link
Contributor Author

The reason can probably be here: https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/engine/media_extractors.py#L325

The container gets created, but never freed.

@nmanovic
Copy link
Contributor

@kirill-sizov , is it something similar to the issue which you are working on?

@azhavoro
Copy link
Contributor

Fixed in #6501

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants