Development Environment Documentation results in error (likely outdated documentation)

[siddagra]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Steps to Reproduce

1. followed steps in docs: https://opencv.github.io/cvat/docs/contributing/development-environment/
2. python manage.py migrate causes the error:

Traceback (most recent call last):
  File "/media/user123/PortableSSD2/cvat/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/media/user123/PortableSSD2/cvat/.env39/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/media/user123/PortableSSD2/cvat/.env39/lib/python3.9/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/media/user123/PortableSSD2/cvat/.env39/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/media/user123/PortableSSD2/cvat/.env39/lib/python3.9/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/media/user123/PortableSSD2/cvat/.env39/lib/python3.9/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/media/user123/PortableSSD2/cvat/cvat/apps/organizations/models.py", line 18, in <module>
    from cvat.apps.engine.models import TimestampedModel
  File "/media/user123/PortableSSD2/cvat/cvat/apps/engine/models.py", line 26, in <module>
    from cvat.apps.engine.utils import parse_specific_attributes
  File "/media/user123/PortableSSD2/cvat/cvat/apps/engine/utils.py", line 416, in <module>
    identifier: str | int,
TypeError: unsupported operand type(s) for |: 'type' and 'type'

This type of operand behaviour is only supported in python3.10 I think. However the documentation explicitly says to use python3.9. Moreover, using python3.10 causes build error and build to fail for python-ldap==3.4.3

Can I please get an updated documentation/guide for the development environment?

Expected Behavior

It should let me set up a superuser and not throw an error

Possible Solution

update the docs. Or fix the operand to use Union typing instead of python3.10 syntax.

Context

I need to make some changes to the repo to get some work done and its causing issues completing my work on deadlines

Environment

Ubuntu 20.04

python==3.9.18
asgiref==3.7.2
astroid==2.11.7
async-timeout==4.0.3
attrs==21.4.0
autopep8==2.0.4
av==9.2.0
azure-core==1.29.4
azure-storage-blob==12.13.0
black==24.1.1
boto3==1.17.61
botocore==1.20.112
cachetools==5.3.1
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.0
click==8.1.7
clickhouse-connect==0.6.8
contourpy==1.1.1
coreapi==2.3.3
coreschema==0.0.4
crontab==1.0.1
cryptography==42.0.2
cycler==0.12.1
datumaro @ git+https://github.com/cvat-ai/datumaro.git@8a14a99fe17f19d98595a2a4a74ab459051cc23b
defusedxml==0.7.1
Deprecated==1.2.14
dill==0.3.7
dj-pagination==2.5.0
dj-rest-auth==2.2.7
Django==4.2.6
django-allauth==0.52.0
django-appconf==1.0.5
django-auth-ldap==2.2.0
django-compressor==4.3.1
django-cors-headers==3.5.0
django-crum==0.7.9
django-extensions==3.0.8
django-filter==2.4.0
django-health-check==3.18.1
django-rq==2.8.1
django-sendfile2==0.7.0
django-silk==5.0.3
djangorestframework==3.14.0
drf-spectacular==0.26.2
EasyProcess==1.1
entrypoint2==1.1
fonttools==4.43.1
freezegun==1.2.2
furl==2.1.0
google-api-core==2.12.0
google-auth==2.23.3
google-cloud-core==2.3.3
google-cloud-storage==1.42.0
google-crc32c==1.5.0
google-resumable-media==2.6.0
googleapis-common-protos==1.60.0
gprof2dot==2022.7.29
h5py==3.10.0
idna==3.4
importlib-metadata==6.8.0
importlib-resources==6.1.0
inflection==0.5.1
isodate==0.6.1
isort==5.12.0
itypes==1.2.0
Jinja2==3.1.3
jmespath==0.10.0
jsonschema==4.17.3
kiwisolver==1.4.5
lazy-object-proxy==1.9.0
limits==3.6.0
lxml==4.9.3
lz4==4.3.2
MarkupSafe==2.1.3
matplotlib==3.8.0
mccabe==0.7.0
msrest==0.7.1
mypy-extensions==1.0.0
natsort==8.0.0
networkx==3.1
nibabel==5.1.0
numpy==1.22.4
oauthlib==3.2.2
opencv-python==4.9.0.80
opencv-python-headless==4.8.1.78
orderedmultidict==1.0.1
orjson==3.9.8
packaging==23.2
pandas==2.1.1
pathspec==0.11.2
patool==1.12
pdf2image==1.14.0
Pillow==10.1.0
platformdirs==3.11.0
protobuf==3.19.6
psutil==5.9.4
psycopg2-binary==2.9.5
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycocotools==2.0.7
pycodestyle==2.11.0
pycparser==2.21
PyJWT==2.8.0
pylint==2.14.5
pylint-django==2.5.3
pylint-plugin-utils==0.7
pylogbeat==2.0.1
pyparsing==3.1.1
pyrsistent==0.19.3
python-dateutil==2.8.2
python-ldap==3.4.3
python-logstash-async==2.5.0
python3-openid==3.2.0
pytz==2020.1
pyunpack==0.2.1
PyYAML==6.0.1
rcssmin==1.1.1
redis==4.5.4
requests==2.31.0
requests-oauthlib==1.3.1
rjsmin==1.2.1
rope==0.17.0
rq==1.15.1
rq-scheduler==0.13.1
rsa==4.9
ruamel.yaml==0.17.35
ruamel.yaml.clib==0.2.8
rules==3.3
s3transfer==0.4.2
scipy==1.11.3
Shapely==1.7.1
six==1.16.0
snakeviz==2.1.0
sqlparse==0.4.4
tensorboardX==2.6
tomli==2.0.1
tomlkit==0.12.1
tornado==6.3.3
tqdm==4.66.1
typing_extensions==4.8.0
tzdata==2023.3
uritemplate==4.1.1
urllib3==1.26.18
wrapt==1.15.0
zipp==3.17.0
zstandard==0.21.0

[adkbbx]

Hello @bsekachev and @siddagra,

I faced a similar issue to the one mentioned earlier but managed to resolve it by updating my Python version to Python 3.10.0 from Python 3.9

However, after fixing that, I encountered a new error when running python manage.py migrate:

(.env) adkbbx@LAPTOP-V56U8GG8:/mnt/e/GSOC/CVAT/cvat$ python manage.py migrate
Traceback (most recent call last):
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: fe_sendauth: no password supplied


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/e/GSOC/CVAT/cvat/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 117, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/migrations/loader.py", line 58, in __init__
    self.build_graph()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/migrations/loader.py", line 235, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 81, in applied_migrations
    if self.has_table():
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/migrations/recorder.py", line 57, in has_table
    with self.connection.cursor() as cursor:
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
    with self.wrap_database_errors:
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/mnt/e/GSOC/CVAT/cvat/.env/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "localhost" (127.0.0.1), port 5432 failed: fe_sendauth: no password supplied

Based on my limited understanding, it seems that the PostgreSQL server is expecting authentication (a password), but Django did not provide one during the connection attempt. However, I'm unsure about how to proceed in resolving this issue.

Could you please provide some insight into what might be causing this error and how I can fix it? Additionally, if there are any best practices or common pitfalls to be aware of when connecting to databases like PostgreSQL using Django, I would greatly appreciate any advice or resources you can share.

Thank you for your assistance and patience with my questions, and I hope changing the Python version solves your issue.

[bsekachev]

Postgres does not really require password authentification. It is internal, not exposed outside.
Maybe you already have another postgres instance running on this port?

Usually for development I deploy necessary services in Docker:
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build cvat_opa cvat_db cvat_redis_inmem cvat_redis_ondisk cvat_server

And then run server:django, server: RQ - export, server: RQ - import configurations in vscode. It works.
Additionally you may run UI server using yarn start:cvat-ui

[siddagra]

For anyone who this may help, I got it running with the python 3.10 by running the following first:

sudo apt-get update && sudo apt-get --no-install-recommends install -y build-essential curl git redis-server python3.10-dev python3.10-pip python3.10-venv python3.10-tk libldap2-dev libsasl2-dev

python3.10 needs to be explicitly mentioned or else it will install build tools for python 3.8 if python3 is used in the install command by default on Ubuntu.

[siddagra]

However, now I am unable to get SAM model to work in development environment.

[adkbbx]

@bsekachev

Thank you for your prompt response. I successfully set up the development environment locally and managed to run the UI server using the commands you provided! 🎉

First, I set up the Docker services using:

docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build cvat_opa cvat_db cvat_redis_inmem cvat_redis_ondisk cvat_server

Then, in Visual Studio Code, I ran the following configurations:

• server:django
• server:RQ - export
• server:RQ - import

Additionally, I executed yarn start:cvat-ui to start the UI server.

The final output resembled the image below, and the server successfully ran on my localhost port 3000:

I've also created a PR to resolve this issue. Let me know if you have any suggestions for it.

Thank you for your time!

[nmanovic]

@adkbbx , thank you for your contribution!

@siddagra , I will close the issue. It looks like it was resolved by our community!