Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

make test Failures #145

Closed
AyanSinhaMahapatra opened this issue Apr 9, 2021 · 2 comments
Closed

make test Failures #145

AyanSinhaMahapatra opened this issue Apr 9, 2021 · 2 comments

Comments

@AyanSinhaMahapatra
Copy link
Member

On latest main and a fresh configure

  • Ubuntu 20.04,
  • python 3.8.5, and
  • psql 12.6

Traceback:

▶ make test
-> Run the test suite
bin/python manage.py test --noinput
/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/cluecode/copyrights.py:3122: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/nltk/tag/sequential.py:540: FutureWarning: Possible nested set at position 7
  self._regexps = [(re.compile(regexp), tag,) for regexp, tag in regexps]
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...............................................................................................................................EE.E............................
======================================================================
ERROR: test_scanpipe_pipes_scancode_scan_for_files (scanpipe.tests.test_pipes.ScanPipePipesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/mock.py", line 1325, in patched
    return func(*newargs, **newkeywargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/scanpipe/tests/test_pipes.py", line 323, in test_scanpipe_pipes_scancode_scan_for_files
    resource.refresh_from_db()
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 637, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1169, in execute_sql
    cursor.execute(sql, params)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL SYSCALL error: EOF detected


======================================================================
ERROR: test_scanpipe_pipes_scancode_virtual_codebase (scanpipe.tests.test_pipes.ScanPipePipesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/scanpipe/tests/test_pipes.py", line 328, in test_scanpipe_pipes_scancode_virtual_codebase
    project = Project.objects.create(name="asgiref")
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/scanpipe/models.py", line 213, in save
    super().save(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
    updated = self._save_table(
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
    return manager._insert(
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1408, in execute_sql
    with self.connection.cursor() as cursor:
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

======================================================================
ERROR: test_scanpipe_pipes_tag_not_analyzed_codebase_resources (scanpipe.tests.test_pipes.ScanPipePipesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/scanpipe/tests/test_pipes.py", line 75, in test_scanpipe_pipes_tag_not_analyzed_codebase_resources
    p1 = Project.objects.create(name="Analysis")
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/scanpipe/models.py", line 213, in save
    super().save(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
    updated = self._save_table(
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
    return manager._insert(
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1408, in execute_sql
    with self.connection.cursor() as cursor:
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/base/base.py", line 237, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/home/ayan/Desktop/nexB/main_repos/scancode.io/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

----------------------------------------------------------------------
Ran 159 tests in 111.855s

FAILED (errors=3)
Destroying test database for alias 'default'...
make: *** [Makefile:116: test] Error 1

Though running the scan_codebase pipeline works perfectly on a couple of files.
tdruez added a commit that referenced this issue Apr 21, 2021
@tdruez
Refactor scan_for_files to extract the database transaction from the …
f88e334 
…pool processes #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>
@tdruez
Copy link
Contributor

tdruez commented Apr 21, 2021

@AyanSinhaMahapatra could you run the tests again using the latests code from https://github.com/nexB/scancode.io/compare/145-multiprocessing

git pull
git checkout 145-multiprocessing
make test

Let me know if we get better results.

tdruez added a commit that referenced this issue Apr 21, 2021
@tdruez
Refine logging message formatting #145
2d0e4a2 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Apr 21, 2021
@tdruez
Log the scan_for_files progress to the database Run object #145
ee74d46 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
@AyanSinhaMahapatra
Copy link
Member Author

@tdruez This solves the test failure issue. Thanks.

tdruez added a commit that referenced this issue Apr 22, 2021
@tdruez
Apply multiprocessing refactoring to the scan_for_application_package…
49708b7 
…s pipe #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Apr 22, 2021
@tdruez
Add entry in the CHANGELOG #145
2a8ba8a 
Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Apr 22, 2021
@tdruez
Remove the logging to the database for scancode pipe scanning function …
41935f6 
…#145

This has a negative impact on performances

Signed-off-by: Thomas Druez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Apr 22, 2021
@tdruez
Refactor multiprocessing code to extract the database operations from…
7c4cf04 
… the sub-processes #145 (#159)

* Refactor scan_for_files to extract the database transaction from the pool processes #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Refine logging message formatting #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Log the scan_for_files progress to the database Run object #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Apply multiprocessing refactoring to the scan_for_application_packages pipe #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Add entry in the CHANGELOG #145

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Remove the logging to the database for scancode pipe scanning function #145

This has a negative impact on performances

Signed-off-by: Thomas Druez <tdruez@nexb.com>
@tdruez tdruez closed this as completed Apr 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tdruez @AyanSinhaMahapatra