Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: sqlalche braekes after finishing... #346

Closed
spirillen opened this issue Dec 16, 2023 · 0 comments
Closed

BUG: sqlalche braekes after finishing... #346

spirillen opened this issue Dec 16, 2023 · 0 comments
Assignees
Labels

Comments

@spirillen
Copy link
Contributor

Description

After the test have "completed" sqlalchemy complains about broken mysql connection

Configuration

cli_decoding:
    adblock: false
    aggressive: false
    rpz: false
    wildcard: false
cli_testing:
    autocontinue: false
    chancy_tester: false
    ci:
        active: false
        branch: master
        command: null
        commit_message: PyFunceble - AutoSave
        distribution_branch: master
        end_command: null
        end_commit_message: PyFunceble - Results
        max_exec_minutes: 15
    cidr_expand: false
    complements: false
    cooldown_time: 0.0
    days_between:
        db_clean: 28
        db_retest: 1
    db_type: csv
    display_mode:
        all: false
        colour: true
        dots: false
        execution_time: false
        less: true
        max_registrar: 15
        percentage: true
        quiet: false
        registrar: false
        simple: false
        status: all
    file_filter: null
    file_generation:
        analytic: true
        hosts: false
        merge_output_dirs: false
        no_file: false
        plain: true
        unified_results: false
    hosts_ip: 0.0.0.0
    inactive_db: true
    local_network: false
    max_workers: null
    mining: false
    preload_file: false
    sorting_mode:
        hierarchical: false
        standard: true
    testing_mode:
        availability: true
        reputation: false
        syntax: false
    whois_db: true
collection:
    preferred_status_origin: recommended
    push: false
    url_base: https://collection.dead-hosts.funilrys.com
debug:
    active: false
    level: info
dns:
    delay: 0.0
    follow_server_order: true
    protocol: UDP
    server: null
    trust_server: false
http_codes:
    list:
        potentially_down:
        - 400
        - 402
        - 404
        - 409
        - 410
        - 412
        - 414
        - 415
        - 416
        - 451
        potentially_up:
        - 0
        - 300
        - 301
        - 302
        - 303
        - 304
        - 305
        - 307
        - 308
        - 403
        - 405
        - 406
        - 407
        - 408
        - 411
        - 413
        - 417
        - 418
        - 421
        - 422
        - 423
        - 424
        - 426
        - 428
        - 431
        - 500
        - 501
        - 502
        - 503
        - 504
        - 505
        - 506
        - 507
        - 508
        - 510
        - 511
        up:
        - 100
        - 101
        - 102
        - 200
        - 201
        - 202
        - 203
        - 204
        - 205
        - 206
        - 207
        - 208
        - 226
        - 429
    self_managed: false
links:
    api_date_format: https://pyfunceble.funilrys.com/api/date-format
    api_no_referrer: https://pyfunceble.funilrys.com/api/no-referrer
lookup:
    collection: false
    dns: true
    http_status_code: true
    netinfo: true
    reputation: false
    special: true
    timeout: 5
    whois: true
max_http_retries: 3
proxy:
    global:
        http: null
        https: null
    rules: []
share_logs: false
user_agent:
    browser: chrome
    custom: null
    platform: linux
verify_ssl_certificate: false

Reproduction

  1. pyfunceble -f matrix/source/porn_filters/explicit_content/wildcard.list

Expected behavior

No errors

Screenshots

image

Versions

OS: Ubuntu 22.04.3

Python Version: 3.11

PyFunceble Version: pyfunceble 4.2.4.dev (Blue Duckling: Ixora)

Additional context

Fatal Error: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
[SQL: DELETE FROM pyfunceble_continue WHERE pyfunceble_continue.session_id = %(session_id_1)s]
[parameters: {'session_id_1': '9d6747bf4ab0fa349c99f4ff'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Traceback (most recent call last):
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 803, in _write_bytes
self._sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
result = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 557, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 861, in _execute_command
self._write_bytes(packet)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 806, in _write_bytes
raise err.OperationalError(
pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/system/launcher.py", line 1060, in start
self.run_standard_end_instructions()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/system/launcher.py", line 924, in run_standard_end_instructions
self.remove_unwanted_files()
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/system/launcher.py", line 901, in remove_unwanted_files
remove_continue_dataset(protocol)
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/cli/system/launcher.py", line 858, in remove_continue_dataset
self.continue_dataset.cleanup(  # pylint: disable=unexpected-keyword-arg
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/dataset/db_base.py", line 104, in wrapper
return func(self, *args, **kwargs)  # pylint: disable=not-callable
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/PyFunceble/dataset/autocontinue/sql.py", line 86, in cleanup
).delete(synchronize_session=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 3176, in delete
result: CursorResult[Any] = self.session.execute(
^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2262, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2144, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1946, in orm_execute_statement
return super().orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1412, in execute
return meth(
^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context
self._handle_dbapi_exception(
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/home/$USER/.local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/cursors.py", line 153, in execute
result = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 557, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 861, in _execute_command
self._write_bytes(packet)
File "/home/$USER/.local/lib/python3.11/site-packages/pymysql/connections.py", line 806, in _write_bytes
raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
[SQL: DELETE FROM pyfunceble_continue WHERE pyfunceble_continue.session_id = %(session_id_1)s]
[parameters: {'session_id_1': '9d6747bf4ab0fa349c99f4ff'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
@spirillen spirillen added the bug label Dec 16, 2023
@funilrys funilrys moved this from 🆕 New to 📋 Backlog in PyFunceble Backlog Dec 19, 2023
funilrys added a commit that referenced this issue Dec 19, 2023
Indeed, before this patch, we weren't retrying to reconnect properly.
This patch fixes the issue by using the pre_ping function of sqlalchemy.

This patch fixes #346.

Contributors:
  * @spirillen
@github-project-automation github-project-automation bot moved this from 📋 Backlog to ✅ Done in PyFunceble Backlog Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants