Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NRTM responses can hang on PyPy #774

Closed
mxsasha opened this issue Apr 17, 2023 · 0 comments · Fixed by #777 or #801
Closed

NRTM responses can hang on PyPy #774

mxsasha opened this issue Apr 17, 2023 · 0 comments · Fixed by #777 or #801
Assignees
@mxsasha

Comments

@mxsasha
Copy link
Collaborator

mxsasha commented Apr 17, 2023

When using PyPy (confirmed with 3.9), larger but reasonable NRTM requests can entirely hang the whois worker. The query never returns, the worker does not recover. This starts to occur around a few thousand serials. The same queries respond fine with CPython in reasonable time with reasonable response sizes.

Temporary workaround: use CPython.

Sample traceback from SIGUSR1:

## Thread: MainThread(140687221291968) ##
  File "/home/irrd/irrd-venv/bin/irrd", line 8, in <module>
    sys.exit(main())
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/daemon/main.py", line 81, in main
    run_irrd(mirror_frequency=mirror_frequency,
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/daemon/main.py", line 104, in run_irrd
    whois_process.start()
  File "/opt/pypy/lib/pypy3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/utils/process_support.py", line 21, in run
    super().run()
  File "/opt/pypy/lib/pypy3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/server.py", line 35, in start_whois_server
    server = WhoisTCPServer(
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/server.py", line 78, in __init__
    worker.start()
  File "/opt/pypy/lib/pypy3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/opt/pypy/lib/pypy3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/server.py", line 145, in run
    self.handle_connection()
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/server.py", line 198, in handle_connection
    if not self.handle_query(query):
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/server.py", line 212, in handle_query
    response = self.query_parser.handle_query(query)
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/query_parser.py", line 77, in handle_query
    return self.handle_ripe_command(query)
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/query_parser.py", line 381, in handle_ripe_command
    result = self.handle_nrtm_request(components.pop(0))
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/server/whois/query_parser.py", line 480, in handle_nrtm_request
    return NRTMGenerator().generate(
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/mirroring/nrtm_generator.py", line 71, in generate
    output += '\n' + operation['operation'].value
  File "/home/irrd/irrd-venv/lib/pypy3.9/site-packages/irrd/utils/process_support.py", line 48, in sigusr1_handler
    code += traceback.format_list(traceback.extract_stack(stack))

Sample strace:

munmap(0x7ff419381000, 95555584)        = 0
munmap(0x7ff41eea2000, 95551488)        = 0
mmap(NULL, 95555584, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41eea1000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff40dd37000
munmap(0x7ff41eea1000, 95555584)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41ee99000
munmap(0x7ff40dd37000, 95588352)        = 0
munmap(0x7ff413860000, 95555584)        = 0
munmap(0x7ff4249c2000, 95551488)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff42a4d7000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff419370000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff413847000
munmap(0x7ff419370000, 95588352)        = 0
munmap(0x7ff42a4d7000, 95588352)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff42a4d7000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff419370000
munmap(0x7ff42a4d7000, 95588352)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff42a4d7000
munmap(0x7ff419370000, 95588352)        = 0
munmap(0x7ff413847000, 95588352)        = 0
munmap(0x7ff41ee99000, 95588352)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4249ae000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41ee85000
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41935c000
munmap(0x7ff41ee85000, 95588352)        = 0
munmap(0x7ff4249ae000, 95588352)        = 0
mmap(NULL, 95588352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4249ae000
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff413832000
munmap(0x7ff4249ae000, 95588352)        = 0
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff4249ad000
munmap(0x7ff413832000, 95592448)        = 0
munmap(0x7ff41935c000, 95588352)        = 0
munmap(0x7ff42a4d7000, 95588352)        = 0
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41ee83000
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff419359000
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41382f000
munmap(0x7ff419359000, 95592448)        = 0
munmap(0x7ff41ee83000, 95592448)        = 0
mmap(NULL, 95592448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41ee83000
mmap(NULL, 95617024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff40dcff000
munmap(0x7ff41ee83000, 95592448)        = 0
mmap(NULL, 95617024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41ee7d000
munmap(0x7ff40dcff000, 95617024)        = 0
munmap(0x7ff41382f000, 95592448)        = 0
munmap(0x7ff4249ad000, 95592448)        = 0
mmap(NULL, 95617024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff42a4d0000
mmap(NULL, 95617024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41934d000
mmap(NULL, 95617024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff41381d000
munmap(0x7ff41934d000, 95617024)        = 0
munmap(0x7ff42a4d0000, 95617024)        = 0
@mxsasha mxsasha self-assigned this Apr 17, 2023
mxsasha added a commit that referenced this issue Apr 19, 2023
@mxsasha
attempted fix of #774
ba05d0a
@mxsasha mxsasha linked a pull request Apr 19, 2023 that will close this issue
Fix #774 - Fix NRTM generator performance on PyPy #777
Merged
mxsasha added a commit that referenced this issue Apr 19, 2023
@mxsasha
Fix #774 - Fix NRTM generator performance on PyPy (#777)
87bceb8 
Apparently, PyPy deals quite poorly when extending the same string many times, as the old code did. More details in #774 / #777.
mergify bot pushed a commit that referenced this issue Apr 19, 2023
@mxsasha @mergify
Fix #774 - Fix NRTM generator performance on PyPy (#777)
2ddb8a0 
Apparently, PyPy deals quite poorly when extending the same string many times, as the old code did. More details in #774 / #777.

(cherry picked from commit 87bceb8)

# Conflicts:
#	irrd/mirroring/nrtm_generator.py
mergify bot pushed a commit that referenced this issue Apr 19, 2023
@mxsasha @mergify
Fix #774 - Fix NRTM generator performance on PyPy (#777)
78b1f7c 
Apparently, PyPy deals quite poorly when extending the same string many times, as the old code did. More details in #774 / #777.

(cherry picked from commit 87bceb8)
mxsasha added a commit that referenced this issue Apr 19, 2023
@mergify @mxsasha
[4.3.x] Fix #774 - Fix NRTM generator performance on PyPy (#777) (#779)
e275171 
Apparently, PyPy deals quite poorly when extending the same string many times, as the old code did. More details in #774 / #777.

(cherry picked from commit 87bceb8)

Co-authored-by: Sasha Romijn <github@mxsasha.eu>
mxsasha added a commit that referenced this issue Apr 19, 2023
@mergify @mxsasha
[4.2.x] Fix #774 - Fix NRTM generator performance on PyPy (backport #777
c2951ed 
) (#778)

Apparently, PyPy deals quite poorly when extending the same string many times, as the old code did. More details in #774 / #777.

(cherry picked from commit 87bceb8)

Co-authored-by: Sasha Romijn <github@mxsasha.eu>
@mxsasha mxsasha mentioned this issue Jun 5, 2023
Merged
@mxsasha mxsasha mentioned this issue Aug 22, 2023
Merged
mxsasha added a commit that referenced this issue Aug 22, 2023
@mxsasha
Remove PyPy support.
dfb3768 
PyPy support has been dropped due to repeated complex issues at a quite limited performance improvement. The PyPy specific issues are often highly complicated to debug, e.g. #578, #774, #848. Latest issue was infinite memory usage growth in the ROA importer. The time spent on all these issues is not worth the slight performance boost.

This commit removes CI and updates the docs, but does not remove the few lines of PyPy compatibility code.
mxsasha added a commit that referenced this issue Aug 22, 2023
@mxsasha
Remove PyPy support.
6a7c9e8 
PyPy support has been dropped due to repeated complex issues at a quite limited performance improvement. The PyPy specific issues are often highly complicated to debug, e.g. #578, #774, #848. Latest issue was infinite memory usage growth in the ROA importer. The time spent on all these issues is not worth the slight performance boost.

This commit removes CI and updates the docs, but does not remove the few lines of PyPy compatibility code.
@mxsasha mxsasha mentioned this issue Aug 22, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mxsasha mxsasha

Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@mxsasha