PyPy-related issues

[JSCU-CNI]

When using PyPy to run dissect, we encountered two issues:

• Autocomplete is not working in target-shell

• The performance of reverse_readlines (dissect/target/helpers/fsutil:1011) is terrible

  For both issues, we haven't got a clue what's causing them. Since dissect is tested against PyPy, you might want to look into these issues.

[JSCU-CNI]

Thanks for looking into the broken autocomplete @cecinestpasunepipe. For future reference: pypy/pypy#5006 tracks this apparent PyPy issue and #829 for dissect.

[cfbolz]

@JSCU-CNI could you tell us how to reproduce the slowness of reverse_readlines? I'm happy to look into this on the PyPy side of things, but would need some more information on how to reproduce it.

[JSCU-CNI]

Thanks for taking the time to look into this @cfbolz!

The slow performance we mentioned is about the reverse_readlines function located at https://github.com/fox-it/dissect.target/blob/main/dissect/target/helpers/fsutil.py#L535.

The easiest method to test this function is probably by running pytest tests/helpers/test_fsutil.py which should invoke the test_reverse_readlines function located at https://github.com/fox-it/dissect.target/blob/main/tests/helpers/test_fsutil.py#L676.

The snippet below also could get you started. You might need some additional dependencies on your system such as liblzo2-dev and libyara-dev.

git clone https://github.com/fox-it/dissect.target.git
cd dissect.target/
pypy3.9 -m venv venv
. venv/bin/activate
pip install -e .[dev]

And then in the repl or some local script:

from dissect.target.helpers import fsutil
fh = open("some-large-file", "rt")
result = list(fsutil.reverse_readlines(fh))

Hopefully this can get you started in debugging this strange performance hit in PyPy.

[mattip]

Thanks for reporting. The reverse_readlines performance problem should be solved on latest nightly builds of pypy3.10. Is there a way you can verify the problem is solved (using tomorrows builds? I see you are using a separate repo for the CI templates that uses pypy3.10. The setup python action supports using nightly builds of pypy by specifying pypy3.10-nightly, but maybe just wait for the new version

[JSCU-CNI]

Thanks @cfbolz and @mattip for your effort in fixing these issues.

I can confirm that autocomplete now works as expected in PyPy 3.10.14 7.3.17. The tabbing behaviour does seem to be slightly different from CPython, but that is mainly a UX difference and fine with me.

The reverse readlines function also seems to be significantly faster now:

$ time pypy example.py 
real	0m31.707s
user	0m31.534s
sys	0m0.068s

$ time pypy example.py
real	0m0.311s
user	0m0.295s
sys	0m0.016s

[cfbolz]

awesome, thanks for the feedback!