Moving from threads to asyncio

[maurosoria]

Hello folks,

I am opening this issue because I do not have the time to do it myself, but I really think that now is the momentot move from threads to an asyncio based application (at least for requests and some callbacks).

I planned from the beginning to do this, but back then asyncio wasn't widely supported by most of linux distros and wasn't as mature like it is today.

This is a very important step, because it would open the door to add extra scanning capabilities to dirsearch, that now seems very difficult with a normal thread scheme.

What is the feature?

Moving from a regular thread-based system to an asynchronous based requests system would improve the performance.

Take into consideration

• The idea is using an asyncio executor for processing Jobs.
• It is important to follow pep8 guidelines and validate the code.
• It might be a good idea considering a model who may, in the future, support scanning multiple hosts at the same time (like meg)
• I guess the proper library for doing this may be aiohttp, but I guess for now it is important to stop using threads and keep using requests.
• It can be very time consuming to code, but maybe using CPU bound executor for certain tasks could make a difference (maybe not).

Obviously, I will assist you if you want to cooperate in all I can, and I will review the code myself.

Thanks for cooperating, and spread the voice!

Regards,
Mauro

[shelld3v]

Should we fully stop using threads and turn into async, or support both?

[shelld3v]

And the fact that it will be really hard to can control the speed if we use asyncio

[maurosoria]

it is possible to mix asyncio with threads, using the proper asyncio executor:

• Python 3 ThreadPoolExecutor

[dbfreem]

@maurosoria I actually have a lot of experience moving code over to asyncio. One thing to keep in mind is it can grow legs really quickly and get complex. Do you have a suggestion on where would be a good section to pull off and move over to asyncio. I would expect to use aiohttp instead of request though, so you get the true benefit of asyncio.

Anyways, thought I would reach out and see if you wall were still moving forward with this, if so if you needed some help.

[maurosoria]

Hello @dbfreem ! I am very grateful for your interest !

IMO Fuzzer class is the best candidate for starting to move code. You can rewrite that class, create a super class BaseFuzzer, and just move the old class to a subclass (e.g. TheadingFuzzer) and you can write another (eg. AsyncioFuzzer).

In that way we can write universal unit tests for both fuzzers (and make other ones in the future!).

the other candidate (which I'm also looking forward to doing that) is the Controller class. But, in that case, is much more complex because you need to completely rewrite everything (report writing, user input, etc).

If you are really into this please do not hesitate in use this or another communication method you prefer (like twitter!)

[dbfreem]

Hey @maurosoria that sounds like a great plan. I plan on working on this over the holidays, so I will keep you up to date on the progress.

[willjprice]

Following

[maurosoria]

Hello! Any news about this ?

[Prady18]

No

…

On Fri, Feb 2, 2024, 1:05 AM Mauro Soria ***@***.***> wrote: Hello! Any news about this ? — Reply to this email directly, view it on GitHub <#643 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR5BMYBGBR6EEJNX4ZAQ5PTYRPU7DAVCNFSM4UJ7WCNKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJSGIYDQMZYGA2A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[DylanTiger]

Hey what are you going to use asyncio and aiohttp or asyncio and httpx? or something else that i ignore? I think that with chatgpt help it won't need much to accomplish it :D

[Prady18]

Good question! ⁉️

…

On Thu, May 16, 2024, 2:52 AM DylanTiger ***@***.***> wrote: Hey what are you going to use asyncio and aiohttp or asyncio and httpx? or something else that i ignore? I think that with chatgpt help it won't need much to accomplish it :D — Reply to this email directly, view it on GitHub <#643 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR5BMYGLINVZYVHH2RTFWVDZCPG2XAVCNFSM4UJ7WCNKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGM2DONZTGUYQ> . You are receiving this because you commented.Message ID: ***@***.***>

[asadmins]

Is there any progress on this project?

[shelld3v]

Is there any progress on this project?

Not yet

[zrquan]

Hey guys, I've submitted a draft PR to support asynchronous mode, testing and reviews are welcome. The speedup from using coroutines is significant, hopefully it can be merged into the master branch soon 😄