Added recursion support

[amadejpapez]

Want to search the whole folder and see what all of the files in that folder contain? pyWhat can now do exactly that!

• if input is a directory, recursively search through it (Using a folder path as input #43)
• now that pyWhat is searching for multiple files at once, if a file is identified with magic numbers, it will also print the name of the file, so it will be easier to see which one is identified
• it pyWhat is printing matches from multiple files, it will also print in which file each match is located in a new Filename row (if input is only text, this row is not there)
• API now by default also checks if input is a file or a folder. If you want to disable this and only treat input as text, add True to identify() like this identify("file.txt", True)
• hidden file found with recursion has a CTF flag in the name? pyWhat is now also searching for matches inside of filenames!
• added recursion support to the README and to --help command
• added links to all of the badges in README (Strange behaviour of README badges #60)
• Discover Number can no longer be matched inside of a string

[bee-san]

Can I get a screenshot to see what this looks like, pls? 💖

[amadejpapez]

Can I get a screenshot to see what this looks like, pls?

[bee-san]

Do you think it'd be better if we printed what file it came from in the table as a new column?

[amadejpapez]

Do you think it'd be better if we printed what file it came from in the table as a new column?

Oo that's a good idea but I don't think there is enough space to add another column. Maybe a separate table for each file would be better?

[bee-san]

Do you think it'd be better if we printed what file it came from in the table as a new column?

Oo that's a good idea but I don't there is enough space to add another column. Maybe a separate table for each file would be better?

I don't there is enough space to add another column

Why not? Most lines continue onto the next line if its too long? 😄

[amadejpapez]

Why not? Most lines continue onto the next line if its too long?

Yeah you are right, there is enough space 😁

[amadejpapez]

Okay @bee-san I think this PR is done now. I have reverted all of the changes to tests, so they all work as before. Also pyWhat is now searching filenames, so if there is a file found with recursion that has a CTF flag in the name, pyWhat will be able to identify it. :D

[bee-san]

Conflicting merge issues 😢 Did you say that links didn't work on this PR? 🙏🏻

[amadejpapez]

Conflicting merge issues Did you say that links didn't work on this PR? 🙏🏻

Just saw the new release yes, so will fix them! 😁

About the links, this issue is affecting the latest version of pyWhat and not only this PR. Links only work for some terminals - I tested in Arch and Kali, and they do not open the full URL but only the visible part. This PR is not causing this but when tests broke, I noticed that links do not open for me. Here is a screenrecording from Kali, running pyWhat from PyPi, so you can see what is happening:

github-issue.mp4

[amadejpapez]

I have fixed almost every test, but as soon as I add recursion support, filtration brakes. I just cannot get my head around why.. Could you @piatrashkakanstantinass please take a look at this as you have written filtration? 🥺

If I run pywhat --include_tags finance fixtures/file, distribution in regex-identifier.py prints as Distribution({'Tags': {'finance'}... (what it should be) but in my version it is Distribution({'Tags': {'password', 'credentials', 'hacking', 'geo-location', 'token', 'media', 'finance', 'internet', 'cryptocurrency', 'username', 'networking', 'website', 'security', 'aws', 'cyber security', 'identifiers', 'credit card'}... so it is not seeing what to filter and it just returns all of the regexes.

[bee-san]

they do not open the full URL but only the visible part.

This is actually very normal and quite expected, hyperlinks are not supported in all terminals.

This is why it is essential that we show the full link with no cut-off or new lines to make it work everywhere :)

[bee-san]

What if we removed:

Click here to analyse in the browser

So there was more space? Or we changed the wording?

[amadejpapez]

So there was more space? Or we changed the wording?

This would sadly not fix it. The only way I was able to get it to work for me is if the URL is from the start to the end in one line, without ....

If we did something like this:

This makes links to work unless they again go over the line... 😢 Maybe we could add recursion to pyWhat and make this into an issue 😄

Fixed.