User data collection

[Rot127]

I was talking with a designer the other day and talked about how to build systems which follow a "biological pattern". We used "biological pattern" to describe a system which allows to let things die which are no longer useful or are too complex to use (in the latter case it could be rebuild).

The question is how to find out what can be removed/needs a rebuild and what not.
Simple Github issues and PRs don't necessarily solve the problem, because user might request/build a solution for their problem which might not be the best solution ("best" is relative but I leave it undefined here for brevity).

Apparently in the design world the way to build such a system is to collect data from the user to figure out:

• Which features are used and which not
• How they are used or for what they are used for
• Who the user is who generates this data (this is important since for example a professional reverse engineer possibly uses Rizin features much different than a CTF player and has different needs).

The question I wanted to put out there is:

Should we implement a data collection system for this kind of data?
It could greatly improve Rizin in the long term since we can add or remove features on the basis of measurable reason.
Also we can improve the user experience but also can more easily control which features have priority (depending on what user basis we want to focus on or what is in high use).

Obviously this system must be 100% opt-in and users should be the only one who can donate their usage data.

First idea would be something like this:

• A config flag is set if data should be collected.
• Users can anonymously upload the log file somewhere.
• Add details about there profession and their use cases for Rizin.

What do you think?

[ret2libc]

It's a while I think about something like this but I'm always afraid it won't be too well received by people tbh. These kind of systems where you capture what users do are usually seen with a bit of skepticism afaik.

That said, it would really help a lot... even just knowing the number of times each command is executed, without even their arguments, could help

[Rot127]

These kind of systems where you capture what users do are usually seen with a bit of skepticism afaik.

They are for sure and for a good reason IMO.
Still, couldn't it be designed/build in a way that:

• It only collects the data the user chooses to (so it is adaptable to his/her threat model)
• Data collection code can be reviewed/verified easily.
• Does not collect certain identifiable data e.g.: binary names, function names, comments etc. (although what is identifiable and what not is hard to get right).

... even just knowing the number of times each command is executed, without even their arguments, could help

Maybe this could be implemented at first and, if deemed as useful, be extended.

[Heersin]

Interesting! It could be a separate plugin for collecting data. There is a paper[1] describe an experiment about collecting info when user do reverse tasks. You may take it as a reference.

[1] Burk, Kevin, et al. "Decomperson: How Humans Decompile and What We Can Learn From It." 31st USENIX Security Symposium (USENIX Security 22). 2022.

[Rot127]

To give this topic another spin.

It would be really good to know what part of the API is used by developers for their scripts/tools.
Open sourced tools (especially the small hacky ones) suffer often from becoming out of date.
Because they only run on some old API (sorry this works only with IDA pre-v7 etc.).

None-public software on the other hand suffers from changing APIs, but no way to communicate that they rely on it.

But some API changes are imminent at some point. Because Rizin is so big.

Now, we could add a script which scans your code for use of API functions and reports them to our server. The server can have a public statistic which APIs are used the most.

This way we can make informed decisions in the future which APIs we can change and which we can't (or have to be careful with).

The script can be very simple (basically a grep over all c files) and hence easy to audit.
The data can also be reported anonymously.

Problem could be forgery of course.