radiff2 diff offsets too!!! not just opcodes. a recompiled file will be "different"

[john-peterson]

https://github.com/radareorg/radare2/issues?q=radiff2

I scanned the link above for general usage

But still didn't understand anything apparently. Had to delete most of what I wrote

The basic improvements are

• Look for the corresponding function in ADJACENT ADDRESSES ONLY. The compiler is not creative! It won't rearrange the function into a complete jumbled. It's still ALMOST the same

• FREE MEMORY AFTER EVERY FUNCTION I hope I didn't embarrass or turn everyone off here. A diff should use ten meg not ten gig

• DO AN ACTUAL DIFF this is the most severe error radiff2 is diff ONLY IN NAME it's like naming a downy Einstein. It doesn't actually diff any function. It will print UNMATCHED (or even NEW?) for unchanged functions. It's confused by offsets of the same code. It can't do a pure opcode diff. Yet. I hope to fix this if I can.

!! NOTICE that I AM aware of -O and used that. Still a gazillion false positives !! I have not tried everything like

asm.imm.trim

???? Can I a scan only a portion? For testing it's impossible to wait for the whole file. WHY IS IT NOT WRITING INCREMENTALLY????

The command I used is

radiff2 -G 'e anal.vars=0;e anal.hasnext=1;afr;aac' -COj 0/Metropolis_x64_vk.exe 1/Metropolis_x64_vk.exe 2>/dev/null >exe.json.all&

In the unlikely event that someone actually cares about this you can obtain the files from

steamcmd
download_depot 644830 644832 3895879220004142024
download_depot 644830 644832 7595559457580866760

It's one of these annoying cases where the debug flags are leaked and the imbecile managers ask for MORE SECRET DEBUG FLAGS. Like that has any financial impact whatsoever. They needed to print the collision mesh to have any chance at testing the game. But they added more secret flags in the input files to block that from curious outsiders... Pointless waste of time. And a god mode to skip past, run past, portions of the game. The teleport function is so bad it's essentially useless for moving between levels. Not to mention that it hangs if you leave a level... Etc

I used to work in Fatburen Södermalm Stockholm but left professional work years ago for obvious reasons

It took six hours and hanged the machine with four gig. I had to reboot with six gig and it completed. Several hundred UNMATCHED including the culprit that was matched 1% except having only four new imports in 3000 opcodes or whatever the afi "size" unit is. (Could be 3000 bananas as far as I know)

    "addr": 5370580832,
    "name": "fcn.1401c8f60",                    "size": 26530,                              "diff_addr": 5370582896,                    "diff_name": "fcn.1401c9770",               "diff_size": 27068,                         "match": "UNMATCH",                         "dist": 0.024924

A function summary diff was pointless. I tried to manually do a function reference diff but it was impossible to manually excluded all false positives

In the end a reference or summary diff would have revealed the diff. Of hundred thousand functions one function had four new references to a single external function. That was the culprit. But it was buried in hundreds of false positives. So I had to discover it by painstaking debugging in gdb (it's an exe fully wine compatible but only mingw gdb has officially working support for that at the moment. I very much hope to port mingw gdb win32 ABI patches to radare. Believe me I am your friend. It's painful to listen to me but don't let that confuse you about my intentions)

And the fact that aac is non functional. aar FROM ENTRY was needed to discover the function at all... So whatever you gave radiff. No matter how many A you passed it it would have been futile. -G aar was necessary. Unfortunately that Information would have helped me. Maybe you can put that among the welcome message (not being sarcastic)

---

This is a portion of my original confused message. Mostly deleted

50% of the time I am MISSING something so please correct me

The target is a large file with few differences

Maybe a picture wilyhelp someone scanning similar discussion. A picture says a thousand words right

I think this is really simple

---------- ~~ qqqq~~~~~

Loop over symbols and show

1 a table over changed functions
2 highlighted difference. Only basic ASM

~~---->>>>>

It seems to kind of do tha already from the pictures I've seen but the big difference is NO ANALYSIS just read the data

As you can tell almost out of 25 gig ram. There is nothing to analyse. Most opcodes are exactly the same. But a few needles in a haystack that I need to dig up. But I always look at the plank in my own eye before the straw in yours

Bla bl bl. ...

...

Bla bla bl

Blablabla

[trufae]

There's an issue with analysis-less diffing, which is, that you don't know before hand how large a symbol or a function is. You can take the assumption that a symbol will end when the next symbol is found or before the next section is found.

You can diff the listings of symbols/imports/.. without analysing the binary at all with the -i flag:

-i [ifscm] diff imports | fields | symbols | classes | methods

Also you can do a full disasm diffing with -D which also doesnt require analysis, but in order to find the differences between two symbols you need to know the following:

• symbol size
• basic blocks

So i agree that it will be desirable to have a way to perform a fast symbol code diffing that doesnt require basic block analysis, and asuming symbol size = R_MIN (next symbol, next_section) - symbol_addr it can be possible in the same way as -D works right now.

Another option is to perform the full -D diffing (using delta diffing, which is quite expensive compared to lineaar diffing) and them map each offset with the bin symbol associated (the nearest above).

So yeah, thanks for opening the discussion before jumping into the code :)