[Hexagon] RzIL uplifiting #3837

Rot127 · 2023-09-08T00:01:01Z

SQUASH ME

commit message: Uplift Hexagon architecture to RzIL

The general structure is, that every (sub-)instruction has a getter for it's RzIL code.
Calling the getter will return the RzIL operation.
If RzIL for an instruction is requested, the plugin makes a decision. Because Hexagon only executes whole instruction packets. If the instruction is not the last instruction in a packet, it will simply return EMPTY(). If the RzIL for the last instruction in a packet is requested, it will get the RzIL operations for all instructions in the packet, shuffles them into the correct execution order (according to some rules) and returns the complete operation for the packet.

The RzIL code was entirely generated with the rzil-compiler, using the semantic definition of the QEMU Hexagon module.

Currently successful compile instructions (and tested):

[*] 1581/1733 standard instructions compiled.
[*] 431/643 HVX instructions compiled.
[*] In total: 2012/2376 instructions compiled.

It was tested with:

(Semantic tests) rz-tracetest against the execution trace of the QEMU Hexagon test binaries.
(Bug free and semi-semtantic test) Adding tests which simply execute the test binaries to ensure leak and segfault free execution. Also it is executed until a certain instruction is reached (end of main or loc.pass symbol), partially testing it executes correctly.

For the uplifting several changes and modernization had to be made:

Enhance consistency of decoding
- Allow to disassemble an instruction without copying the result. This is used if the given buffer of instruction bytes is larger than one instruction width. In this case, as many instructions as the buffer can hold are disassembled and buffered for later.
- Generally enhance buffering of instructions.
- Allow to mark a packet as valid before it is completely decoded (in case we know it must be valid, e.g. if it is a jump target of a valid packet).
Fix (hopefully) all memory leaks of the Hexagon plugin.
Changes to register getters, because RzIL needs finer control to translate alias or explicit register names to their real register.
- Getter for register name is now done by table, so for future distinction between DSP version we can just select another table.
- Translation functions from register alias or explicit name to their real register.
- Each operand contains now it's variable ID (e.g. d for register Rd) as in the ISA (for mapping in the RzIL code).
Ease debugging by tracking in more precision, if an instruction is added to a stale, active or new packet.
Add registers C20 - C29 (not yet present in LLVM)
Some renaming to make the code more readable.

Your checklist for this pull request

I've read the guidelines for contributing to this repository
I made sure to follow the project's coding style
I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
I've added tests that prove my fix is effective or that my feature works (if possible)
I've updated the rizin book with the relevant information (if needed)

Detailed description

Hexagon uplifting PR.

TODO after merge

rz-rzilcompiler
- Transfer ownership to RizinOrg
- Tag rz-rzilcompile as v1.0
rz-hexagon
- Change submodule to use RizinOrg repo
- Use tag of rz-rzilcompiler
- Refactor rz-hexagon to build as proper python package.
- Tag rz-hexagon as v1.1

Test plan

Added, all green

Rot127 · 2023-11-13T19:29:52Z

Open this for the first round of review now.

There are still the rz-test cases missing. Due to inner working of the RzIL generation it is not really possible to add the asm style tests (so I check it again to be sure). Instead I would simply execute most test bins to the address where they can be considered passed.

Currently the tests fail due to missing #3973

XVilka

The code looks great. Amazing job!

librz/analysis/arch/hexagon/hexagon_il.c

XVilka · 2023-11-14T13:49:48Z

librz/asm/arch/hexagon/hexagon_arch.c

+			newest = i;
+		}
+	}
+	RZ_LOG_DEBUG("╭─────┬──────────────┬─────┬──────────────────┬───────────────╮\n");


We should have general API for that, inside RzTable.

Couldn't do the last column which is split into 4, which makes it easier to read IMHO. Do you think it is still better? Then I change it. Though it is only debug printing.

It was just a remark, not a requirement. But it would be nice to figure out what's missing in RzTable.

librz/asm/p/asm_hexagon.c

librz/il/il_routines.c

librz/include/rz_il/rz_il_opbuilder_begin.h

Rot127 · 2023-11-21T20:13:40Z

In order that the tests succeed we need rizinorg/rizin-testbins#127 merged though.
Before that maybe look at https://github.com/rizinorg/rizin/pull/3837/files#diff-a1583f59ca1cc7b3c25cde35c24fade6df510112bcf21174d5d2995a5baa14e2

Rot127 · 2023-11-21T20:20:23Z

@thestr4ng3r To your notice: 722e5c0

wargio

mostly is ok. just a stupid detail for me.

librz/arch/asm.c

XVilka

LGTM but it would be nice to extract non-Hexagon changes in a separate PR and merge it first.

Rot127 · 2024-03-21T16:30:05Z

Please let me run all rz-tracetest tests after #4373. Just to be sure.

Add hexagon il code to rouce files. Update generated files Add undocumented ops file. Fix build: Add missing effects. Update generated files. Degrade/remove very common log messages. Rename: p -> pkt and k -> slot Reverse renaming of k, since it is indeed the location in the packet and not the slot. Add slot to HexInsn (needed for RzIL) Uppdate generated files. Disassemble a full packet if it is in the buffer. Allow to get the packet IL ops if they are ready. Only copy disassembly result if requested. Fix typo Fix packet recognition of first and jump target packets. Remove check for theoretically unreachable effects. The assumptions made in this line doesn't need to be true. For Hexagon every write to a register first happens to a temporary register file. Once the whole packet is checked and is valid, the operations are performed. The last step is to sync the the temporary register file with the default one. Update generated files (16 passed test_*) Restructure generated rzil code Update with optimization by resolving compares and tenaries All test_* cases succeed Update source files with new sub-routines Remove pseudo instructions Update generated files (all tests pass). Update source after rebase Fix: Check rw overlap only for x registers. Add RzIL tests for hexagon. Update generated files Remove unit test for further effect after control effect. Update generated files. Fix unit test. After instructions with invalid decoded registers are decoded as invalid, the unit test failed. Fix syntactical changes in tests. Use upper case register names in CC Fix many many mem leaks. Only generate IL code if requested. Enable load_align test. Fix syntax: Remove trailing ; for invalid decodes with parse_bits == 0. Add angle brakcets type annotations. Remove many ressource leaks. Note that not RzAsm and RzAnalysis for Hexagon cannot be freed, because there is currrently no reasonable way to make the plugin thread safe. Add bitmaps Use RzBitmap instead of manually setting bits. Fix type annotations. Triple test timeout for ASAN tests. Fix type annotation. Use PVector instead of Vector to save the HexILOp pointer. Ignore dwarf error for now. Apparently, mixing allocated and static memory in a vector leaks the allocated memory. rip. Replace bitmaps with bitvectors. Ensure packet il stats reset. Remove warning from test

It is dictated by BAP theory that the jumps come last.

Rot127 · 2024-03-22T05:53:29Z

Ok, all good now:

./run-test-sets.sh -i -t essentials,float
test_vspliceb [PASS]
test_vpmpyh [PASS]
test_vminh [PASS]
test_vmaxh [PASS]
test_vlsrw [PASS]
test_vcmpw [PASS]
test_vcmpb [PASS]
test_vavgw [PASS]
test_round [PASS]
test_reorder [PASS]
test_packet [PASS]
test_mpyi [PASS]
test_lsr [PASS]
test_jmp [PASS]
test_hwloops [PASS]
test_hl [PASS]
test_fibonacci [PASS]
test_ext [PASS]
test_dotnew [PASS]
test_cmp [PASS]
test_clobber [PASS]
test_call [PASS]
test_bitsplit [PASS]
test_bitcnt [PASS]
test_abs [PASS]
usr [PASS]
v68_scalar [PASS]
v73_scalar [PASS]
test-vma [PASS]
load_align [PASS]
multi_result [PASS]
overflow [PASS]
first [PASS]
mem_noshuf [PASS]
preg_alias [PASS]
dual_stores [PASS]
mem_noshuf_exception [PASS]
read_write_overlap [PASS]
reg_mut [PASS]
misc [PASS]
fpstuff [PASS]

Matheus-Garbelini · 2024-09-09T09:15:26Z

Hi, just a question. Is the curent dev branch of rizin using latest changes from https://github.com/rizinorg/rz-hexagon and https://github.com/rizinorg/rz-rzilcompiler ?

Rot127 · 2024-09-09T13:30:20Z

@Matheus-Garbelini Yes. If you plan doing research on Hexagon firmware you can ping me on our Mattermost as well.

Rot127 added this to the 0.7.0 milestone Sep 8, 2023

github-actions bot added rz-test RzAnalysis RzAsm-Disassembler labels Sep 8, 2023

github-actions bot added the API label Sep 11, 2023

Rot127 force-pushed the rzil-hexagon branch from 3f283a4 to 16d421e Compare September 22, 2023 23:07

github-actions bot added the RZIL label Oct 1, 2023

wargio changed the title ~~[Hexagon] RzIL uplifitng~~ [Hexagon] RzIL uplifiting Oct 29, 2023

github-actions bot added the RzIO label Nov 13, 2023

Rot127 marked this pull request as ready for review November 13, 2023 19:29

Rot127 requested review from wargio, ret2libc, thestr4ng3r, XVilka and kazarmy as code owners November 13, 2023 19:29

XVilka requested changes Nov 14, 2023

View reviewed changes

Rot127 mentioned this pull request Nov 14, 2023

Add several common bit operations to RzIL. #3977

Merged

5 tasks

Rot127 force-pushed the rzil-hexagon branch from d27ff98 to 662e4d5 Compare November 21, 2023 20:12

Rot127 force-pushed the rzil-hexagon branch from 30f7a36 to 563b98c Compare November 23, 2023 13:52

github-actions bot added RzCore RzUtil labels Nov 29, 2023

Rot127 force-pushed the rzil-hexagon branch 2 times, most recently from 73071a3 to 8f18e32 Compare November 29, 2023 20:48

github-actions bot removed RzCore RzUtil labels Nov 29, 2023

wargio approved these changes Mar 20, 2024

View reviewed changes

wargio reviewed Mar 20, 2024

View reviewed changes

librz/arch/asm.c Outdated Show resolved Hide resolved

XVilka approved these changes Mar 20, 2024

View reviewed changes

Rot127 force-pushed the rzil-hexagon branch from e3e5975 to b4c065f Compare March 21, 2024 16:29

Rot127 added 13 commits March 22, 2024 00:51

Add rz_vector_assign_at()

e0e8dd1

Use rz_asm_parse_param_free and fix two mem leaks

21b523c

Sync with generator.

8176eef

Add Hexagon arch to labeler

6a78b27

Sync with newest generated files

29141a6

Init slot_cancelled

847e7f4

Increase timeout for Windows tests.

0010632

Revert rizinorg@722e5c0

fc7f7c5

It is dictated by BAP theory that the jumps come last.

Sync with lated generated files.

9864429

Remove old bitmap implementation.

8ff2864

Remove derefence after rizinorg#4373

0fcead0

Fix shuffling and setup of jump_target

aa40433

Rot127 force-pushed the rzil-hexagon branch from a74b80c to aa40433 Compare March 22, 2024 05:53

github-actions bot removed RzCore visual panel labels Mar 22, 2024

Fix test

19073e3

Rot127 merged commit fb6efca into rizinorg:dev Mar 22, 2024
44 checks passed

Rot127 deleted the rzil-hexagon branch March 22, 2024 07:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Hexagon] RzIL uplifiting #3837

[Hexagon] RzIL uplifiting #3837

Rot127 commented Sep 8, 2023 •

edited

Loading

Rot127 commented Nov 13, 2023 •

edited

Loading

XVilka left a comment

XVilka Nov 14, 2023

Rot127 Nov 15, 2023 •

edited

Loading

XVilka Nov 15, 2023

Rot127 commented Nov 21, 2023

Rot127 commented Nov 21, 2023

wargio left a comment

XVilka left a comment

Rot127 commented Mar 21, 2024

Rot127 commented Mar 22, 2024

Matheus-Garbelini commented Sep 9, 2024

Rot127 commented Sep 9, 2024

[Hexagon] RzIL uplifiting #3837

[Hexagon] RzIL uplifiting #3837

Conversation

Rot127 commented Sep 8, 2023 • edited Loading

SQUASH ME

Rot127 commented Nov 13, 2023 • edited Loading

XVilka left a comment

Choose a reason for hiding this comment

XVilka Nov 14, 2023

Choose a reason for hiding this comment

Rot127 Nov 15, 2023 • edited Loading

Choose a reason for hiding this comment

XVilka Nov 15, 2023

Choose a reason for hiding this comment

Rot127 commented Nov 21, 2023

Rot127 commented Nov 21, 2023

wargio left a comment

Choose a reason for hiding this comment

XVilka left a comment

Choose a reason for hiding this comment

Rot127 commented Mar 21, 2024

Rot127 commented Mar 22, 2024

Matheus-Garbelini commented Sep 9, 2024

Rot127 commented Sep 9, 2024

Rot127 commented Sep 8, 2023 •

edited

Loading

Rot127 commented Nov 13, 2023 •

edited

Loading

Rot127 Nov 15, 2023 •

edited

Loading