Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] implement match backend #22

Merged
merged 68 commits into from
Nov 8, 2016
Merged
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
a6db5b2
[WIP] implement match backend
nirizr Aug 18, 2016
bd3f58c
add celery and empty tasks file
nirizr Aug 25, 2016
a878d5d
Try my multi-table inheritence idea
nirizr Aug 25, 2016
09db5cd
Merge Task model with async tasks and create proper views
nirizr Aug 25, 2016
ad24f3a
Create task when function data upload is done
nirizr Aug 26, 2016
7eb3470
Create base MatchAction class both MatchAll and MatchFunction inherit
nirizr Aug 26, 2016
ccf5a83
call get_functions() in MatchAllAction.get_functions_count
nirizr Aug 27, 2016
34a4280
initialize MatchAction params in __init__
nirizr Aug 27, 2016
36126ae
add match dialog
nirizr Aug 28, 2016
25243e0
fix minor codacy issues
nirizr Aug 29, 2016
3122721
fix flake8 issues
nirizr Aug 29, 2016
8d863f8
Implement simple match sources
nirizr Aug 29, 2016
a290903
Fix single source function match
nirizr Aug 29, 2016
f2ec53e
Add file_id to vectors and an empty loop that pulls the vectors
nirizr Aug 31, 2016
25ddae6
Add progress fields to task
nirizr Aug 31, 2016
1401ac1
fix spaces around '+' in tasks.py
nirizr Aug 31, 2016
e202739
Fix vector file field population
nirizr Aug 31, 2016
1d95efb
Add match task exection's progress bar
nirizr Sep 1, 2016
35f2a4a
Fix target_vectors filter
nirizr Sep 2, 2016
19cffe4
Add task execution progress report and progress bar
nirizr Sep 2, 2016
69b48b6
use BaseDialog's bottom layout
nirizr Sep 2, 2016
d7d17af
explicitly catch all exceptions and raise on MatchAction.perform_task
nirizr Sep 2, 2016
d7b4100
Add task to admin panel
nirizr Sep 2, 2016
eac0bea
fix two base_layout issues
nirizr Sep 5, 2016
a7f6de7
accept null project when creating task
nirizr Sep 5, 2016
f73894d
Accept progress bar when task is complete
nirizr Sep 5, 2016
b02ddd8
Bulk delayed instance requests to speed up the collection phase
nirizr Sep 6, 2016
f2244cd
simplify get_functions_count
nirizr Sep 7, 2016
94a7d75
Create serverside vector classes to reduce boilerplate
nirizr Sep 7, 2016
b05bf87
safer timer use in match action
nirizr Sep 7, 2016
4df4f20
fix codacy minor issues
nirizr Sep 7, 2016
4578c45
add type and score to Match Model
nirizr Sep 13, 2016
975b184
implement match for identity functions
nirizr Sep 15, 2016
4ad58b4
make Vector class inherit and remove VectorHash from list of avaiable…
nirizr Sep 15, 2016
b8958ee
Add space
nirizr Sep 16, 2016
d341729
HistogramVector skeleton
nirizr Sep 16, 2016
042a105
create Match objects when successfully matching
nirizr Sep 16, 2016
8915bfe
Split id to match_type and vector_type, to allow several matches usin…
nirizr Sep 17, 2016
3b4a9d8
Begining of histogram matching
nirizr Sep 18, 2016
9c71fc5
Add scipy and sklearn to requirements
nirizr Sep 20, 2016
85b690e
Mark task as failed on exception (and reraise it)
nirizr Sep 20, 2016
f3e9aac
move Match object creation to tasks.py to avoid cyclic import
nirizr Sep 21, 2016
eb3320e
Add miniconda install to travis when testing server
nirizr Sep 21, 2016
5e24543
finish HistogramVector.match
nirizr Sep 22, 2016
60b8e11
Remove python 3.2 from travis, fix codacy issues
nirizr Sep 22, 2016
afd5b63
Better seperation between server matching algos and database vectors
nirizr Sep 23, 2016
144e51a
Fix new action/dialog related issues introduced in 6913865
nirizr Sep 23, 2016
547b62f
Ditch my through replacement attempt, make matching flow work
nirizr Sep 25, 2016
f529d39
make task progress bar visible without requiring actual progress
nirizr Sep 25, 2016
db2b09b
more progress on making histogram matches work
nirizr Sep 28, 2016
775bd41
Time each match type
nirizr Oct 5, 2016
fbc8078
set Task model progress_max nullable and handle optional None value i…
nirizr Oct 5, 2016
072ccc9
more fixes to histogram match backend
nirizr Oct 5, 2016
d8edd71
Another attempt at speeding up the matches process
nirizr Oct 16, 2016
6633342
more robust status handling in task progress query
nirizr Oct 17, 2016
e1bb374
send more parameters in a match task creation
nirizr Oct 19, 2016
7718f39
More adjustments to the source/target/methods parameters
nirizr Oct 21, 2016
13b696f
Make item selection combobox a dialog.Base method
nirizr Oct 22, 2016
e489af6
Add optional choice-specific additional input to the radio group crea…
nirizr Oct 22, 2016
27da9ff
Send selected target_project and target_file to server
nirizr Oct 22, 2016
90ceb59
Handle target_project and target_file match parameters on backend
nirizr Oct 22, 2016
2c7bf08
Allow excuding values in create_item_select and exclude current file …
nirizr Oct 26, 2016
1b80518
import qt from idasix
nirizr Oct 26, 2016
2d4521a
Add support for range and single function sources
nirizr Oct 27, 2016
b8ef12f
self review fixes part one
nirizr Oct 28, 2016
c31224f
Prevent editing target and source info in existing task
nirizr Oct 28, 2016
6cb948c
Replace create_item_select and create_radio_select with QItemSelect a…
nirizr Oct 30, 2016
41ccc63
Split matches.py to entire directory of match type definitions
nirizr Nov 2, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Create task when function data upload is done
Signed-off-by: Nir Izraeli <nirizr@gmail.com>
nirizr authored and Nir Izraeli committed Nov 8, 2016
commit ad24f3ae703dfd81a76420ee4b5e71b31ccea757
28 changes: 21 additions & 7 deletions idaplugin/rematch/actions/match.py
Original file line number Diff line number Diff line change
@@ -14,37 +14,51 @@ class MatchAllAction(base.BoundFileAction):
group = "Match"

def activate(self, ctx):
self.file_id = netnode.bound_file_id

self.function_gen = enumerate(Functions())
pd = QtWidgets.QProgressDialog(labelText="Processing functions...",
pd = QtWidgets.QProgressDialog(labelText="Processing...\nYou may continue "
"working but avoid ground-"
"breaking changes.",
minimum=0, maximum=len(list(Functions())))
self.progress = pd
self.progress.canceled.connect(self.cancel)

self.timer = QtCore.QTimer()
self.timer.timeout.connect(self.perform)
self.timer.timeout.connect(self.perform_upload)
self.timer.start()

def perform(self):
self.progress.accepted.connect(self.accepted_upload)

def perform_upload(self):
try:
i, offset = self.function_gen.next()

func = instances.FunctionInstance(self.file_id, offset)
func = instances.FunctionInstance(netnode.bound_file_id, offset)
network.query("POST", "collab/instances/", params=func.serialize(),
json=True)

i = i + 1
self.progress.setValue(i)
if (i >= self.progress.maximum()):
self.timer.stop()
self.progress.accept()
except:
self.timer.stop()
raise

def cancel(self):
self.timer.stop()

def accepted_upload(self):
self.timer.stop()
self.timer.disconnect()

# TODO: ask for project to compare against
task_params = {'action': 'commit', 'file': netnode.bound_file_id,
'project': None}
r = network.query("POST", "collab/tasks/", params=task_params, json=True)
print(r)


# TODO: inherit logic in MatchAllAction
class MatchFunctionAction(base.BoundFileAction):
name = "Match &Function"
group = "Match"