local_run refactoring #508

chkeita · 2021-02-04T19:58:37Z

Summary of the Pull Request

This is a refactoring of the local debugging.

The input queue as been abstracted and can now be locally
The SyncDir can now sync a local directory
Added the ability to monitor a local directory with a queue

Reviewers Notes

The most import part of the PR are

The queue abstraction and implementations
- src/agent/storage-queue/src/azure_queue.rs
- src/agent/storage-queue/src/local_queue.rs
- src/agent/storage-queue/src/lib.rs
Changes to support local files in
- src/agent/onefuzz/src/syncdir.rs
Example of how those abstractions are used to link task together
- src/agent/onefuzz-agent/src/local/libfuzzer_fuzz.rs
- src/agent/onefuzz-agent/src/local/common.rs

Validation Steps Performed

How does someone test & validate?

- Receipt i now an internal class - message instance can now delete themselves -

adding retry to local queue operations fix storage type detection

fix queue monitoring

wiring local libfuzzer, libfuzzer_crash_report, libfuzzer_coverage

src/agent/onefuzz/src/syncdir.rs

src/agent/onefuzz/src/fs.rs

bringing back renam instead of copy for moving files

src/agent/onefuzz-agent/src/local/libfuzzer.rs

src/agent/reqwest-retry/src/lib.rs

…-run add send_retry_default to local run's download_input

ghost · 2021-03-17T19:00:37Z

Hello @bmc-msft!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

chkeita force-pushed the chkeita/local_run branch from aa003eb to 16be7a4 Compare February 9, 2021 02:53

chkeita force-pushed the chkeita/local_run branch from 16be7a4 to 084d0d9 Compare February 16, 2021 18:31

chkeita added 28 commits February 22, 2021 18:54

moving sync operation inside SyncDir

1faaa67

Refactor queue management

19e3fae

- Receipt i now an internal class - message instance can now delete themselves -

more refactoring

359969e

WIP local queue implementation

216571e

Fix build

7f54250

build fix

d6ae489

removed unused imports

ebc797c

move auth to onefuzz

74e6c9b

handle queue authentication

10f2b2e

WIP converting local fuzzing templates

8be75a8

build fix

68a5fd9

local queue sender and receiver are now create per requests

b3d352e

adding retry to local queue operations fix storage type detection

fix url scheme check

bacb202

fixing unit tests

ad12f53

formatting

04a0de2

utlis::download_inputs supports file url

c8c4951

fix queue monitoring

support local files in BlobUrl

a2a11a2

updating local lib fuzzer crash report

9503f0d

build fix

b14ab1a

formatting

6de303a

formatting

bb8d459

fix tests

a6ee53f

convert blobUrl to enum

1d5485a

formatting

21bf38b

fix clippy issues

9d7f395

fix test

6e1802d

introducing DirectoryMonitorQueue struct

b55f5a3

wiring local libfuzzer, libfuzzer_crash_report, libfuzzer_coverage

bring back message ids

2cd5231

bmc-msft reviewed Mar 12, 2021

View reviewed changes

src/agent/onefuzz/src/syncdir.rs Outdated Show resolved Hide resolved

bmc-msft reviewed Mar 12, 2021

View reviewed changes

src/agent/onefuzz/src/fs.rs Show resolved Hide resolved

chkeita mentioned this pull request Mar 12, 2021

Do an early-checking the platform specific tools (robocoby or rsync) from here, and cdb or gdb for the crash reporting tasks during initialization. #669

Open

chkeita and others added 10 commits March 11, 2021 18:28

rename _sync to sync_impl

b3aac9b

added unit tests

7a503e6

Merge branch 'main' into chkeita/local_run

9b3b730

bringing back bail when upload fail but

741660c

introduced a type to distinguish between dir and file path

6a5dd24

bring back comment

ca62a9b

update to yaque 0.5.1

2bde835

Merge remote-tracking branch 'upstream/main' into chkeita/local_run

4c4286f

Creating temporary folder inside the libfuzzer task dir

747fa6d

bringing back renam instead of copy for moving files

Merge branch 'main' into chkeita/local_run

0fe1240

chkeita commented Mar 15, 2021

View reviewed changes

src/agent/onefuzz-agent/src/local/libfuzzer.rs Show resolved Hide resolved

Merge branch 'main' into chkeita/local_run

f911db0

bmc-msft reviewed Mar 15, 2021

View reviewed changes

src/agent/reqwest-retry/src/lib.rs Outdated Show resolved Hide resolved

bmc-msft and others added 6 commits March 15, 2021 19:20

Update src/agent/reqwest-retry/src/lib.rs

3f7aee3

merge fix

43d11fb

Merge branch 'main' into chkeita/local_run

53ab0b7

add send_retry_default to local run's download_input

0c05a2f

Merge pull request #2 from bmc-msft/add-retry-to-download_input-local…

e9f53a4

…-run add send_retry_default to local run's download_input

Merge remote-tracking branch 'upstream/main' into chkeita/local_run

44bd9bb

bmc-msft added auto-merge passed-integration-tests labels Mar 17, 2021

ghost merged commit 4b07fdc into microsoft:main Mar 17, 2021

chkeita deleted the chkeita/local_run branch March 31, 2021 21:53

chkeita linked an issue Apr 19, 2021 that may be closed by this pull request

Local run refactoring #670

Closed

chkeita mentioned this pull request Apr 19, 2021

Local run refactoring #670

Closed

ghost locked as resolved and limited conversation to collaborators May 1, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

local_run refactoring #508

local_run refactoring #508

chkeita commented Feb 4, 2021 •

edited

Loading

ghost commented Mar 17, 2021

local_run refactoring #508

local_run refactoring #508

Conversation

chkeita commented Feb 4, 2021 • edited Loading

Summary of the Pull Request

Reviewers Notes

Validation Steps Performed

ghost commented Mar 17, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

chkeita commented Feb 4, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.