Event export serializer #3543

tevoinea · 2023-10-03T15:19:25Z

Summary of the Pull Request

What is this about?

This PR introduces a new JsonConverter for sending events. It closes #3076 and closes #3381.

Info on Pull Request

What does this include?

A new feature flag EnableSlimEventSerialization which allows us to toggle the new behavior
A new JsonConverter EventExportConverter
Send 'slim' events to signalr queue and to webhooks queue
Events logged to AppInsights are not slim (useful for debugging)
Events stored in the events container are not slim (so that they can be retrieved in full capacity later via event id)
Tests

Validation Steps Performed

How does someone test & validate?

Unit tests for the JsonConverter
Manual testing for receiving webhooks

* Remove the retention policy setting (microsoft#3452) --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

This reverts commit c69deed.

* Redo-8.7.1-hotfix --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

…t#3467) * Add field to ado config for checking duplicate work items * Make duplicate fields nullable and add it to python models * Update broken tests * Update docs to include new ado_duplicate_fields property

Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [elsa](https://github.com/manishearth/elsa) from 1.8.1 to 1.9.0. - [Commits](Manishearth/elsa@v1.8.1...v1.9.0) --- updated-dependencies: - dependency-name: elsa dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0. - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](Stebalien/tempfile@v3.7.1...v3.8.0) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Updating requirements.txt to accept >= onefuzztypes. * Trying to loosen restriction.

Bumps [notify](https://github.com/notify-rs/notify) from 6.0.1 to 6.1.1. - [Release notes](https://github.com/notify-rs/notify/releases) - [Changelog](https://github.com/notify-rs/notify/blob/main/CHANGELOG.md) - [Commits](notify-rs/notify@notify-6.0.1...notify-6.1.1) --- updated-dependencies: - dependency-name: notify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Release 8.8.0

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [gimli](https://github.com/gimli-rs/gimli) from 0.27.3 to 0.28.0. - [Changelog](https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md) - [Commits](gimli-rs/gimli@0.27.3...0.28.0) --- updated-dependencies: - dependency-name: gimli dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [winreg](https://github.com/gentoo90/winreg-rs) from 0.50.0 to 0.51.0. - [Release notes](https://github.com/gentoo90/winreg-rs/releases) - [Changelog](https://github.com/gentoo90/winreg-rs/blob/master/CHANGELOG.md) - [Commits](gentoo90/winreg-rs@v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: winreg dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

* Starting integration tests * Ready to test the test * Parametrize test * checkpoint * Test works * Run integration tests in pipeline * fmt * . * -p * Install clang * quotes not required in yaml? * Hopefully fixed windows? * Try without killondrop * lint * small test * another test * Reuse core name * Wrong step * bump tokio? * Try with rust * make build happy * Bump pete and small clean up * Clean up and make the test pass regularly * fix broken ci * Lower the poll timeout * Set the timeout in a nicer way * fix windows * fmt * Include and copy pdbs * Ignore if pdb is missing on linux * It takes too long for coverage to be generated * lint * Only warn on missing coverage since it's flaky * Fix windows build * Small clean up * Try lowering the poll delay * fix coverage * PR comments * . * Apparently make is missing? * Remove aggressive step skipping in CI

* Fix sed checks for CLI versioning * Fix. * Fix. * Changing build_cli * Trying greater than * Tring once more. * Trying major minor * trying to replace major minor * Using major minor

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add more comprehensive checks and better error messages to area/iteration path validation * Join invalid chars with space instead of comma * Make tree path validation more testable * Add error code for invalid ADO project in config * Write unit tests for tree path validation * Format tree path unit tests * Merge escape character and control character checks and clarify error message

* Improve handling of unexpected breakpoints * fmt

* Fuzz coverage recording * Update cargo.toml * Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs Co-authored-by: George Pollard <porges@porg.es> * Fix fuzz --------- Co-authored-by: George Pollard <porges@porg.es>

* Reporting coverage on task start up * Moving metric up.

* Compare task version to service version * Swallow output when looking for appropriate name

* Create directories if they don't exist in the template * fmt

- [x] ability to specify a retention period on a container, which applies to newly-created blobs - [x] specify default retention periods in templates from CLI side There's a small breaking change to the Python JobHelper class.

Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.7.0 to 1.8.0. - [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md) - [Commits](rayon-rs/rayon@rayon-core-v1.7.0...rayon-core-v1.8.0) --- updated-dependencies: - dependency-name: rayon dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [insta](https://github.com/mitsuhiko/insta) from 1.31.0 to 1.32.0. - [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md) - [Commits](mitsuhiko/insta@1.31.0...1.32.0) --- updated-dependencies: - dependency-name: insta dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Disable and VM CLI commands. * Formatting. * More formatting. * More formatting. * Removing Repro check.

* Make modules and coverage allowlist case insensitive on Windows * Tests and fmt * PR comments * fmt * Debugging missing file coverage * fmt * Broken linux test * Add a case insensitive transformer for better perf * cargo fix

codecov-commenter · 2023-10-03T15:37:15Z

Codecov Report

Merging #3543 (49c0c54) into main (e7b0dd5) will increase coverage by 0.07%.
The diff coverage is 83.07%.

@@            Coverage Diff             @@
##             main    #3543      +/-   ##
==========================================
+ Coverage   38.97%   39.05%   +0.07%     
==========================================
  Files         301      302       +1     
  Lines       36848    36903      +55     
==========================================
+ Hits        14363    14413      +50     
- Misses      22485    22490       +5

Files	Coverage Δ
...Service/ApiService/onefuzzlib/WebhookOperations.cs	`37.43% <77.77%> (+1.72%)`	⬆️
...Service/ApiService/onefuzzlib/events/Converters.cs	`91.66% <91.66%> (ø)`
src/ApiService/ApiService/onefuzzlib/Events.cs	`69.23% <37.50%> (-1.46%)`	⬇️

... and 4 files with indirect coverage changes

…inea/CustomEventSerializer

Default off

…evoinea/onefuzz into tevoinea/CustomEventSerializer

AdamL-Microsoft and others added 30 commits August 29, 2023 19:57

Release 8.7.1 (hotfix) (microsoft#3459)

c69deed

* Remove the retention policy setting (microsoft#3452) --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

Revert "Release 8.7.1 (hotfix) (microsoft#3459)" (microsoft#3468)

c8986aa

This reverts commit c69deed.

Redo 8.7.1 (microsoft#3469)

7b40402

* Redo-8.7.1-hotfix --------- Co-authored-by: Cheick Keita <chkeita@microsoft.com>

Support custom ado fields that mark work items as duplicate (microsof…

d999603

…t#3467) * Add field to ado config for checking duplicate work items * Make duplicate fields nullable and add it to python models * Update broken tests * Update docs to include new ado_duplicate_fields property

Update readme with archive message (microsoft#3408)

b2435b1

Co-authored-by: Adam <103067949+AdamL-Microsoft@users.noreply.github.com>

Remove unnecessary method argument (microsoft#3473)

f141050

Updating requirements.txt to accept >= onefuzztypes. (microsoft#3477)

b2e6a07

* Updating requirements.txt to accept >= onefuzztypes. * Trying to loosen restriction.

Bump azure_* crates (microsoft#3478)

74475cc

Release 8.8.0 (microsoft#3466)

64699ed

* Release 8.8.0

Fix sed checks for CLI versioning (microsoft#3486)

830b479

* Fix sed checks for CLI versioning * Fix. * Fix. * Changing build_cli * Trying greater than * Tring once more. * Trying major minor * trying to replace major minor * Using major minor

Improve handling of unexpected breakpoints (microsoft#3493)

d009476

* Improve handling of unexpected breakpoints * fmt

Update azure_* crates (microsoft#3503)

18f2b4a

Fuzz coverage recording (microsoft#3322)

9ede0de

* Fuzz coverage recording * Update cargo.toml * Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs Co-authored-by: George Pollard <porges@porg.es> * Fix fuzz --------- Co-authored-by: George Pollard <porges@porg.es>

Reporting coverage on task start up (microsoft#3502)

cde6a19

* Reporting coverage on task start up * Moving metric up.

Remove feature flag from heartbeat metrics. (microsoft#3505)

1fb1563

Update archive notice. (microsoft#3507)

c7a9827

Add onefuzz service version to job created events (microsoft#3504)

58da7b4

tevoinea and others added 10 commits September 21, 2023 17:31

Tevoinea/add version checking in local tasks (microsoft#3517)

60766e6

* Compare task version to service version * Swallow output when looking for appropriate name

Create directories if they don't exist in the template (microsoft#3522)

e3c4a40

* Create directories if they don't exist in the template * fmt

Disable repro and debug VM CLI commands. (microsoft#3494)

d2ba170

* Disable and VM CLI commands. * Formatting. * More formatting. * More formatting. * Removing Repro check.

Make modules case insenstive on windows (microsoft#3527)

2c8ecc9

* Make modules and coverage allowlist case insensitive on Windows * Tests and fmt * PR comments * fmt * Debugging missing file coverage * fmt * Broken linux test * Add a case insensitive transformer for better perf * cargo fix

Update windows interceptor list (microsoft#3528)

e12b41e

Starting to work on custom converter

f8f4461

Add slim serializer for events export

c1ee35b

chkeita force-pushed the main branch from 434a435 to 7b79dd6 Compare October 3, 2023 18:56

tevoinea added 2 commits October 4, 2023 09:10

.

252e795

Merge branch 'main' of https://github.com/microsoft/onefuzz into tevo…

7387ada

…inea/CustomEventSerializer

tevoinea marked this pull request as ready for review October 4, 2023 17:50

tevoinea and others added 2 commits October 5, 2023 10:30

Cleanup and better tests

701025b

Merge branch 'main' into tevoinea/CustomEventSerializer

9a214e3

tevoinea requested review from chkeita and Porges October 6, 2023 14:17

chkeita approved these changes Oct 6, 2023

View reviewed changes

Merge branch 'main' into tevoinea/CustomEventSerializer

9899410

tevoinea added the passed-integration-tests label Oct 6, 2023

tevoinea added 2 commits October 10, 2023 09:21

Update feature-flags.bicep

5ea3ccc

Default off

Merge branch 'main' into tevoinea/CustomEventSerializer

9cd0dfa

tevoinea enabled auto-merge (squash) October 10, 2023 13:21

tevoinea added 2 commits October 10, 2023 17:15

Fix api.py

bbdcb8d

Merge branch 'tevoinea/CustomEventSerializer' of https://github.com/t…

49c0c54

…evoinea/onefuzz into tevoinea/CustomEventSerializer

tevoinea merged commit 2f2a649 into microsoft:main Oct 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Event export serializer #3543

Event export serializer #3543

tevoinea commented Oct 3, 2023 •

edited

Loading

codecov-commenter commented Oct 3, 2023 •

edited

Loading

Event export serializer #3543

Event export serializer #3543

Conversation

tevoinea commented Oct 3, 2023 • edited Loading

Summary of the Pull Request

Info on Pull Request

Validation Steps Performed

codecov-commenter commented Oct 3, 2023 • edited Loading

Codecov Report

tevoinea commented Oct 3, 2023 •

edited

Loading

codecov-commenter commented Oct 3, 2023 •

edited

Loading