Defender functional test plan fixes #1057
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
schrolla
added
bug
1 task
schrolla
force-pushed
the
1056-defender-functional-test-plan-tests-not-configured-correctly
branch
from
9b9e759 to
bf8b2b5
Compare
Sloane4
approved these changes
Apr 11, 2024
|
@schrolla @Dylan-MITRE I can't run against the g3 tenant, so if someone can confirm that it runs without errors, that would be great! |
Dylan-MITRE
approved these changes
Apr 11, 2024
schrolla
force-pushed
the
1056-defender-functional-test-plan-tests-not-configured-correctly
branch
from
bf8b2b5 to
544009c
Compare
|
@nanda-katikaneni Reviewed, rebased, and ready for merge. All tests passed. No smoke test run as this does not change any ScubaGear code, only the functional tests separate from it. |
nanda-katikaneni
deleted the
1056-defender-functional-test-plan-tests-not-configured-correctly
branch
ahuynhMITRE
added a commit
that referenced
this pull request
May 6, 2024
* (New) Add Licensing Information to AAD Report (#1011) * Copied Changes form broken branch for new PR * removed trailing whitespace * Removed more trailing white space * updated licensing table to have unique id and added case in smoke test for licensing table * remove white space * added check for licensing information to functional tests * added smoke test for licensing * Add a UTC timestamp to the Provider JSON (#1009) * add a quick timestamp * add milliseconds * timestamp_zulu * add timestampzulu to metadata in mergejson * fix unit tests * fix bug found in previous PR * Add config file for GitHub Pages (#1001) * add YML for theme * add config for pages * lint * put regal yaml back * Create a workflow for bumping the ScubaGear module version (#989) * create workflow for automated version bumping * clean up whitespace * bump checkout version * fix variable substitution and regex match output * remove testing variable * Fix the Handful of AAD unit tests that are broken (#1019) * fixed unit tests :) * Resolved Linter Warnings --------- Co-authored-by: Devesh Agarwal <dagarwal@mitre.org> * Remove default vars from test configs (#996) * minor edits to aad.5.3v1 and aad5.4v1 instructions (#1028) * uncenter the text (#1031) * Add option to generate a config file template (#984) * config file generation * Adding defender * Final generate config file * Ready for pull request * moved to support module * added to manifest file * pascal case and whitespace * linter hacks * linter test * "unused" variables * linter hacks part II * linter hacks part III * Unit Test added * added default value for unit testing various syntax fixes as well * fixed unit test, mock converting to yaml * fixed unit tests? * fixed? * more debugging * debugging * Update README.md for generating config * Update ScubaGear version to 1.2.0 (#1029) Co-authored-by: GitHub Action <action@github.com> * Fix bug with msaad52v1 only admins consent to apps (#1043) * uploading for Cassey debugging session * modified Rego policy 5.2 and unit tests to account for Microsoft updates to tenant output data * modified 5.2 functional tests based on Microsoft changes to JSON output * Update sample reports to latest version output examples (#1058) * Fix publish package errors (#1042) * inject write-error * comment out publish * show manifest error * remove fail silently * debug path version * fix id typo * fixed list * fix needs * fix release prep again * write prereleasetag * get types * fix typo * fix typos * set tags as array * reset tags * debug w string * remove array * gc manifest * get childitem * write module path * Debug manifest * debug more * typecast to string * debug string concat * display hashtable * more debug details * tweak debugging * update install w prerelease * commenting * install required version * add debug * add debug * add version * use 1 if * hard code * install only * force it * remove install * add debug statements * see params * fix parameter set * fix comment typo * use binding * add comments * fix @ * remove cmdlet * uncomm param set * remove param sets * split into 2 * fix push paths * add env back * clean debug statement * fix pipeline issues * add output test * use write-output * clean up minor * back to previous version * use output * more output * use debug * more output * fix ps lint * test returning false * return to old error * hardcode params * fix step name * improve debug statements * back to write host * debug prerelease * changed info to host * switch tag version * write manifest out * update version * fix temp print * bump to 8 * add import back * require version * bump to 10 * bump vers * without tag * bump tag * use find mod * fix unit test * bump tag * trivial change to trigger tests * hide error * v 03 * fix pipeline * return false * add write error * remove details * be silent * be false * cleanup * unhardcode * Exclude Write-Host locally * commented out push trigger * comment out push trigger * update description * Add a Workflow to check for OPA version updates (#1004) * add parseable language * add OPA Update workflow * remove white space * grammar * Update Pull Request Description Co-authored-by: Addam Schroll <108814318+schrolla@users.noreply.github.com> --------- Co-authored-by: Addam Schroll <108814318+schrolla@users.noreply.github.com> * Defender functional test plan fixes (#1057) * Remove IsNotChecked flag from G3 3.1 Non-compliant - No defender license test * Move impersonation protection tests from standard to g5 variant * Fix Markdown in readme (#1068) * fix table * fix links * minor changes * Add Policy Group Names to ScubaResults.json (#1041) * package Report Results by control group * finish comment * upgrade the ScubaGear ScubaResults metadata with even more context * make comment clearer * make another comment clearer * Bump OPA version from v0.61.0 to v0.63.0 (#1070) * Bump OPA version from v0.61.0 to v0.63.0 * also add support for 0.62.1 * add omit comments * add back in the space * add back in the link --------- Co-authored-by: GitHub Action <action@github.com> Co-authored-by: buidav <105074908+buidav@users.noreply.github.com> * Add GitLeaks to CI/CD Pipeline (#1066) * add if * update pipeline * remove push * set permissions * Enhance AAD Provider and Rego Code to automate checks for policy MS.AAD.3.3v1 (#1014) * Update provider json for auth method config * Added new json for MS auth feature settings * Combined authentication method calls into one json object * Updates to JSON for aad 3.4 and 3.5 * updates to aad 3.3 rego * Include all auth method configs in json export * Updates to aad 3.3 to check msauth settings * Updates to unit tests for aad 3.4 * Update AADConfig.rego 3.3v1 status check * fix N/A criticality for 3.5 and adjust unit and functional tests for new JSON structure * changed JSON structure for 3.4, 3.5 because it had redundant objects and updated all tests * fix run unit test * revised 3.3 unit tests * update 3.3 code & added unit tests * refactored names and added comments to 3.3 policy * refactored 3.4 to reduce redundant code and bad variable naming * streamlined code for 3.5 to reduce redundancy and returned ActualValue which was missing * renamed MFAPolicies to PhishingResistantMFAPolicies for accuracy * initial version 3.3 functional test wip * added functional test cases * removed NotCheckedDetails import since linter complained * made AuthenticationPolicyMigrationIsComplete shorter due to linter complaints * changed AuthenticationPolicyMigrationIsComplete from boolean assignment to if due to linter * removed redundant call to Get-MgBetaPolicyAuthenticationMethodPolicy which was left by mistake --------- Co-authored-by: Ted Kolovos <107076927+tkol2022@users.noreply.github.com> Co-authored-by: Sloane4 <cdiaz@mitre.org> * Bump OPA version from v0.63.0 to v0.64.1 (#1079) Co-authored-by: GitHub Action <action@github.com> * fixed functional test for 5.3 so that it only produces a single EnableAdminConsentRequests field (#1081) --------- Co-authored-by: Devesh Agarwal <147526644+dagarwal-mitre@users.noreply.github.com> Co-authored-by: David Bui <105074908+buidav@users.noreply.github.com> Co-authored-by: James Garriss <52328727+james-garriss@users.noreply.github.com> Co-authored-by: Ted Kolovos <107076927+tkol2022@users.noreply.github.com> Co-authored-by: Devesh Agarwal <dagarwal@mitre.org> Co-authored-by: Addam Schroll <108814318+schrolla@users.noreply.github.com> Co-authored-by: amart241 <106177534+amart241@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: GitHub Action <action@github.com> Co-authored-by: Shanti Satyapal <78565245+ssatyapal123@users.noreply.github.com> Co-authored-by: Sloane4 <cdiaz@mitre.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗣 Description
Applies fixes to the Defender functional test plans to align with latest licensing and assessment check requirements. Specifically, the updates:
IsNotCheckedflag from one MS.DEFENDER.3.1v1 test since Defender can check this value, even in a tenant without the appropriate license💭 Motivation and context
Functional test plans need to function correctly to help identify potential bugs in ScubaGear. Incorrect test plans waste developer time tracking down non-existent bugs and may "cover" actual bugs.
Closes #1056
🧪 Testing
To test the moved tests, run the G5 test plan variant against a G5 test tenant to validate all impersonation protection tests pass. You can use a test tag filter of
MS.DEFENDER.2*to limit the test scope to the affected items.To test the
IsNotCheckedtest removal, run the G3 test plan in a G3 test tenant with a test tag filter ofMS.DEFENDER.3*.✅ Pre-approval checklist
✅ Pre-merge checklist
PR passed smoke test check.
Feature branch has been rebased against changes from parent branch, as needed
Use
Rebase branchbutton below or use this reference to rebase from the command line.Resolved all merge conflicts on branch
Notified merge coordinator that PR is ready for merge via comment mention
✅ Post-merge checklist