❓👂 Question/Feedback - Make e2e pipelines and results public

[olljanat]

Question/Feedback

I noticed that e2e tests have been moved to Azure DevOps on some point. However for me it looks that those are now hosted on project which type is "Private".

Possible Answers/Solutions?

As this is open source project it would be nice if project type "Public" can used on that Azure DevOps project side too so pipeline definitions and build results would be visible on here too.

[jtracey93]

Hey @olljanat,

Thanks for raising this and we completely agree however, we have to run the tests in ADO that do actual real deployments as it's the only way you can get PRs from forks to access secrets. The secrets are only available to PRs from branches from the same repo and owners.

Unfortunately, this feature isnt available in GitHub yet but is in ADO hence we moved our deployment E2E tests to ADO.

But we do as much testing as possible in GitHub actions like the linting and the builds and soon to be PSRule also 👍 so you can see these, just the deployment tests have to be private in ADO today as explained above.

I'll close this off but know when available we will move to GitHub actions 👍👍👍

Thanks

Jack

[olljanat]

Well as ADO user I would be more interested to see how things are done in that side, what exactly is tested and how? and maybe even to be able to contribute to those tests.

we have to run the tests in ADO that do actual real deployments

To be exact. Afaiu biggest thing missing from current e2e is test with WhatIf mode. I mean that test should contain steps:

1. Deploy
2. Deploy again with WhatIf mode
3. Check output from WhatIf to make sure that it does not find differences between code and environment.

I see that ability to be able to run pull requests with WhatIf mode against of production very critical as other why it is not possible to have GIT repo with CI to production and multiple persons contributing to code without high risk to break things.

[jtracey93]

Hi @olljanat,

Thanks for coming back and sharing your thoughts. Really insightful.

The ADO tests are visible in the 'tests' folder. We don't want to hide anything just have to for the secret exposure reasons I shared above but it's only the logs.

You will see the results in the PR once we trigger the tests just not the detailed logs.

As for what if we are currently refactoring our tests so please raise a feature request for that and we will triage. @jfaurskov for awareness👍

Thanks

Jack

[jtracey93]

@olljanat we are also working on an accelerator repo for ALZ bicep that will contain production ready pipelines for ADO and GitHub for consumer usage rather than for the maintenance of the repo.

@ejhenry is working on these

[olljanat]

Cool. Sounds that I'm on rights place on right time :)

@ejhenry how far you have got with that one? Any draft which you can already share?

I have been working with something similar for our use case lately. I copied cleaned version of it to https://github.com/olljanat/alz-bicep-ci

Basically it solves already couple of issues:

• On local development it always points to "lab" environment to make sure that those can be safely deployed (I use my Azure subscription on MSDN for that purpose).
• Scripts defaults to WhatIf mode so they can be also run against of production when needed to check if code is in sync with environment.
• Updating latest ALZ-Bicep version can be done easily by just running script standard/scripts/update-alz-bicep.ps1
• Pull requests will run WhatIf mode against of Acceptance and Production environments and raise warning to build log if changes have been found (which is useful when you just update ALZ-Bicep version and except that there is no changes)
• Scripts use combination of PowerShell + Azure CLI and works on both Windows and Linux machines.
• Scripts will block any deployment where WhatIf detects that any resources would be removed.
• After merge to master CI is triggered and extra checks/approvals can be later included to pipeline happening after Acceptance deployment (which I assume are needed after setup is complex enough).

[ejhenry]

@olljanat nothing to share yet, but will loop you in when we do 👍

Thanks for sharing, I'll definitely take a look at your repo!