Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline V3 #37428

Merged
merged 191 commits into from
Sep 25, 2024
Merged

Pipeline V3 #37428

merged 191 commits into from
Sep 25, 2024

Conversation

scbedd
Copy link
Member

@scbedd scbedd commented Sep 17, 2024

This PR:

  • Updates PullRequests to utilize git to run CI only packages that were changed
    • Packages that have been changed may optionally include other packages for sparse validation. For instance azure-core configures an additional 15 or so packages that must be brought in for sparse validation. This is configured solely in get_package_properties.py.
  • Affected packages are now batched, so even if a PR touches 50+ packages, we will simply expand the test matrix by limiting each test run to a max of 8 packages (arbitrary limit). We just...add more test runs. Analyze will still have an issue with this many packages, I will likely multiplex that in follow-up PRs as well.
    • In the above case, when additional validation packages are pulled in due to a large number of directly affected packages, we will also sparsely include those validation packages into the test matrix. ("sparsely" translating to "on a single platform and python combination, we don't really care which")

scbedd and others added 30 commits July 19, 2024 15:20
@scbedd
update venv reference methodology
38aeb2c
@scbedd
remove platform from build step
8a2b75c
@scbedd
Merge branch 'update-venv-reference' into always-build
217d394
@scbedd
a _bunch_ of build changes. this is to get our builds all running the…
1c8ddf8 
… same way
@scbedd
ok we aren't honoring the conditionals, but we can test the actual bu…
354213e 
…ild runs.
@scbedd
update indentation
d84253d
@scbedd
update condition usage in use-venv
0442b24
@scbedd
remove wonky condition
2cf398e
@scbedd
adjust packages usage in build-extended-artifacts
c0ffa9c
@scbedd
Merge remote-tracking branch 'upstream/main' into simplify-build-agen…
a2ea84b 
…t-selection
@scbedd
why is this thing having such a hard time processing getting corehttp…
1a7dcc1 
… package properties?
@scbedd
Merge remote-tracking branch 'upstream/main' into simplify-build-agen…
10a6127 
…t-selection
@scbedd
waiting for confirmation, but pretty certain this will work
425b744
@scbedd @weshaggard
Update eng/pipelines/templates/steps/build.yml
871e662 
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
@scbedd
update the conditions in build-package-artifacts. the intent is to sa…
0c0f06d 
…ve effort where possible
@scbedd
correct all the variable references
b358429
@scbedd
correct condition in set dev build
79f95f7
@scbedd
simplify the conditional
9753eb8
@scbedd
cant remove a directory we're operating in
7ff3b89
@scbedd
leave behind the credscan suppression file though
105edfe
@scbedd
clean up all the files in the repo
2ba740d
@scbedd
buncha updates!
c205cb6
@scbedd
clean up the outputs in build-package-artifacts
873ffd6
@scbedd
fix the artifact download call
56ec683
@scbedd
restore the python release stages to what they were. my changes were …
387d3d0 
…not required
@scbedd
Merge remote-tracking branch 'upstream/main' into pipelinev3-v2
b550519
@scbedd
rationalizing our pipeline v3 work into the new simplified build. ver…
c3ac400 
…y close to being ready to go. got a lot of simplifying to work through
@scbedd
merge main
91e00e4
@scbedd
update usage of resolve-package-targeting everywhere which will honor…
393d610 
… the packageproperties folder
@scbedd
update our commit reference
cca4e9f
@scbedd scbedd requested a review from a team as a code owner September 20, 2024 22:56
scbedd added 16 commits September 20, 2024 17:20
@scbedd
only run build_extended against packages directly included
5d80d5f
@scbedd
grab the changes from eng/common from azure-sdk-tools to check it
ad46436
@scbedd
use the same variable name
9ea9a20
@scbedd
fix the condition for the values being scanned
9b323bf
@scbedd
Merge remote-tracking branch 'upstream/main' into pipelinev3-v2
1384c80
@scbedd
remove comments
def8ec6
@scbedd
use AsHashtable
cc309b5
@scbedd
remove the ci.yml transformation
0259e48
@scbedd
looks like I had changed a core file when it was entirely unnecessary!
98f7b85
@scbedd
remove the os.pop stuff
2de24a7
@scbedd
remove extraneous work
bdbd604
@scbedd
remove the parallel sphinx invocation. we gain effectively no time fr…
b0260ce 
…om it
@scbedd
Merge remote-tracking branch 'upstream/main' into pipelinev3-v2
9466c63
@scbedd
merge main, then ignore the samples directory when invoking pytest
79736e3
@scbedd
Merge remote-tracking branch 'upstream/main' into pipelinev3-v2
0e524a2
@scbedd
add samples directory to the pytest ignore_args config that is used e…
91a357d 
…verywhere
@scbedd
Copy link
Member Author

scbedd commented Sep 25, 2024 •
edited
Loading

The failing python - pullrequest is a CI build for the build definition. The error is correct. I am going to ignore this failure and simply submit a follow-up PR to azure-template that resolves it. Not worth another commit and retrigger of all the pipelines.

@scbedd
Copy link
Member Author

scbedd commented Sep 25, 2024

/check-enforcer override

@scbedd scbedd merged commit da4820a into main Sep 25, 2024
306 of 308 checks passed
@scbedd scbedd deleted the pipelinev3-v2 branch September 25, 2024 20:44
w-javed pushed a commit to w-javed/azure-sdk-for-python that referenced this pull request Oct 3, 2024
@scbedd @weshaggard
@hallipr @w-javed
Pipeline V3 (Azure#37428)
1e9f86f 
* refactor package selection methodology to utilize Save-Package-Properties for package targeting
* introduce the python - pullrequest build definition that is triggered against all sdk/ directories, and builds/tests the packages which were changed
  * this also includes abstractions to reduce runtime by distributing sets of packages and duplicating matrix definition
* refactor all analyze checks which used to operate on ServiceDirectory to instead operate upon the packages present within a PackageInfoFolder from Save-Package-Properties
* miscellaneous tooling changes to account for new matrix possibilities

---------

Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hallipr hallipr hallipr left review comments

@weshaggard weshaggard weshaggard approved these changes

@mccoyp mccoyp Awaiting requested review from mccoyp mccoyp is a code owner

@benbp benbp Awaiting requested review from benbp benbp is a code owner

@paulshealy1 paulshealy1 Awaiting requested review from paulshealy1 paulshealy1 is a code owner

@achauhan-scc achauhan-scc Awaiting requested review from achauhan-scc achauhan-scc is a code owner

@kingernupur kingernupur Awaiting requested review from kingernupur kingernupur is a code owner

@lmazuel lmazuel Awaiting requested review from lmazuel lmazuel is a code owner

@annatisch annatisch Awaiting requested review from annatisch annatisch is a code owner

@johanste johanste Awaiting requested review from johanste johanste is a code owner

@xiangyan99 xiangyan99 Awaiting requested review from xiangyan99 xiangyan99 is a code owner

@iscai-msft iscai-msft Awaiting requested review from iscai-msft iscai-msft is a code owner

@kashifkhan kashifkhan Awaiting requested review from kashifkhan kashifkhan is a code owner

@pvaneck pvaneck Awaiting requested review from pvaneck pvaneck is a code owner

@christothes christothes Awaiting requested review from christothes christothes was automatically assigned from Azure/azure-sdk-write-identity

@billwert billwert Awaiting requested review from billwert billwert was automatically assigned from Azure/azure-sdk-write-identity

@ahsonkhan ahsonkhan Awaiting requested review from ahsonkhan ahsonkhan was automatically assigned from Azure/azure-sdk-write-identity

@KarishmaGhiya KarishmaGhiya Awaiting requested review from KarishmaGhiya KarishmaGhiya was automatically assigned from Azure/azure-sdk-write-identity

@chlowell chlowell Awaiting requested review from chlowell chlowell was automatically assigned from Azure/azure-sdk-write-identity

Assignees

@scbedd scbedd

Labels
Azure.Core Machine Learning
Projects
Azure SDK EngSys 🤖🧠
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@scbedd @azure-sdk @hallipr @weshaggard