Improve testing for updating from 'develop' branch to 'master' branch by querying CDash results #380
Comments
bartlettroscoe
added
Framework tasks
bartlettroscoe
added this to the Improve productivity, stability, and quality of Trilinos milestone
|
The below email indicates that we should have at least one OSX build in the set of queried build on CDash. There are several other builds that will need to be added as well obviously. As stated above, adding these extra builds are likely to be separate stories. The story will be just to set up the infrastructure to query CDash correctly and get just a small number of extra builds in the initial set. From: Trilinos-developers [mailto:trilinos-developers-bounces@trilinos.org] On Behalf Of Tobias Wiesner Thanks, Stefan. Tobias Did anyone else see this build error? This occurred on Mac clang this morning. At this point, I have not looked at my other tests. Regards, Stefan |
|
Thanks Ross!
I proposed in #370 that these builds should enable as many TPLs as possible, since enabling a TPL in Trilinos generally enables more code than it disables. (Amesos(2) and Zoltan(2) have good examples -- each TPL has interface code and tests.) We should at least enable the latest or "most common" versions of all the TPLs that we claim to support in the following list: |
|
My biggest concern with having all of the jobs enable as many TPLs as possible is that makes it very easy for optional dependencies to become required dependencies. I agree we should do some testing with as many TPLs as is reasonable, but we should also have some builds that are minimal to protect against dependency creep. |
That's a good point. It would make sense to have both a minimal TPL build (BLAS, LAPACK, MPI), and a "typical set of enables for apps" build. |
|
Here is another build we might consider adding the the criteria to update from 'develop' to 'master'.
|
|
The issue is compiler versions in this case. At some point last week stuff got broken for GCC 4.7.2 with an internal compiler error because Mark was using only 4.9.2 to check in. Now its the other way around, he used 4.7.2 to check in and didn't notice that it broke 4.9 and newer with internal compiler errors. What that clearly means is that we need multi compiler tests rather sooner than later. |
|
I would like to have an Albany build (flavor TBD) to be part of the criteria for moving Trilinos code from develop to master. Albany catches issues with header installation, issues to do with headers being included in multiple files, namespace collisions, unexpected non-backward-compatible changes, CMake issues with seacas and stk, and sometimes provides code coverage beyond individual package tests. I believe that Albany catches the vast majority of issues that slip through the current process and break Nalu. |
|
@bartlettroscoe you might want to speak to Aaron Levine (@allevin) in CSRI since he has working Jenkins remote querying scripts which work with test checks and automated Pull Request generation from devel to master branches. This was developed for the SST project and has been successfully running for around 4 months already. |
@agsalin, that would be good if this could be made solid. First, someone would need to set up an automated build of Trilinos for the exact configuration of Trilinos used by Albany and post to the Trilinos CDash dashboard. It would be good if this exact build was then used to install Trilinos and the Albany build(s) would build against that installed version of Trilinos. Then that Trilinos build (or builds) could also be queried in the |
@nmhamster, it would be good what was set up and how hard it is to work with the GitHub and Jenkins APIs. I will be at SNL/ABQ next week. Perhaps we can arrange a meeting then? I know @bmpersc and @jwillenbring would be interested since they mentioned this at the Trilinos Leaders Meeting today. |
|
@bartlettroscoe (re 2 comments ago): The master branch of Albany on Linux is at or near 100% much of the time, and many of the failures we do get are from Trilinos. However, there are a few times per month where master gets broken based on an Albany commit. I suggest that, as a starting point for the automated Trilinos develop->master process, we get Albany testing into the scripts but, at the beginning, don't let an Albany failure cancel the update of master. If I and a few others can get an email with the Albany test results, we can do triage and figure out if we are detecting a problem with Trilinos or not. Over time, we can work to make the process robust enough to where we are comfortable making the automated Trilinos update contingent on success of Albany. (I expect we might need to either do what you did with Sierra, or create a "stable" branch that only gets updated from master when 100% pass.) |
|
@william76, this issue was opened a long time ago that proposed using the CDash API to determine pass/fail for a bunch of builds to update the 'master' branch from the 'develop' branch like we discussed on Monday 2/26/2018. |
|
@bartlettroscoe, ah ok. Thanks! I think we'll be waiting on the newer CDash since I can get the SHA1 hashes of the builds on the dashboard... once that's in place, I already have about 99% of what I'd need in a script to check the set of builds that we care about for promoting dev->master and could set up an automatic process pretty easily. |
|
The @trilinos/framework team went with the PR testing infrastructure. Closing as wontfix |
bartlettroscoe
added
the
resolved: wontfix
Next Action Status:
Waiting for #370 to be completed before getting started on this ...
Blocked By: #370
CC: @jwillenbring, @bmpersc, @mhoemmen, @trilinos/framework
Description:
The Story #370 implemented the initial transition to the 'develop'/'master' branch workflow using just a simple automated job that just does one build with the checkin-test.py script and then pushes the updated 'master' branch. This is a follow-on Story to put the infrastructure and the process in place to beef up the level of testing that needs to pass before updating the 'master' branch. The approach will be to take advantage of the new CDash API that allows you to download CDash query results as a JSON data-structure and then use a Python script to inspect it and make sure all of the targeted builds and packages all passed.
This has already been implemented for CASL VERA in the Python script
vera_cdash_pass_fail.py. This script is run as:This script uses the reusable Python module:
The script
vera_cdash_pass_fail.pydoes one query of theVERACDash project and one of theVERADriverCDash project (created using the TriBITS Dashboard Driver (TDD) system). TheVERACDash query is shown above and it is examined to make sure that all of the expected builds are present and that there are no failing configures, builds or tests. TheVERADriverquery is done to make sure that all of the outer CTest driver builds ran and there was no failures (e.g. timeouts). Without querying theVERADriverproject to make sure that none of the CTest drivers timed out, then having all passingVERAbuilds is not sufficient (because packages may have never been tested).Because Trilinos uses Jenkins to run CTest build drivers, a query of Jenkins may be needed to ensure that none of the selected builds timed out. However, it appears that Jenkins supports a remote API so developing queries of Jenkins should be possible as well.
Tasks:
trilinos_cdash_pass_fail.pythat will run and inspect one or more queries of Trilinos CDash project and return false if any of these queries show failures. Also, do queries of the Jenkins CDash drivers to make sure that the outer CTest driver invocations don't timeout or otherwise fail.The text was updated successfully, but these errors were encountered: