Travis: Build new binding pull requests with tests #5389

davidgraeff · 2019-04-04T11:29:55Z

travisbuddy

This PR adds travisbuddy to .travis.yml. That service add (update) a comment to a PR on a build failure with the log attached. This is an intermediate step until we have full github checks api support and a curl command figured out to report SAT findings and Test results to the "Checks" tab. See also #4935

Partial build

The PR commits are analysed and if exactly one new directory is added to bundles/ then a build with tests and checks is performed for that new bundle otherwise the full repo build without tests and checks is executed as before.

WIP

This PR includes an absolute irrelevant commit for testing the bundles/ changed detection
Karaf feature test is not yet performed in the partial build
SAT should fail if a new binding is not added to the bom or the parent pom.xml or the karaf feature file
Maven is still showing downloads although I have -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn set. That clutters the build log which would otherwise only show SAT and test reports

Advantage for a new binding pull request

Build time of under 5 minutes
Clean log with only relevant SAT and test findings for that new PR. We can even increase the SAT report level and make more warnings fail the build.
Review time can be spend on more important questions like architecture and technology patterns

Signed-off-by: David Graeff david.graeff@web.de

TravisBuddy · 2019-04-04T20:04:44Z

Travis tests have failed

Hey @davidgraeff,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

2nd Build

View build log

buildci.sh

buildci.sh: command not found

TravisBuddy Request Identifier: e0396110-5714-11e9-979a-6d93f48c24a0

TravisBuddy · 2019-04-04T20:38:51Z

Hey @davidgraeff,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: a6775c70-5719-11e9-979a-6d93f48c24a0

martinvw · 2019-04-05T18:13:28Z

.. out to report SAT findings ...

That used to work briefly on Jenkins.

@kaikreuzer did you set this up?

kaikreuzer · 2019-04-05T20:39:26Z

@martinvw Yes, Jenkins is configured to publish the SAT reports of the PR builds - but it looks in target/*, where the report does not exist any longer. Where is it now?

martinvw · 2019-06-27T20:55:25Z

@davidgraeff is this something which can be wrapped up, it looks promising.

davidgraeff · 2019-06-28T09:09:56Z

I have outlined what is still missing in the WIP section. I would like to have karaf feature tests, but I do not know the maven goal and target to do that. And the karaf maven plugin page is absolutely not helpful for a non-maven guy :/

A second issue is that maven still outputs download progress. I really don't know what is going on in maven developers mind and judging by the code that I have inspected I'd say not much at all. They could at least have used different log classes for different types of output.
Maybe they expect people to use piped log filters. I honestly don't know.

I wish to have a clean build log for travisbuddy, so that even newbies can easily identify and tell apart notes, warnings and build step logs.

martinvw · 2019-06-28T09:58:33Z

I cant solve the karaf feature check for you but i can look into the second. I found https://codecraft.vaamo.de/2014/08/22/less-is-more-when-it-comes-to-ci-logs.html is this a road you already explored? Regards Op vr 28 jun. 2019 om 11:10 schreef David Gräff <notifications@github.com>

…

I have outlined what is still missing in the WIP section. I would like to have karaf feature tests, but I do not know the maven goal and target to do that. And the karaf maven plugin page is absolutely not helpful for a non-maven guy :/ The second think is that maven still shows the download progress. I really don't know what is going on in maven developers mind and judging by the code that I have inspected I'd say not much at all. I guess they imagine to use maven with a piped log filter. I really don't know. But I wish to have a clean build log for travisbuddy, so that even newbies can easily identify and tell apart notes, warnings and build step logs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/openhab/openhab2-addons/pull/5389?email_source=notifications&email_token=AAUA577G5UDZU6TZOI5MG6TP4XIO5A5CNFSM4HDSI2C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZRFVA#issuecomment-506663636>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUA577ASPSYJQTBVAJ5XV3P4XIO5ANCNFSM4HDSI2CQ> .

davidgraeff · 2019-06-28T11:17:03Z

What you have found looks promising, that's what I had in mind

martinvw · 2019-06-28T11:49:20Z

Karaf feature test is not yet performed in the partial build
SAT should fail if a new binding is not added to the bom or the parent pom.xml or the karaf feature file

I suppose to separate that and merge what you already included, shall I also remove the new fake binding?

martinvw · 2019-06-28T11:50:50Z

@davidgraeff shall I also rebase it? You might then be better of removing your local copy I IMHO

Signed-off-by: David Graeff <david.graeff@web.de>

…uration part. Use daily snapshot updates Signed-off-by: David Graeff <david.graeff@web.de>

Signed-off-by: David Graeff <david.graeff@web.de>

…ngs from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl>

martinvw · 2019-06-28T12:15:21Z

@openhab/2-x-add-ons-maintainers

The two running builds should help us decide whether its time to merge:

Full build:

https://travis-ci.org/openhab/openhab2-addons/jobs/551749200

Single-binding build: (Ran for 5 min 13 sec)

https://travis-ci.org/openhab/openhab2-addons/jobs/551752246

TravisBuddy · 2019-06-28T12:19:02Z

Hey @davidgraeff,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: e8583c50-999e-11e9-8610-39232d9ecaa4

martinvw · 2019-06-28T12:20:49Z

I think that is such a big improvement that we should just do it. We could create issues for the open points and we SHOULD NOT FORGET to remove the fake binding :-D

Could we also use this moment to re-enable the tests (not the itTests)

martinvw

I'm happy but we shouild not forget to remove the fake binding

TravisBuddy · 2019-06-28T12:42:55Z

Hey @davidgraeff,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 3eee7f40-99a2-11e9-8610-39232d9ecaa4

davidgraeff · 2019-06-28T13:13:40Z

Perfect. I agree to merge asap.

J-N-K · 2019-06-28T13:16:13Z

Not directly related but similar topic: we should get SAT working again, this especially important for new contributions.

martinvw · 2019-06-28T13:25:41Z

Not directly related but similar topic: we should get SAT working again, this especially important for new contributions.

AFAIK it just works, I was at least able to run it for the single binding. I didn’t test the full build though.

J-N-K · 2019-06-28T13:30:00Z

SAT is not including all reports and not working for bindings with internal libraries (PR Open)

davidgraeff · 2019-06-28T13:33:09Z

AFAIK it just works, I was at least able to run it for the single binding. I didn’t test the full build though.

The full build without SAT (and tests?) apparently already takes 40 minutes. Because maven doesn't perform SAT checks in parallel to build steps, that will probably increase to an unhandable long time.

So I think it never make sense to run SAT for a full build in this repo. (Not even the distro would need that. I mean what will the distro guys do if a binding in here fails on SAT checks.)

With this PR merged, every binding that will be touched from then on will run SAT (except the ones with internal libraries as J-N-K has pointed out).

martinvw · 2019-06-28T15:16:05Z

@wborn / @Hilbrand / @cweitkamp / @J-N-K I think that if you are okay, then its ready to be merged, I stripped of the test-commit.

J-N-K · 2019-06-28T15:23:15Z

Manually verified Signed-Off.

TravisBuddy · 2019-06-28T15:59:13Z

Hey @davidgraeff,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: ab623700-99bd-11e9-8610-39232d9ecaa4

martinvw · 2019-06-28T19:34:55Z

@davidgraeff there is a little flaw, if the author does not keep the master branch of his own repo up-to-date it still triggers a full rebuild. We should try to look into that.

On my feature/4633-improve-thermostat3-support branch, git diff --diff-filter=A --dirstat=files,0 master...HEAD returns:

   0.0% .github/ISSUE_TEMPLATE/
   0.0% .github/
   0.0% bom/openhab-addons/
   0.0% bom/openhab-core-index/
   0.0% bom/runtime-index/
   0.0% bom/test-index/
   0.0% bom/
   0.0% bundles/org.openhab.binding.airquality/src/main/feature/
   0.0% bundles/org.openhab.binding.airquality/src/main/java/org/openhab/binding/airquality/internal/discovery/
   0.0% bundles/org.openhab.binding.airquality/src/main/java/org/openhab/binding/airquality/internal/handler/
   0.0% bundles/org.openhab.binding.airquality/src/main/java/org/openhab/binding/airquality/internal/json/
   0.0% bundles/org.openhab.binding.airquality/src/main/java/org/openhab/binding/airquality/internal/
   0.0% bundles/org.openhab.binding.airquality/src/main/resources/ESH-INF/binding/
   0.0% bundles/org.openhab.binding.airquality/src/main/resources/ESH-INF/thing/
   0.0% bundles/org.openhab.binding.airquality/
   0.0% bundles/org.openhab.binding.airvisualnode/src/main/feature/
   0.0% bundles/org.openhab.binding.airvisualnode/src/main/java/org/openhab/binding/airvisualnode/internal/config/
   0.0% bundles/org.openhab.binding.airvisualnode/src/main/java/org/openhab/binding/airvisualnode/internal/discovery/
......
   0.0% itests/org.openhab.io.hueemulation.tests/
   0.0% itests/org.openhab.persistence.mapdb.tests/src/main/java/org/openhab/persistence/mapdb/
   0.0% itests/org.openhab.persistence.mapdb.tests/
   0.0% itests/
   0.0% licenses/epl-2.0/
   0.0% src/etc/
   0.0% tools/static-code-analysis/checkstyle/
warning: inexact rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 7601 and retry the command.

martinvw · 2019-06-28T19:43:48Z

buildci.sh

+}
+
+# Determine if this is a new addon -> Perform tests + integration tests and all SAT checks with increased warning level
+CHANGED_DIR=`git diff --diff-filter=A --dirstat=files,0 master...HEAD bundles/ | sed 's/^[ 0-9.]\+% bundles\///g' | grep -o -P "^([^/]*)" | uniq`


Should we also have a special build for people changing only files in a single module, now it only focuses on either new contributions or it does the complete build.

martinvw · 2019-06-28T19:56:19Z

@davidgraeff there is a little flaw, if the author does not keep the master branch of his own repo up-to-date it still triggers a full rebuild. We should try to look into that.

I found a possible solution, there is the option to get the commit range which is part of the PR as an input: $TRAVIS_COMMIT_RANGE I'm adapting the scripts and will also add support for only changes in a single module, I think that I would just handle it as one, unless someone disagrees?

martinvw · 2019-06-28T20:45:33Z

I included the changes mentioned above in #5765

* Travis: Build new binding pull requests with tests * Move shell script into own file and keep the yaml file for the configuration part. Use daily snapshot updates * Overwrite travis default behaviour Signed-off-by: David Graeff <david.graeff@web.de> * Some finetuning to filter out download progress and include all warnings from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl>

* Travis: Build new binding pull requests with tests * Move shell script into own file and keep the yaml file for the configuration part. Use daily snapshot updates * Overwrite travis default behaviour Signed-off-by: David Graeff <david.graeff@web.de> * Some finetuning to filter out download progress and include all warnings from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl> Signed-off-by: Maximilian Hess <mail@ne0h.de>

* Travis: Build new binding pull requests with tests * Move shell script into own file and keep the yaml file for the configuration part. Use daily snapshot updates * Overwrite travis default behaviour Signed-off-by: David Graeff <david.graeff@web.de> * Some finetuning to filter out download progress and include all warnings from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl>

* Travis: Build new binding pull requests with tests * Move shell script into own file and keep the yaml file for the configuration part. Use daily snapshot updates * Overwrite travis default behaviour Signed-off-by: David Graeff <david.graeff@web.de> * Some finetuning to filter out download progress and include all warnings from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl> Signed-off-by: Tim Roberts <timmarkroberts@gmail.com>

davidgraeff requested a review from a team as a code owner April 4, 2019 11:29

davidgraeff force-pushed the travisex branch 2 times, most recently from 240e34a to 203a57d Compare April 4, 2019 20:34

wborn added the work in progress A PR that is not yet ready to be merged label Apr 6, 2019

davidgraeff mentioned this pull request May 4, 2019

[infrastructure] auto-create addons-BOM #5551

Merged

davidgraeff added the infrastructure Build system and Karaf related issues and PRs label May 6, 2019

martinvw removed the work in progress A PR that is not yet ready to be merged label Jun 28, 2019

martinvw changed the title ~~[WIP] Travis: Build new binding pull requests with tests~~ Travis: Build new binding pull requests with tests Jun 28, 2019

David Graeff added 3 commits June 28, 2019 13:54

Travis: Build new binding pull requests with tests

e9a0be2

Signed-off-by: David Graeff <david.graeff@web.de>

Move shell script into own file and keep the yaml file for the config…

a24a822

…uration part. Use daily snapshot updates Signed-off-by: David Graeff <david.graeff@web.de>

Overwrite travis default behaviour

c7fb80b

Signed-off-by: David Graeff <david.graeff@web.de>

martinvw force-pushed the travisex branch from 947ec47 to fe19725 Compare June 28, 2019 12:03

Some finetuning to filter out download progress and include all warni…

9934de1

…ngs from the reporting utility Signed-of-by: Martin van Wingerden <martin@martinvw.nl>

martinvw force-pushed the travisex branch from cc3bf9e to 01b3aaa Compare June 28, 2019 12:13

martinvw approved these changes Jun 28, 2019

View reviewed changes

martinvw force-pushed the travisex branch from 01b3aaa to 9934de1 Compare June 28, 2019 15:14

J-N-K approved these changes Jun 28, 2019

View reviewed changes

J-N-K merged commit f9fa7f6 into openhab:master Jun 28, 2019

davidgraeff deleted the travisex branch June 28, 2019 18:51

martinvw reviewed Jun 28, 2019

View reviewed changes

wborn added this to the 2.5 milestone Jul 30, 2019

wborn added the test label Oct 1, 2019

wborn removed this from the 2.5 milestone Dec 8, 2019

wborn added this to the 2.5 milestone Dec 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Travis: Build new binding pull requests with tests #5389

Travis: Build new binding pull requests with tests #5389

davidgraeff commented Apr 4, 2019 •

edited

Loading

TravisBuddy commented Apr 4, 2019

TravisBuddy commented Apr 4, 2019

martinvw commented Apr 5, 2019

kaikreuzer commented Apr 5, 2019

martinvw commented Jun 27, 2019

davidgraeff commented Jun 28, 2019 •

edited

Loading

martinvw commented Jun 28, 2019 via email

davidgraeff commented Jun 28, 2019

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019 •

edited

Loading

TravisBuddy commented Jun 28, 2019

martinvw commented Jun 28, 2019 •

edited

Loading

martinvw left a comment

TravisBuddy commented Jun 28, 2019

davidgraeff commented Jun 28, 2019

J-N-K commented Jun 28, 2019

martinvw commented Jun 28, 2019

J-N-K commented Jun 28, 2019

davidgraeff commented Jun 28, 2019 •

edited

Loading

martinvw commented Jun 28, 2019

J-N-K commented Jun 28, 2019

TravisBuddy commented Jun 28, 2019

martinvw commented Jun 28, 2019 •

edited

Loading

martinvw Jun 28, 2019

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019

Travis: Build new binding pull requests with tests #5389

Travis: Build new binding pull requests with tests #5389

Conversation

davidgraeff commented Apr 4, 2019 • edited Loading

travisbuddy

Partial build

WIP

Advantage for a new binding pull request

TravisBuddy commented Apr 4, 2019

Travis tests have failed

2nd Build

TravisBuddy Request Identifier: e0396110-5714-11e9-979a-6d93f48c24a0

TravisBuddy commented Apr 4, 2019

TravisBuddy Request Identifier: a6775c70-5719-11e9-979a-6d93f48c24a0

martinvw commented Apr 5, 2019

kaikreuzer commented Apr 5, 2019

martinvw commented Jun 27, 2019

davidgraeff commented Jun 28, 2019 • edited Loading

martinvw commented Jun 28, 2019 via email

davidgraeff commented Jun 28, 2019

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019 • edited Loading

TravisBuddy commented Jun 28, 2019

TravisBuddy Request Identifier: e8583c50-999e-11e9-8610-39232d9ecaa4

martinvw commented Jun 28, 2019 • edited Loading

martinvw left a comment

Choose a reason for hiding this comment

TravisBuddy commented Jun 28, 2019

TravisBuddy Request Identifier: 3eee7f40-99a2-11e9-8610-39232d9ecaa4

davidgraeff commented Jun 28, 2019

J-N-K commented Jun 28, 2019

martinvw commented Jun 28, 2019

J-N-K commented Jun 28, 2019

davidgraeff commented Jun 28, 2019 • edited Loading

martinvw commented Jun 28, 2019

J-N-K commented Jun 28, 2019

TravisBuddy commented Jun 28, 2019

TravisBuddy Request Identifier: ab623700-99bd-11e9-8610-39232d9ecaa4

martinvw commented Jun 28, 2019 • edited Loading

martinvw Jun 28, 2019

Choose a reason for hiding this comment

martinvw commented Jun 28, 2019

martinvw commented Jun 28, 2019

davidgraeff commented Apr 4, 2019 •

edited

Loading

davidgraeff commented Jun 28, 2019 •

edited

Loading

martinvw commented Jun 28, 2019 •

edited

Loading

martinvw commented Jun 28, 2019 •

edited

Loading

davidgraeff commented Jun 28, 2019 •

edited

Loading

martinvw commented Jun 28, 2019 •

edited

Loading