Enhancement for travis: Record TRAVIS_BUILD_NUMBER per TRAVIS_BRANCH

[johngluckmdsol]

Travis has the ability to rerun nodes. We have sporadic failures and for a while, we have been using the Travis node rerun functionality. However, we figured out the other day that Knapsack doesn't necessarily return the same tests that it returned the last time the node was run. Which is a good thing because then it would never optimize. However, ideally, we'd like knapsack to recognize when we're rerunning a node and return the set of tests it ran for that node.

I'd propose that, in order to make it do this, you could accept the TRAVIS_BUILD_NUMBER and into knapsack and record it for each one and, if you already have that build number, send back the same set of tests. That would make travis re-run behave as expected. To minimize the amount of data you need to archive, you could do it per branch (TRAVIS_BRANCH) and only save the last run per branch and that would still be extremely useful.

What do you say?

[ArturT]

That's interesting use case. It might require a lot of changes on API and gem side. I'm wondering about simple solution. Maybe we could solve this on knapsack_pro gem level. My friend wrote once the article about re-running tests: http://blog.lunarlogic.io/2015/randomly-failing-cucumber-scenarios-in-continuous-integration-system/
Maybe we could remember failing test files and then rerun them again. I'm not sure how hard it would be with RSpec.

Your failing tests are in RSpec or Cucumber? Maybe you will be able to prepare some simple rake task and check if that could work.

[johngluckmdsol]

Both our rspec and cukes fail sporadically, but our cukes are failing more often. It's up to you how to implement if you think it's a good idea.

I liked the article and that's a good idea. I'll run it past my team but, that said, I still think that knapsack's interaction with Travis' rerun node is not obvious and it would be nice if the implicit behavior of rerunning the tests from that node is what it actually did. Either that or call out in the documentation under travis that it won't do what is expected.

[ArturT]

I'm going to look into that and learn more about Travis rerun feature probably at the end of the week when I will find more time. Maybe I will come up with a better solution. In the meantime if you will have any idea, samples of code from your team please share. Thanks!

[johngluckmdsol]

This is what travis looks like. If you see the circular arrow button on the right hand side, this is what we're using. On first page, here's what it looks like. The button here will rerun all three nodes

If you click into one of the nodes in the list above, you will see the following screen. Clicking on the circular arrow on this screen on this will rerun just the specific node

[ArturT]

@johngluckmdsol Sorry for being later. I have recently very busy weeks and a lot of work during weekends related to knapsack pro but I will try to look at this issue as soon as possible. I will get back to you.

[johngluckmdsol]

@ArturT , we aren't in a big hurry for this. It's a nice-to-have but in the interim, we are working on our wobbly tests so that we don't need to use this feature ;)

[ArturT]

@johngluckmdsol There is similar problem here #15 and I described an idea how to handle it there.

[johngluckmdsol]

I think that could work for our situation. There's nothing we need to do on our side but wait for a new version of knapsack_pro, right?

[ArturT]

@johngluckmdsol Correct. I prepared yesterday a new working API version but it's not on production yet. I need to update version of the knapsack_pro gem and do some tests first. I will let you know once it will be available.

[ArturT]

@johngluckmdsol Hey, I've just released a new knapsack_pro version 0.10.0. It should solve the problem. Just update the gem and that's it.

You can learn more how it works now in readme:
https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_test_suite_splite-test-suite-split-based-on-seed

or see the changelog:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/CHANGELOG.md#0100

Thank you for reporting the issue and for being patient. It took a while but we came up with the more generic solution. 😃

[johngluckmdsol]

Awesome, I'll give it shot right now

[johngluckmdsol]

So, I tried this out. I re-ran one node and it worked almost perfectly. I ended up getting one more test on the restart than I had in the first run. I'm guessing you have a one-off problem. If you need me to send you more details in email, I can do so.

[ArturT]

Please try to create a new commit and run tests for it and if you will see duplicates then please send me knapsack pro logging output via email. I will try to check it tomorrow. Thanks!

[johngluckmdsol]

Sure, I may not get to this until tomorrow

[johngluckmdsol]

Actually, nm, just started it.

[johngluckmdsol]

I wasn't able to reproduce this. I'll keep an eye on it, but you can go head and close it. In otherwords, this now works as expected. Thanks for the new feature!

[ArturT]

@johngluckmdsol Great. Thanks!