Fix jenkins agent check when build does not yet have results

[jzoldak]

This includes a test (currently marked as expected to fail) which exposes the KeyError: 'result' error that has been reported in #567.

The logic of how highwater marks are done in the jenkins agent check is flawed:

1. Jenkins will create build.xml files before the build is finished, which does not yet include the result in the XML. I have seen this on long-running matrix jobs. This will cause the agent to throw the KeyError, as shown in the testCheckWithRunningBuild added in this PR. Also the agent will already have set the highwater mark to that build, so its results will never be processed even after jenkins finishes and updates the build.xml file.
2. Note also that for concurrent builds, if a build that started later finishes first then the agent will set the highwater mark at the timestamp of the later build and the earlier one will never be processed. I didn't write a test for that, but you can see it by reading through the jenkins.py code.

[jzoldak]

@remh you might be interested in this.

Also, looks like the Travis build failed due to unrelated causes - it seems like it couldn't start up couchdb as the service was already up.

[remh]

Thanks a lot @jzoldak .
I'll review that really soon.

[jzoldak]

@remh, cool thx.

I submitted a fix for the incomplete build.xml files and pushed. FWIW I've installed the new code on our prod jenkins server and it is working as expected and reporting in metrics correctly.

With regard to the 2nd problem noted above - I'm guessing that this agent was written before jenkins could do concurrent builds, so it assumes that builds will finish consecutively. Should I put in another issue for that, so we can get this one through first? I'm not sure I will have time to work on that thornier issue right now.

[remh]

Thanks a lot @jzoldak !

The master branch is now frozen until release of Agent 5.0.0. We'll review and merge for the next release.

[remh]

Closing in favor of #1068