Jenkins check should collect some basic metrics

[conorbranagan]

Including build times (tagged by build name?), a counter on build failures, etc.

[conorbranagan]

It looks like we're actually already capturing the build duration in _get_build_metadata so it's just a matter of turning this into a metric.

[adamjt]

Sorting by node would be great as well. For example, we only really care about build times on master, and want to filter out feature branch builds. This functionality exists in the Global Build Stats plugin.

[remh]

taking it

[remh]

@adamjt about the branch builds, are you talking about git branches ?

[adamjt]

@remh Yes, sorry, nodes was a poor choice of words. My use case is that for alerting, the only thing I care about is a significant change in build time on the master branch. It would be very noisy if that alert metric was polluted with failures and long build times from feature branches, which are expected to be fairly unstable early on.

[adamjt]

@remh We got job duration metrics only once, for a handful of builds that happened a long time ago, and now we're seeing this in the logs:

ERROR (init.py:454): Check 'jenkins' instance #0 failed#012Traceback (most recent call last):#12 File "/usr/share/datadog/agent/checks/init.py", line 445, in run#012 self.check(instance)#12 File "/usr/share/datadog/agent/checks.d/jenkins.py", line 141, in check#012 if output['result'] == 'SUCCESS':#012KeyError: 'result'

[remh]

Hum not sure it's related to this new metric.
Can you send me the content of the build.xml file for this job ?
Make sure you get rid off any private information in it.

[adamjt]

I'm not sure how to tell which build it's failing on, how would I go about that?

[benpatterson]

Hi - We also see this error in our infrastructure. Log below.

2014-06-17 12:49:05 UTC | INFO | dd.collector | checks.collector(collector.py:347) | Finished run #4330. Collection time: 4.47s. Emit time: 0.09s
2014-06-17 12:51:43 UTC | INFO | dd.collector | checks.collector(agent_metrics.py:42) | CPU consumed (%) is high: 17.8, metrics count: 13, events count: 1
2014-06-17 12:52:22 UTC | INFO | dd.collector | checks.collector(collector.py:347) | Finished run #4340. Collection time: 4.72s. Emit time: 0.04s
2014-06-17 12:53:02 UTC | ERROR | dd.collector | checks.jenkins(__init__.py:515) | Check 'jenkins' instance #0 failed
Traceback (most recent call last):
  File "/usr/share/datadog/agent/checks/__init__.py", line 506, in run
    self.check(copy.deepcopy(instance))
  File "/usr/share/datadog/agent/checks.d/jenkins.py", line 159, in check
    if output['result'] == 'SUCCESS':
KeyError: 'result'

It appears to be intermittent (as you can see from previous runs captured in the log).

@adamjt @remh