[status_check] Unhandled exception when calling a custom check with syntax errors

[insideClaw]

**Output of the info page **

sudo service datadog-agent info
2017-08-09 15:31:26,268 | WARNING | dd.collector | utils.dockerutil(dockerutil.py:94) | Error while detecting orchestrator: Error while fetching server API version: ('Connection aborted.', error(2, 'No such file or directory'))
2017-08-09 15:31:26,304 | ERROR | dd.collector | collector(agent.py:610) | Uncaught error running the Agent
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/agent.py", line 606, in <module>
    sys.exit(main())
  File "/opt/datadog-agent/agent/agent.py", line 532, in main
    return Agent.info(verbose=options.verbose)
  File "/opt/datadog-agent/agent/agent.py", line 219, in info
    return CollectorStatus.print_latest_status(verbose=verbose)
  File "/opt/datadog-agent/agent/checks/check_status.py", line 282, in print_latest_status
    message = module_status.render()
  File "/opt/datadog-agent/agent/checks/check_status.py", line 180, in render
    ] + ["", ""]
  File "/opt/datadog-agent/agent/checks/check_status.py", line 546, in body_lines
    '  ' + '-' * (len(cs.name) + 3 + len(cs.check_version))
TypeError: object of type 'NoneType' has no len()
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/agent.py", line 606, in <module>
    sys.exit(main())
  File "/opt/datadog-agent/agent/agent.py", line 532, in main
    return Agent.info(verbose=options.verbose)
  File "/opt/datadog-agent/agent/agent.py", line 219, in info
    return CollectorStatus.print_latest_status(verbose=verbose)
  File "/opt/datadog-agent/agent/checks/check_status.py", line 282, in print_latest_status
    message = module_status.render()
  File "/opt/datadog-agent/agent/checks/check_status.py", line 180, in render
    ] + ["", ""]
  File "/opt/datadog-agent/agent/checks/check_status.py", line 546, in body_lines
    '  ' + '-' * (len(cs.name) + 3 + len(cs.check_version))
TypeError: object of type 'NoneType' has no len()
====================
Dogstatsd (v 5.16.0)
====================
 
  Status date: 2017-08-09 15:31:18 (9s ago)
  Pid: 14984
  Platform: Linux-4.4.0-81-generic-x86_64-with-Ubuntu-16.04-xenial
  Python Version: 2.7.13, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log, syslog:/dev/log
 
  Flush count: 0
  Packet Count: 0
  Packets per second: 0
  Metric count: 0
  Event count: 0
  Service check count: 0
 
====================
Forwarder (v 5.16.0)
====================
 
  Status date: 2017-08-09 15:31:27 (0s ago)
  Pid: 14983
  Platform: Linux-4.4.0-81-generic-x86_64-with-Ubuntu-16.04-xenial
  Python Version: 2.7.13, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log, syslog:/dev/log
 
  Queue Size: 606 bytes
  Queue Length: 1
  Flush Count: 3
  Transactions received: 2
  Transactions flushed: 1
  Transactions rejected: 0
  API Key Status: API Key is valid
 
 
======================
Trace Agent (v 5.16.0)
======================
 
  Pid: 14982
  Uptime: 13 seconds
  Mem alloc: 1028744 bytes
 
  Hostname: [redacted]
  Receiver: localhost:8126
  API Endpoint: https://trace.agent.datadoghq.com
 
  Bytes received (1 min): 0
  Traces received (1 min): 0
  Spans received (1 min): 0
 
  Bytes sent (1 min): 0
  Traces sent (1 min): 0
  Stats sent (1 min): 0

Additional environment details (Operating System, Cloud provider, etc):
4.4.0-81-generic #104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.3 LTS (Xenial Xerus)"

Steps to reproduce the issue:

1. Write a custom check such as /etc/dd-agent/checks.d/example.py with a syntax error:

from checks import AgentCheck

class ExampleCheck(AgentCheck):
    def check(self, instance):
    self.gauge('hello.world', 1)

2. Create the corresponding yaml file in /etc/dd-agent/conf.d/example.yaml:

init_config:

instances:
  [{}]

3. sudo /etc/init.d/datadog-agent restart
4. sudo /etc/init.d/datadog-agent info

Describe the results you received:
The above-described error output, ultimately leading only as far as error in line 546 of check_status.py:
TypeError: object of type 'NoneType' has no len()

Describe the results you expected:
A handled exception with meaningful output that stipulates there is a syntax error in a custom check.
Ideally it either raises the root-cause custom check exception that shows the responsible line of code, or provides information which custom check triggered the exception to allow separate debugging.

Additional information you deem important (e.g. issue happens only occasionally):
While this should be encountered with all types of syntax errors, in some specific cases where checking with Python for syntax errors on one host succeeds, but it fails on another, can result in confusion about what has caused the issue, leading to many hours of "black-box" debugging that can be avoided with an indication of the custom check having a syntax error when ran by the agent.

[truthbk]

@insideClaw man, that was one awesome issue description. Thank you so much for the breakdown and investigation 🙇

All we have to do know is fix it. I'll make sure we do so - should be a small quick fix - and ship in the next release.

[insideClaw]

Excellent, I'm glad you found it well put.
Also that I'm not misguided of it being an issue - the Datadog support person who I spoke to was originally convincing me that's normal Python behaviour :)

Good luck with the fix and I'll be looking forward to your release.

[truthbk]

@insideClaw a fix was merged, should be available in the upcoming release, due very soon. Thanks again! 👍

[truthbk]

[info] unknown ver if failed module load, custom if custom check.