Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] add flare feature (contact support) #1422

Merged
merged 6 commits into from
Mar 18, 2015
Merged

[core] add flare feature (contact support) #1422

merged 6 commits into from
Mar 18, 2015

Conversation

degemer
Copy link
Member

@degemer degemer commented Mar 12, 2015

  • Create a Flare class to collect all relevant files (logs &
    configuration files), and then upload them to datadog, which will
    transfer them to support
  • Put all code relative to the configcheck in a function for reusability
  • add the flare option to init.d script

As I said, it adds some imports and probably belongs somewhere else. (at least to be easily accessible from the GUI for Windows/Mac)

@degemer degemer added this to the 5.3.0 milestone Mar 12, 2015
@LeoCavaille
Copy link
Member

What about starting to have a real utils/ subclass where we can migrate utils. I hate this toplevel util thingie.
from utils.flare import Flare ?

from checks.check_status import CollectorStatus, DogstatsdStatus, ForwarderStatus
from config import get_config, get_system_stats, get_parsed_args,\
load_check_directory, get_logging_config, check_yaml,\
get_config, get_config_path, get_confd_path
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_config duped. To avoid that even better:

from config import (
     sorted symbols a-z
     ...
)

@LeoCavaille
Copy link
Member

Info that could be added:

  • pip freeze from the self-contained agent directory ?
  • version string from config.py (in case the pickle stuff doesn't work)

# Save logs file paths
def _save_logs_path(self, config):
prefix = ''
if get_os() == 'windows':
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use Platform.is_windows() instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also you should probably use the get_logging_config function https://github.com/DataDog/dd-agent/blob/master/config.py#L900

@LeoCavaille
Copy link
Member

Awesome project @degemer! Looking forward to using it and saving a lot of time for our customers and ourselves. cc @DBLaw

@remh
Copy link
Contributor

remh commented Mar 12, 2015

+1 great job overall @degemer !

degemer added a commit that referenced this pull request Mar 12, 2015
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
@degemer
Copy link
Member Author

degemer commented Mar 12, 2015

Thanks @LeoCavaille & @remh !
Léo: even if the pickle isn't there, the version string is shown. I will have a look tomorrow for pip freeze.
I think I addressed all your comments, here is an example output:

2015-03-12 22:51:06,106 | INFO | dd.collector | flare(flare.py:91) | Collecting logs and configuration files:
2015-03-12 22:51:06,107 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/collector.log
2015-03-12 22:51:06,111 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/forwarder.log.1
2015-03-12 22:51:07,767 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/forwarder.log
2015-03-12 22:51:08,330 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/dogstatsd.log
2015-03-12 22:51:08,637 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/dogstatsd.log.1
2015-03-12 22:51:08,650 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/jmxfetch.log
2015-03-12 22:51:08,876 | INFO | dd.collector | flare(flare.py:163) |   * /var/log/datadog/supervisord.log
2015-03-12 22:51:08,890 | INFO | dd.collector | flare(flare.py:172) |   * /etc/dd-agent/datadog.conf
2015-03-12 22:51:08,892 | INFO | dd.collector | flare(flare.py:183) |   * /etc/dd-agent/supervisor.conf
2015-03-12 22:51:08,907 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/fluentd.yaml
2015-03-12 22:51:08,912 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/jmx.yaml
2015-03-12 22:51:08,920 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/activemq.yaml - this file contains a password which has been removed in the version collected
2015-03-12 22:51:08,924 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/etcd.yaml
2015-03-12 22:51:08,927 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/mysql.yaml - this file contains a password which has been removed in the version collected
2015-03-12 22:51:08,934 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/consul.yaml
2015-03-12 22:51:08,936 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/nginx.yaml
2015-03-12 22:51:08,939 | INFO | dd.collector | flare(flare.py:211) |   * /etc/dd-agent/conf.d/network.yaml
2015-03-12 22:51:08,940 | INFO | dd.collector | flare(flare.py:95) |   * datadog-agent configcheck output
2015-03-12 22:51:09,017 | INFO | dd.collector | flare(flare.py:97) |   * datadog-agent status output
2015-03-12 22:51:10,102 | INFO | dd.collector | flare(flare.py:99) |   * datadog-agent info output
2015-03-12 22:51:10,286 | INFO | dd.collector | flare(flare.py:102) | Saving all files to /tmp/datadog-agent-2015-03-12-22-51-06.tar.bz2
/tmp/datadog-agent-2015-03-12-22-51-06.tar.bz2 is going to be uploaded to Datadog.
Do you want to continue [Y/n]?
Please enter your email: *****@datadoghq.com
2015-03-12 22:51:19,913 | INFO | dd.collector | flare(flare.py:114) | Uploading /tmp/datadog-agent-2015-03-12-22-51-06.tar.bz2 to Datadog Support
2015-03-12 22:51:20,360 | INFO | dd.collector | flare(flare.py:300) | Your logs were successfully uploaded. For future reference, your internal case id is 666

degemer added a commit that referenced this pull request Mar 13, 2015
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
degemer added a commit that referenced this pull request Mar 13, 2015
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
- add pip freeze output
@degemer
Copy link
Member Author

degemer commented Mar 13, 2015

I added the pip freeze output.
I didn't use urljoin since it doesn't do what I want:

> urljoin('/zendesk/flare', '146')
'/zendesk/146'

degemer added a commit that referenced this pull request Mar 13, 2015
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
- add pip freeze output
'hostname': self._hostname,
'email': email
}
r = requests.post(url, files=files, data=data)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pass the apikey to the url as a query string parameter?

@degemer
Copy link
Member Author

degemer commented Mar 16, 2015

  • Send api_key as a query parameter. (@remh comment)
  • Update flare url to something more generic


# Print output (success/error) of the request
def _analyse_result(self, resp):
if resp.status_code == 200:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick but you should likely use:

resp.raise_for_status():

It will:

  • Make sure that if one day we change the response code from 200 to 202 (which we should), your code would still work
  • Format an exception message in a nicer way so you won't have to do it yourself.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I try to send explicit errors (like 'Apikey unknown' , 'This case is not yours') from the server, which I then catch and display whereas raise_for_status only gives the status code and the standard error associated. I should use ranges for status codes though.

@remh
Copy link
Contributor

remh commented Mar 16, 2015

Looks almost good! Could you add a test to make sure that the endpoint exist please?

@degemer degemer force-pushed the quentin/flare branch 2 times, most recently from 2bfd747 to 9d5a639 Compare March 17, 2015 20:07
degemer added a commit that referenced this pull request Mar 17, 2015
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
- add pip freeze output
@degemer
Copy link
Member Author

degemer commented Mar 18, 2015

I added some test for the upload, and another on testing the existence of the endpoint (marked as failing for now).
First try with unittest, there is probably a better way to write them.

# Print output (success/error) of the request
def _analyse_result(self):
try:
json_resp = json.loads(self._resp.text)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You coud just do:

json_resp = self._resp.json()

degemer added 6 commits March 18, 2015 19:21
- Create a Flare class to collect all relevant files (logs &
  configuration files), and then upload them to datadog, which will
  transfer them to support
- Put all code relative to the configcheck in a function for reusability
- add the flare option to init.d script
- create a directory utils
- move configcheck and flare into utils
- address various comments #1422
- add pip freeze output
& rename support URL into something more generic
Create a function in config to get new endpoints, and use it also in the
forwarder
- test the upload part (mocking requests.post )
- test the existence of the endpoint (marked as failing for now)
- clean configcheck
- add flare option to centos and source install commands
- rewrite the analyse_result part (dealing first with our custom 400
  errors, then with standard errors, and then with success)
- fix endpoint test
@remh
Copy link
Contributor

remh commented Mar 18, 2015

Awesome!

remh added a commit that referenced this pull request Mar 18, 2015
[core] add flare feature (contact support)
@remh remh merged commit 68d7e4a into master Mar 18, 2015
@LeoCavaille
Copy link
Member

Sweeeeet! 💯
I am so looking forward to seeing it used on support cases!

Also protip @degemer : I did it for you but don't forget to delete your branches once they're merged to master.

@LeoCavaille LeoCavaille deleted the quentin/flare branch March 19, 2015 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants