Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vSphere new implementation #5251

Merged
merged 23 commits into from
Feb 3, 2020
Merged

vSphere new implementation #5251

merged 23 commits into from
Feb 3, 2020

Conversation

FlorianVeaux
Copy link
Member

@FlorianVeaux FlorianVeaux commented Dec 19, 2019

Re-implementing the vSphere integration for multiple reasons:

  • Simplifying the logic
  • Removing legacy (deprecated configuration field, deprecated agent 5 logic with a single check instance...)
  • Improving performance (less api calls as we don't query the list of metrics anymore, making every metric configurable)
  • Stop relying on a 10 years old non-maintained ThreadPool implementation
  • Include bug fixes that couldn't be added in the previous implementation without making the integration unusable for existing users.

Missing parts that will come in separate PRs:

  • Collecting tags from vCenter (requires an additional library/REST calls). Will be done in a separate PR.
  • Collecting per-instance metrics (instance is basically an extra dimension for metrics. For example, vsphere.cpu.usage can be tagged with instance:%d where the integer is the CPU core identifier). For now we do not collect those as it is expensive but we should add the option to collect them. Will be in a separate PR.

Major things to look for:

  • The way filters are implemented
  • The way concurrency is implemented
  • Any opinion regarding the config or implementation designs.

Note: Because we moved the legacy vsphere.py file to vsphere_legacy.py and because the new implementation also has to be named vsphere.py, git tries to show a diff between the two implementation which is a bit confusing. You can look the latest version of the vsphere.py file just here
Note2: The integration is running against our lab in our dddev environment.
Note3: I haven't checked yet if metadata.csv is up to date but in theory it should.
Note4: Don't mind the number of lines, most of it is either big fixture files or just moving the legacy code somewhere else.

@FlorianVeaux FlorianVeaux force-pushed the florian/vsphere_v2 branch 2 times, most recently from dc51c68 to 2e1e520 Compare December 27, 2019 12:35
@FlorianVeaux FlorianVeaux marked this pull request as ready for review December 27, 2019 12:35
@FlorianVeaux FlorianVeaux requested review from a team as code owners December 27, 2019 12:35
@codecov
Copy link

codecov bot commented Dec 27, 2019

@codecov
Copy link

codecov bot commented Jan 3, 2020

Copy link
Contributor

@ofek ofek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So great! Amazing docs too 👍

Comment on lines 52 to 64
if not self.ssl_verify:
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.verify_mode = ssl.CERT_NONE
elif self.ssl_capath:
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.verify_mode = ssl.CERT_REQUIRED
context.load_verify_locations(capath=self.ssl_capath)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to copy what you can from

if self._tls_verify: # no cov
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext
# https://docs.python.org/3/library/ssl.html#ssl.PROTOCOL_TLS
tls_context = ssl.SSLContext(protocol=PROTOCOL_TLS_CLIENT)
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext.verify_mode
tls_context.verify_mode = ssl.CERT_REQUIRED
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext.check_hostname
tls_context.check_hostname = self._validate_hostname
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations
if self._cafile or self._capath:
tls_context.load_verify_locations(self._cafile, self._capath, None)
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_default_certs
else:
tls_context.load_default_certs(ssl.Purpose.SERVER_AUTH)
# https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain
if self._cert:
tls_context.load_cert_chain(self._cert, keyfile=self._private_key)

Copy link
Member

@AlexandreYang AlexandreYang Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's something we do over and over. Can we extract that as a reusable util? (Probably in another PR)

vsphere/datadog_checks/vsphere/api.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/data/conf.yaml.example Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/__init__.py Outdated Show resolved Hide resolved
Clean up config options

Add dependency

Re-implement excluded_host_tags work

Fix tests

Fetch remaining metrics

Clean-up

Fix dependencies

fix style

[review] max_query_metrics exception

[review] Document self._conn.content type

[review] Address minor comments
Copy link
Member

@AlexandreYang AlexandreYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Part 1)

Still need to review few more things :)

vsphere/datadog_checks/vsphere/api.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Show resolved Hide resolved
vsphere/datadog_checks/vsphere/api.py Show resolved Hide resolved
vsphere/datadog_checks/vsphere/utils.py Outdated Show resolved Hide resolved
Copy link
Member

@AlexandreYang AlexandreYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Part 2/2)

Let me know if some comments are not clear :)

vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/cache.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/utils.py Outdated Show resolved Hide resolved
vsphere/datadog_checks/vsphere/metrics.py Show resolved Hide resolved
vsphere/datadog_checks/vsphere/vsphere.py Outdated Show resolved Hide resolved
@FlorianVeaux
Copy link
Member Author

FlorianVeaux commented Jan 10, 2020

Question: Should ressource_filters expect metrics to be prefixed with vsphere or not?

EDIT: Added documentation and leaving it as is

@FlorianVeaux
Copy link
Member Author

FlorianVeaux commented Jan 10, 2020

Question2: Should we allow collection_type: both
Edit: Added

vsphere/tests/test_filters.py Outdated Show resolved Hide resolved
vsphere/tests/test_filters.py Outdated Show resolved Hide resolved
AlexandreYang
AlexandreYang previously approved these changes Jan 24, 2020
Copy link
Member

@AlexandreYang AlexandreYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome looks good to me.

A PR desc of what changed and motivation for this refactoring can help.

ofek
ofek previously approved these changes Jan 24, 2020
@FlorianVeaux FlorianVeaux dismissed stale reviews from ofek and AlexandreYang via 55b49cb January 24, 2020 17:59
ofek
ofek previously approved these changes Jan 24, 2020
Copy link
Contributor

@ofek ofek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

mor_type = type(mor)
if mor_type not in self._content:
self._content[mor_type] = {}
self._content[mor_type][mor] = mor_data
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use something else than a MOR object as cache key for self._content[mor_type][mor] ?

Maybe use a "MOR Id".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into pyvmomi code https://github.com/vmware/pyvmomi/blob/master/pyVmomi/VmomiSupport.py#L596-L610.

What this means is that is it safe to use the mor directly as the cache key, the string representation of it should be unique so there should be no hash collision, and even in case of the same string for two different mors, the __eq__ method should prevent collisions.

So another possibility would be to use str(mor) as the cache key, but note that we would lose a bit in case two mors have the same string representation. Note also that the str method can be quite complex in regard of the very simple __hash__ and __eq__ methods.

Copy link
Member

@AlexandreYang AlexandreYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants