Add an autoinstrumentation mechanism and an instrumentor for Flask #327

ocelotl · 2019-12-11T00:39:25Z

Fixes #300

ocelotl · 2019-12-11T00:58:01Z

This PR includes an autoinstrumentation mechanism and a Flask instrumentor (an instrumentor is a class that implements the _instrument and _uninstrument methods).

It works like this:

A console command is defined here. This makes it possible to run a command named opentelemetry-auto-instrumentation that will execute this function.
When the opentelemetry-auto-instrumentation command is executed, then the instrument method of the different instrumentors is called.
2.In the case of the Flask instrumentor, the original flask.Flask gets replaced with _InstrumentedFlask (in this case, the Flask instrumentor uses monkey patching to perform the instrumentation, nevertheless, monkey patching is not always the method used to do this, so the name instrumentor is preferred over patcher).
Once this is done, the app gets executed here.

ocelotl · 2019-12-20T17:26:36Z

Please leave your comments 👍

toumorokoshi

Nice! this is a big step, thanks for focusing on this.

My two cents (and I think we need some more opinions here) are:

Enable programmatic control of patching

I'm not a huge fan of the patch executable (like auto_agent) myself, because I think that inverts control away from the application author and toward some other entity. These are hard to chain (e.g. if something else wants to go modify the import and startup logic), and hard to customize.

Looking at dd-agent-py, which we're taking inspiration from, customization of the dd agent happens via several environment variables. https://docs.datadoghq.com/tracing/setup/python/#environment-variable. I've personally found environment variables (in essence a dictionary of key-value pairs) fairly limiting, as it doesn't enable things like more complex types or programmatic control.

Also I don't know how robust or flexible sitecustomize is as an interface. Will it impact it's use with other systems that modify sitecustomize, such as zc.buildout?

Patching Interface

I think the patch interface looks good. I feel like some extensions will probably want to expose more primitives (e.g. flasks' instrument_app) to allow finer control of the extensions themselves.

Overall I think this achieves standardization of patching for auto-instrumentation, which is great, as well as it's utilization of setuptools entry points which is best practice.

ext/opentelemetry-ext-flask/src/opentelemetry/ext/flask/__init__.py

ext/opentelemetry-ext-flask/tests/test_flask_integration.py

toumorokoshi · 2019-12-30T22:48:34Z

ext/opentelemetry-ext-flask/tests/test_flask_integration.py

@@ -36,7 +36,9 @@ def setspanattr(key, value):

        self.span.set_attribute = setspanattr

-        self.app = Flask(__name__)
+        FlaskPatcher().patch()


will this cause issues with other unit tests that might want the raw Flask, as it will permanently Flask with an instrumented variety?

We may also use Flask apps to bring up example test servers, or we may in the future.

There's a technique used to reload modules we could use during teardown to ensure it gets set back: f328909

It may cause issues, but I think it is necessary for patching purposes. I can investigate a way that the patching is done in a way that only happens for the testing purposes of the patching agent. Also, I can think about a unpatching method that gets called automatically when patching is no longer needed.

Does the reload function not work for your purposes? I believe it should reset everything you've patched: https://docs.python.org/3/library/importlib.html#importlib.reload

Although maybe a best-effort unpatch API is also a good idea as it'll be hard for people to be aware of the new patches that were added from version to version, so they may find their previous custom unpatching function not unpatching new patches added by a newer version of the auto-instrumentation.

toumorokoshi · 2019-12-30T22:50:14Z

opentelemetry-api/setup.py

@@ -40,6 +40,14 @@
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
    ],
+    entry_points={
+        "console_scripts": [
+            "auto_agent = opentelemetry.commands.auto_agent:run"


I think "agent" is probably the wrong word. how about Instrument? auto_instrument would also match the opentelemetry spec name (autoinstrumentation) that we're trying to support here.

also we should probably prefix any commands with opentelemetry, as console_scripts is a global namespace.

so "opentelemetry-auto-instrument" as a final suggestion.

Ok, I'll have to read the documentation on console_scripts thoroughly, I am under the impression it must have this name for our purposes.

Never mind, I understand now what you mean.

opentelemetry-api/src/opentelemetry/commands/auto_agent.py

toumorokoshi · 2019-12-31T05:12:29Z

opentelemetry-api/src/opentelemetry/commands/initialize/sitecustomize.py

+
+_LOG = getLogger(__file__)
+
+for entry_point in iter_entry_points("opentelemetry_patcher"):


can this be also be available as a public method? I know we're looking at ddtrace for inspiration, and that codebase includes a way to patch things via patch_all.

ocelotl · 2020-01-02T17:49:49Z

Thanks for your comments, @toumorokoshi. 👍 Regarding the patch executable, I think there is value in having a fire-and-forget auto instrumentation executable that allows the user to just take non-instrumented code and make it instrumented automatically. This seems useful for situations when quick instrumentation is just needed an not much else needs consideration.

That being said, I see our users in need of configuration soon after they get the simplest of automatic instrumentations done. Of course, this means supporting both scenarios.

I'm not much a fan of environment variables myself, they are vulnerable to them being lost after a reboot or a new shell being started and they require usually long prefixes to keep them separated from each other. I prefer configuration files myself, since it is possible to keep them in version control. That being said, other people prefer command line arguments because they make it very easy to adjust stuff when making many runs of a command 🤷‍♂️. I'll look into this and will propose a solution.

You raise a valid point when you mention the lack of a programmatic approach to auto instrumentation. This looks like a job for a well defined and documented API that can be used to create customized auto instrumentation tools. Also, will look into this and will propose a solution.

Please let me know if you have more design-level ideas for me to consider.

Thanks!

codecov-io · 2020-01-08T23:07:37Z

Codecov Report

Merging #327 into master will increase coverage by 0.14%.
The diff coverage is 88.46%.

@@            Coverage Diff             @@
##           master     #327      +/-   ##
==========================================
+ Coverage   89.50%   89.64%   +0.14%     
==========================================
  Files          43       43              
  Lines        2229     2240      +11     
  Branches      248      248              
==========================================
+ Hits         1995     2008      +13     
  Misses        159      159              
+ Partials       75       73       -2

Impacted Files	Coverage Δ
...-ext-flask/src/opentelemetry/ext/flask/__init__.py	`90.16% <88.00%> (+0.16%)`	⬆️
...app/src/opentelemetry_example_app/flask_example.py	`100.00% <100.00%> (ø)`
...emetry-sdk/src/opentelemetry/sdk/trace/__init__.py	`92.38% <0.00%> (+0.34%)`	⬆️
...elemetry-api/src/opentelemetry/context/__init__.py	`95.45% <0.00%> (+4.54%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b1d866...5bb20d1. Read the comment docs.

ocelotl · 2020-01-08T23:18:07Z

I have fixed the last issues I had in this PR, @toumorokoshi, please let me know what you think.

Oberon00

Requesting changes because:

Please put everything new, maybe except BasePatcher, inside a new distribution, not in API.
Do not make unrelated style changes here, e.g. https://github.com/open-telemetry/opentelemetry-python/pull/327/files#r362112308 and related.

opentelemetry-api/src/opentelemetry/patcher/__init__.py

opentelemetry-api/tests/patcher/test_patcher.py

opentelemetry-api/src/opentelemetry/patcher/__init__.py

ext/opentelemetry-ext-flask/src/opentelemetry/ext/flask/__init__.py

opentelemetry-api/src/opentelemetry/auto_instrument/auto_instrument.py

opentelemetry-api/src/opentelemetry/auto_instrument/sitecustomize.py

ext/opentelemetry-ext-flask/tests/test_flask_integration.py

Oberon00 · 2020-01-31T15:35:01Z

Something like the BasePatcher interface is much-needed! 👍

ocelotl · 2020-01-31T23:11:42Z

Thank you for your review, @Oberon00!

codeboten

Requested a few changes, but this is looking pretty good! Excited to see this functionality in the library.

opentelemetry-sdk/src/opentelemetry/sdk/trace/__init__.py

opentelemetry-api/src/opentelemetry/patcher/__init__.py

examples/auto_instrumentation/README.md

examples/auto_instrumentation/publisher.py

examples/auto_instrumentation/publisher_untraced.py

examples/auto_instrumentation/utils.py

codeboten · 2020-02-04T23:49:41Z

opentelemetry-api/setup.py

@@ -40,6 +40,15 @@
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
    ],
+    entry_points={
+        "console_scripts": [


+1 on moving the console script to a separate package. Maybe it ends up moving into the https://github.com/open-telemetry/opentelemetry-auto-instr-python/ repo, just a thought

opentelemetry-api/setup.py

examples/auto_instrumentation/README.md

codeboten

Thanks for addressing my feedback! LGTM

mauriciovasquezbernal

Overall looks good to me.
I think the documentation needs a little bit of rework, specially because it has been changed a lot after this PR was opened.

About the example, I think the right place for that is docs/examples.

opentelemetry-auto-instrumentation/README.rst

...elemetry-auto-instrumentation/src/opentelemetry/auto_instrumentation/auto_instrumentation.py

opentelemetry-auto-instrumentation/tests/__init__.py

docs/index.rst

opentelemetry-auto-instrumentation/example/publisher_instrumented.py

ext/opentelemetry-ext-flask/tests/test_flask_integration.py

opentelemetry-auto-instrumentation/example/formatter.py

opentelemetry-auto-instrumentation/example/publisher_instrumented.py

mauriciovasquezbernal · 2020-03-25T20:06:16Z

opentelemetry-auto-instrumentation/README.rst

+
+The code in ``program.py`` needs to use one of the packages for which there is
+an OpenTelemetry extension. For a list of the available extensions please check
+`here <https://opentelemetry-python.readthedocsio/>`_.


It's better to use an "internal" target for this link. So if we decide to host our documentation somewhere else the generated link is not broken.

Makes sense, but what do you mean with "internal"? Can you provide an example, please?

Co-Authored-By: Mauricio Vásquez <mauricio@kinvolk.io>

toumorokoshi

I think there's still a lot of additional work that will have to go into auto-instrumentation in general, but I think the patterns you've laid our here look really good, and put us in a great position to actually build this out.

Also great idea to try to find a pattern that allows extensions to be leveraged for auto-instrumentation as well. I see that as as huge win for maintainability in the future.

c24t

LGTM! Thanks for pushing through this long review cycle @ocelotl.

My only remaining (and nit-level) comment is that opentelemetry_auto_instrumentation_instrumentor is a mouthfull, opentelemetry_instrumentor seems to describe the same thing and is much shorter.

c24t · 2020-03-30T18:58:37Z

opentelemetry-auto-instrumentation/src/opentelemetry/auto_instrumentation/__init__.py

@@ -0,0 +1,28 @@
+# Copyright The OpenTelemetry Authors


FYI it looks like this isn't included in the built docs.

Sorry, which docs are the built docs?

The ones you get running tox -e docs or make html in the docs dir.

ocelotl · 2020-03-30T19:24:25Z

LGTM! Thanks for pushing through this long review cycle @ocelotl.

My only remaining (and nit-level) comment is that opentelemetry_auto_instrumentation_instrumentor is a mouthfull, opentelemetry_instrumentor seems to describe the same thing and is much shorter.

👍 Thanks for the review! I have shortened the entry point name

ocelotl force-pushed the issue_300 branch 2 times, most recently from 1b300de to 577d5c6 Compare December 19, 2019 21:53

ocelotl added discussion Issue or PR that needs/is extended discussion. ext WIP Work in progress labels Dec 20, 2019

toumorokoshi reviewed Dec 31, 2019

View reviewed changes

ocelotl force-pushed the issue_300 branch 3 times, most recently from d55604c to a81cbc4 Compare January 7, 2020 19:51

ocelotl marked this pull request as ready for review January 8, 2020 23:17

ocelotl requested a review from a team January 8, 2020 23:17

toumorokoshi mentioned this pull request Jan 14, 2020

Research integrating auto- and manual instrumentations #300

Closed

ocelotl self-assigned this Jan 27, 2020

ocelotl force-pushed the issue_300 branch from 415ad10 to 41b308f Compare January 27, 2020 19:48

c24t requested a review from codeboten January 30, 2020 16:23

Oberon00 previously requested changes Jan 31, 2020

View reviewed changes

ocelotl force-pushed the issue_300 branch from 20888f9 to b222198 Compare February 3, 2020 18:08

codeboten suggested changes Feb 4, 2020

View reviewed changes

codeboten reviewed Feb 5, 2020

View reviewed changes

examples/auto_instrumentation/README.md Outdated Show resolved Hide resolved

codeboten suggested changes Feb 5, 2020

View reviewed changes

examples/auto_instrumentation/README.md Outdated Show resolved Hide resolved

ocelotl force-pushed the issue_300 branch from 90df1ef to 6d04ee5 Compare February 5, 2020 20:08

ocelotl force-pushed the issue_300 branch from 2665fb6 to 23e6c7e Compare February 17, 2020 23:47

ocelotl removed the WIP Work in progress label Feb 19, 2020

codeboten approved these changes Mar 24, 2020

View reviewed changes

ocelotl changed the title ~~Add autoinstrumentation prototype for Flask~~ Add an autoinstrumentation and an instrumentor for Flask Mar 24, 2020

ocelotl changed the title ~~Add an autoinstrumentation and an instrumentor for Flask~~ Add an autoinstrumentation mechanism and an instrumentor for Flask Mar 24, 2020

fix documents

31ec6c3

mauriciovasquezbernal reviewed Mar 25, 2020

View reviewed changes

ocelotl and others added 10 commits March 25, 2020 16:14

Add comment

b18efe3

Update opentelemetry-auto-instrumentation/README.rst

bc7a9a7

Co-Authored-By: Mauricio Vásquez <mauricio@kinvolk.io>

Update opentelemetry-auto-instrumentation/README.rst

06e5a2b

Co-Authored-By: Mauricio Vásquez <mauricio@kinvolk.io>

Move description to the package docstring

f23d004

Update opentelemetry-auto-instrumentation/example/README.md

3218bff

Co-Authored-By: Mauricio Vásquez <mauricio@kinvolk.io>

Use python instead of python3

6f36a33

Remove formatter

f0f182e

Change logger

0f65460

Update opentelemetry-auto-instrumentation/tests/__init__.py

4907663

Co-Authored-By: Mauricio Vásquez <mauricio@kinvolk.io>

Remove year

94f6167

ocelotl mentioned this pull request Mar 26, 2020

Add Flask instrumentor open-telemetry/opentelemetry-python-contrib#7

Closed

ocelotl added 4 commits March 26, 2020 14:09

Remove line

91803f9

Move examples

410817e

Remove blank line

fdf1774

Rename directory

c8df74b

toumorokoshi approved these changes Mar 29, 2020

View reviewed changes

Merge branch 'master' into issue_300

7172275

c24t approved these changes Mar 30, 2020

View reviewed changes

Merge branch 'master' into issue_300

2a56a92

Shorten entry point

5bb20d1

toumorokoshi merged commit a137bc2 into open-telemetry:master Mar 30, 2020

toumorokoshi mentioned this pull request Mar 30, 2020

docs: updating changelogs for 0.6 release #533

Merged

ocelotl mentioned this pull request Apr 6, 2020

Fix example in documentation #551

Merged

c24t mentioned this pull request Apr 6, 2020

Add opentelemetry-auto-instrumentation to build.sh #556

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an autoinstrumentation mechanism and an instrumentor for Flask #327

Add an autoinstrumentation mechanism and an instrumentor for Flask #327

ocelotl commented Dec 11, 2019

ocelotl commented Dec 11, 2019 •

edited

Loading

ocelotl commented Dec 20, 2019

toumorokoshi left a comment

toumorokoshi Dec 30, 2019

ocelotl Jan 1, 2020

toumorokoshi Jan 5, 2020

toumorokoshi Dec 30, 2019

toumorokoshi Dec 30, 2019

ocelotl Jan 1, 2020

ocelotl Jan 2, 2020

toumorokoshi Dec 31, 2019

ocelotl commented Jan 2, 2020

codecov-io commented Jan 8, 2020 •

edited

Loading

ocelotl commented Jan 8, 2020

Oberon00 left a comment

Oberon00 commented Jan 31, 2020

ocelotl commented Jan 31, 2020

codeboten left a comment

codeboten Feb 4, 2020

codeboten left a comment

mauriciovasquezbernal left a comment

mauriciovasquezbernal Mar 25, 2020

ocelotl Mar 25, 2020

toumorokoshi left a comment

c24t left a comment

c24t Mar 30, 2020

ocelotl Mar 30, 2020

c24t Mar 30, 2020

ocelotl commented Mar 30, 2020


		_LOG = getLogger(__file__)

		for entry_point in iter_entry_points("opentelemetry_patcher"):

Add an autoinstrumentation mechanism and an instrumentor for Flask #327

Add an autoinstrumentation mechanism and an instrumentor for Flask #327

Conversation

ocelotl commented Dec 11, 2019

ocelotl commented Dec 11, 2019 • edited Loading

ocelotl commented Dec 20, 2019

toumorokoshi left a comment

Choose a reason for hiding this comment

Enable programmatic control of patching

Patching Interface

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocelotl commented Jan 2, 2020

codecov-io commented Jan 8, 2020 • edited Loading

Codecov Report

ocelotl commented Jan 8, 2020

Oberon00 left a comment

Choose a reason for hiding this comment

Oberon00 commented Jan 31, 2020

ocelotl commented Jan 31, 2020

codeboten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

mauriciovasquezbernal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toumorokoshi left a comment

Choose a reason for hiding this comment

c24t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ocelotl commented Mar 30, 2020

ocelotl commented Dec 11, 2019 •

edited

Loading

codecov-io commented Jan 8, 2020 •

edited

Loading