Merge branch 'main' into unwrap-celery-exceptioninfo

CHANGELOG.md

@@ -6,11 +6,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## Unreleased
+- `opentelemetry-instrumentation-asgi` Add `http.server.request.size` metric
+  ([#1867](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1867))
+
+### Fixed
+
+- Fix elastic-search instrumentation sanitization to support bulk queries
+  ([#1870](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1870))
+- Update falcon instrumentation to follow semantic conventions
+  ([#1824](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1824))
+- Fix sqlalchemy instrumentation wrap methods to accept sqlcommenter options([#1873](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1873))
 
 - `opentelemetry-instrumentation-celery` Unwrap Celery's `ExceptionInfo` errors and report the actual exception that was raised. ([#1863](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1863))
 
 ### Added
 
+- Fix async redis clients not being traced correctly ([#1830](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1830))
 - Make Flask request span attributes available for `start_span`. 
   ([#1784](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1784))
 - Fix falcon instrumentation's usage of Span Status to only set the description if the status code is ERROR.
@@ -22,12 +33,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   ([#1679](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1679))
 - `opentelemetry-instrumentation-asgi` Add `http.server.response.size` metric
   ([#1789](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1789))
+- `opentelemetry-instrumentation-grpc` Allow gRPC connections via Unix socket
+  ([#1833](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1833))
+- Fix elasticsearch `Transport.perform_request` instrument wrap for elasticsearch >= 8
+  ([#1810](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1810))
 
 ## Version 1.18.0/0.39b0 (2023-05-10)
 
 - `opentelemetry-instrumentation-system-metrics` Add `process.` prefix to `runtime.memory`, `runtime.cpu.time`, and `runtime.gc_count`. Change `runtime.memory` from count to UpDownCounter. ([#1735](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1735))
 - Add request and response hooks for GRPC instrumentation (client only)
   ([#1706](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1706))
+- Fix memory leak in SQLAlchemy instrumentation where disposed `Engine` does not get garbage collected
+  ([#1771](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1771)
 - `opentelemetry-instrumentation-pymemcache` Update instrumentation to support pymemcache >4
   ([#1764](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1764))
 - `opentelemetry-instrumentation-confluent-kafka` Add support for higher versions of confluent_kafka

README.md

@@ -49,6 +49,8 @@ depend on `opentelemetry-sdk` or another package that implements the API.
 **Please note** that these libraries are currently in _beta_, and shouldn't
 generally be used in production environments.
 
+Unless explicitly stated otherwise, any instrumentation here for a particular library is not developed or maintained by the authors of such library.
+
 The
 [`instrumentation/`](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation)
 directory includes OpenTelemetry instrumentation packages, which can be installed
@@ -97,7 +99,6 @@ Approvers ([@open-telemetry/python-approvers](https://github.com/orgs/open-telem
 - [Aaron Abbott](https://github.com/aabmass), Google
 - [Jeremy Voss](https://github.com/jeremydvoss), Microsoft
 - [Sanket Mehta](https://github.com/sanketmehta28), Cisco
-- [Shalev Roda](https://github.com/shalevr), Cisco
 
 Emeritus Approvers:
 
@@ -112,6 +113,7 @@ Maintainers ([@open-telemetry/python-maintainers](https://github.com/orgs/open-t
 
 - [Diego Hurtado](https://github.com/ocelotl), Lightstep
 - [Leighton Chen](https://github.com/lzchen), Microsoft
+- [Shalev Roda](https://github.com/shalevr), Cisco
 
 Emeritus Maintainers:
 

dev-requirements.txt

@@ -16,5 +16,5 @@ mypy-protobuf>=1.23
 protobuf~=3.13
 markupsafe>=2.0.1
 codespell==2.1.0
-requests==2.28.1
+requests==2.31.0
 ruamel.yaml==0.17.21

instrumentation/opentelemetry-instrumentation-asgi/src/opentelemetry/instrumentation/asgi/__init__.py

@@ -511,6 +511,11 @@ def __init__(
             unit="By",
             description="measures the size of HTTP response messages (compressed).",
         )
+        self.server_request_size_histogram = self.meter.create_histogram(
+            name=MetricInstruments.HTTP_SERVER_REQUEST_SIZE,
+            unit="By",
+            description="Measures the size of HTTP request messages (compressed).",
+        )
         self.active_requests_counter = self.meter.create_up_down_counter(
             name=MetricInstruments.HTTP_SERVER_ACTIVE_REQUESTS,
             unit="requests",
@@ -603,6 +608,16 @@ async def __call__(self, scope, receive, send):
                     self.server_response_size_histogram.record(
                         self.content_length_header, duration_attrs
                     )
+                request_size = asgi_getter.get(scope, "content-length")
+                if request_size:
+                    try:
+                        request_size_amount = int(request_size[0])
+                    except ValueError:
+                        pass
+                    else:
+                        self.server_request_size_histogram.record(
+                            request_size_amount, duration_attrs
+                        )
             if token:
                 context.detach(token)
 

instrumentation/opentelemetry-instrumentation-asgi/tests/test_asgi_middleware.py

@@ -47,16 +47,19 @@
     "http.server.active_requests",
     "http.server.duration",
     "http.server.response.size",
+    "http.server.request.size",
 ]
 _recommended_attrs = {
     "http.server.active_requests": _active_requests_count_attrs,
     "http.server.duration": _duration_attrs,
     "http.server.response.size": _duration_attrs,
+    "http.server.request.size": _duration_attrs,
 }
 
 
 async def http_app(scope, receive, send):
     message = await receive()
+    scope["headers"] = [(b"content-length", b"128")]
     assert scope["type"] == "http"
     if message.get("type") == "http.request":
         await send(
@@ -99,6 +102,7 @@ async def error_asgi(scope, receive, send):
     assert isinstance(scope, dict)
     assert scope["type"] == "http"
     message = await receive()
+    scope["headers"] = [(b"content-length", b"128")]
     if message.get("type") == "http.request":
         try:
             raise ValueError
@@ -592,6 +596,8 @@ def test_basic_metric_success(self):
                                 )
                             elif metric.name == "http.server.response.size":
                                 self.assertEqual(1024, point.sum)
+                            elif metric.name == "http.server.request.size":
+                                self.assertEqual(128, point.sum)
                         elif isinstance(point, NumberDataPoint):
                             self.assertDictEqual(
                                 expected_requests_count_attributes,
@@ -630,7 +636,7 @@ async def target_asgi(scope, receive, send):
                                 expected_target,
                             )
                             assertions += 1
-        self.assertEqual(assertions, 2)
+        self.assertEqual(assertions, 3)
 
     def test_no_metric_for_websockets(self):
         self.scope = {

instrumentation/opentelemetry-instrumentation-elasticsearch/src/opentelemetry/instrumentation/elasticsearch/__init__.py

@@ -98,6 +98,12 @@ def response_hook(span, response):
 
 from .utils import sanitize_body
 
+# Split of elasticsearch and elastic_transport in 8.0.0+
+# https://www.elastic.co/guide/en/elasticsearch/client/python-api/master/release-notes.html#rn-8-0-0
+es_transport_split = elasticsearch.VERSION[0] > 7
+if es_transport_split:
+    import elastic_transport
+
 logger = getLogger(__name__)
 
 
@@ -137,16 +143,28 @@ def _instrument(self, **kwargs):
         tracer = get_tracer(__name__, __version__, tracer_provider)
         request_hook = kwargs.get("request_hook")
         response_hook = kwargs.get("response_hook")
-        _wrap(
-            elasticsearch,
-            "Transport.perform_request",
-            _wrap_perform_request(
-                tracer,
-                self._span_name_prefix,
-                request_hook,
-                response_hook,
-            ),
-        )
+        if es_transport_split:
+            _wrap(
+                elastic_transport,
+                "Transport.perform_request",
+                _wrap_perform_request(
+                    tracer,
+                    self._span_name_prefix,
+                    request_hook,
+                    response_hook,
+                ),
+            )
+        else:
+            _wrap(
+                elasticsearch,
+                "Transport.perform_request",
+                _wrap_perform_request(
+                    tracer,
+                    self._span_name_prefix,
+                    request_hook,
+                    response_hook,
+                ),
+            )
 
     def _uninstrument(self, **kwargs):
         unwrap(elasticsearch.Transport, "perform_request")

instrumentation/opentelemetry-instrumentation-elasticsearch/src/opentelemetry/instrumentation/elasticsearch/utils.py

@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import json
 
 sanitized_keys = (
     "message",
@@ -51,6 +52,9 @@ def _unflatten_dict(d):
 
 
 def sanitize_body(body) -> str:
+    if isinstance(body, str):
+        body = json.loads(body)
+
     flatten_body = _flatten_dict(body)
 
     for key in flatten_body:

instrumentation/opentelemetry-instrumentation-elasticsearch/tests/helpers_es8.py

@@ -0,0 +1,41 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from elasticsearch_dsl import Document, Keyword, Text
+
+
+class Article(Document):
+    title = Text(analyzer="snowball", fields={"raw": Keyword()})
+    body = Text(analyzer="snowball")
+
+    class Index:
+        name = "test-index"
+
+
+dsl_create_statement = {
+    "mappings": {
+        "properties": {
+            "title": {
+                "analyzer": "snowball",
+                "fields": {"raw": {"type": "keyword"}},
+                "type": "text",
+            },
+            "body": {"analyzer": "snowball", "type": "text"},
+        }
+    }
+}
+dsl_index_result = (1, {}, '{"result": "created"}')
+dsl_index_span_name = "Elasticsearch/test-index/_doc/2"
+dsl_index_url = "/test-index/_doc/2"
+dsl_search_method = "POST"

instrumentation/opentelemetry-instrumentation-elasticsearch/tests/test_elasticsearch.py

@@ -37,7 +37,9 @@
 
 major_version = elasticsearch.VERSION[0]
 
-if major_version == 7:
+if major_version == 8:
+    from . import helpers_es8 as helpers  # pylint: disable=no-name-in-module
+elif major_version == 7:
     from . import helpers_es7 as helpers  # pylint: disable=no-name-in-module
 elif major_version == 6:
     from . import helpers_es6 as helpers  # pylint: disable=no-name-in-module
@@ -479,3 +481,7 @@ def test_body_sanitization(self, _):
             sanitize_body(sanitization_queries.filter_query),
             str(sanitization_queries.filter_query_sanitized),
         )
+        self.assertEqual(
+            sanitize_body(json.dumps(sanitization_queries.interval_query)),
+            str(sanitization_queries.interval_query_sanitized),
+        )

instrumentation/opentelemetry-instrumentation-falcon/src/opentelemetry/instrumentation/falcon/__init__.py

@@ -428,7 +428,6 @@ def process_resource(self, req, resp, resource, params):
 
         resource_name = resource.__class__.__name__
         span.set_attribute("falcon.resource", resource_name)
-        span.update_name(f"{resource_name}.on_{req.method.lower()}")
 
     def process_response(
         self, req, resp, resource, req_succeeded=None
@@ -483,6 +482,12 @@ def process_response(
                 response_headers = resp.headers
 
             if span.is_recording() and span.kind == trace.SpanKind.SERVER:
+                # Check if low-cardinality route is available as per semantic-conventions
+                if req.uri_template:
+                    span.update_name(f"{req.method} {req.uri_template}")
+                else:
+                    span.update_name(f"{req.method}")
+
                 custom_attributes = (
                     otel_wsgi.collect_custom_response_headers_attributes(
                         response_headers.items()

instrumentation/opentelemetry-instrumentation-falcon/tests/app.py

@@ -61,6 +61,13 @@ def on_get(self, _, resp):
         resp.set_header("my-secret-header", "my-secret-value")
 
 
+class UserResource:
+    def on_get(self, req, resp, user_id):
+        # pylint: disable=no-member
+        resp.status = falcon.HTTP_200
+        resp.body = f"Hello user {user_id}"
+
+
 def make_app():
     _parsed_falcon_version = package_version.parse(falcon.__version__)
     if _parsed_falcon_version < package_version.parse("3.0.0"):
@@ -76,4 +83,6 @@ def make_app():
     app.add_route(
         "/test_custom_response_headers", CustomResponseHeaderResource()
     )
+    app.add_route("/user/{user_id}", UserResource())
+
     return app

instrumentation/opentelemetry-instrumentation-falcon/tests/test_falcon.py

@@ -110,7 +110,7 @@ def _test_method(self, method):
         spans = self.memory_exporter.get_finished_spans()
         self.assertEqual(len(spans), 1)
         span = spans[0]
-        self.assertEqual(span.name, f"HelloWorldResource.on_{method.lower()}")
+        self.assertEqual(span.name, f"{method} /hello")
         self.assertEqual(span.status.status_code, StatusCode.UNSET)
         self.assertEqual(
             span.status.description,
@@ -145,7 +145,7 @@ def test_404(self):
         spans = self.memory_exporter.get_finished_spans()
         self.assertEqual(len(spans), 1)
         span = spans[0]
-        self.assertEqual(span.name, "GET /does-not-exist")
+        self.assertEqual(span.name, "GET")
         self.assertEqual(span.status.status_code, StatusCode.UNSET)
         self.assertSpanHasAttributes(
             span,
@@ -177,7 +177,7 @@ def test_500(self):
         spans = self.memory_exporter.get_finished_spans()
         self.assertEqual(len(spans), 1)
         span = spans[0]
-        self.assertEqual(span.name, "ErrorResource.on_get")
+        self.assertEqual(span.name, "GET /error")
         self.assertFalse(span.status.is_ok)
         self.assertEqual(span.status.status_code, StatusCode.ERROR)
         self.assertEqual(
@@ -206,6 +206,33 @@ def test_500(self):
                 span.attributes[SpanAttributes.NET_PEER_IP], "127.0.0.1"
             )
 
+    def test_url_template(self):
+        self.client().simulate_get("/user/123")
+        spans = self.memory_exporter.get_finished_spans()
+        self.assertEqual(len(spans), 1)
+        span = spans[0]
+        self.assertEqual(span.name, "GET /user/{user_id}")
+        self.assertEqual(span.status.status_code, StatusCode.UNSET)
+        self.assertEqual(
+            span.status.description,
+            None,
+        )
+        self.assertSpanHasAttributes(
+            span,
+            {
+                SpanAttributes.HTTP_METHOD: "GET",
+                SpanAttributes.HTTP_SERVER_NAME: "falconframework.org",
+                SpanAttributes.HTTP_SCHEME: "http",
+                SpanAttributes.NET_HOST_PORT: 80,
+                SpanAttributes.HTTP_HOST: "falconframework.org",
+                SpanAttributes.HTTP_TARGET: "/",
+                SpanAttributes.NET_PEER_PORT: "65133",
+                SpanAttributes.HTTP_FLAVOR: "1.1",
+                "falcon.resource": "UserResource",
+                SpanAttributes.HTTP_STATUS_CODE: 200,
+            },
+        )
+
     def test_uninstrument(self):
         self.client().simulate_get(path="/hello")
         spans = self.memory_exporter.get_finished_spans()