Exporters: add prometheus remote write exporter #177

oktocat · 2021-10-13T14:50:56Z

closes: #122

Co-authored-by: Kevin Labesse kevin@labesse.me
Signed-off-by: Kevin Labesse kevin@labesse.me
Signed-off-by: Denis Zaitcev oktocat@yandex.com

oktocat · 2021-10-14T16:45:19Z

cc @ocervell

google-cla · 2021-10-15T10:14:03Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-10-15T10:22:38Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-10-15T10:29:22Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

oktocat · 2021-10-15T10:30:48Z

@googlebot I consent.

google-cla · 2021-10-15T10:30:48Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2021-10-15T10:30:51Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

oktocat · 2021-10-15T10:31:20Z

pffff whatever. Help me out here, I signed my soul over to the CLA already : )

eraac · 2021-10-15T11:31:10Z

@googlebot I consent.

closes: google#122 Co-authored-by: Kevin Labesse <kevin@labesse.me> Signed-off-by: Kevin Labesse <kevin@labesse.me> Signed-off-by: Denis Zaitcev <oktocat@yandex.com>

ocervell · 2021-11-03T09:15:26Z

Hi @oktocat , sorry for your issues with the bot ! I'll review this shortly, quite a big PR :)

Signed-off-by: Denis Zaitcev <oktocat@yandex.com>

ocervell · 2021-11-26T13:25:55Z

Makefile

+ @echo "running on Linux, assuming Ubuntu, installing snappy headers with apt"
+ sudo apt install libsnappy-dev -y || apt install libsnappy-dev -y
+endif
+ $(PIP) install -e ."[api, datadog, prometheus, elasticsearch, pubsub, cloud_monitoring, bigquery, dev, prometheus_remote_write]"


I'd prefer to keep the install clean of additional dependencies, can you move this to install_snappy function and call it after clean ?

ocervell

Looking great so far, thanks for adding support for this !
One thing is missing: adding the snappy install to the Dockerfile so that Docker installs can work with your exporter: could you please take care of this ?
Please also address the few points and merge the latest master into your branch so that tests can pass before we merge :)

ocervell · 2021-11-26T13:37:15Z

Makefile

@@ -74,11 +85,11 @@ coverage:
 lint: flake8 pylint

 flake8:
- flake8 --ignore=$(FLAKE8_IGNORE) $(NAME)/ --max-line-length=80
+ flake8 --ignore=$(FLAKE8_IGNORE) --exclude slo_generator/exporters/gen/ $(NAME)/ --max-line-length=80


can you please move the slo_generator/exporters/gen folder to slo_generator/proto so that we can split it from the rest of the exporters code ?

ocervell · 2021-11-26T13:38:38Z

slo_generator/exporters/prometheus_remote_write.py

+import requests
+import snappy
+from slo_generator.exporters.base import MetricsExporter
+from slo_generator.exporters.gen.remote_pb2 import (


ditto, this import will become from slo_generator.proto.remote_pb2 import after moving the generated files to the proto folder

skinlayers · 2023-08-03T05:49:49Z

@oktocat It's unfortunate that this PR has been closed, as we'd rather not deploy prometheus pushgateway just to support slo-generator. Would you have any objections if I continued work on your code?

ocervell · 2023-09-04T09:23:26Z

@skinlayers feel free to continue this work and open a new PR ;)

skinlayers · 2024-02-29T20:01:18Z

@oktocat @lvaylet
I was preparing a pull request from our updated branch, but ran into an issue rebasing/merging on v2.6.0. I encountered this issue several months ago while attempting to rebase, but never got a chance to go back and look at it again. Currently, the version based on slo-generator v2.3.4 seems to work, though we have been having issues with Cloud Run 503 timeouts [correction: 504 errors] that come up intermittently that seem to stick to a single instance until that instances goes away. So far, we've been unable to find the source of the problem even after engaging GCP Support, so it may be unrelated to v2.3.4 + prometheus_remotewrite_exporter. However, we do have a few additional customizations to the deployment such as using a k8s CronJob + python script to list all the SLO yaml files in the GCS bucket, and submit them as an HTTP request to Cloud Run. We currently use the terraform-slo-generator module to manage the deployment. Unfortunately, do to the 504 errors, we will probably be switching to a different solution, but I wanted to contribute our work back upstream.

ERROR - myservice-http-requests-availability | 1h       | PrometheusRemoteWriteExporter "prometheus_remote_write" failed. | ImportError: Exporter not found in shared config.
ERROR - Exporter not found in shared config.

{
  "textPayload": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.9/site-packages/slo_generator/compute.py\", line 130, in export\n    raise ImportError(\"Exporter not found in shared config.\")\nImportError: Exporter not found in shared config.",
  "insertId": "XXXXXXXX",
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "my-org-slo-generator",
      "location": "us-central1",
      "revision_name": "slo-generator-01234-xyz",
      "service_name": "slo-generator",
      "configuration_name": "slo-generator"
    }
  },
  "timestamp": "2024-02-29T06:29:16.324301Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "YYYYYYYY"
  },
  "logName": "projects/my-org-slo-generator/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2024-02-29T06:29:16.599553118Z",
  "errorGroups": [
    {
      "id": "XXXXXXXX"
    }
  ]
}

lvaylet · 2024-03-11T09:58:27Z

Hi @skinlayers, sorry to hear that. I am not aware of recurrent 503 errors on my test setup, but I will take a look. I have been really busy over the last two weeks. Hopefully I can get a look in two days. To this rebasing issue, too.

skinlayers · 2024-03-11T11:42:56Z

@lvaylet Correction: We've been getting 504 timeouts (not 503). I tried increasing the timeout to 1hr, but it didn't seem to help. Since I've seen no one else reporting this issue, I assume its something specific to our setup and customizations.

{
  "textPayload": "The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout",
  "insertId": "XXXXXXXX",
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "https://slo-generator-YYYYYYYY-uc.a.run.app/",
    "requestSize": "1620",
    "status": 504,
    "responseSize": "716",
    "userAgent": "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)",
    "remoteIp": "X.X.X.X",
    "serverIp": "Y.Y.Y.Y",
    "latency": "299.969843416s",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "revision_name": "slo-generator-00048-ZZZ",
      "service_name": "slo-generator",
      "location": "us-central1",
      "project_id": "myproject-slo-generator-prd",
      "configuration_name": "slo-generator"
    }
  },
  "timestamp": "2024-03-11T11:28:02.730899Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "XXXXXXXXXXXXXXXXXXX"
  },
  "logName": "projects/myproject-slo-generator-prd/logs/run.googleapis.com%2Frequests",
  "trace": "projects/myproject-slo-generator-prd/traces/ccc9adf2edcc45cab16f06e8378477e0",
  "receiveTimestamp": "2024-03-11T11:33:02.832155707Z",
  "spanId": "AAAAAAAAAAAAAA"
}

lvaylet · 2024-03-27T11:30:42Z

Hi @skinlayers, not sure this is good news but I am also getting a lot of 504 errors after setting up synthetic probes on my CI instance of the SLO Generator. Let me investigate a bit more.

GeniyX · 2024-04-27T19:06:09Z

During testing, I discovered that the error 'Exporter not found in shared config' was caused by proto files for the exporter. If you rebuild them using a new version of protoc and correct the cross imports, then the problem disappears.

Linking to the version of the compiler used in prometheus is not necessary: https://github.com/skinlayers/slo-generator/blob/prometheus-remotewrite-exporter/slo_generator/proto/Makefile#L5
Binary compatibility of the protocol is not dependent on the compiler version.

I'm not sure that I have enough skills to create a correct PR (rarely work with python). I had to edit the import in the already generated _pb2.py files to make it startup. But I hope that this will help to add this functionality to the main branch.

oktocat marked this pull request as draft October 13, 2021 14:51

oktocat mentioned this pull request Oct 13, 2021

exporter: Prometheus Remote Write API #175

Closed

oktocat force-pushed the prometheus-remotewrite-exporter branch 3 times, most recently from d932125 to 7834804 Compare October 14, 2021 16:44

oktocat changed the title ~~WIP: prometheus remote write exporter~~ prometheus remote write exporter Oct 15, 2021

oktocat marked this pull request as ready for review October 15, 2021 09:23

oktocat force-pushed the prometheus-remotewrite-exporter branch 2 times, most recently from a17ec4a to 2f97e9e Compare October 15, 2021 10:01

oktocat force-pushed the prometheus-remotewrite-exporter branch from 0d7b368 to 35a886f Compare October 15, 2021 10:29

oktocat changed the title ~~prometheus remote write exporter~~ Exporters: add prometheus remote write exporter Oct 15, 2021

oktocat force-pushed the prometheus-remotewrite-exporter branch from 35a886f to a76a2bc Compare October 18, 2021 15:09

Exporters: add prometheus remote write exporter

804204c

closes: google#122 Co-authored-by: Kevin Labesse <kevin@labesse.me> Signed-off-by: Kevin Labesse <kevin@labesse.me> Signed-off-by: Denis Zaitcev <oktocat@yandex.com>

oktocat force-pushed the prometheus-remotewrite-exporter branch from a76a2bc to 804204c Compare October 20, 2021 13:35

ocervell added the feature New feature or request label Nov 3, 2021

oktocat added 2 commits November 3, 2021 11:35

install snappy headers during make install

868a585

Signed-off-by: Denis Zaitcev <oktocat@yandex.com>

install snappy headers before python wrapper 🤦

7c029f5

Signed-off-by: Denis Zaitcev <oktocat@yandex.com>

oktocat force-pushed the prometheus-remotewrite-exporter branch from 3b597a1 to 7c029f5 Compare November 3, 2021 12:28

ocervell reviewed Nov 26, 2021

View reviewed changes

ocervell requested changes Nov 26, 2021

View reviewed changes

oktocat closed this Jul 21, 2023

skinlayers mentioned this pull request Aug 17, 2023

[New exporter] Prometheus Remote Write #122

Open

lvaylet reopened this Oct 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exporters: add prometheus remote write exporter #177

Exporters: add prometheus remote write exporter #177

oktocat commented Oct 13, 2021 •

edited

Loading

oktocat commented Oct 14, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

oktocat commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

oktocat commented Oct 15, 2021

eraac commented Oct 15, 2021

ocervell commented Nov 3, 2021

ocervell Nov 26, 2021 •

edited

Loading

ocervell left a comment •

edited

Loading

ocervell Nov 26, 2021

ocervell Nov 26, 2021

skinlayers commented Aug 3, 2023 •

edited

Loading

ocervell commented Sep 4, 2023

skinlayers commented Feb 29, 2024 •

edited

Loading

lvaylet commented Mar 11, 2024

skinlayers commented Mar 11, 2024

lvaylet commented Mar 27, 2024

GeniyX commented Apr 27, 2024 •

edited

Loading

Exporters: add prometheus remote write exporter #177

Are you sure you want to change the base?

Exporters: add prometheus remote write exporter #177

Conversation

oktocat commented Oct 13, 2021 • edited Loading

oktocat commented Oct 14, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

oktocat commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

google-cla bot commented Oct 15, 2021

oktocat commented Oct 15, 2021

eraac commented Oct 15, 2021

ocervell commented Nov 3, 2021

ocervell Nov 26, 2021 • edited Loading

Choose a reason for hiding this comment

ocervell left a comment • edited Loading

Choose a reason for hiding this comment

ocervell Nov 26, 2021

Choose a reason for hiding this comment

ocervell Nov 26, 2021

Choose a reason for hiding this comment

skinlayers commented Aug 3, 2023 • edited Loading

ocervell commented Sep 4, 2023

skinlayers commented Feb 29, 2024 • edited Loading

lvaylet commented Mar 11, 2024

skinlayers commented Mar 11, 2024

lvaylet commented Mar 27, 2024

GeniyX commented Apr 27, 2024 • edited Loading

oktocat commented Oct 13, 2021 •

edited

Loading

ocervell Nov 26, 2021 •

edited

Loading

ocervell left a comment •

edited

Loading

skinlayers commented Aug 3, 2023 •

edited

Loading

skinlayers commented Feb 29, 2024 •

edited

Loading

GeniyX commented Apr 27, 2024 •

edited

Loading