Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Pyfunc] Add ability to push metrics to Prometheus push gateway #311

Merged
merged 8 commits into from
Feb 6, 2023
Merged

[Pyfunc] Add ability to push metrics to Prometheus push gateway #311

merged 8 commits into from
Feb 6, 2023

Conversation

pradithya
Copy link
Member

@pradithya pradithya commented Oct 28, 2022 •
edited
Loading

What this PR does / why we need it:

Due to limitation in knative (knative/serving#8471) . It's currently not possible to expose prometheus metrics server endpoint in separate port from the grpc port. As such, custom metrics from pyfunc server has to be pushed using prometheus push gateway.

This PR add 3 new configuration to support that

Environment Variable Description
PUSHGATEWAY_ENABLED Enable pushing metrics to prometheus push gateway, only available when CARAML_PROTOCOL is set to UPI_V1 (default = false)
PUSHGATEWAY_URL URL of the prometheus push gateway (default = localhost:9091)
PUSHGATEWAY_PUSH_INTERVAL_SEC Interval in seconds for pushing metrics to prometheus push gateway (default = 30)

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Checklist

  • Tested locally
  • Updated documentation

@pradithya pradithya changed the title Add push metrics implementation Add ability to push metrics to Prometheus push gateway Oct 28, 2022
@pradithya pradithya changed the title Add ability to push metrics to Prometheus push gateway [Pyfunc] Add ability to push metrics to Prometheus push gateway Oct 28, 2022
@pradithya pradithya marked this pull request as ready for review October 28, 2022 04:16
@ariefrahmansyah
Copy link
Contributor

Any updates for this PR?

Can you rebase this first? The deploy-merlin.sh has already been updated to use CaraML's mlp charts by this PR #318.

@pradithya
Copy link
Member Author

@ariefrahmansyah Rebased

@pradithya pradithya merged commit 72616dd into main Feb 6, 2023
@pradithya pradithya deleted the push_prom branch February 6, 2023 06:21
shydefoo added a commit that referenced this pull request Feb 21, 2023
@shydefoo
Merge remote-tracking branch 'upstream/main' into enable-wi-kaniko
427fba3 
* upstream/main:
  Refactor deployed model labels (#346)
  [Pyfunc] Add ability to push metrics to Prometheus push gateway (#311)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ariefrahmansyah ariefrahmansyah ariefrahmansyah approved these changes

@tiopramayudi tiopramayudi Awaiting requested review from tiopramayudi

@karzuo karzuo Awaiting requested review from karzuo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pradithya @ariefrahmansyah