-
Notifications
You must be signed in to change notification settings - Fork 43
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Pyfunc] Add ability to push metrics to Prometheus push gateway (#311)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. Run unit tests and ensure that they are passing 2. If your change introduces any API changes, make sure to update the e2e tests 3. Make sure documentation is updated for your PR! --> **What this PR does / why we need it**: <!-- Explain here the context and why you're making the change. What is the problem you're trying to solve. ---> Due to limitation in knative (knative/serving#8471) . It's currently not possible to expose prometheus metrics server endpoint in separate port from the grpc port. As such, custom metrics from pyfunc server has to be pushed using prometheus push gateway. This PR add 3 new configuration to support that | Environment Variable | Description | | ------------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | PUSHGATEWAY_ENABLED | Enable pushing metrics to prometheus push gateway, only available when `CARAML_PROTOCOL` is set to `UPI_V1` (default = false) | | PUSHGATEWAY_URL | URL of the prometheus push gateway (default = localhost:9091) | | PUSHGATEWAY_PUSH_INTERVAL_SEC | Interval in seconds for pushing metrics to prometheus push gateway (default = 30) | **Which issue(s) this PR fixes**: <!-- *Automatically closes linked issue when PR is merged. Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> Fixes # **Does this PR introduce a user-facing change?**: <!-- If no, just write "NONE" in the release-note block below. If yes, a release note is required. Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required". For more information about release notes, see kubernetes' guide here: http://git.k8s.io/community/contributors/guide/release-notes.md --> ```release-note ``` **Checklist** - [X] Tested locally - [X] Updated documentation
- Loading branch information
aria
authored
Feb 6, 2023
1 parent
f37de15
commit 72616dd
Showing
10 changed files
with
129 additions
and
64 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
import logging | ||
import socket | ||
import time | ||
from threading import Thread | ||
|
||
from prometheus_client import push_to_gateway | ||
|
||
from pyfuncserver.config import Config | ||
|
||
|
||
def start_metrics_pusher(push_gateway, registry, target_info, interval_sec): | ||
""" | ||
Start periodic job to push metrics to prometheus push gatewat | ||
""" | ||
logging.info(f"starting metrics pusher, url: {push_gateway} with interval {interval_sec} s") | ||
daemon = Thread(target=push_metrics, args=(push_gateway, registry, interval_sec, target_info), | ||
daemon=True, name='metrics_push') | ||
daemon.start() | ||
|
||
|
||
def push_metrics(gateway_url, registry, interval_sec, grouping_keys): | ||
""" | ||
push metrics to prometheus push gateway every interval_sec | ||
Should be called in separate thread | ||
""" | ||
while True: | ||
push_to_gateway(gateway_url, "merlin_pyfunc_upi", registry=registry, grouping_key=grouping_keys) | ||
time.sleep(interval_sec) | ||
|
||
|
||
def labels(config: Config): | ||
""" | ||
Labels to be added to all metrics | ||
""" | ||
return { | ||
"merlin_model_name": config.model_manifest.model_name, | ||
"merlin_model_version": config.model_manifest.model_version, | ||
"host": socket.getfqdn() | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters