Use StackDriver + BigQuery to log predictions #79
Comments
|
Issue-Label Bot is automatically applying the label Links: app homepage, dashboard and code for this bot. |
|
I setup a biguery sink in the project issue-label-bot-dev to begin experimenting with. |
|
There's some information here about doing structured logging with the logging module. It looks like this relies on the caller of I think we want to do something like the json formatter to automatically format all entries as json |
|
I created a new sync. It looks like before when I created a sync I used a filter expression that wouldn't include the new prod deployment. I created a new sync with the filter |
* worker.py should format logs as json entries. This will make it easier to query the data in BigQuery and stackdriver to measure performance. * Related to kubeflow#79 * To deal with workload identity flakiness (kubeflow#88) test that we can get application default credentials on startup and if not exit. * As a hack to deal with multi-threading issues with Keras models (kubeflow#89) have the predict function load a new model on each call * It looks like the way pubsub works there is actually a thread pool so predict calls won't be handled in the same thread even though we throttle it to handle one item at a time.
* worker.py should format logs as json entries. This will make it easier to query the data in BigQuery and stackdriver to measure performance. * Related to #79 * To deal with workload identity flakiness (#88) test that we can get application default credentials on startup and if not exit. * As a hack to deal with multi-threading issues with Keras models (#89) have the predict function load a new model on each call * It looks like the way pubsub works there is actually a thread pool so predict calls won't be handled in the same thread even though we throttle it to handle one item at a time.
|
Logs are now in stackdriver. Here's a sample query |
|
it looks like logs are streamed in nearly real time to BigQuery. I observed log entries showing up almost immediately. So it looks as though the sync is much more frequent then once a day. |
We should use bigquery and stackdriver to log predictions.
This should work as follows
The text was updated successfully, but these errors were encountered: