[code_search] Script to load model metrics into Katib/ModelDB

[jlewi]

We'd like to take advantage of Katib's integration with model db. To track the performance of models.
More detail can be found in the doc.

The first step is probably a simple script that can load relevant data about a model into Katib. The data we probably want is

• GCS location of the model
• Evaluation metrics for the model

[jlewi]

Here are some instructions on how to use ModelDB for this.
https://github.com/mitdbg/modeldb/blob/master/client/python/light_api.md

I looked at the Katib CLI but I'm not sure the Katib CLI is that useful in this case.

[jlewi]

I used the ModelDB sample code to create a model and store data in ModelDB
https://github.com/jlewi/kubeflow-dev/tree/master/modeldb

This appears to have worked

• In the projects UI tab I can see the newly created project and it lists two models
• It also shows some statistics that lead me to believe the data was actually loaded

But when I click on the projects tab I just get a screen that says loading models

@mpvartak Any thoughts?

I'd like to be able to use ModelDB to store metadata about a model e.g.

• Location of a model
• Data used to train
• Various metrics and parameters

And then be able to use ModelDB's UI to browse the models.

[mpvartak]

@jlewi this is the same workflow that's being used in Katib afaik. So that should work the same. I will try to repro the bug and ping back here.

[jlewi]

Thanks.

[mpvartak]

See the BasicWorkflow.ipynb example in http://modeldb.csail.mit.edu:8000
This is the same workflow you are using afaik.
I am not seeing the error in the frontend UI: http://modeldb.csail.mit.edu:3000

Do you have a different setup?

[jlewi]

@mpvartak What do you mean by setup? I'm running on GKE using Katib so I assume the setup is different (possibly different versions of modeldb from what is running in your setup).

I copied the code from https://github.com/mitdbg/modeldb/blob/master/client/python/light_api.md so I expect its the same.

Can you provide suggestions on how to debug this?

Is there a way to query the database directly e.g. through an API or a CLI so I can see whether the data is there and whether its just a UI issue?

[mpvartak]

Here's my setup: pip install modeldb + the docker versions of the frontend and backend. I'm on Python2.7. I use the same code as the on the webpage (Method 2 specifically) and am able to log models/visualize correctly. So I can confirm that works. I haven't tried on GKE, so can't comment on what might be going on there.

For debugging, the sqlite file lives in server/codegen/sqlite and can be read directly using sqlite3; you can query the mongo data via mongod. I'd recommend making sure that the sqlite DB shows the models in FitEvent table + ExperimentRun tables. If that looks good, then next step is to check on the key-value data stored in mongodb.

I can try to dig further if you can provide more info.

[bhack]

When we talk about GCS, i.e. for the location of the model, can we use a more generic approach like GoogleContainerTools/kaniko#121 (comment)?

[jlewi]

@mpvartak Can you provide your syncer config and a link to your python code?

[jlewi]

@mpvartak I can see data in the sqllite DB but I'm not sure what I'm looking for. Would you be willing to try it out on our dev cluster (dev.kubeflow.org)? I'd be happy to give you access; I just need a Google account for you.

[jlewi]

It looks like Katib added an API to upload model data; see
kubeflow/katib#43

I don't think there's a CLI yet; but we might have more luck with Katib.

[lluunn]

cc @lluunn

[jlewi]

@garganubhav is looking into this.

@garganubhav I think it would be fantastic if we can get ModelDB working so that we can load data in via API and use the API to browse and explore models.

[mpvartak]

Hey @jlewi @garganubhav my team is rewriting the guts of modeldb to make it more robust, add features, and make easy to deploy on kubernetes. let me share the roadmap and get the kubeflow team's feedback too. on slack, is katib the best channel to have this discussion?

i'm happy to help @garganubhav on this issue, let's figure out if it makes sense to integrate into current modeldb or can wait for next version.

[garganubhav]

@jlewi yes, would update very soon. @mpvartak thank you, the channel on slack is model-management

[chrisheecho]

Hi @garganubhav do you have any update on this one? we are trying to cut down issues on 0.4 and would love to get your take

[mpvartak]

I have a solution for the persistence issue. Let me push to MDB repo.

…

On Mon, Oct 29, 2018 at 8:46 AM chrisheecho ***@***.***> wrote: Hi @garganubhav <https://github.com/garganubhav> do you have any update on this one? we are trying to cut down issues on 0.4 and would love to get your take — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFeWw3-0TXzfYLiay-YUPP7PxUJrRe5_ks5upyLLgaJpZM4Uh6q0> .

[garganubhav]

@chrisheecho this is tested. Modeldb is working with Katib.

[jlewi]

@garganubhav So do we have a script and docs somewhere that we can point customers to? What is the remaining work to close this out?

[garganubhav]

As I know, ModelDb is currently deleted from Katib.

[jlewi]

@garganubhav @mpvartak has a PR open to add modeldb as a separate package. So it would still be good to have examples and instructions for how people can take advantage of modeldb to track their models.

[garganubhav]

Yes, I was checking it before and was getting some error. Will try today again.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.