Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Providing user control over scope of experiment tracking functionality #131

Open
ekhl opened this issue Apr 29, 2020 · 1 comment
Open
Labels
feature-request Use this label to indicate a feature request needs design A design is needed in order to proceed

Comments

@ekhl
Copy link
Contributor

ekhl commented Apr 29, 2020

(forking #63 (comment) into its own issue)

Context

Currently all tracked entities (metrics, parameters, artifacts, etc.) are associated with a job and a job starts with import foundations and ends with the exit hook. Though convenient in some cases, this can be restricting when a user wants to have more granular control over when a job should start and end (for example, when a user wants to define multiple jobs in a single python process, or when a user is developing interactively like in a notebook).

Proposal

Open to design ideas. We can close this and create more concrete issues after arriving at an agreed upon design
Some considerations for the design:

  • maintain backwards compatibility with the existing behaviour
  • embrace uniqueness of notebook user workflow (e.g. difficult to anticipate kernel restarts, context managers don't span across cells, execution order of cells is not guaranteed, etc.)
  • interplay between allowing for multiple jobs in a process and the current recommended approach for hyperparameter and model searches
@ekhl ekhl added feature-request Use this label to indicate a feature request needs design A design is needed in order to proceed labels Apr 29, 2020
@ekhl
Copy link
Contributor Author

ekhl commented Apr 29, 2020

@shazraz @mohammedri bringing the conversation about tracking functionality from #63 to a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request Use this label to indicate a feature request needs design A design is needed in order to proceed
Projects
None yet
Development

No branches or pull requests

1 participant