Benchmarking toolkit wrap up #2462
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
jppgks
reviewed
Sep 19, 2022
jppgks
reviewed
Sep 19, 2022
justinxzhao
reviewed
Sep 19, 2022
justinxzhao
reviewed
Sep 27, 2022
abidwael
force-pushed
the
benchmarking-toolkit-wrap-up
branch
from
2a8f7d1 to
4c64fd6
Compare
|
Using the |
jppgks
approved these changes
Sep 28, 2022
|
EDIT: Looks like the LudwigProfiler is not serializable because of the thread that is started when entering its context. Debugging serializability: >>> from ray.util import inspect_serializability
>>> from ludwig.benchmarking.profiler_callbacks import LudwigProfilerCallback
>>> cb = LudwigProfilerCallback({'experiment_name': '/tmp', 'profiler': {'use_torch_profiler': False, 'logging_interval': 0}})
>>> inspect_serializability(cb, name="LudwigProfilerCallback")
================================================================================
Checking Serializability of <ludwig.benchmarking.profiler_callbacks.LudwigProfilerCallback object at 0xffff9d867b20>
================================================================================
(True, set())
>>> cb.on_preprocess_start()
>>> inspect_serializability(cb, name="LudwigProfilerCallback")
================================================================================
Checking Serializability of <ludwig.benchmarking.profiler_callbacks.LudwigProfilerCallback object at 0xffff9d6c4d90>
================================================================================
!!! FAIL serialization: cannot pickle '_thread.lock' object
Serializing 'preload' <function Callback.preload at 0xffff7cdbf670>...
Serializing '_abc_impl' <_abc_data object at 0xffff9d5b0300>...
!!! FAIL serialization: cannot pickle '_abc_data' object
WARNING: Did not find non-serializable object in <_abc_data object at 0xffff9d5b0300>. This may be an oversight.
================================================================================
Variable:
FailTuple(_abc_impl [obj=<_abc_data object at 0xffff9d5b0300>, parent=<ludwig.benchmarking.profiler_callbacks.LudwigProfilerCallback object at 0xffff9d6c4d90>])
was found to be non-serializable. There may be multiple other undetected variables that were non-serializable.
Consider either removing the instantiation/imports of these variables or moving the instantiation into the scope of the function/class.
If you have any suggestions on how to improve this error message, please reach out to the Ray developers on github.com/ray-project/ray/issues/
================================================================================
(False, {FailTuple(_abc_impl [obj=<_abc_data object at 0xffff9d5b0300>, parent=<ludwig.benchmarking.profiler_callbacks.LudwigProfilerCallback object at 0xffff9d6c4d90>])}) |
|
@jppgks Thanks for investigating! That was the issue I encountered with a ray backend. I looked at other options of how we can still achieve this (using mutliprocessing instead of threading, etc.) but didn't get to a working solution. For now, let's just default to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes
LudwigProfiler.