better core dump for debugging #184
Conversation
| f'gcloud storage cp -r /tmp/xla_dump/ {args.debug_dump_gcs}/$WORKER_ID;' | ||
| f'gcloud storage cp /tmp/*.dump {args.debug_dump_gcs}/$WORKER_ID;' | ||
| f'gcloud storage cp -r /tmp/tpu_logs {args.debug_dump_gcs}/$WORKER_ID;' | ||
| 'sleep 120;' |
There was a problem hiding this comment.
This is a workaround, not an ideal solution. The main host pod requires a significant amount of time to upload /tmp/*.dump which are approximately 10 GiB in size.
I did find the main host pod was terminated earlier before finishing upload. I just want to add a buffer time to each pod's lifecycle, ensuring the host pod has sufficient time to finish uploading.
Let me know if you have better solution.
There was a problem hiding this comment.
Hmm buffer times are scary to put into production code because you don't know if you have set the time for too long or too short!
It looks like gcloud storage cp .. a blocking call? Why would the main host terminate earlier is strange... do you mind looking into this more? Thank you Zhiyu!
| @@ -2104,8 +2104,12 @@ def get_main_container(args, system, docker_image, resource_type) -> str: | |||
| ' is required but not installed. Aborting"; exit 24;};' | |||
There was a problem hiding this comment.
I don't think you need the gsutil_test_command anymore since you move to gcloud storage
There was a problem hiding this comment.
Good point. Let me change this part accordingly.
| 'WORKER_ID=$HOSTNAME;' | ||
| f'gsutil -m cp -r /tmp/xla_dump/ {args.debug_dump_gcs}/$WORKER_ID;' | ||
| f'gcloud storage cp -r /tmp/xla_dump/ {args.debug_dump_gcs}/$WORKER_ID;' | ||
| f'gcloud storage cp /tmp/*.dump {args.debug_dump_gcs}/$WORKER_ID;' |
There was a problem hiding this comment.
Do we need additional flags set to find the /tmp/*.dump and /tmp/tpu_logs? just want to make sure these additional dumps always exist and don't crash a users job if they can't be found (like only available with additional flags)
There was a problem hiding this comment.
- No flags for dumping /tmp/tpu_logs.
- As for
/tmp/*.dump, this file will be dumped in case of some specific hardware failure. We can also force core dump with some flags as well.
There will be an error without finding the specific path. And the pipeline will still continue after the error since we didn't set -e;.
There was a problem hiding this comment.
We can also force core dump with some flags as well.
SG, Should we explicitly set the flags for the core dumps?
| @@ -2104,8 +2104,12 @@ def get_main_container(args, system, docker_image, resource_type) -> str: | |||
| ' is required but not installed. Aborting"; exit 24;};' | |||
| ) | |||
| xpk_internal_commands += ( | |||
| 'set -x;' | |||
There was a problem hiding this comment.
Curious why we add set -x; here? Is it related to the debug_dump_gcs feature or a more general feature? If it is a more general feature, maybe it should be part of --enable-debug-logs arg?
Features
upload more detailed info to gcs bucket in failures
Switch to
gcloud storage cp, since it around twice as fast asgsutil -m cp.Testing / Documentation