Get trouble with Local mode using SageMaker Debugger

[chaeeon-lim]

Abstract

I'm trying to test SageMaker Debugger examples in 'local mode' from amazon-sagemaker-examples. I'm trying to see the same debugging result in 'local mode' with AWS SageMaker Notebook instance.

What i've done

I put a few lines in one of the examples to use it in 'local mode' referenced from amazon-sagemaker-local-mode-example. The example is tf-mnist-builtin-rule.ipynb.
Commented line is original code in the example

import subprocess
# import boto3
from sagemaker.local import LocalSession
from sagemaker.tensorflow import TensorFlow

instance_type = 'local'
try:
    if subprocess.call("nvidia-smi") == 0:
        instance_type = "local_gpu"
except:
    pass
session = LocalSession()
session.config = {'local': {'local_code': True}}
# session = boto3.session.Session()
# region = session.region_name
role = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'
estimator = TensorFlow(
    # role=sagemaker.get_execution_role(),
    role=role,
    instance_count=1,
    # instance_type="ml.p3.8xlarge",
    instance_type=instance_type,
    # image_uri=f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04",
    framework_version='2.4.1',
    py_version="py37",
    max_run=3600,
    source_dir="./src",
    entry_point="tf-resnet50-cifar10.py",
    # Debugger Parameters
    rules=built_in_rules,
    sagemaker_session=session,
)

Problem:

I can't see debugging report in realtime unlike AWS SageMaker Notebook instance.
The example said wait=false option (see the figure) makes notebook proceed though it is still running but not working in 'local mode'.

Question:

I tried to use the 'Thread' built-in function to make it work concurrently, however, the estimator was not updated until the thread.

from threading import Thread
Thread(target=estimator.fit).start()
...
type(estimator.latest_training_job)
# => NoneType

Any idea make wait=False option working in 'local mode'?

Thanks for reading.