Fixed potential insufficient samples due to num_files is not divisible by comm.size #200
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…buffer size to be 2 for cosmoflow
|
I get that error message: Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/main.py", line 394, in main
benchmark.run()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/main.py", line 333, in run
self.framework.get_loader(dataset_type=DatasetType.TRAIN).read()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/data_loader/tf_data_loader.py", line 98, in read
epoch_number=0).next()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/reader/tf_reader.py", line 110, in next
total = math.float(len(self._file_list)/self._args.comm_size / self.batch_size * self._args.num_samples_per_file)
AttributeError: module 'math' has no attribute 'float'
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.If I replace math.float with float i get that: Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/main.py", line 394, in main
benchmark.run()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/main.py", line 333, in run
self.framework.get_loader(dataset_type=DatasetType.TRAIN).read()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/data_loader/tf_data_loader.py", line 98, in read
epoch_number=0).next()
File "/usr/local/lib/python3.10/dist-packages/dlio_profiler/logger.py", line 183, in wrapper
x = func(*args, **kwargs)
File "/root/measure/storage/dlio_benchmark/dlio_benchmark/reader/tf_reader.py", line 111, in next
return self._dataset.take(total*self._args.epochs).prefetch(buffer_size=self._args.prefetch_size)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 1561, in take
return take_op._take(self, count, name=name)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/take_op.py", line 24, in _take
return _TakeDataset(self, count, name=name)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/data/ops/take_op.py", line 33, in __init__
self._count = ops.convert_to_tensor(count, dtype=dtypes.int64, name="count")
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/profiler/trace.py", line 183, in wrapped
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py", line 696, in convert_to_tensor
return tensor_conversion_registry.convert(
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/tensor_conversion_registry.py", line 234, in convert
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/constant_op.py", line 335, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/weak_tensor_ops.py", line 142, in wrapper
return op(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/constant_op.py", line 271, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/constant_op.py", line 284, in _constant_impl
return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/constant_op.py", line 296, in _constant_eager_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/constant_op.py", line 103, in convert_to_eager_tensor
return ops.EagerTensor(value, ctx.device_name, dtype)
TypeError: Cannot convert 39.09375 to EagerTensor of dtype int64
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
|
|
@LouisDDN Fixed the typo |
|
It seems to work now: ./benchmark.sh run --hosts myhost --workload resnet50 --param reader.read_threads=4 --accelerator-type h100 --num-accelerators 8 --results-dir fakeres --param dataset.num_files_train=100 --param dataset.data_folder=/mnt/nfs_clientshare/resnet50With the PR (1 epochs) [INFO] Averaged metric over all epochs
[METRIC] ==========================================================
[METRIC] Number of Simulated Accelerators: 8
[METRIC] Training Accelerator Utilization [AU] (%): 98.6350 (0.2730)
[METRIC] Training Throughput (samples/second): 13967.4690 (104.5302)
[METRIC] Training I/O Throughput (MB/second): 1527.3199 (11.4302)
[METRIC] train_au_meet_expectation: success
[METRIC] ==========================================================
[/root/measure/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:185]With main: [INFO] 2024-06-07T08:27:18.153579 Maximum number of steps reached [/root/measure/storage/dlio_benchmark/dlio_benchmark/main.py:290]
[INFO] 2024-06-07T08:27:18.153864 Ending block 1 - 39 steps completed in 9.07 s [/root/measure/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:282]
[INFO] 2024-06-07T08:27:18.154736 Epoch 1 - Block 1 [Training] Accelerator Utilization [AU] (%): 97.3939 [/root/measure/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:285]
[INFO] 2024-06-07T08:27:18.154918 Epoch 1 - Block 1 [Training] Throughput (samples/second): 13756.4333 [/root/measure/storage/dlio_benchmark/dlio_benchmark/utils/statscounter.py:286]
(hangs) |
zhenghh04
added a commit
that referenced
this pull request
Jun 8, 2024
* Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py
zhenghh04
added a commit
that referenced
this pull request
Jun 8, 2024
* Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py
zhenghh04
added a commit
that referenced
this pull request
Jun 11, 2024
* Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Bring v1.0 to the most recent commit (#202) (#203) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Fix requirements file (#204) Signed-off-by: Johnu George <johnugeorge109@gmail.com> --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com>
zhenghh04
added a commit
that referenced
this pull request
Jun 11, 2024
* Bring v1.0 to the most recent commit (#202) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * sync up (#205) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Bring v1.0 to the most recent commit (#202) (#203) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Fix requirements file (#204) Signed-off-by: Johnu George <johnugeorge109@gmail.com> --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> * barrier in the beginning --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com>
zhenghh04
added a commit
that referenced
this pull request
Jun 12, 2024
* Bring v1.0 to the most recent commit (#202) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * sync up (#205) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Bring v1.0 to the most recent commit (#202) (#203) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Fix requirements file (#204) Signed-off-by: Johnu George <johnugeorge109@gmail.com> --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> * barrier in the beginning * fixed bugs * fixed MPI initilization issue --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com>
zhenghh04
added a commit
that referenced
this pull request
Aug 6, 2024
* Fixed the MPI initialization issue (#207) * Bring v1.0 to the most recent commit (#202) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * sync up (#205) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Bring v1.0 to the most recent commit (#202) (#203) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Fix requirements file (#204) Signed-off-by: Johnu George <johnugeorge109@gmail.com> --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> * barrier in the beginning * fixed bugs * fixed MPI initilization issue --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> * Switch DLIO Profiler to DFTracer. (#208) * Switch DLIO Profiler to DFTracer. * Github workflow fixed. * DFTracer pip package address fixed. * switch mpi to openmpi * added build debug so that symbols are shown. * added DEBUG logging for DFTRACER * removed explicit install of DFTracer as it is done during pip. * added clanup code. * ci cleanup * Updated for DFTracer changes * switched to 1.0.1 Due to a simple pypi bug * Switch to release 1.0.2 * Package info updated. (#210) * Update dali_tfrecord_reader.py to import PerfTrace and Profile from utils.utility * Update dali_npy_reader.py * Update dali_image_reader.py * Update custom_npz_reader.py * Update custom_torch_data_loader.py [ci-skip] * Update pytorch_checkpointing.py [ci-skip] * Update native_dali_data_loader.py [ci-skip] * Publish on PyPI (#211) * PyPI publish workflow added. * `requirements.txt` cleaned up. * Action names refactored. * Missing requirements fixed. * `setup.py` requirement versions refactored. * CI support for running via `requirements.txt`. * `VENV_PATH` fixed in CI script. * CI script conditional executable fix. * Cleared redundant CI matrix options. --------- Co-authored-by: Izzet Yildirim <iyildirim@anl.gov> Co-authored-by: Izzet Yildirim <izzet@izzet.dev> Co-authored-by: Huihuo Zheng <zhenghh04@gmail.com> * Fix README CI badge (#212) * Ignore file indexing for native data loader. The sample building and native data loader case is needed only for DLIO created data loaders. For native data loaders which provide their own API;s they provide their own indexing and there this sampling can be ignored. --------- Signed-off-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Hariharan Devarajan <mani.hariharan@gmail.com> Co-authored-by: Izzet Yildirim <iyildirim@anl.gov> Co-authored-by: Izzet Yildirim <izzet@izzet.dev> Co-authored-by: hariharandev1 <hariharandev1@llnl.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixed the hang, an alternative solution to #198