benchmark fix #229

kiddyjinjin · 2024-09-26T06:22:30Z

PR Category

Benchmark

Type of Change

New Feature

PR Description

1. New Testing Parameters

This PR introduces new benchmark testing parameters, including:

level
- Type: str
- Description: Marks the level of the benchmark.
- Available levels:
  - comprehensive (default): Comprehensive testing.
  - core: Core testing.
warmup
- Type: int
- Description: The number of warm-up iterations.
- Default value: DEFAULT_WARMUP_COUNT = 1000
iter
- Type: int
- Description: The number of benchmark iterations.
- Default value: DEFAULT_ITER_COUNT = 100
query
- Description: Indicates that the benchmark will only query properties without executing the full benchmark logic.
- Default: This parameter is not set by default.
record
- Type: str
- Description: Specifies the format of the output data.
- Available options:
  - none (default)
  - log: Logs output in JSON format.
dtype
- Type: list[str]
- Description: Specifies the data types for benchmark testing. Available dtypes can be listed using pytest --help.
- Available data types:
  - torch.float16, torch.float32, torch.bfloat16, torch.int16, torch.int32, torch.bool, torch.complex64
metric
- Type: list[str]
- Description: Specifies the metrics covered by the benchmark test.
- Available metrics:
  - latency, speedup, tflops, latency_base, accuracy, utilization

2. Structural Design Adjustments

This section outlines several structural design adjustments:

Added the BenchmarkMetrics abstraction
- Represents the benchmark information to be recorded for specific operations at specific sizes and data types.
Added the BenchmarkResult abstraction
- Represents all test results for a specific operation on specific hardware and at a specified benchmark level.
Adjusted the design of the Benchmark structure
- Changed the per-operator Function-level benchmark to a Class-level benchmark for a category of operators, facilitating unified configuration of default benchmark parameters and allowing for inheritance and overrides.

3. Improvements to Test Data

The previous testing data was based on a specific batch, optional size list, and optional dtype list for combinatorial testing. This approach was somewhat limited in expression. It has now been changed to a more abstract input_generator, which provides a default input generator. Special input scenarios can directly override the corresponding generator.

tongxin

It's a nice rewrite and enhancement to current benchmarking tools!

tongxin · 2024-09-26T09:58:52Z

benchmark/attri_util.py

+class ReadOnly:
+    def __init__(self, value):
+        self._value = value
+
+    @property
+    def value(self):
+        return self._value


Why do we need this abstraction?

This structure is intended to protect data from unexpected changes. Since Python uses reference types by default rather than value types, it is susceptible to unintended modifications.

tongxin · 2024-09-26T10:02:44Z

benchmark/attri_util.py

+
+BLAS_OPS = ReadOnly(["addmm", "mv", "addmm", "mm", "outer"])
+
+DEFAULT_WARMUP_COUNT = 100


My experience is we need a much longer warmup to warrant a stable perf result. I suggest flip the warmup and repeat values.

tongxin · 2024-09-26T10:05:58Z

benchmark/attri_util.py

+# BLAS situation
+# BLAS shapes is defined by (B,M,N,K), it is different from the non blas Shapes
+DEFAULT_BLAS_BENCH_SHAPES = [(1, 1, 1, 32), (4, 15, 160, 1024), (16, 495, 5333, 71)]
+DEFAULT_BLAS_WITHOUT_BATCH_BENCH_SHAPES = [(1, 1, 32), (15, 160, 1024), (495, 5333, 71)]


What about BLAS_DEFAULT_BMNK, BLAS_DEFAULT_MNK?
Also, the larger sizes are rather small.

tongxin · 2024-09-26T10:10:52Z

benchmark/attri_util.py

+
+@dataclass
+class BenckmarkMatrics:
+    # the simple version shape info, this shape setted here just to with the last version.


Sorry I'm a little fussy here. The past tense of set is set still.

…r user-specified dtype and metrics, and abstract input generator.

…AULT_SHAPES_2D_ONLY shapes

Bowen12992

Great Job，how about add some docs to CONTRIBUTING.md

StrongSpoon · 2024-10-28T06:10:37Z

benchmark/performance_utils.py

        fn = lambda: op(*args, **kwargs)
        if self.is_backward:
            out = fn()
            dout = torch.randn_like(out)
            fn = lambda: out.backward(dout, retain_graph=True)
-        if CPU_MODE:
-            for i in range(WARMUP):
+        if Config.cpu_mode:


could we design a new mode which outputs both cpu latency and gpu latency? @tianxiao-baai

* benchmark fix * add seven new testing parameters * move shapes info to yaml file * Added the BenchmarkMetrics & BenchmarkResult abstraction

kiddyjinjin added 2 commits September 26, 2024 06:20

benchmark fix

91724e4

merge upstream to this

13b5800

tongxin reviewed Sep 26, 2024

View reviewed changes

kiddyjinjin added 12 commits September 27, 2024 01:55

change three kinds of benchmark level setting to two kinds

a4d985b

update the basic settings

84fa232

adjust the batch & shape info for all the operators

4d0bc21

merge upstream

d347517

benchmark fix: Refactor benchmark structure design, add interfaces fo…

f155778

…r user-specified dtype and metrics, and abstract input generator.

benchmark fix for special operations

fc5841e

ammend

03cf1de

merge perf shapes

7eb9be4

Merge remote-tracking branch 'upstream/master'

0a07f8d

merge upstream

04cdf48

benchmark fix

412c1f4

specify DEFAULT_SHAPES_EXCLUDE_1D & DEFAULT_SHAPES_EXCLUDE_3D and DEF…

7db32d5

…AULT_SHAPES_2D_ONLY shapes

Bowen12992 reviewed Oct 23, 2024

View reviewed changes

kiddyjinjin added 4 commits October 23, 2024 08:40

merge upstream/master

ea3076e

fix pre-commit

949ec70

update CONTRIBUTING.md & CONTRIBUTING_cn.md

371072c

for pre-commit

182dc0a

StrongSpoon reviewed Oct 28, 2024

View reviewed changes

kiddyjinjin added 6 commits October 29, 2024 07:35

move shapes info to yaml file

7c57c47

move shapes info to yaml file

f69b181

fix record log bug

8cb7204

merge upstream

778fcfb

pre-commit fix

cfecc5e

fix json encode bug: when meeting custom object

4f63e88

tianxiao-baai approved these changes Oct 30, 2024

View reviewed changes

Bowen12992 approved these changes Oct 30, 2024

View reviewed changes

kiddyjinjin merged commit 4e6cb3b into FlagOpen:master Oct 30, 2024
4 checks passed

machuanjiang pushed a commit that referenced this pull request Nov 15, 2024

benchmark fix (#229)

2f22ebc

* benchmark fix * add seven new testing parameters * move shapes info to yaml file * Added the BenchmarkMetrics & BenchmarkResult abstraction

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark fix #229

benchmark fix #229

kiddyjinjin commented Sep 26, 2024 •

edited

Loading

tongxin left a comment

tongxin Sep 26, 2024

kiddyjinjin Sep 30, 2024

tongxin Sep 26, 2024

kiddyjinjin Sep 30, 2024

tongxin Sep 26, 2024

tongxin Sep 26, 2024

Bowen12992 left a comment

StrongSpoon Oct 28, 2024


		BLAS_OPS = ReadOnly(["addmm", "mv", "addmm", "mm", "outer"])

		DEFAULT_WARMUP_COUNT = 100

benchmark fix #229

benchmark fix #229

Conversation

kiddyjinjin commented Sep 26, 2024 • edited Loading

PR Category

Type of Change

PR Description

1. New Testing Parameters

2. Structural Design Adjustments

3. Improvements to Test Data

tongxin left a comment

Choose a reason for hiding this comment

tongxin Sep 26, 2024

Choose a reason for hiding this comment

kiddyjinjin Sep 30, 2024

Choose a reason for hiding this comment

tongxin Sep 26, 2024

Choose a reason for hiding this comment

kiddyjinjin Sep 30, 2024

Choose a reason for hiding this comment

tongxin Sep 26, 2024

Choose a reason for hiding this comment

tongxin Sep 26, 2024

Choose a reason for hiding this comment

Bowen12992 left a comment

Choose a reason for hiding this comment

StrongSpoon Oct 28, 2024

Choose a reason for hiding this comment

kiddyjinjin commented Sep 26, 2024 •

edited

Loading