[feat] Query Memory Consumption and expanded profiler with memory consumption #448

jjfumero · 2024-06-11T14:17:51Z

Description

This PR expands the profiler and the TornadoVM Execution Plan API to obtain the current memory consumption per task-graph, total number of bytes transferred to the device back and forth per execution plan, and the total memory usage. This feature is requested from the GAIA project.

a) getTotalBytesTransferred

This expands the profiler to obtain the total number of bytes transferred in every execution plan launch. The number of bytes transferred depends on READ_ONLY, WRITE_ONLY and READ_WRITE data buffers.

 try (TornadoExecutionPlan executionPlan = new TornadoExecutionPlan(immutableTaskGraph)) {
            TornadoExecutionResult executionResult = executionPlan.execute();
            long totalBytesTransferred = executionResult.getProfilerResult().getTotalBytesTransferred();
            long copyInBytes = executionResult.getProfilerResult().getTotalBytesCopyIn();
            long copyOutBytes = executionResult.getProfilerResult().getTotalBytesCopyOut();
            assertEquals(copyInBytes + copyOutBytes, totalBytesTransferred);
        }

b) getTotalDeviceMemoryUsage

The total device memory usage registers all allocations needed to run an execution plan. This value can be queried using the TornadoVM profiler.

try (TornadoExecutionPlan executionPlan = new TornadoExecutionPlan(immutableTaskGraph)) {
            TornadoExecutionResult executionResult = executionPlan.execute();
            long totalMemoryUsedInBytes = executionResult.getProfilerResult().getTotalDeviceMemoryUsage();

            // 3 Arrays
            final long sizeAllocated = a.getNumBytesOfSegmentWithHeader() * 3;
            assertEquals(sizeAllocated, totalMemoryUsedInBytes);

        }

c) getCurrentMemoryUsage

Value to query at the execution plan level, to obtain the actual memory usage at any point during execution.

try (TornadoExecutionPlan executionPlan = new TornadoExecutionPlan(immutableTaskGraph)) {
            executionPlan.execute();
            long currentMemoryUsageInBytes = executionPlan.getCurrentMemoryUsage();
            final long sizeAllocated = a.getNumBytesOfSegmentWithHeader() * 3;
            assertEquals(sizeAllocated, currentMemoryUsageInBytes);
        }

Problem description

n/ a.

Backend/s tested

Mark the backends affected by this PR.

OpenCL
PTX
SPIRV

OS tested

Mark the OS where this PR is tested.

Linux
OSx
Windows

Did you check on FPGAs?

If it is applicable, check your changes on FPGAs.

Yes
No

How to test the new patch?

make BACKEND=opencl
tornado-test --enableProfiler console -V uk.ac.manchester.tornado.unittests.memory.MemoryConsumptionTest

make BACKEND=ptx
tornado-test --enableProfiler console -V uk.ac.manchester.tornado.unittests.memory.MemoryConsumptionTest

make BACKEND=spirv
tornado-test --enableProfiler console -V uk.ac.manchester.tornado.unittests.memory.MemoryConsumptionTest

tornado-api/src/main/java/uk/ac/manchester/tornado/api/profiler/ProfilerType.java

stratika · 2024-06-12T06:54:23Z

tornado-api/src/main/java/uk/ac/manchester/tornado/api/ImmutableTaskGraph.java

+    public long getTotalBytesTransferred() {
+        return taskGraph.getTotalBytesTransferred();
+    }
+
+    public long getTotalDeviceMemoryUsage() {
+        return taskGraph.getTotalDeviceMemoryUsage();
+    }
+
+    public long getCurrentMemoryUsage() {
+        return taskGraph.getCurrentMemoryUsage();
+    }


let's remove the public access modifier for those methods, since we do not want to expose them from the TaskGraph.

stratika · 2024-06-12T06:54:42Z

tornado-api/src/main/java/uk/ac/manchester/tornado/api/TaskGraph.java

+    public long getTotalBytesTransferred() {
+        return taskGraphImpl.getTotalBytesTransferred();
+    }
+
+    public long getTotalDeviceMemoryUsage() {
+        return taskGraphImpl.getTotalDeviceMemoryUsage();
+    }
+
+    public long getCurrentMemoryUsage() {
+        return taskGraphImpl.getCurrentMemoryUsage();
+    }


Same here, let's remove the public for those methods.

stratika · 2024-06-12T06:55:33Z

tornado-api/src/main/java/uk/ac/manchester/tornado/api/TornadoExecutionPlan.java

@@ -514,6 +524,14 @@ long getDeviceKernelTime() {
            return immutableTaskGraphList.stream().map(ImmutableTaskGraph::getDeviceKernelTime).mapToLong(Long::longValue).sum();
        }

+        long getTotalBytesCopyIn() {


Let's add Java docs to describe what is the functionality of each method. The description should make clear when each method should be used.

stratika · 2024-06-12T06:56:36Z

tornado-assembly/src/bin/tornado-test

@@ -124,6 +124,7 @@ __TEST_THE_WORLD__ = [
    TestEntry("uk.ac.manchester.tornado.unittests.api.TestDevices"),
    TestEntry("uk.ac.manchester.tornado.unittests.tensors.TestTensorTypes"),
    TestEntry("uk.ac.manchester.tornado.unittests.tensors.TestTensorAPIWithOnnx"),
+    TestEntry("uk.ac.manchester.tornado.unittests.memory.TestStressDeviceMemory"),


Shouldn't we add the new unit-test?

MemoryConsumptionTest

stratika · 2024-06-12T07:00:53Z

...s/opencl/src/main/java/uk/ac/manchester/tornado/drivers/opencl/runtime/OCLTornadoDevice.java

    }

    @Override
-    public synchronized int deallocate(DeviceBufferState deviceBufferState) {
+    public synchronized long deallocate(DeviceBufferState deviceBufferState) {
+        long memoryRegionDellocated = 0;


refactor the variable to deallocatedSpace to keep consistency with the allocate method. Also, let's use the same variable names across all backends:
PTXTornadoDevice, SPIRVTornadoDevice.

stratika · 2024-06-12T07:01:21Z

...s/spirv/src/main/java/uk/ac/manchester/tornado/drivers/spirv/runtime/SPIRVTornadoDevice.java

    }

    @Override
-    public synchronized int deallocate(DeviceBufferState deviceBufferState) {
+    public synchronized long deallocate(DeviceBufferState deviceBufferState) {
+        long spaceDeallocated = 0;


Refactor variable to deallocatedSpace

stratika · 2024-06-12T07:01:34Z

...drivers/ptx/src/main/java/uk/ac/manchester/tornado/drivers/ptx/runtime/PTXTornadoDevice.java

    }

    @Override
-    public synchronized int deallocate(DeviceBufferState deviceBufferState) {
+    public synchronized long deallocate(DeviceBufferState deviceBufferState) {
+        long spaceDeallocated = 0;


Refactor variable to deallocatedSpace

…r/ProfilerType.java Co-authored-by: Thanos Stratikopoulos <34061419+stratika@users.noreply.github.com>

…' into feat/query/memory

jjfumero · 2024-06-12T10:06:55Z

All comments applied

stratika

Thanks @jjfumero. LGTM

jjfumero added 7 commits June 11, 2024 10:33

[profiler] Add data size transferred in the profiler

4307fbc

[fix] Constant memory buffer size fixed

467eb42

Use the executionPlanId to create the kernel stack frame

d2fbe71

Merge branch 'develop' into feat/query/memory

f4ac73a

Memory tranferred test

89dfebd

Total memory allocated added into the profiler

e43e0d8

[feat] API query current memory usage

e49712f

jjfumero added API feature New feature proposal labels Jun 11, 2024

jjfumero requested review from mairooni and stratika June 11, 2024 14:17

jjfumero self-assigned this Jun 11, 2024

jjfumero added 2 commits June 11, 2024 16:19

[javadoc] New API methods

ad54abe

MemoryConsumption test added in the test suite

faa9dc5

mairooni approved these changes Jun 11, 2024

View reviewed changes

stratika requested changes Jun 12, 2024

View reviewed changes

jjfumero and others added 3 commits June 12, 2024 11:54

Update tornado-api/src/main/java/uk/ac/manchester/tornado/api/profile…

488af73

…r/ProfilerType.java Co-authored-by: Thanos Stratikopoulos <34061419+stratika@users.noreply.github.com>

[feedback] Fixes from PR beehive-lab#448

e5b3063

Merge remote-tracking branch 'refs/remotes/personal/feat/query/memory…

bd5b279

…' into feat/query/memory

ProfileInterface moved to ProfilerInterface

f1e6342

stratika approved these changes Jun 12, 2024

View reviewed changes

jjfumero merged commit dd4bba7 into beehive-lab:develop Jun 12, 2024
2 checks passed

jjfumero deleted the feat/query/memory branch June 12, 2024 10:18

jjfumero mentioned this pull request Jun 27, 2024

[release] TornadoVM v1.0.6 #464

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Query Memory Consumption and expanded profiler with memory consumption #448

[feat] Query Memory Consumption and expanded profiler with memory consumption #448

jjfumero commented Jun 11, 2024 •

edited by stratika

Loading

stratika Jun 12, 2024

stratika Jun 12, 2024

stratika Jun 12, 2024

stratika Jun 12, 2024

jjfumero Jun 12, 2024

stratika Jun 12, 2024

stratika Jun 12, 2024

stratika Jun 12, 2024

jjfumero commented Jun 12, 2024

stratika left a comment

[feat] Query Memory Consumption and expanded profiler with memory consumption #448

[feat] Query Memory Consumption and expanded profiler with memory consumption #448

Conversation

jjfumero commented Jun 11, 2024 • edited by stratika Loading

Description

Problem description

Backend/s tested

OS tested

Did you check on FPGAs?

How to test the new patch?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjfumero commented Jun 12, 2024

stratika left a comment

Choose a reason for hiding this comment

jjfumero commented Jun 11, 2024 •

edited by stratika

Loading