Skip to content
This repository has been archived by the owner on Jan 26, 2024. It is now read-only.

device memory accounting not correct #36

Open
psychocoderHPC opened this issue Jul 12, 2022 · 0 comments
Open

device memory accounting not correct #36

psychocoderHPC opened this issue Jul 12, 2022 · 0 comments

Comments

@psychocoderHPC
Copy link

After I found #35 I checked where the memory update function is used.
I found places where the accounting is performed after the memory allocation happened.
This is introducing incorrect memory accounting.
If one thread is allocating memory and another thread is checking the free memory it is possible that the following data race is happening and the amount of free memory reported is wrong.

Case:

  • Thread 0 is allocating memory but has not reached the function call updateFreeMemory() e.g. here
  • Thread 1 is querying the amount of free memory, result will be the amount of free memory before Thread 0 allocated memory because the accounting was not performed
  • Thread 0 is calling updateFreeMemory() -> now the internal free memory counter is correct but Thread 1 got the wrong amount of free memory

This issue is not very critical because in a parallel environment the amount of free memory could always be outdated of concurrent threads de/allocating memory.
Never the less IMO the account should be performed before the memory is allocated and in cases where the allocation failed the accounting should be rolled back.

Code snippets where the accounting is called after the real allocation:
https://github.com/ROCm-Developer-Tools/ROCclr/blob/6a62060c30cf910f07c2b98015def633daa7bf62/device/rocm/rocmemory.cpp#L809-L812
https://github.com/ROCm-Developer-Tools/ROCclr/blob/6a62060c30cf910f07c2b98015def633daa7bf62/device/rocm/rocmemory.cpp#L886-L892
https://github.com/ROCm-Developer-Tools/ROCclr/blob/6a62060c30cf910f07c2b98015def633daa7bf62/device/rocm/rocmemory.cpp#L1189-L1200

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant