Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update gather_data function #226

Open
wants to merge 3 commits into
base: development
Choose a base branch
from
Open

Update gather_data function #226

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3e2d75a
Update _utils.py
bknight1 Aug 13, 2024
b37bc20
Update discretisation.py
bknight1 Aug 15, 2024
d651d49
add L2 norm function
bknight1 Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/underworld3/discretisation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ def view(self):
else:
i = 0

ii = uw.utilities.gather_data(np.array([i]), dtype="int")
ii = uw.utilities.gather_data(np.array([i]))

if uw.mpi.rank == 0:
print(
Expand Down
19 changes: 12 additions & 7 deletions src/underworld3/utilities/_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,14 @@ def mem_footprint():
return python_process.memory_info().rss // 1000000


def gather_data(val, bcast=False, dtype="float64"):
def gather_data(val, masked_value=-999999, bcast=False):

"""
gather values on root (bcast=False) or all (bcast = True) processors
Parameters:
vals : Values to combine into a single array on the root or all processors
masked_value : value to use as mask value
bcast : broadcast array/value to all processors

returns:
val_global : combination of values form all processors
Expand All @@ -106,12 +108,15 @@ def gather_data(val, bcast=False, dtype="float64"):
rank = uw.mpi.rank
size = uw.mpi.size

dtype = val.dtype


### make sure all data comes in the same order
with uw.mpi.call_pattern(pattern="sequential"):
if len(val > 0):
val_local = np.ascontiguousarray(val.copy())
else:
val_local = np.array([np.nan], dtype=dtype)
# with uw.mpi.call_pattern(pattern="sequential"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why comment out the uw.mpi.call_pattern part? This was useful for dealing with sequential or parallel hdf5 implementations on various HPCs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From tests I've done it hasn't made a difference to the order of the array. If anything needed to be sequential it would be the gathering of the data on the root processor not the creation of the array on each processor. I'm not sure how Gatherv gathers the data but from those tests thee array is in the same order each time.

if len(val > 0):
val_local = np.ascontiguousarray(val.copy(), dtype=dtype)
else:
val_local = np.ascontiguousarray([masked_value], dtype=dtype)


comm.barrier()
Expand All @@ -134,7 +139,7 @@ def gather_data(val, bcast=False, dtype="float64"):

if uw.mpi.rank == 0:
### remove rows with NaN
val_global = val_global[~np.isnan(val_global)]
val_global = val_global[val_global != masked_value]

comm.barrier()

Expand Down
Loading