Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous History Sampling #511

Merged
merged 5 commits into from
Jan 18, 2018
Merged

Asynchronous History Sampling #511

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
30fc6c2
fixed adding histories of different lengths through addRealization
PaulTalbot-INL Jan 12, 2018
a0a8e58
added fixme comment per reviewer request
PaulTalbot-INL Jan 16, 2018
15911d1
Merge branch 'dataobject-rework' into talbpaul/async-hists
PaulTalbot-INL Jan 17, 2018
e7fb0b6
fixed restart MC typo
PaulTalbot-INL Jan 17, 2018
54b083c
Merge branch 'dataobject-rework' into talbpaul/async-hists
PaulTalbot-INL Jan 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions framework/DataObjects/TestXHistorySet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,6 +644,44 @@ def formatRealization(rlz):
checkRlz('No input space',rlz0,rlz,skip='Timelike')



######################################
# ASYNC HISTORIES #
######################################
xml = createElement('HistorySet',attrib={'name':'test'})
xml.append(createElement('Input',text='a,b'))
xml.append(createElement('Output',text='x,y'))
data = XHistorySet.HistorySet()
data.messageHandler = mh
data._readMoreXML(xml)
rlz1 = {'a': np.array([1.0]),
'b': np.array([2.0]),
'x': np.array([1.0, 2.0, 3.0]),
'y': np.array([6.0, 7.0, 8.0]),
'time': np.array([0.0, 0.1, 0.2])}

rlz2 = {'a': np.array([11.0]),
'b': np.array([12.0]),
'x': np.array([11.0, 12.0]),
'y': np.array([16.0, 17.0]),
'time': np.array([0.05, 0.15])}

data.addRealization(rlz1)
data.addRealization(rlz2)
# check collection in realizations, in collector
checkRlz('Adding asynchronous histories, collector[0]',data.realization(index=0),rlz1,skip=['time'])
checkRlz('Adding asynchronous histories, collector[1]',data.realization(index=1),rlz2,skip=['time'])
# check stored in collector, not in synced histories
idx = data._orderedVars.index('time')
times = data._collector[:,idx]
checkArray('Asynchronous histories, collector, time[0]',times[0],rlz1['time'],float)
checkArray('Asynchronous histories, collector, time[1]',times[1],rlz2['time'],float)
# check as dataset, just for kicks
data.asDataset()
checkRlz('Adding asynchronous histories, dataset[0]',data.realization(index=0),rlz1,skip=['time'])
checkRlz('Adding asynchronous histories, dataset[1]',data.realization(index=1),rlz2,skip=['time'])


print(results)

sys.exit(results["fail"])
Expand Down
15 changes: 10 additions & 5 deletions framework/DataObjects/XDataSet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ def addRealization(self,rlz):
rlz = self._selectiveRealization(rlz)

## check alignment of indexes
self._checkAlignedIndexes(rlz) # TODO implement
self._checkAlignedIndexes(rlz)
# NB If no scalar entry is made, this construction fails. In that case,
# instead of treating each dataarrray as an object, numpy.asarray calls their asarray methods,
# unfolding them and making a full numpy array with more dimensions, instead of effectively
Expand Down Expand Up @@ -689,11 +689,16 @@ def _checkAlignedIndexes(self,rlz,tol=1e-15):
for index in self.indexes:
# if it's aligned so far, check if it still is
if index in self._alignedIndexes.keys():
# if close enough, then keep the aligned values; otherwise, take action
if isinstance(rlz[index][0],(float,int)):
closeEnough = all(np.isclose(rlz[index],self._alignedIndexes[index],rtol=tol))
# first, if lengths don't match, they're not aligned.
if len(rlz[index]) != len(self._alignedIndexes[index]):
closeEnough = False
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a question here. It seems we will not check the duplicate data in the indexes if there length are not the same. If I understand correctly, this will increase the memory and slow down the calculation. Let's talk on Monday.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alfoa I need you comments here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct; if the length of the indexes is not the same, then the old index and the new index cannot be exactly the same, so we have to store them separately. Right?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed in the meeting, can you add "FIXME" here regarding the memory and speed issue?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

else:
closeEnough = all(rlz[index] == self._alignedIndexes[index])
# "close enough" if float/int, otherwise require exactness
if isinstance(rlz[index][0],(float,int)):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about it is a numpy float or int?
You should add numpy.integer and numpy.number for checking all of them

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from my understanding, isinstance(np.integer(6),int) is True. I will run a test to confirm...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do have an issue when the dtype=np.int32 in the past.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

a = np.array([6],dtype=np.integer)
type(a[0])           # result is <type 'numpy.int64'>
isinstance(a[0],int) # result is True

a = np.array([3.14],dtype=np.float)
type(a[0])             # result is <type 'numpy.float64'>
isinstance(a[0],float) # result is True

Inheritance! 😃

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you test np.int32, np.float32, np.int8, ...?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paul.
Firstly I told you that the correct way to check for any float-like types in numpy is numpy.number (from which all the numpy.float* inherits from) and not numpy.float
Second, I agree that this checks should be handled in the utilities where we are going to have control on the additional types we would (if any) need in the future.

>>> a = np.array([6],dtype=np.float32)
>>> b = np.array([6],dtype=np.float64)
>>> c = np.array([6],dtype=np.float128)

>>> isinstance(a[0],np.number)
True
>>> isinstance(b[0],np.number)
True
>>> isinstance(c[0],np.number)
True

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From NumPy:
screen shot 2018-01-16 at 12 11 05 pm

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the fundamental data types from NumPy

Copy link
Collaborator Author

@PaulTalbot-INL PaulTalbot-INL Jan 16, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, do not add more here. This needs to be a seperate discussion from this PR. Let's focus.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is handled in #524.

closeEnough = all(np.isclose(rlz[index],self._alignedIndexes[index],rtol=tol))
else:
closeEnough = all(rlz[index] == self._alignedIndexes[index])
# if close enough, then keep the aligned values; otherwise, take action
if not closeEnough:
dtype = rlz[index].dtype
# TODO add new column to collector, propagate values up to (not incl) current rlz
Expand Down