Question: how to create an epc of properties without including the grid

[mgimhof]

I have an application that creates an EPC file containing only ContinuousProperties and the EpcExternalPartReference to the h5 file. The use case is python as an external property calculator. The grid is not explicitly included because the app already contains the grid and it does not need to be round tripped (it is a very simple-minded calculator that does not account for grid geometry and topology!).

I can pick up properties and perform some calculation to create a new one. The problem is when trying to create an epc for the new property.

Happening to know the number of cells and the grid uuid, I expected that the return problem can be mocked up as

import numpy as np

import resqpy.model as rq
import resqpy.property as rp

CONTENT_TYPE = 'obj_IjkGridRepresentation'
SUPPORT_UUID = '492f069b-888a-4a17-8cc1-cdc845774f18'
NUMCELLS = 64

new_model = rq.Model('d:/new.epc', new_epc=True)
grid = new_model.create_supporting_representation(support_uuid=SUPPORT_UUID, title='grid',
                                                  content_type=CONTENT_TYPE)
new_model.add_part(content_type=CONTENT_TYPE, uuid=SUPPORT_UUID, root=grid)
values = np.arange(start=0, stop=NUMCELLS, dtype=np.float32)
p = rp.Property.from_array(new_model, cached_array=values, source_info='rpm', keyword='PERM_IJ',
                           property_kind='permeability rock', support_uuid=SUPPORT_UUID)

new_model.store_epc()

If the add_part line is included, then I fail with

C:\Users\m9imhof\AppData\Local\ExxonMobil\PetrelPythonPlugin\pytrel\python.exe D:/DEV/resqpy/test_local/import_export.py
Traceback (most recent call last):
  File "D:\DEV\resqpy\test_local\import_export.py", line 15, in <module>
    p = rp.Property.from_array(new_model, cached_array=values, source_info='rpm', keyword='PERM_IJ',
  File "D:\DEV\resqpy\resqpy\property.py", line 3542, in from_array
    prop.create_xml(support_uuid = support_uuid,
  File "D:\DEV\resqpy\resqpy\property.py", line 3736, in create_xml
    self.collection.create_xml_for_imported_list_and_add_parts_to_model(
  File "D:\DEV\resqpy\resqpy\property.py", line 2685, in create_xml_for_imported_list_and_add_parts_to_model
    p_node = self.create_xml(
  File "D:\DEV\resqpy\resqpy\property.py", line 3024, in create_xml
    self.model.create_reciprocal_relationship(p_node, 'destinationObject', support_root, 'sourceObject')
  File "D:\DEV\resqpy\resqpy\model.py", line 2973, in create_reciprocal_relationship
    uuid_b = node_b.attrib['uuid']
  File "src\lxml\etree.pyx", line 2479, in lxml.etree._Attrib.__getitem__
KeyError: 'uuid'
Process finished with exit code 1

Without the add_part line, I fail with

C:\Users\m9imhof\AppData\Local\ExxonMobil\PetrelPythonPlugin\pytrel\python.exe D:/DEV/resqpy/test_local/import_export.py
Traceback (most recent call last):
  File "D:\DEV\resqpy\test_local\import_export.py", line 15, in <module>
    p = rp.Property.from_array(new_model, cached_array=values, source_info='rpm', keyword='PERM_IJ',
  File "D:\DEV\resqpy\resqpy\property.py", line 3522, in from_array
    prop.collection.set_support(model = prop.model,
  File "D:\DEV\resqpy\resqpy\property.py", line 243, in set_support
    assert support_part is not None, 'supporting representation part missing in model'
AssertionError: supporting representation part missing in model
Process finished with exit code 1

[mgimhof]

Now I realize that I confuse things badly in my example, but the question remains: how could I create an epc file with just a couple of properties all pointing to the same UUID for the supporting representation.

I could look over derivedmodel.add_one_grid_property_array() and finally remove grid and crs; but that seems rather heavy.

[andy-beer]

Ah, this is a good question @mgimhof !

In RESQML a property object includes a reference to a supporting representation. The RESQML standard allows such referenced parts to be absent from the dataset. However, resqpy often requires them to be present – and the processing of properties is one such situation, primarily to ascertain the expected shape of the property arrays (and also to determine default and/or valid indexable elements).

Without some major work on the resqpy code, I think that adding in a mock grid is the way to go, for now at least.

If the original grid is an IJK grid, then the resqpy RegularGrid class can be used, initialising with just the extent_kji argument. If the original grid is Unstructured, then you can use the brand new (= minimally tested) resqpy UnstructuredGrid class. Both of these should allow grids without a geometry to be created and used as a supporting representation. The difficult step is to patch the uuid – either inserting the uuid of the mock grid into the reference nodes of the properties (which would leave the xml relationships messed up, though one might get away with that for temporary processing), or changing the uuid of the mock grid to that of the original grid prior to creating the mock grid xml. There is some uuid patching functionality in the Model class but I will have to take a good look before advising further.

In your example the add_part failed whilst trying to create xml relationships (reciprocal relationships) – the create_xml methods in resqpy usually allow this to be skipped by setting an add_relationships argument to False.

[andy-beer]

The Model class now has a method for patching the uuid in xml supporting representation reference blocks. Here is some sketchy code outline showing possible use. Note that the _rels information is not modified in this sequence, which might cause problems for some workflows.

import resqpy.model as rq
import resqpy.grid as grr
import resqpy.property as rqp
import resqpy.crs as rqc

epc = 'props_only.epc'

model = rq.Model(epc)

remote_grid_uuid = '73bdefda-0953-11ec-8eb8-80e650222718'
remote_grid_extent_kji = (100, 250, 300)

# this example is for an IJK grid, something similar could be done for an Unstructured grid
crs = rqc.Crs(model)
crs.create_xml()
temp_grid = grr.RegularGrid(model, extent_kji = remote_grid_extent_kji, title = 'mock grid', crs_uuid = crs.uuid)
temp_grid.create_xml(
    add_relationships = False,
    write_active = False,
    write_geometry = False,
    add_cell_length_properties = False)

# patch in the uuid of the temporary grid as supporting representation
change_count = 0
for p_root in model.roots(obj_type = 'ContinuousProperty'):  # note: extend roots list for other classes of property
    p_root = model.root_for_part(part)
    changed = model.change_uuid_in_supporting_representation_reference(p_root, remote_grid_uuid, temp_grid.uuid)
    if changed: change_count += 1
print(f'supporting representation for {change_count} properties modified')

# rebuild the property collection for the temporary grid
temp_grid.property_collection = None
temp_pc = temp_grid.extract_property_collection()
print(temp_pc.number_of_parts())

# now work with properties as normal (though some _rels based functionality might not work)

# to add new properties to the dataset, use something like the following

new_array = np.random.random(remote_grid_extent_kji)
temp_pc.add_cached_array_to_imported_list(
    new_array,
    'calculated property',  # source info, typically used when importing data from another format
    'random',  # citation title for new property
    discrete = False,
    uom = 'm3',
    time_index = None,
    null_value = None,
    property_kind = 'volume',
    local_property_kind_uuid = None,
    facet_type = None,
    facet = None,
    realization = None,
    indexable_element = 'cells',
    count = 1,
    const_value = None)
# repeat for multiple new properties as required

# append new arrays to the hdf5 file
temp_pc.write_hdf5_for_imported_list()

# create the xml for the new arrays (with temp supporting representation)
temp_pc.create_xml_for_imported_list_and_add_parts_to_model(support_uuid = temp_grid.uuid)

# when array processing is complete, patch the true supporting representation uuid back into the xml
change_count = 0
for part in temp_pc.parts():
    p_root = model.root_for_part(part)
    changed = model.change_uuid_in_supporting_representation_reference(p_root, temp_grid.uuid, remote_grid_uuid)
    if changed: change_count += 1
print(f'supporting representation for {change_count} properties restored')

# delete the temp grid
model.remove_part(model.part(uuid = temp_grid.uuid))

# and finally store the epc
model.store_epc()

[mgimhof]

@andy-beer
Thanks, that is a workable solution.

With regard to the rels files: The effect of model.remove_part(model.part(uuid=temp_grid.uuid), True) mimics what I see RESQML APIs and applications do.

[andy-beer]

Thanks for the feedback @mgimhof

I'll leave this issue open for now, as a reminder that we could make the resqpy property module work better in this situation.

[mgimhof]

@andy-beer
The motivation of for this issue is a calculator workflow where properties are imported from an EPC that does not contain the grid, compute some new properties, and export these to EPC (without grid since it was not transmitted in the first place)

In light of the script above, we need remote_grid_uuid and remote_grid_extent_kji

remote grid extent I can get from the shape of the imported properties.
remote grid uuid I could get from the supporting representation referenced in each property xml

To get the support representation uuid, I see 3 solutions:

1. just call support_uuid = rqet.find_nested_tags_text(property.root, ['SupportingRepresentation', 'UUID'])
2. stick self.support_uuid = rqet.find_nested_tags_text(property.root, ['SupportingRepresentation', 'UUID']) into the definition of Property._load_from_xml(), or
3. try to overwrite self.collection.support_uuid = rqet.find_nested_tags_text(property.root, ['SupportingRepresentation', 'UUID']) in the definition of Property._load_from_xml() since the PropertyCollection contains this attribute

[andy-beer]

@mgimhof
The best long term solution is for the property.py module to be modified so that both PropertyCollection and Property classes work when the supporting representation is not present in the dataset. Then hopefully there would be no need to modify the references. Until such time, I would keep things as simple as possible.

Your 3 options make use of property.root which implies the Property object has been initialised, which might not be possible until after the substitution. So I would suggest for now going with option 1, but instead of property.root you will need to use the root from the Model object prior to instantiating a resqpy Property,. So, something like:

property_root = model.root(obj_type = 'ContinuousProperty', multiple_handling = 'first') # or Discrete or Categorical

then use property_root instead of property.root in your option 1.

[andy-beer]

@mgimhof
An equivalent and slightly cleaner version of the above would be:

property_part = model.part(obj_type = 'ContinuousProperty', multiple_handling = 'first')
support_uuid = model.supporting_representation_for_part(property_part)

or more completely to handle the 3 numerical property types the first line above could be replaced with something like:

for prop_type in ('Continuous', 'Discrete', 'Categorical'):
   property_part = model.part(obj_type = prop_type + 'Property', multiple_handling = 'first')
   if property_part is not None: break