Close #152 Represent Bools as Enums #153

ax3l · 2015-02-07T01:49:34Z

Proposal to change the internal representation of bool values to be compatible with h5py as documented in #152.

increases the major file format version since that won't be backwards compatible, e.g., for restarts from files

To Do

add h5py compatibility tests to .travis.yml / ./run_tests
double-check sizeof (bool) which is 1 Byte vs. real size of 8 Bytes after conversation - correct in all cases (add a read/write test?)
actually this is not an API change so 1.2.5 milestone would be ok, too. But it looks a bit odd to make such a change in a revision - so go for the 1.3 target version?
add enum/bool read support (e.g., for restarts)

ax3l · 2015-02-07T01:51:50Z

@PrometheusPi can you check out this libSplash branch and compile PIConGPU against it to run and test your example again?

PrometheusPi · 2015-02-09T01:24:39Z

@ax3l Yes, I will check this out later today. Would be great if it works.

PrometheusPi · 2015-02-09T10:31:32Z

I compiled PIConGPU with libSplash from this pull. When writing the checkpoint of time step 0, the following exception was thrown:

[1,0]<stderr>:terminate called after throwing an instance of 'splash::DCException'
[1,0]<stderr>:  what():  Exception for DCDataSet [radiationFlag] create: Failed to create dataset
[1,0]<stderr>:[kepler003:08315] *** Process received signal ***
[1,0]<stderr>:[kepler003:08315] Signal: Aborted (6)
[1,0]<stderr>:[kepler003:08315] Signal code:  (-6)
[1,0]<stderr>:[kepler003:08315] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f3f12921340]
[1,0]<stderr>:[kepler003:08315] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f3f10bc2cc9]
[1,0]<stderr>:[kepler003:08315] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f3f10bc60d8]
[1,0]<stderr>:[kepler003:08315] [ 3] /opt/pkg/compiler/gnu/gcc/4.6.2/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x11d) [0x7f3f11227c6d]
[1,0]<stderr>:[kepler003:08315] [ 4] /opt/pkg/compiler/gnu/gcc/4.6.2/lib64/libstdc++.so.6(+0xbedd6) [0x7f3f11225dd6]
[1,0]<stderr>:[kepler003:08315] [ 5] /opt/pkg/compiler/gnu/gcc/4.6.2/lib64/libstdc++.so.6(+0xbee03) [0x7f3f11225e03]
[1,0]<stderr>:[kepler003:08315] [ 6] /opt/pkg/compiler/gnu/gcc/4.6.2/lib64/libstdc++.so.6(+0xbeefe) [0x7f3f11225efe]
[1,0]<stderr>:[kepler003:08315] [ 7] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN6splash9DCDataSet6createERKNS_14CollectionTypeEiNS_10DimensionsEjbb+0x92c) [0x76866c]
[1,0]<stderr>:[kepler003:08315] [ 8] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN6splash21ParallelDataCollector12writeDataSetEiNS_10DimensionsES1_RKNS_14CollectionTypeEjNS_9SelectionEPKcPKv+0x146) [0x751926]
[1,0]<stderr>:[kepler003:08315] [ 9] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN6splash21ParallelDataCollector5writeEiNS_10DimensionsES1_RKNS_14CollectionTypeEjNS_9SelectionEPKcPKv+0x28c) [0x755e3c]
[1,0]<stderr>:[kepler003:08315] [10] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN6splash23ParallelDomainCollector11writeDomainEiNS_10DimensionsES1_RKNS_14CollectionTypeEjNS_9SelectionEPKcNS_6DomainENS_16IDomainCollector12DomDataClassEPKv+0x121) [0x75d181]
[1,0]<stderr>:[kepler003:08315] [11] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN6splash23ParallelDomainCollector11writeDomainEiRKNS_14CollectionTypeEjNS_9SelectionEPKcNS_6DomainES7_NS_16IDomainCollector12DomDataClassEPKv+0x235) [0x75c905]
[1,0]<stderr>:[kepler003:08315] [12] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN8picongpu4hdf517ParticleAttributeINS_24placeholder_definition1813radiationFlagEEclIN5PMacc5FrameINS_23OperatorCreateVectorBoxENS6_19ParticleDescriptionIN5boost3mpl6stringILi101ELi0ELi0ELi0ELi0ELi0ELi0ELi0EEENS6_4math2CT6VectorIN4mpl_10integral_cIiLi8EEESJ_NSH_2naEEENSB_6v_itemINS_24placeholder_definition1313globalCellIdxINS_24placeholder_definition2017globalCellIdx_picENS6_24placeholder_definition1113pmacc_isAliasEEENSM_IS3_NSM_INS_24placeholder_definition1613momentumPrev1ENSM_INS_24placeholder_definition179weightingENSM_INS_24placeholder_definition158momentumENSM_INS_24placeholder_definition128positionINS_24placeholder_definition1412position_picESS_EENSB_7vector0ISK_EELi0EEELi0EEELi0EEELi0EEELi0EEELi0EEENSB_6vectorINS_24placeholder_definition2214particlePusherINS_9particles6pusher5BorisESS_EENS_24placeholder_definition215shapeINS1G_6shapes3TSCESS_EENS_24placeholder_definitio??????Y ? Q????E???E?@???
[1,0]<stderr>:??? ? Q????E???E?{ { ?Y???????????(A???,??3@=????[kepler003:08315] [13] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN8picongpu4hdf512WriteSpeciesINS_9ParticlesIN5PMacc19ParticleDescriptionIN5boost3mpl6stringILi101ELi0ELi0ELi0ELi0ELi0ELi0ELi0EEENS3_4math2CT6VectorIN4mpl_10integral_cIiLi8EEESE_NSC_2naEEENS6_6v_itemINS_24placeholder_definition1813radiationFlagENSH_INS_24placeholder_definition1613momentumPrev1ENSH_INS_24placeholder_definition179weightingENSH_INS_24placeholder_definition158momentumENSH_INS_24placeholder_definition128positionINS_24placeholder_definition1412position_picENS3_24placeholder_definition1113pmacc_isAliasEEENS6_7
[1,0]<stderr>:vector0ISF_EELi0EEELi0EEELi0EEELi0EEELi0EEENS6_6vectorINS_24placeholder_definition2214particlePusherINS_9particles6pusher5BorisESV_EENS_24placeholder_definition215shapeINS17_6shapes3TSCESV_EENS_24placeholder_definition2313interpolationINS_28FieldToParticleInterpolationIS1E_NS_30AssignedTrilinearInterpolationEEESV_EENS_24placeholder_definition247currentINS_13curre??????Y ? Q????E???E?@???
[1,0]<stderr>:??? ? Q????E???E?{ { ?Y?????[kepler003:08315] [14] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN8picongpu4hdf510HDF5Writer9writeHDF5EPv+0x6f1) [0x711831]
[1,0]<stderr>:[kepler003:08315] [15] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN8picongpu4hdf510HDF5Writer20notificationReceivedEjb+0x36c) [0x71206c]
[1,0]<stderr>:[kepler003:08315] [16] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN5PMacc16SimulationHelperILj2EE11dumpOneStepEj+0x121) [0x6ab471]
[1,0]<stderr>:[kepler003:08315] [17] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN5PMacc16SimulationHelperILj2EE15startSimulationEv+0x130) [0x6a6d40]
[1,0]<stderr>:[kepler003:08315] [18] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(_ZN8picongpu17SimulationStarterINS_21InitialiserControllerENS_16PluginControllerENS_12MySimulationEE5startEv+0x72) [0x6f6a42]
[1,0]<stderr>:[kepler003:08315] [19] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu(main+0x65) [0x652945]
[1,0]<stderr>:[kepler003:08315] [20] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f3f10badec5]
[1,0]<stderr>:[kepler003:08315] [21] /data/hplsim/production/HHG_frontSurfaceInstability/011_Kluge_A_radiation_libSplashFix/picongpu/bin/picongpu() [0x652def]
[1,0]<stderr>:[kepler003:08315] *** End of error message ***

ax3l · 2015-02-09T12:33:02Z

fixed, let us add the h5py read example now on h5/testWriteRead_0_0_0.h5 data: deep/folders/data_bool (steps 10 and 20) -> added

ax3l · 2015-02-09T13:32:07Z

@PrometheusPi tests added and it should work now. can you please give it a try with your PIConGPU example again?

PrometheusPi · 2015-02-09T13:32:57Z

@ax3l I am already on it.

ax3l · 2015-02-09T13:33:00Z

@f-schmitt-zih ready for review! :)

ax3l · 2015-02-09T13:44:03Z

src/include/splash/basetypes/ColTypeBool.hpp

@@ -35,8 +35,11 @@ class ColTypeBool : public CollectionType

    ColTypeBool()
    {
-        const hsize_t dim[] = {sizeof (bool)};
-        this->type = H5Tarray_create(H5T_NATIVE_B8, 1, dim);
+        this->type = H5Tenum_create(H5T_NATIVE_INT8);


same 1byte memory footprint as before, but now an 8-bit native (short) int: link

add comment above?

PrometheusPi · 2015-02-09T14:32:54Z

👍 great job @ax3l - it works !!!

ax3l · 2015-02-09T15:06:49Z

awesome, thanks!

f-schmitt · 2015-02-10T05:29:55Z

tests/SimpleDataTest.cpp

+        if (boolRead[i] != boolWrite[i])
+        {
+#if defined TESTS_DEBUG
+            std::cout << i << ": " << boolRead[i] << " != exptected " << boolWrite[i] << std::endl;


typo, "expected"

Ha, copy paste typo! Thanks! :)

ax3l · 2015-02-10T14:02:47Z

I also added travis testing with a simple, serial h5py read now.
That is enough to verify the bool compatibility.

on a side note: if someone wants to test parallel hdf5 reads, there are two issues in h5py (h5py/h5py#542) and pip (pypa/pip#18) that make automated installs a bit tricky (if at least one of them get's implemented it will be easier).

ax3l · 2015-02-11T12:27:49Z

@f-schmitt-zih ready for final review/merge if you like. Run-Time tests are all clean :)

- read/write a bool field in C++ - read a bool field with h5py - copy test scripts to build dir

Close #152 Represent Bools as Enums

bussmann · 2015-03-20T12:48:37Z

Great job!

Else it does not identify those as `bool_` but as `int8`. Regression for ComputationalRadiationPhysics#153 (still in `dev`).

Else it does not identify those as `bool_` but as `int8`. Regression for ComputationalRadiationPhysics#153 (still in `dev`). Increases format by a minor since the structure is still the same but the identification in `h5py` is improved (and it is still not in a stable release).

ax3l added the feature label Feb 7, 2015

ax3l added this to the Version 1.3 milestone Feb 7, 2015

ax3l assigned f-schmitt Feb 7, 2015

ax3l force-pushed the topic-boolAsEnum branch 3 times, most recently from cfc9765 to a9b9bce Compare February 9, 2015 12:30

ax3l force-pushed the topic-boolAsEnum branch 6 times, most recently from d975023 to 00d4907 Compare February 9, 2015 13:29

ax3l mentioned this pull request Feb 9, 2015

Tests: Better Documentation #156

Closed

2 tasks

ax3l reviewed Feb 9, 2015
View reviewed changes

ax3l force-pushed the topic-boolAsEnum branch 2 times, most recently from af6779b to 42fe24f Compare February 9, 2015 13:52

f-schmitt reviewed Feb 10, 2015
View reviewed changes

ax3l force-pushed the topic-boolAsEnum branch 12 times, most recently from 9b12e46 to e2a9ec4 Compare February 10, 2015 13:53

ax3l force-pushed the topic-boolAsEnum branch from e2a9ec4 to fe01bae Compare February 11, 2015 12:30

ax3l added 2 commits February 11, 2015 13:30

Add Bool SimpleTest

59d2087

- read/write a bool field in C++ - read a bool field with h5py - copy test scripts to build dir

Add h5py & numpy to travis

fe01bae

f-schmitt pushed a commit that referenced this pull request Feb 12, 2015

Merge pull request #153 from ax3l/topic-boolAsEnum

029c9b8

Close #152 Represent Bools as Enums

f-schmitt merged commit 029c9b8 into ComputationalRadiationPhysics:dev Feb 12, 2015

ax3l deleted the topic-boolAsEnum branch February 12, 2015 10:21

ax3l mentioned this pull request Mar 31, 2015

Revert Attributes: Offset, Size #162

Open

1 task

ax3l mentioned this pull request Apr 14, 2015

Add Char & String Data Types #167

Merged

2 tasks

ax3l mentioned this pull request May 4, 2015

Add Recognition for new DCDataTypes #174

Closed

2 tasks

ax3l mentioned this pull request Aug 10, 2015

SDC_ATTR_COMPRESSION: New Bool Type #184

Closed

ax3l added the format change label Aug 12, 2015

ax3l added a commit to ax3l/libSplash that referenced this pull request Oct 6, 2015

Bool: CAPS Enum Labels for h5py

a0aa309

Else it does not identify those as `bool_` but as `int8`. Regression for ComputationalRadiationPhysics#153 (still in `dev`).

ax3l mentioned this pull request Oct 6, 2015

Bool: CAPS Enum Labels for h5py #198

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Close #152 Represent Bools as Enums #153

Close #152 Represent Bools as Enums #153

ax3l commented Feb 7, 2015

ax3l commented Feb 7, 2015

PrometheusPi commented Feb 9, 2015

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

ax3l commented Feb 9, 2015

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

ax3l Feb 9, 2015

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

f-schmitt Feb 10, 2015

ax3l Feb 10, 2015

ax3l commented Feb 10, 2015

ax3l commented Feb 11, 2015

bussmann commented Mar 20, 2015

Close #152 Represent Bools as Enums #153

Close #152 Represent Bools as Enums #153

Conversation

ax3l commented Feb 7, 2015

To Do

ax3l commented Feb 7, 2015

PrometheusPi commented Feb 9, 2015

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

ax3l commented Feb 9, 2015

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

ax3l Feb 9, 2015

Choose a reason for hiding this comment

PrometheusPi commented Feb 9, 2015

ax3l commented Feb 9, 2015

f-schmitt Feb 10, 2015

Choose a reason for hiding this comment

ax3l Feb 10, 2015

Choose a reason for hiding this comment

ax3l commented Feb 10, 2015

ax3l commented Feb 11, 2015

bussmann commented Mar 20, 2015