Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix warnings, commenting off 'find_path' and 'find_library' for Mercury in src/commons/CMakeLists.txt #93

Merged
merged 253 commits into from
Jun 23, 2023

Conversation

zhangwei217245
Copy link
Collaborator

@zhangwei217245 zhangwei217245 commented Jun 23, 2023

I fixed all the warnings that I can fix throughout the entire project, and also suppressed some trivial warnings (they are either not accurate in reflecting the errors or only help little in ensuring code correctness.)

Also, per Dr. Tang's request, I commented off 'find_path' and 'find_library' directives in src/commons/CMakeLists.txt. But we need to discuss about the dependency management of the project with CMake, especially the standard of using CMake find_package directive.

Additionally, this PR fixes some of the length issues for server address strings, temp dir strings, hostnames and object/app names. This will ensure that no potential memory leak for snprintf.

@zhangwei217245
Copy link
Collaborator Author

image

…NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4
@jeanbez
Copy link
Member

jeanbez commented Jun 23, 2023

@zhangwei217245 what is the dependency problem we are having that CMake cannot identify?

@zhangwei217245
Copy link
Collaborator Author

@zhangwei217245 what is the dependency problem we are having that CMake cannot identify?

By default, find_package() directive should be able to capture XXX_INCLUDE_DIR (suppose XXX is the package name we are trying to find). However, for Mercury, calling find_package(MERCURY) doesn't materialize MERCURY_INCLUDE_DIR variable but this variable is included as part of PDC_EXT_INCLUDE_DEPENDENCIES (line 29 in src/commons/CMakeLists.txt and line 45 of src/api/CMakeLists.txt in the current develop branch) .

I have encountered an issue when compiling on my own machine which shows that mercury_atomic.h could not be found when the compiler was trying to process pdc_id_pkg.h. (Though this is not reproduced now). And this is why I noticed that MERCURY_INCLUDE_DIR is not materialized.

Though currently the project can compile without a problem showing "mercury.h" cannot be found, it doesn't mean we will never need MERCURY_INCLUDE_DIR to be materialized. Plus,

With the CMake convention that XXX_INCLUDE_DIR should be addressed when package is found, I suggest to

  1. make sure XXX_INCLUDE_DIR is materialized whenever find_package is called
  2. containing FindXXX.cmake file in our project to help with the search of XXX packages in CMake.

@houjun houjun merged commit 46abeed into hpc-io:develop Jun 23, 2023
houjun added a commit that referenced this pull request Aug 23, 2023
…ry in src/commons/CMakeLists.txt (#93)

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

* update script

* fix some warnings

* fix some warnings

* update

* update

* fix warning

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* update

* update

* update

* server address and file paths using 1024, TMP_DIR path using 1024/2, NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4

* update

* update

* update

* update

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>
houjun added a commit that referenced this pull request Aug 23, 2023
* updates in documentation

* update docs

* trigger update

* trigger update

* remove extension

* include docs build dependencies

* update file

* update metrics

* Update .gitlab-ci.yml

* Update .gitlab-ci.yml

* update logo

* Update .gitlab-ci.yml

* adding recovered documentation

* update text

* fix typo

* update CSS

* increase content width

* remove build docs

* Refer documentation to readthedocs website (#95)

* Update README.md

* Update getting_started.rst

* Update getting_started.rst

* include updated text

* fix links

* fix typo

* Update README.md

* update output

* Revert "update output"

This reverts commit fe1f8b4.

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* update code

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* clang format

* clang format

* clang-format-10

* change file name

* update llsm importer

* update llsm importer

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update tag names

* update tag names

* update query startingpos

* update job scripts

* fix iteration count in final report

* update job scripts and benckmark program

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* Remove unnecessary fflush call

Signed-off-by: Chen Wang <wangvsa@gmail.com>

* Fix Issue #85, server segfault when another client application with different number of ranks connects to it

* Committing clang-format changes

* update VPIC output timing precision (#88)

* update VPIC output timing precision

* update timing to make consistent

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* add FindMERCURY.cmake

* LLSM Importer update: new job script + new data type update on kvtags (#92)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* fix warnings, commenting off 'find_path' and 'find_library' for Mercury in src/commons/CMakeLists.txt  (#93)

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

* update script

* fix some warnings

* fix some warnings

* update

* update

* fix warning

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* update

* update

* update

* server address and file paths using 1024, TMP_DIR path using 1024/2, NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4

* update

* update

* update

* update

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* Update clang-format-fix.yml

* Update clang-format-fix.yml

* Increase the default server cache size to 32GB and flush frequency to 30s

* Committing clang-format changes

* update commons/CMakeLists.txt

* Fix unnecessary memory allocation (#103)

* Fix an issue with opening a deleted container, added test (#101)

* Fix an issue with opening a deleted container, added test

* Refactor the query aggregation process

* Fix container tag delete error (#102)

* Fix container tag delete error

* Committing clang-format changes

* Update tag delete function

* Refactor metdata lookup process for tag deletion

* Committing clang-format changes

* Formatting and comment

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix the server cache issue when cache becomes full and needs flush (#113)

* Fix the server cache issue when cache becomes full and need flush

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix a wrong option description. (#115)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update option message

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* Install header files needed by PDCpy (#114)

* Support unlimited object dimension size (#117)

* Support unlimited object dimension szie

* Add function description for PDC_SIZE_UNLIMITED

* Fix obj_round_robin_io_all test code

* Committing clang-format changes

* More header files fix for PDCpy

* Fix cmake path (#121)

* Fix cmake path

* Fix cmake path

* Kvtag query (#122)

* Add a new collective kvtag query api to return full/aggregated results to all clients

* Committing clang-format changes

* Add test code

* Committing clang-format changes

* Add an optimization when multiple clients issue different queries

* Add test program

* Fix free issue

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* fix conflict issue

* fix conflict issue

---------

Signed-off-by: Chen Wang <wangvsa@gmail.com>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>
Co-authored-by: Wei Zhang <zhangwei217245@gmail.com>
Co-authored-by: Chen Wang <wangvsa@gmail.com>
Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Wei Zhang <zhangwei217245@lbl.gov>
@zhangwei217245 zhangwei217245 deleted the feature/llsm_importer branch September 1, 2023 14:23
jeanbez added a commit that referenced this pull request Jan 9, 2024
…ry in src/commons/CMakeLists.txt (#93)

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

* update script

* fix some warnings

* fix some warnings

* update

* update

* fix warning

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* update

* update

* update

* server address and file paths using 1024, TMP_DIR path using 1024/2, NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4

* update

* update

* update

* update

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>
jeanbez added a commit that referenced this pull request Jan 9, 2024
* updates in documentation

* update docs

* trigger update

* trigger update

* remove extension

* include docs build dependencies

* update file

* update metrics

* Update .gitlab-ci.yml

* Update .gitlab-ci.yml

* update logo

* Update .gitlab-ci.yml

* adding recovered documentation

* update text

* fix typo

* update CSS

* increase content width

* remove build docs

* Refer documentation to readthedocs website (#95)

* Update README.md

* Update getting_started.rst

* Update getting_started.rst

* include updated text

* fix links

* fix typo

* Update README.md

* update output

* Revert "update output"

This reverts commit fe1f8b4.

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* update code

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* clang format

* clang format

* clang-format-10

* change file name

* update llsm importer

* update llsm importer

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update tag names

* update tag names

* update query startingpos

* update job scripts

* fix iteration count in final report

* update job scripts and benckmark program

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* Remove unnecessary fflush call

Signed-off-by: Chen Wang <wangvsa@gmail.com>

* Fix Issue #85, server segfault when another client application with different number of ranks connects to it

* Committing clang-format changes

* update VPIC output timing precision (#88)

* update VPIC output timing precision

* update timing to make consistent

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* add FindMERCURY.cmake

* LLSM Importer update: new job script + new data type update on kvtags (#92)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* fix warnings, commenting off 'find_path' and 'find_library' for Mercury in src/commons/CMakeLists.txt  (#93)

* Revert "update output"

This reverts commit fe1f8b4.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

* update script

* fix some warnings

* fix some warnings

* update

* update

* fix warning

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* update

* update

* update

* server address and file paths using 1024, TMP_DIR path using 1024/2, NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4

* update

* update

* update

* update

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* Update clang-format-fix.yml

* Update clang-format-fix.yml

* Increase the default server cache size to 32GB and flush frequency to 30s

* Committing clang-format changes

* update commons/CMakeLists.txt

* Fix unnecessary memory allocation (#103)

* Fix an issue with opening a deleted container, added test (#101)

* Fix an issue with opening a deleted container, added test

* Refactor the query aggregation process

* Fix container tag delete error (#102)

* Fix container tag delete error

* Committing clang-format changes

* Update tag delete function

* Refactor metdata lookup process for tag deletion

* Committing clang-format changes

* Formatting and comment

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix the server cache issue when cache becomes full and needs flush (#113)

* Fix the server cache issue when cache becomes full and need flush

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix a wrong option description. (#115)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update option message

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* Install header files needed by PDCpy (#114)

* Support unlimited object dimension size (#117)

* Support unlimited object dimension szie

* Add function description for PDC_SIZE_UNLIMITED

* Fix obj_round_robin_io_all test code

* Committing clang-format changes

* More header files fix for PDCpy

* Fix cmake path (#121)

* Fix cmake path

* Fix cmake path

* Kvtag query (#122)

* Add a new collective kvtag query api to return full/aggregated results to all clients

* Committing clang-format changes

* Add test code

* Committing clang-format changes

* Add an optimization when multiple clients issue different queries

* Add test program

* Fix free issue

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* fix conflict issue

* fix conflict issue

---------

Signed-off-by: Chen Wang <wangvsa@gmail.com>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>
Co-authored-by: Wei Zhang <zhangwei217245@gmail.com>
Co-authored-by: Chen Wang <wangvsa@gmail.com>
Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Wei Zhang <zhangwei217245@lbl.gov>
jeanbez added a commit that referenced this pull request Jan 23, 2024
* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* Merge tools/llsm_importer (#81)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* Adding tutorial for llsm_importer tool. (#84)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* make sure the line feed is included for string attribute

* update timing for overall completion time

* Update .gitlab-ci.yml removing Cori

Remove Cori CI in advance of Cori's decommission by the end of the month.

* Remove unnecessary fflush call

Signed-off-by: Chen Wang <wangvsa@gmail.com>

* update formatting

* LLSM_importer Tutorial and Timing for job completion time. (#86)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* Fix Issue #85, server segfault when another client application with different number of ranks connects to it

* Committing clang-format changes

* update metrics

* Update .gitlab-ci.yml

* update VPIC output timing precision (#88)

* update VPIC output timing precision

* update timing to make consistent

* llsm_importer (#1)

formatter on llsm_importer

* Tiff Parallel Reader sync to latest version (#89)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* llsm_importer (#1)

formatter on llsm_importer

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* Update .gitlab-ci.yml

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* Data type for new kvtag (including refactoring and serde framework) (#90)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* LLSM Importer update: new job script + new data type update on kvtags (#92)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* Update .gitlab-ci.yml

* fix warnings, commenting off 'find_path' and 'find_library' for Mercury in src/commons/CMakeLists.txt  (#93)

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

* update script

* fix some warnings

* fix some warnings

* update

* update

* fix warning

* update

* update

* update

* update

* update

* update

* update

* update

* update

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* fix warnings

* update

* update

* update

* server address and file paths using 1024, TMP_DIR path using 1024/2, NAME_MAX for appname and objname takes 1024/2, HOSTNAME takes 1024/8, NA_INFO_STRING takes 1024/4

* update

* update

* update

* update

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* Update clang-format-fix.yml

* Update clang-format-fix.yml

* Increase the default server cache size to 32GB and flush frequency to 30s

* Committing clang-format changes

* add FindMERCURY.cmake

* update commons/CMakeLists.txt

* Fix unnecessary memory allocation (#103)

* Fix an issue with opening a deleted container, added test (#101)

* Fix an issue with opening a deleted container, added test

* Refactor the query aggregation process

* Fix container tag delete error (#102)

* Fix container tag delete error

* Committing clang-format changes

* Update tag delete function

* Refactor metdata lookup process for tag deletion

* Committing clang-format changes

* Formatting and comment

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix the server cache issue when cache becomes full and needs flush (#113)

* Fix the server cache issue when cache becomes full and need flush

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix a wrong option description. (#115)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update option message

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>

* Install header files needed by PDCpy (#114)

* Support unlimited object dimension size (#117)

* Support unlimited object dimension szie

* Add function description for PDC_SIZE_UNLIMITED

* Fix obj_round_robin_io_all test code

* Committing clang-format changes

* More header files fix for PDCpy

* Fix cmake path (#121)

* Fix cmake path

* Fix cmake path

* Kvtag query (#122)

* Add a new collective kvtag query api to return full/aggregated results to all clients

* Committing clang-format changes

* Add test code

* Committing clang-format changes

* Add an optimization when multiple clients issue different queries

* Add test program

* Fix free issue

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Rebase develop to stable (#125)

* updates in documentation

* update docs

* trigger update

* trigger update

* remove extension

* include docs build dependencies

* update file

* update metrics

* Update .gitlab-ci.yml

* Update .gitlab-ci.yml

* update logo

* Update .gitlab-ci.yml

* adding recovered documentation

* update text

* fix typo

* update CSS

* increase content width

* remove build docs

* Refer documentation to readthedocs website (#95)

* Update README.md

* Update getting_started.rst

* Update getting_started.rst

* include updated text

* fix links

* fix typo

* Update README.md

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* update code

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* clang format

* clang format

* clang-format-10

* change file name

* update llsm importer

* update llsm importer

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update tag names

* update tag names

* update query startingpos

* update job scripts

* fix iteration count in final report

* update job scripts and benckmark program

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* Remove unnecessary fflush call

Signed-off-by: Chen Wang <wangvsa@gmail.com>

* Fix Issue #85, server segfault when another client application with different number of ranks connects to it

* Committing clang-format changes

* update VPIC output timing precision (#88)

* update VPIC output timing precision

* update timing to make consistent

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* add FindMERCURY.cmake

* LLSM Importer update: new job script + new data type update on kvtags (#92)

* remove unnecessary install block from CMakeLists.txt

* update output

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribute value

* change the max length of attribute value

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* llsm tiff import test

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update code

* update cmake and llsm_importer

* update cmake and llsm_importer

* close if in cmake

* cmake fix tiff

* cmake policy to suppress warning

* add pdc include dir

* update code

* update code

* update code

* update code

* update code

* update code

* update array generating method

* update array generating method

* update array generating method

* update array generating method

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* update CMakeLists

* fix return type

* fix return type

* add timing

* add timing

* fix output

* llsm tiff importer 1st version: read csv and import tiff files to PDC, adding metadata available in CSV files and TIFF loader

* fix vairable name

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* fix cmake

* add scripts

* add scripts

* add scripts

* debugging for nonMPI program

* debugging for nonMPI program

* debugging for nonMPI program

* clang format, without PDC, everything works perfectly. program fails at PDC init stage where PDCprop_create(PDC_CONT_CREATE, pdc) is being created

* enable MPI

* enable MPI

* enlarge BCase size

* enlarge BCase size

* enlarge BCase size

* resolve bcast count

* llsm data path in script

* llsm data path in script

* update csv reader

* update csv reader

* update csv reader

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc

* enlarge max write

* update pdc

* update pdc

* update pdc

* update pdc

* update pdc_import.c

* update pdc_import.c

* update pdc_export.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update pdc_import.c

* update tools/cmake

* clang format

* clang format

* added a tutorial for llsm_importer

* added a tutorial for llsm_importer

* make sure the line feed is included for string attribute

* update timing for overall completion time

* update formatting

* update metrics

* forcibly enable openmp

* adding C flags from the mex compiler

* Update .gitlab-ci.yml

* updated code

* clang format

* llsm_importer (#1)

formatter on llsm_importer

* add type for kvtag structure (#2)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* Feature/metadata type (#3)

* upate metadata type system

* update serde framework to coupe with the new data type system

* replace unnecessary data types

* adding type for pdc_kvtag_t, all occurances are fixed

* update new commons CMake for publishing commons

* commons compilation passed

* compiled

* remove unnecessary header files from installation

* resolve conflict

* add important files

* clang formatting

* update cmake

* update

* print numWorkers

* update scripts

* update script

* update script

* formatting

* update llsm_tools.c

* remove unnecessary hash table init

---------

Co-authored-by: Houjun Tang <htang4@lbl.gov>
Co-authored-by: Jean Luca Bez <jlbez@lbl.gov>

* fix warnings, commenting off 'find_path' and 'find_library' for Mercury in src/commons/CMakeLists.txt  (#93)

* Revert "update output"

This reverts commit fe1f8b44995bc0dabd3b957e1032c2da26f56fdd.

* build kvtag_add_get_scale

* comment off free

* update code

* 1. kvtag_scale_add_get added \n 2. uint64_t support for obj/tag/query count \n 3. moving work assigning block downwards right before creating objects \n 4. everything is tested working

* do while loop added, tested with 1m object and works

* 1m objects test works, 10m object test fail as the original also fails

* add new executable to test set

* enlarge PDC_SERVER_ID_INTERVAL

* update code

* update console args

* add p search test

* add console arg for changing number of attributes per object

* free allocated memory

* fix query count issue

* fix attr length definition

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* code refactored

* fix data type

* fix data type

* fix data type

* add client side statistics

* add client side statistics

* fix format

* clang formatter

* update CMake

* update CMake

* update CMake

* free allocated memory properly

* clang format

* clang format

* clang-format-10

* change file name

* address review comments

* update llsm importer

* update llsm importer

* update server checkpoint intervals

* update gitignore

* adding job scripts

* adding one debugging msg

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update container creation to collective mode for debugging purpose

* update output for uint64_t

* add scripts

* update output for uint64_t

* update output for uint64_t

* update output for uint64_t

* update scripts

* update scripts

* delete debugging message

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* make Cmake to publish scripts directory

* update tag names

* update tag names

* update query startingpos

* update query startingpos

* update job scripts

* add progressive timing for kvtag_add_get_scale

* fix iteration count in final report

* update job scripts and benckmark program

* update message format

* update message format

* update message format

* update message format

* clang format

* update job scripts

* comment off object/container close procedure in  benchmark to save node hours

* change the max number of object to 1M

* change the max length of attribut…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants