feat(c++,spark): support json payload file format #518

amygbAI · 2024-06-11T03:44:30Z

Reason for this PR

This is a rebased pull request, so to speak of ( due to the issue with JSONOptions being private in spark 3.2 and 3.3 ) ..also added test artifacts separately in incubator-graphar-testing. Was able to compile with spark 3.2 and 3.3 ( in both datasources-32 and datasources-33 )

What changes are included in this PR?

Changes in datasources-32 and datasources-33
Changes in cpp
Changes in pyspark

Are these changes tested?

yes, added the generated ldbc samples in incubator-graphar-testing/ldbc_sample/json

Are there any user-facing changes?

not that i am aware of

None

SemyonSinchenko · 2024-06-11T18:08:49Z

@amygbAI I see that all the tests are passed except Neo4j example and the error is about missing test data:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: file:/datadrive/APACHE_GRAPHAR/incubator-graphar/testing/ldbc_sample/person_0_0.csv

I mean, that code is compiled and there is no problem you mentioned in #488

amygbAI · 2024-06-12T00:56:22Z

thanks at @SemyonSinchenko .. pulled and removed the local reference .. is there some way to run all of these builds before opening a PR ? as per directions on the thread and README's i built all the directories where i had made changes and tested ..

cpp/src/filesystem.cc

cpp/test/test_arrow_chunk_reader.cc

maven-projects/spark/datasources-32/src/main/scala/org/apache/spark/sql/graphar/GarTable.scala

maven-projects/target/.plxarc

maven-projects/target/maven-shared-archive-resources/META-INF/DEPENDENCIES

maven-projects/target/maven-shared-archive-resources/META-INF/LICENSE

maven-projects/target/maven-shared-archive-resources/META-INF/NOTICE

acezen · 2024-06-12T06:15:51Z

thanks at @SemyonSinchenko .. pulled and removed the local reference .. is there some way to run all of these builds before opening a PR ? as per directions on the thread and README's i built all the directories where i had made changes and tested ..

Hi, @amygbAI, The C++ library you can reference to the https://github.com/apache/incubator-graphar/tree/main/cpp#building to check the test, and for format error, remember that we use the clang-format-8 for format check.

amygbAI · 2024-06-12T08:54:45Z

resolved all comments, formatted, linted and built using instructions .. also removed the "build" folder from cpp since it wasn't in the repository ( much like target folders )

SemyonSinchenko · 2024-06-12T09:21:11Z

May we add also a test for graphar spark+json?

amygbAI · 2024-06-12T09:46:42Z

please send me an example for spark + csv and i ll refactor it

…

On Wed, Jun 12, 2024 at 2:51 PM Semyon ***@***.***> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-12T11:09:19Z

please send me an example for spark + csv and i ll refactor it
…
On Wed, Jun 12, 2024 at 2:51 PM Semyon @.> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU . You are receiving this because you were mentioned.Message ID: @.>

You can refer to the existed test case an just add a test with updated json test data like:

Spark

incubator-graphar/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala

Lines 99 to 117 in ff057cf

    
           test("read vertex chunks") { 
        
             // construct the vertex information 
        
             val prefix = testData + "/ldbc_sample/parquet/" 
        
             val vertex_yaml = prefix + "person.vertex.yml" 
        
             val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) 
        
             // construct the vertex reader 
        
             val reader = new VertexReader(prefix, vertex_info, spark) 
        
             // test reading the number of vertices 
        
             assert(reader.readVerticesNumber() == 903) 
        
             val property_group = vertex_info.getPropertyGroup("gender") 
        
             // test reading a single property chunk 
        
             val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) 
        
             assert(single_chunk_df.columns.length == 4) 
        
             assert(single_chunk_df.count() == 100) 
        
             val cond = "gender = 'female'" 
        
             var df_pd = single_chunk_df.select("firstName", "gender").filter(cond)

json test can be:

test("read vertex chunks") {
// construct the vertex information
    val prefix = testData + "/ldbc_sample/json/"
    val vertex_yaml = prefix + "Person.vertex.yml"
    val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark)

    // construct the vertex reader
    val reader = new VertexReader(prefix, vertex_info, spark)

    // test reading the number of vertices
    assert(reader.readVerticesNumber() == 903)
    val property_group = vertex_info.getPropertyGroup("gender")

    // test reading a single property chunk
    val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)
    assert(single_chunk_df.columns.length == 4)
    assert(single_chunk_df.count() == 100)
    val cond = "gender = 'female'"
    var df_pd = single_chunk_df.select("firstName", "gender").filter(cond)

pyspark

incubator-graphar/pyspark/tests/test_reader.py

Lines 29 to 67 in ff057cf

    
           def test_vertex_reader(spark): 
        
               initialize(spark) 
        
               vertex_info = VertexInfo.load_vertex_info( 
        
                   GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph") 
        
                   .joinpath("person.vertex.yml") 
        
                   .absolute() 
        
                   .__str__() 
        
               ) 
        
               vertex_reader = VertexReader.from_python( 
        
                   GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph").absolute().__str__(), 
        
                   vertex_info, 
        
               ) 
        
               assert VertexReader.from_scala(vertex_reader.to_scala()) is not None 
        
               assert vertex_reader.read_vertices_number() > 0 
        
               assert ( 
        
                   vertex_reader.read_vertex_property_group( 
        
                       vertex_info.get_property_group("name") 
        
                   ).count() 
        
                   > 0 
        
               ) 
        
               assert ( 
        
                   vertex_reader.read_vertex_property_chunk( 
        
                       vertex_info.get_property_groups()[0], 0 
        
                   ).count() 
        
                   > 0 
        
               ) 
        
               assert ( 
        
                   vertex_reader.read_all_vertex_property_groups().count() 
        
                   >= vertex_reader.read_vertex_property_group( 
        
                       vertex_info.get_property_group("age") 
        
                   ).count() 
        
               ) 
        
               assert ( 
        
                   vertex_reader.read_multiple_vertex_property_groups( 
        
                       [vertex_info.get_property_group("name")] 
        
                   ).count() 
        
                   > 0 
        
               )

json test can be:

def test_vertex_reader_with_json(spark):
    initialize(spark)

    vertex_info = VertexInfo.load_vertex_info(
        GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/")
        .joinpath("Person.vertex.yml")
        .absolute()
        .__str__()
    )
    vertex_reader = VertexReader.from_python(
        GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().__str__(),
        vertex_info,
    )
    assert VertexReader.from_scala(vertex_reader.to_scala()) is not None
    assert vertex_reader.read_vertices_number() > 0
    assert (
        vertex_reader.read_vertex_property_group(
            vertex_info.get_property_group("name")
        ).count()
        > 0
    )
    assert (
        vertex_reader.read_vertex_property_chunk(
            vertex_info.get_property_groups()[0], 0
        ).count()
        > 0
    )
    assert (
        vertex_reader.read_all_vertex_property_groups().count()
        >= vertex_reader.read_vertex_property_group(
            vertex_info.get_property_group("age")
        ).count()
    )
    assert (
        vertex_reader.read_multiple_vertex_property_groups(
            [vertex_info.get_property_group("name")]
        ).count()
        > 0
    )

amygbAI · 2024-06-12T15:51:35Z

alritey folks .. done On Wed, Jun 12, 2024 at 4:39 PM Weibin Zeng ***@***.***> wrote:

…

please send me an example for spark + csv and i ll refactor it … <#m_-7895995846438427138_> On Wed, Jun 12, 2024 at 2:51 PM Semyon *@*.*> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 (comment) <#518 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> . You are receiving this because you were mentioned.Message ID: @.*> You can refer to the existed test case an just add a test with updated json test data like: - Spark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala#L99-L117 json test can be: test("read vertex chunks") {// construct the vertex information val prefix = testData + "/ldbc_sample/json/" val vertex_yaml = prefix + "Person.vertex.yml" val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) // construct the vertex reader val reader = new VertexReader(prefix, vertex_info, spark) // test reading the number of vertices assert(reader.readVerticesNumber() == 903) val property_group = vertex_info.getPropertyGroup("gender") // test reading a single property chunk val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) assert(single_chunk_df.columns.length == 4) assert(single_chunk_df.count() == 100) val cond = "gender = 'female'" var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) - pyspark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/pyspark/tests/test_reader.py#L29-L67 json test can be: def test_vertex_reader_with_json(spark): initialize(spark) vertex_info = VertexInfo.load_vertex_info( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") .joinpath("Person.vertex.yml") .absolute() .__str__() ) vertex_reader = VertexReader.from_python( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().__str__(), vertex_info, ) assert VertexReader.from_scala(vertex_reader.to_scala()) is not None assert vertex_reader.read_vertices_number() > 0 assert ( vertex_reader.read_vertex_property_group( vertex_info.get_property_group("name") ).count() > 0 ) assert ( vertex_reader.read_vertex_property_chunk( vertex_info.get_property_groups()[0], 0 ).count() > 0 ) assert ( vertex_reader.read_all_vertex_property_groups().count() >= vertex_reader.read_vertex_property_group( vertex_info.get_property_group("age") ).count() ) assert ( vertex_reader.read_multiple_vertex_property_groups( [vertex_info.get_property_group("name")] ).count() > 0 ) — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-17T03:13:01Z

alritey folks .. done On Wed, Jun 12, 2024 at 4:39 PM Weibin Zeng @.> wrote:
…
please send me an example for spark + csv and i ll refactor it … <#m_-7895995846438427138_> On Wed, Jun 12, 2024 at 2:51 PM Semyon @.> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 (comment) <#518 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU . You are receiving this because you were mentioned.Message ID: @.> You can refer to the existed test case an just add a test with updated json test data like: - Spark

incubator-graphar/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala

Lines 99 to 117 in ff057cf

test("read vertex chunks") {

// construct the vertex information

val prefix = testData + "/ldbc_sample/parquet/"

val vertex_yaml = prefix + "person.vertex.yml"

val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark)

// construct the vertex reader

val reader = new VertexReader(prefix, vertex_info, spark)

// test reading the number of vertices

assert(reader.readVerticesNumber() == 903)

val property_group = vertex_info.getPropertyGroup("gender")

// test reading a single property chunk

val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)

assert(single_chunk_df.columns.length == 4)

assert(single_chunk_df.count() == 100)

val cond = "gender = 'female'"

var df_pd = single_chunk_df.select("firstName", "gender").filter(cond)

json test can be: test("read vertex chunks") {// construct the vertex information val prefix = testData + "/ldbc_sample/json/" val vertex_yaml = prefix + "Person.vertex.yml" val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) // construct the vertex reader val reader = new VertexReader(prefix, vertex_info, spark) // test reading the number of vertices assert(reader.readVerticesNumber() == 903) val property_group = vertex_info.getPropertyGroup("gender") // test reading a single property chunk val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) assert(single_chunk_df.columns.length == 4) assert(single_chunk_df.count() == 100) val cond = "gender = 'female'" var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) - pyspark

incubator-graphar/pyspark/tests/test_reader.py

Lines 29 to 67 in ff057cf

def test_vertex_reader(spark):

initialize(spark)

vertex_info = VertexInfo.load_vertex_info(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph")

.joinpath("person.vertex.yml")

.absolute()

.__str__()

)

vertex_reader = VertexReader.from_python(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph").absolute().__str__(),

vertex_info,

)

assert VertexReader.from_scala(vertex_reader.to_scala()) is not None

assert vertex_reader.read_vertices_number() > 0

assert (

vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("name")

).count()

> 0

)

assert (

vertex_reader.read_vertex_property_chunk(

vertex_info.get_property_groups()[0], 0

).count()

> 0

)

assert (

vertex_reader.read_all_vertex_property_groups().count()

>= vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("age")

).count()

)

assert (

vertex_reader.read_multiple_vertex_property_groups(

[vertex_info.get_property_group("name")]

).count()

> 0

)

json test can be: def test_vertex_reader_with_json(spark): initialize(spark) vertex_info = VertexInfo.load_vertex_info( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") .joinpath("Person.vertex.yml") .absolute() .str() ) vertex_reader = VertexReader.from_python( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().str(), vertex_info, ) assert VertexReader.from_scala(vertex_reader.to_scala()) is not None assert vertex_reader.read_vertices_number() > 0 assert ( vertex_reader.read_vertex_property_group( vertex_info.get_property_group("name") ).count() > 0 ) assert ( vertex_reader.read_vertex_property_chunk( vertex_info.get_property_groups()[0], 0 ).count() > 0 ) assert ( vertex_reader.read_all_vertex_property_groups().count() >= vertex_reader.read_vertex_property_group( vertex_info.get_property_group("age") ).count() ) assert ( vertex_reader.read_multiple_vertex_property_groups( [vertex_info.get_property_group("name")] ).count() > 0 ) — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU . You are receiving this because you were mentioned.Message ID: @.**>

Hi, @amygbAI Can you add me to your graphar fork repo's collaborator? I can help you fix the format and test.

amygbAI · 2024-06-17T04:36:18Z

done ..kindly let me know what changes u make .. i am compiling a document of all the communication we have had so that it becomes a good starting point for noobs like me On Mon, Jun 17, 2024 at 8:43 AM Weibin Zeng ***@***.***> wrote:

…

alritey folks .. done On Wed, Jun 12, 2024 at 4:39 PM Weibin Zeng *@*. *> wrote: … <#m_-8019765962861073975_> please send me an example for spark + csv and i ll refactor it … <#m_-7895995846438427138_> On Wed, Jun 12, 2024 at 2:51 PM Semyon @.> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 <#518> (comment) <#518 (comment) <#518 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> . You are receiving this because you were mentioned.Message ID: @.> You can refer to the existed test case an just add a test with updated json test data like: - Spark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala#L99-L117 <https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala#L99-L117> json test can be: test("read vertex chunks") {// construct the vertex information val prefix = testData + "/ldbc_sample/json/" val vertex_yaml = prefix + "Person.vertex.yml" val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) // construct the vertex reader val reader = new VertexReader(prefix, vertex_info, spark) // test reading the number of vertices assert(reader.readVerticesNumber() == 903) val property_group = vertex_info.getPropertyGroup("gender") // test reading a single property chunk val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) assert(single_chunk_df.columns.length == 4) assert(single_chunk_df.count() == 100) val cond = "gender = 'female'" var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) - pyspark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/pyspark/tests/test_reader.py#L29-L67 <https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/pyspark/tests/test_reader.py#L29-L67> json test can be: def test_vertex_reader_with_json(spark): initialize(spark) vertex_info = VertexInfo.load_vertex_info( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") .joinpath("Person.vertex.yml") .absolute() .str() ) vertex_reader = VertexReader.from_python( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().str(), vertex_info, ) assert VertexReader.from_scala(vertex_reader.to_scala()) is not None assert vertex_reader.read_vertices_number() > 0 assert ( vertex_reader.read_vertex_property_group( vertex_info.get_property_group("name") ).count() > 0 ) assert ( vertex_reader.read_vertex_property_chunk( vertex_info.get_property_groups()[0], 0 ).count() > 0 ) assert ( vertex_reader.read_all_vertex_property_groups().count() >= vertex_reader.read_vertex_property_group( vertex_info.get_property_group("age") ).count() ) assert ( vertex_reader.read_multiple_vertex_property_groups( [vertex_info.get_property_group("name")] ).count() > 0 ) — Reply to this email directly, view it on GitHub <#518 (comment) <#518 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU <https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU> . You are receiving this because you were mentioned.Message ID: @.***> Hi, @amygbAI <https://github.com/amygbAI> Can you add me to your graphar folk repo's collaborator? I can help you fix the format and test. — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSC245NKCVPWVESRACTZHZH5JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGA4TIMRZGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-17T07:51:11Z

done ..kindly let me know what changes u make .. i am compiling a document of all the communication we have had so that it becomes a good starting point for noobs like me On Mon, Jun 17, 2024 at 8:43 AM Weibin Zeng @.> wrote:
…
alritey folks .. done On Wed, Jun 12, 2024 at 4:39 PM Weibin Zeng @. > wrote: … <#m_-8019765962861073975_> please send me an example for spark + csv and i ll refactor it … <#m_-7895995846438427138_> On Wed, Jun 12, 2024 at 2:51 PM Semyon @.> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <#518 <#518> (comment) <#518 (comment) <#518 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU . You are receiving this because you were mentioned.Message ID: @.> You can refer to the existed test case an just add a test with updated json test data like: - Spark

incubator-graphar/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala

Lines 99 to 117 in ff057cf

test("read vertex chunks") {

// construct the vertex information

val prefix = testData + "/ldbc_sample/parquet/"

val vertex_yaml = prefix + "person.vertex.yml"

val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark)

// construct the vertex reader

val reader = new VertexReader(prefix, vertex_info, spark)

// test reading the number of vertices

assert(reader.readVerticesNumber() == 903)

val property_group = vertex_info.getPropertyGroup("gender")

// test reading a single property chunk

val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)

assert(single_chunk_df.columns.length == 4)

assert(single_chunk_df.count() == 100)

val cond = "gender = 'female'"

var df_pd = single_chunk_df.select("firstName", "gender").filter(cond)

incubator-graphar/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala

Lines 99 to 117 in ff057cf

test("read vertex chunks") {

// construct the vertex information

val prefix = testData + "/ldbc_sample/parquet/"

val vertex_yaml = prefix + "person.vertex.yml"

val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark)

// construct the vertex reader

val reader = new VertexReader(prefix, vertex_info, spark)

// test reading the number of vertices

assert(reader.readVerticesNumber() == 903)

val property_group = vertex_info.getPropertyGroup("gender")

// test reading a single property chunk

val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0)

assert(single_chunk_df.columns.length == 4)

assert(single_chunk_df.count() == 100)

val cond = "gender = 'female'"

var df_pd = single_chunk_df.select("firstName", "gender").filter(cond)

json test can be: test("read vertex chunks") {// construct the vertex information val prefix = testData + "/ldbc_sample/json/" val vertex_yaml = prefix + "Person.vertex.yml" val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) // construct the vertex reader val reader = new VertexReader(prefix, vertex_info, spark) // test reading the number of vertices assert(reader.readVerticesNumber() == 903) val property_group = vertex_info.getPropertyGroup("gender") // test reading a single property chunk val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) assert(single_chunk_df.columns.length == 4) assert(single_chunk_df.count() == 100) val cond = "gender = 'female'" var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) - pyspark

incubator-graphar/pyspark/tests/test_reader.py

Lines 29 to 67 in ff057cf

def test_vertex_reader(spark):

initialize(spark)

vertex_info = VertexInfo.load_vertex_info(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph")

.joinpath("person.vertex.yml")

.absolute()

.__str__()

)

vertex_reader = VertexReader.from_python(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph").absolute().__str__(),

vertex_info,

)

assert VertexReader.from_scala(vertex_reader.to_scala()) is not None

assert vertex_reader.read_vertices_number() > 0

assert (

vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("name")

).count()

> 0

)

assert (

vertex_reader.read_vertex_property_chunk(

vertex_info.get_property_groups()[0], 0

).count()

> 0

)

assert (

vertex_reader.read_all_vertex_property_groups().count()

>= vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("age")

).count()

)

assert (

vertex_reader.read_multiple_vertex_property_groups(

[vertex_info.get_property_group("name")]

).count()

> 0

)

incubator-graphar/pyspark/tests/test_reader.py

Lines 29 to 67 in ff057cf

def test_vertex_reader(spark):

initialize(spark)

vertex_info = VertexInfo.load_vertex_info(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph")

.joinpath("person.vertex.yml")

.absolute()

.__str__()

)

vertex_reader = VertexReader.from_python(

GRAPHAR_TESTS_EXAMPLES.joinpath("modern_graph").absolute().__str__(),

vertex_info,

)

assert VertexReader.from_scala(vertex_reader.to_scala()) is not None

assert vertex_reader.read_vertices_number() > 0

assert (

vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("name")

).count()

> 0

)

assert (

vertex_reader.read_vertex_property_chunk(

vertex_info.get_property_groups()[0], 0

).count()

> 0

)

assert (

vertex_reader.read_all_vertex_property_groups().count()

>= vertex_reader.read_vertex_property_group(

vertex_info.get_property_group("age")

).count()

)

assert (

vertex_reader.read_multiple_vertex_property_groups(

[vertex_info.get_property_group("name")]

).count()

> 0

)

json test can be: def test_vertex_reader_with_json(spark): initialize(spark) vertex_info = VertexInfo.load_vertex_info( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") .joinpath("Person.vertex.yml") .absolute() .str() ) vertex_reader = VertexReader.from_python( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().str(), vertex_info, ) assert VertexReader.from_scala(vertex_reader.to_scala()) is not None assert vertex_reader.read_vertices_number() > 0 assert ( vertex_reader.read_vertex_property_group( vertex_info.get_property_group("name") ).count() > 0 ) assert ( vertex_reader.read_vertex_property_chunk( vertex_info.get_property_groups()[0], 0 ).count() > 0 ) assert ( vertex_reader.read_all_vertex_property_groups().count() >= vertex_reader.read_vertex_property_group( vertex_info.get_property_group("age") ).count() ) assert ( vertex_reader.read_multiple_vertex_property_groups( [vertex_info.get_property_group("name")] ).count() > 0 ) — Reply to this email directly, view it on GitHub <#518 (comment) <#518 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU . You are receiving this because you were mentioned.Message ID: @.> Hi, @amygbAI https://github.com/amygbAI Can you add me to your graphar folk repo's collaborator? I can help you fix the format and test. — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSC245NKCVPWVESRACTZHZH5JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGA4TIMRZGY . You are receiving this because you were mentioned.Message ID: @.**>

the GraphAr C++ CI is failed in format check. Can you double check the clang-format is version 8.x.x with command

clang-format --version

acezen · 2024-06-17T07:52:35Z

cpp/test/tmp

@@ -0,0 +1,465 @@
+/*


what's the file for? It seems this is a temporary file?

acezen · 2024-06-17T07:54:04Z

cpp/test/test_arrow_chunk_reader.cc

+      std::string vertex_info_path =
+          test_data_dir + "ldbc_sample/json/Person.vertex.yml";
+      std::cout << "Vertex info path: " << vertex_info_path << std::endl;
+      auto fs = FileSystemFromUriOrPath(prefix).value();


You are not define the prefix variable, see the CI report

acezen · 2024-06-17T07:54:31Z

cpp/test/test_arrow_chunk_reader.cc

+      auto new_pg = CreatePropertyGroup({new_property}, pg->GetFileType(),
+                                        pg->GetPrefix());
+      auto maybe_reader =
+          VertexPropertyArrowChunkReader::Make(vertex_info, new_pg, prefix);


Ditto, see report

acezen · 2024-06-17T08:00:28Z

BTW, You can watch the Details, there are the reports for the CI/CD, you can check the report and find the reason why the CI failed.

amygbAI · 2024-06-17T12:54:58Z

ok, pushed .. though i am not sure why this would fail for certain OS and not the others ..also can this test directory also please be included in the CMakeLists.txt

…

On Mon, Jun 17, 2024 at 1:30 PM Weibin Zeng ***@***.***> wrote: BTW, You can watch the Details, there are the reports for the CI/CD, you can check the report and find the reason why the CI failed. 2024-06-17.15.54.59.png (view on web) <https://github.com/apache/incubator-graphar/assets/11835645/388c7151-6a2f-4795-a9e9-204dde95350a> — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSAJ5WQQN4DZCQKRRJLZH2JTHAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZSGU3DKNJQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Signed-off-by: amygbAI <80807752+amygbAI@users.noreply.github.com>

Signed-off-by: acezen <qiaozi.zwb@alibaba-inc.com>

acezen · 2024-06-20T11:18:14Z

Hi, @amygbAI , I have fix the c++ format with clang-format-8, and do some changes:

simplify the test, we do not need to repeat the test of other format again, just test read json chunk is ok
Complement json code of datasource.

...park/datasources-32/src/main/scala/org/apache/spark/sql/graphar/json/JSONWriterBuilder.scala

...park/datasources-33/src/main/scala/org/apache/spark/sql/graphar/json/JSONWriterBuilder.scala

maven-projects/spark/datasources-33/src/main/scala/org/apache/spark/sql/graphar/GarTable.scala

acezen · 2024-06-21T01:47:04Z

Hi， @amygbAI， I think the the PR is overall LGTM, you may fix the remaining problem base on Sem's comments.

amygbAI · 2024-06-21T05:59:48Z

after making the changes when i tried to push git showed me some conflicts and after gpt + google search i followed the below

git stash
git pull origin 170-feat-support-json-payload-file-format
git stash pop
git add .
git commit -m "msg"
git push origin 170-feat-support-json-payload-file-format

for some reason now it shows some conflict in the scripts file AND also in the cpp file filesystem.cc .. totally confused

acezen · 2024-06-21T06:08:22Z

after making the changes when i tried to push git showed me some conflicts and after gpt + google search i followed the below

git stash git pull origin 170-feat-support-json-payload-file-format git stash pop git add . git commit -m "msg" git push origin 170-feat-support-json-payload-file-format

for some reason now it shows some conflict in the scripts file AND also in the cpp file filesystem.cc .. totally confused

That's because your branch and main modified the same file. You need to rebase the main first and fix the conflict:
refer: https://www.atlassian.com/git/tutorials/merging-vs-rebasing

acezen · 2024-06-21T07:37:50Z

after making the changes when i tried to push git showed me some conflicts and after gpt + google search i followed the below

git stash git pull origin 170-feat-support-json-payload-file-format git stash pop git add . git commit -m "msg" git push origin 170-feat-support-json-payload-file-format

for some reason now it shows some conflict in the scripts file AND also in the cpp file filesystem.cc .. totally confused

Or just revert the maven-projects/spark/scripts/run-ldbc-sample2graphar.sh to csv, we don't need to change this file.

amygbAI · 2024-06-22T10:47:32Z

thanks .. done

…

On Fri, Jun 21, 2024 at 1:08 PM Weibin Zeng ***@***.***> wrote: after making the changes when i tried to push git showed me some conflicts and after gpt + google search i followed the below git stash git pull origin 170-feat-support-json-payload-file-format git stash pop git add . git commit -m "msg" git push origin 170-feat-support-json-payload-file-format for some reason now it shows some conflict in the scripts file AND also in the cpp file filesystem.cc .. totally confused Or just revert the maven-projects/spark/scripts/run-ldbc-sample2graphar.sh to csv, we don't need to change this file. — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSHO7OVVWRTKIFXGY4TZIPJ6JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSGE4DGMJRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-24T02:49:16Z

origin

Hi, @amygbAI, seems that you have reverted the changes that I made ╥﹏╥

amygbAI · 2024-06-24T03:00:18Z

dang .. can u just give me the exact version on which u made the changes, i ll do a hard reset to that version and then add my final changes on it ..

…

On Mon, Jun 24, 2024 at 8:19 AM Weibin Zeng ***@***.***> wrote: origin Hi, @amygbAI <https://github.com/amygbAI>, seems that you have reverted the changes that I made ╥﹏╥ — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSBZPJQX7X6WZFE5OQTZI6CMDAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGQ4DOMZUHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-24T03:04:31Z

dang .. can u just give me the exact version on which u made the changes, i ll do a hard reset to that version and then add my final changes on it ..
…
On Mon, Jun 24, 2024 at 8:19 AM Weibin Zeng @.> wrote: origin Hi, @amygbAI https://github.com/amygbAI, seems that you have reverted the changes that I made ╥﹏╥ — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSBZPJQX7X6WZFE5OQTZI6CMDAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGQ4DOMZUHE . You are receiving this because you were mentioned.Message ID: @.>

It's OK now. I have reset to the commit and apply your final change to it. And it should be OK now.

acezen

LGTM, thanks for the work!
Hi, @SemyonSinchenko, this change is OK to me, could you give a review again?

amygbAI · 2024-06-24T03:46:28Z

appreciate all your patience .. i think i made it harder than it actually should have been

…

On Mon, Jun 24, 2024 at 8:42 AM Weibin Zeng ***@***.***> wrote: ***@***.**** approved this pull request. LGTM, thanks for the work! Hi, @SemyonSinchenko <https://github.com/SemyonSinchenko>, this change is OK to me, could you give a review again? — Reply to this email directly, view it on GitHub <#518 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSAAHSH74FYMCWJG7H3ZI6FCVAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMZUGU2DGOJVGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

acezen · 2024-06-24T05:15:55Z

appreciate all your patience .. i think i made it harder than it actually should have been
…
On Mon, Jun 24, 2024 at 8:42 AM Weibin Zeng @.> wrote: @.* approved this pull request. LGTM, thanks for the work! Hi, @SemyonSinchenko https://github.com/SemyonSinchenko, this change is OK to me, could you give a review again? — Reply to this email directly, view it on GitHub <#518 (review)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSAAHSH74FYMCWJG7H3ZI6FCVAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMZUGU2DGOJVGQ . You are receiving this because you were mentioned.Message ID: @.***>

Don't be bothered about that. Since this is your first time to contribute to a open source project, your continue contribution and the final work are excellent. Hope you can join the community and continue to contribute to GraphAr.:)

amygbAI · 2024-06-24T06:10:57Z

appreciate it.. will be on the lookout to help with further dev and bug fixes ! On Mon, Jun 24, 2024 at 10:46 AM Weibin Zeng ***@***.***> wrote:

…

appreciate all your patience .. i think i made it harder than it actually should have been … <#m_-4654679496454462024_> On Mon, Jun 24, 2024 at 8:42 AM Weibin Zeng *@*.*> wrote: @.** approved this pull request. LGTM, thanks for the work! Hi, @SemyonSinchenko <https://github.com/SemyonSinchenko> https://github.com/SemyonSinchenko, this change is OK to me, could you give a review again? — Reply to this email directly, view it on GitHub <#518 (review) <#518 (review)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATIQOSAAHSH74FYMCWJG7H3ZI6FCVAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMZUGU2DGOJVGQ . You are receiving this because you were mentioned.Message ID: *@*.***> Don't be bothered about that. Since this is your first time to contribute to a open source project, your continue contribution and the final work are excellent. Hope you can join the community and continue to contribute to GraphAr.:) — Reply to this email directly, view it on GitHub <#518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSHGEOEWXGDWCZOPKXDZI6TSDAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVGYZDGOBYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

SemyonSinchenko · 2024-06-24T08:58:08Z

I will review it again in the evening.

SemyonSinchenko

LGTM. Thanks for the contribution @amygbAI !

Reason for this PR This is a rebased pull request, so to speak of ( due to the issue with JSONOptions being private in spark 3.2 and 3.3 ) ..also added test artifacts separately in incubator-graphar-testing. Was able to compile with spark 3.2 and 3.3 ( in both datasources-32 and datasources-33 ) What changes are included in this PR? Changes in datasources-32 and datasources-33 Changes in cpp Changes in pyspark Are these changes tested? yes, added the generated ldbc samples in incubator-graphar-testing/ldbc_sample/json Are there any user-facing changes? None --------- Signed-off-by: amygbAI <80807752+amygbAI@users.noreply.github.com> Signed-off-by: acezen <qiaozi.zwb@alibaba-inc.com> Co-authored-by: Ubuntu <bithika@amygb.ai> Co-authored-by: acezen <qiaozi.zwb@alibaba-inc.com>

acezen reviewed Jun 12, 2024

View reviewed changes

acezen reviewed Jun 17, 2024

View reviewed changes

cpp/test/tmp Outdated

@@ -0,0 +1,465 @@

/*

Copy link

Contributor

acezen Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the file for? It seems this is a temporary file?

acezen reviewed Jun 17, 2024

View reviewed changes

lepto2014 and others added 13 commits June 20, 2024 16:40

Added changes for reading in c++, pyspark and READ/WRITE in scala

932c1ed

170 cleared JSONOptions issue

d3d1140

170 changes in ds-33

83e8a04

Update test_arrow_chunk_reader.cc

a901a5a

Signed-off-by: amygbAI <80807752+amygbAI@users.noreply.github.com>

removed local references from run-ldbc-sample2graphar.sh

efe5bb8

resolved PR comments by @acezen

bec290a

170 PR change comments

471a925

170 PR comment changes + clang-format-8

2a85d95

added examples for spark and pyspark testing

bc42773

made changes to test/test_arrow_chunk_reader.cc

9f7b0b7

Fix format with clang-format-8

df24677

Signed-off-by: acezen <qiaozi.zwb@alibaba-inc.com>

Refactor test

636448c

Fix the json reader of spark

16b61bc

Signed-off-by: acezen <qiaozi.zwb@alibaba-inc.com>

acezen force-pushed the 170-feat-support-json-payload-file-format branch from 04518a2 to 16b61bc Compare June 20, 2024 09:57

acezen added 2 commits June 20, 2024 18:05

Update testing submodule

c6ac40c

Signed-off-by: acezen <qiaozi.zwb@alibaba-inc.com>

fix pyspark test

b6c5cfa

acezen changed the title ~~170 feat support json payload file format~~ feat(c++,spark): support json payload file format Jun 20, 2024

SemyonSinchenko reviewed Jun 20, 2024

View reviewed changes

amygbAI pushed a commit to amygbAI/incubator-graphar that referenced this pull request Jun 21, 2024

final fixes for apache#518

04c9026

amygbAI force-pushed the 170-feat-support-json-payload-file-format branch from b6c5cfa to 04c9026 Compare June 21, 2024 05:41

amygbAI pushed a commit to amygbAI/incubator-graphar that referenced this pull request Jun 21, 2024

final push for apache#518

082373c

Fix the file name and class name

ccfe05e

acezen force-pushed the 170-feat-support-json-payload-file-format branch from 86727fd to ccfe05e Compare June 24, 2024 03:03

acezen approved these changes Jun 24, 2024

View reviewed changes

SemyonSinchenko approved these changes Jun 24, 2024

View reviewed changes

acezen merged commit 73e0702 into apache:main Jun 25, 2024
7 checks passed

acezen mentioned this pull request Jun 25, 2024

chore(docs): update the implementation status #529

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(c++,spark): support json payload file format #518

feat(c++,spark): support json payload file format #518

amygbAI commented Jun 11, 2024

SemyonSinchenko commented Jun 11, 2024 •

edited

Loading

amygbAI commented Jun 12, 2024

acezen commented Jun 12, 2024

amygbAI commented Jun 12, 2024

SemyonSinchenko commented Jun 12, 2024

amygbAI commented Jun 12, 2024 via email

acezen commented Jun 12, 2024

amygbAI commented Jun 12, 2024 via email

acezen commented Jun 17, 2024 •

edited

Loading

amygbAI commented Jun 17, 2024 via email

acezen commented Jun 17, 2024

acezen Jun 17, 2024

acezen Jun 17, 2024

acezen Jun 17, 2024

acezen commented Jun 17, 2024

amygbAI commented Jun 17, 2024 via email

acezen commented Jun 20, 2024

acezen commented Jun 21, 2024

amygbAI commented Jun 21, 2024

acezen commented Jun 21, 2024 •

edited

Loading

acezen commented Jun 21, 2024

amygbAI commented Jun 22, 2024 via email

acezen commented Jun 24, 2024

amygbAI commented Jun 24, 2024 via email

acezen commented Jun 24, 2024

acezen left a comment

amygbAI commented Jun 24, 2024 via email

acezen commented Jun 24, 2024

amygbAI commented Jun 24, 2024 via email

SemyonSinchenko commented Jun 24, 2024

SemyonSinchenko left a comment

feat(c++,spark): support json payload file format #518

feat(c++,spark): support json payload file format #518

Conversation

amygbAI commented Jun 11, 2024

Reason for this PR

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

SemyonSinchenko commented Jun 11, 2024 • edited Loading

amygbAI commented Jun 12, 2024

acezen commented Jun 12, 2024

amygbAI commented Jun 12, 2024

SemyonSinchenko commented Jun 12, 2024

amygbAI commented Jun 12, 2024 via email

acezen commented Jun 12, 2024

amygbAI commented Jun 12, 2024 via email

acezen commented Jun 17, 2024 • edited Loading

amygbAI commented Jun 17, 2024 via email

acezen commented Jun 17, 2024

acezen Jun 17, 2024

Choose a reason for hiding this comment

acezen Jun 17, 2024

Choose a reason for hiding this comment

acezen Jun 17, 2024

Choose a reason for hiding this comment

acezen commented Jun 17, 2024

amygbAI commented Jun 17, 2024 via email

acezen commented Jun 20, 2024

acezen commented Jun 21, 2024

amygbAI commented Jun 21, 2024

acezen commented Jun 21, 2024 • edited Loading

acezen commented Jun 21, 2024

amygbAI commented Jun 22, 2024 via email

acezen commented Jun 24, 2024

amygbAI commented Jun 24, 2024 via email

acezen commented Jun 24, 2024

acezen left a comment

Choose a reason for hiding this comment

amygbAI commented Jun 24, 2024 via email

acezen commented Jun 24, 2024

amygbAI commented Jun 24, 2024 via email

SemyonSinchenko commented Jun 24, 2024

SemyonSinchenko left a comment

Choose a reason for hiding this comment

SemyonSinchenko commented Jun 11, 2024 •

edited

Loading

acezen commented Jun 17, 2024 •

edited

Loading

acezen commented Jun 21, 2024 •

edited

Loading