-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rgw/s3select: json output format #548
base: master
Are you sure you want to change the base?
Conversation
9d6af2f
to
2bb3332
Compare
|
||
@pytest.mark.s3select | ||
def test_json_column_sum_min_max(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why did you make this conversion (remove JSON)?
26c4133
to
c9f0e0a
Compare
s3select_assert_result( res_s3select_like, res_s3select ) | ||
|
||
@pytest.mark.s3select | ||
def test_json_like_expressions(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@albin
test_json_like_expressions
is duplicated name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Updated the name.
c9f0e0a
to
9d10291
Compare
the following functions they have much in common, it is possible to combine them into one function. |
list_int = create_list_of_int( 1 , csv_obj ) | ||
print(list_int) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove the debug printout
res_target = min( list_int ) | ||
print("target") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the same
@albin-antony, please review comments in the case of Parquet, it needs to make sure that RGW is correctly built with arrow-library In the JSON output format, it needs to ensure that |
9d10291
to
81bd7af
Compare
@galsalomon66 Thanks for the review. Addressed the comments. Please take a look |
@@ -235,30 +239,45 @@ def create_random_json_object(rows,columns,col_delim=",",record_delim="\n",csv_s | |||
|
|||
return result | |||
|
|||
def csv_to_json(obj, field_split=",",row_split="\n",csv_schema=""): | |||
def create_parquet_object(parquet_size): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@albin-antony
please add other types such as string and float.
output_serialization = {"CSV": {}} | ||
if input == "JSON": | ||
input_serialization = {"JSON": {"Type": "DOCUMENT"}} | ||
output_serialization = {"JSON": {}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
run_s3select
should support output=JSON
or CSV
elif(input == "PARQUET"): | ||
input_serialization = {'Parquet': {}} | ||
output_serialization = {"JSON": {}} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the same
run_s3select
should support output=JSON
or CSV
res = remove_xml_tags_from_result( run_s3select(bucket_name,csv_obj_name,"select count(0) from s3object;") ).replace(",","") | ||
|
||
res = remove_xml_tags_from_result( run_s3select(bucket_name,csv_obj_name,"select count(0) from s3object;","CSV") ).replace(",","") | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the "CSV" parameter is default.
no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, default is CSV. Don't need to override the same. Will remove it.
81bd7af
to
62a75e3
Compare
@albin-antony |
Signed-off-by: Albin Antony <albinantony20@gmail.com>
62a75e3
to
2e53973
Compare
No description provided.