Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set the default charset as UTF-8 for REST calls with application/json MediaType #1437

Closed
lqiu96 opened this issue Mar 3, 2023 · 0 comments · Fixed by #1477 or #1571
Closed

Set the default charset as UTF-8 for REST calls with application/json MediaType #1437

lqiu96 opened this issue Mar 3, 2023 · 0 comments · Fixed by #1477 or #1571
Assignees
@lqiu96
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@lqiu96
Copy link
Contributor

lqiu96 commented Mar 3, 2023 •
edited
Loading

Discovered while implementing the Showcase Compliance Suite:

2023/03/03 17:19:42   request: {
  "name":  "Extreme values",
  "info":  {
    "fString":  "non-ASCII+non-printable string ? ? ? \"\\/\b\f\r\t? works, not newlines yet",
    "fInt32":  2147483647,
    "fSint32":  2147483647,
    "fSfixed32":  2147483647,
    "fUint32":  4294967295,
    "fFixed32":  4294967295,
    "fInt64":  "9223372036854775807",
    "fSint64":  "9223372036854775807",
    "fSfixed64":  "9223372036854775807",
    "fUint64":  "18446744073709551615",
    "fFixed64":  "18446744073709551615",
    "fDouble":  1.7976931348623157e+308,
    "fFloat":  3.4028235e+38,
    "fBool":  false,
    "fBytes":  "",
    "fKingdom":  "LIFE_KINGDOM_UNSPECIFIED",
    "fChild":  null,
    "pString":  "Goodbye",
    "pInt32":  2147483647,
    "pDouble":  1.7976931348623157e+308,
    "pBool":  false
  },
  "serverVerify":  true,
  "fInt32":  0,
  "fInt64":  "0",
  "fDouble":  0
}
Received Info:  f_string:"non-ASCII+non-printable string ? ? ? \"\\/\x08\x0c\r\t? works, not newlines yet"  f_int32:2147483647  f_sint32:2147483647  f_sfixed32:2147483647  f_uint32:4294967295  f_fixed32:4294967295  f_int64:9223372036854775807  f_sint64:9223372036854775807  f_sfixed64:9223372036854775807  f_uint64:18446744073709551615  f_fixed64:18446744073709551615  f_double:1.7976931348623157e+308  f_float:3.4028235e+38  p_string:"Goodbye"  p_int32:2147483647  p_double:1.7976931348623157e+308  p_bool:false
Expected Info:  f_string:"non-ASCII+non-printable string ☺ → ← \"\\/\x08\x0c\r\tሴ works, not newlines yet"  f_int32:2147483647  f_sint32:2147483647  f_sfixed32:2147483647  f_uint32:4294967295  f_fixed32:4294967295  f_int64:9223372036854775807  f_sint64:9223372036854775807  f_sfixed64:9223372036854775807  f_uint64:18446744073709551615  f_fixed64:18446744073709551615  f_double:1.7976931348623157e+308  f_float:3.4028235e+38  p_string:"Goodbye"  p_int32:2147483647  p_double:1.7976931348623157e+308  p_bool:false
2023/03/03 17:19:42 server error: (ComplianceSuiteRequestMismatchError) contents of request "Extreme values" do not match test suite

The unicode values aren't send properly as part of the request. The correct values should be ☺ → ← but they get sent as ? ? ?.

Issue:

  1. The MediaType for REST requests is set here:
    https://github.com/googleapis/gapic-generator-java/blob/c23f981e2ac3c573bed51e725dc7061551179400/gax-java/gax-httpjson/src/main/java/com/google/api/gax/httpjson/HttpRequestRunnable.java#L175

It only set the MediaType's charset as application/json when it could be application/json; charset=utf-8. This ensures that the unicode values are set to be encoded properly.

  1. The default charset is not set to be UTF-8:
    https://github.com/googleapis/google-http-java-client/blob/84216c5cfe2f1600464d34661a208cf165e11b9b/google-http-client/src/main/java/com/google/api/client/http/AbstractHttpContent.java#L94-L98

Instead, it's set to StandardCharsets.ISO_8859_1, which only represents the first 256 characters of Unicode.

This seems to due to this comment: googleapis/google-http-java-client#300 (comment)
@lqiu96 lqiu96 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Mar 3, 2023
@lqiu96 lqiu96 self-assigned this Mar 3, 2023
@lqiu96 lqiu96 transferred this issue from googleapis/google-http-java-client Mar 6, 2023
@lqiu96 lqiu96 changed the title Use Charset UTF-8 for application/json MedaType Set the default charset as UTF-8 for REST calls with application/json MediaType Mar 9, 2023
This was referenced Mar 10, 2023
Closed
Merged
Closed
gcf-merge-on-green bot pushed a commit that referenced this issue Mar 20, 2023
@lqiu96
fix: Use UTF-8 as default charset for HttpJson requests (#1477)
79d986b 
Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://togithub.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes #1437
@release-please release-please bot mentioned this issue Mar 20, 2023
Closed
lqiu96 added a commit to renovate-bot/gapic-generator-java that referenced this issue Mar 20, 2023
@lqiu96
fix: Use UTF-8 as default charset for HttpJson requests (googleapis#1477
da5780f 
)

Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://togithub.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes googleapis#1437
lqiu96 added a commit to renovate-bot/gapic-generator-java that referenced this issue Mar 21, 2023
@lqiu96
fix: Use UTF-8 as default charset for HttpJson requests (googleapis#1477
2609510 
)

Thank you for opening a Pull Request! For general contributing guidelines, please refer to [contributing guide](https://togithub.com/googleapis/gapic-generator-java/blob/main/CONTRIBUTING.md)

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

- [x] Make sure to open an issue as a [bug/issue](https://togithub.com/googleapis/gapic-generator-java/issues/new/choose) before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
- [x] Ensure the tests and linter pass
- [x] Code coverage does not decrease (if any source code was changed)
- [x] Appropriate docs were updated (if necessary)

Fixes googleapis#1437
@lqiu96 lqiu96 mentioned this issue Mar 22, 2023
Closed
@release-please release-please bot mentioned this issue Mar 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lqiu96 lqiu96

Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
1 participant
@lqiu96