Saving "special" emoji in text fields causes 'Control character error, possibly incorrectly encoded.'

[sevfurneaux]

When saving "special" emoji within text fields (such as 🦄), causes this error:

Invalid Argument – yii\base\InvalidArgumentException
Control character error, possibly incorrectly encoded.

This error is raised in in /app/vendor/yiisoft/yii2/helpers/BaseJson.php and /app/vendor/craftcms/cms/src/services/EntryRevisions.php.

At line 390 in EntryRevisions.php:

$result['data'] = Json::decode($result['data']);

After further investigation looking in the database, this is an issue specifically with the entryversions table and what gets saved in the data column when a "special" emoji is used.

You can see a comparison below, with the data column content showing an entry with normal text and an example when I saved the entry with a 🦄:

Normal Data

{"typeId":"14","authorId":"1","title":"This is a test entry","slug":"this-is-a-test-entry","postDate":1523459184,"expiryDate":null,"enabled":true,"newParentId":null,"fields":{"181":"Normal content"}}

When saved with 🦄

{"typeId":"14","authorId":"1","title":"This is a test emoji","slug":"this-is-a-test-emoji","postDate":1523459260,"expiryDate":null,"enabled":true,"newParentId":null,"fields":{"181":"

You can see above, that the data was not saved correctly when the special emoji is used. It cuts off at {"181":" and therefore causes a JSON decoding error.

[brandonkelly]

I’m not able to even save an entry with a 🦄 when using MySQL (get a SQL error), and not able to reproduce the the exception when using PostgreSQL. Have you made any changes to your environment to get past this error in MySQL?

SQLSTATE[HY000]: General error: 1366 Incorrect string value: '\xF0\x9F\xA6\x84' for column 'field_<field_handle>' at row 1

[sevfurneaux]

I've managed to recreate this in two different environments: Nanobox (locally and on a staging site) and also in MAMP Pro 4.2.

In both environments when saving the 🦄, it produced the malformed data in the data column of the entryversions table.

[brandonkelly]

We fixed this at the root, by adding a validation error for textual custom fields that contain emojis (or any other 4+ byte characters), if the DB doesn't support 4+ byte characters. (Any characters that are converted to HTML entities or emoji shortcodes by the field type before being saved won’t be affected.)

[sevfurneaux]

Great, thanks!

[bossanova808]

Hmm, I have jsut hit this issue with a customer adding some order notes (to some line item custom data) with an emoji in it.

I am wondering what the best fix is in this case - db end (unicode mb4?) - js (strip emjois - but inevitably people will be putting them in and expect them to work I expect) - ... or somewhere else?

@brandonkelly Any tips on this?

[brandonkelly]

@bossanova808 Can you repost on the Commerce repo?

[bossanova808]

Issue here - craftcms/commerce#886 (comment)

I commented on the fix too, because it's not complete for my use case yet: craftcms/commerce@11c5dc2 - but maybe it's too hard in that case? Not sure how universal you want to be with accepting this data...