Auto-coerce top-level results and concat arguments to strings #7280

jfirebaugh · 2018-09-13T20:34:35Z

Fixes #6190, implementing auto-coercion for "concat" and string-valued properties. The latter change affects the behavior of coalesce as well, for instance "text-field": ["coalesce", ["get", "a"], ["get", "b"]] will now never produce a type error.

Launch Checklist

briefly describe the changes in this PR
write tests for all new functionality
document any changes to public APIs
manually test the debug page
tagged @mapbox/studio and/or @mapbox/maps-design if this PR includes style spec changes

samanpwbb

Awesome, excited for this change – just had one piece of feedback about the expression migration script.

samanpwbb · 2018-09-13T20:40:52Z

src/style-spec/function/convert.js

-        return get;
+        // By default, expressions for string-valued properties get coerced. To preserve
+        // legacy function semantics, insert an explicit assertion instead.
+        return propertySpec.type === 'string' ? ['string', get] : get;


Should the to-string expression be dropped here: https://github.com/mapbox/mapbox-gl-js/pull/7280/files#diff-4b96c63e8fcfc96468a92c7a73e53b5eR229

👍 , updated.

samanpwbb · 2018-09-13T20:41:46Z

src/style-spec/expression/index.js

+    // For string-valued properties, coerce to string at the top level rather than asserting.
+    const parsed = parser.parse(expression, undefined, undefined, undefined,
+        propertySpec.type === 'string' ? {typeAnnotation: 'coerce'} : undefined);
+


No longer need to-string when using concat. However, it is still needed if the token string consists of a single token.

jfirebaugh · 2018-09-13T23:24:16Z

The recently added format operator (#6994) adds a couple wrinkles here:

As written, this PR no longer actually applies to text-field, since its type changed from string to formatted. This should be okay though, because the default annotation behavior for the formatted type is to coerce.
We should probably extend the auto-coercion behavior to the format operator's arguments, as format will be used in very similar circumstances to concat.

ChrisLoer · 2018-09-13T23:56:23Z

src/style-spec/expression/index.js

    const types = {
        color: ColorType,
        string: StringType,
        number: NumberType,
        enum: StringType,
-        boolean: BooleanType
+        boolean: BooleanType,
+        formatted: FormattedType


I think with this change, every place in the code that checks text instanceof Formatted has a dead branch. I can go clean those up in a later PR. There may also be some performance impact here because of the extra bookkeeping the "formatted" path uses, although I'd guess it's pretty minimal.

This avoids a troublesome circular import.

ChrisLoer · 2018-09-17T17:19:53Z

No obvious change in the Layout benchmark, which should exercise the extra string->formatted coercion (although I've been getting pretty variable results between runs):

ChrisLoer · 2018-09-17T21:36:04Z

I went ahead and made the changes to strip out the instanceof Formatted dead code, and as part of testing that realized that the implicit coercion strategy John used in 85fce01#diff-7341a1d97068bd8c5a153f89bc9e4b1aL117 wouldn't work on this relatively simple case:

[ "coalesce", [ "get", "name_en" ], [ "get", "name" ] ]

The reason was that the coalesce introduced a typeAnnotation of omit, triggering the assert at:

 assert(!options.typeAnnotation);
 parsed = new FormatExpression([{text: parsed, scale: null, font: null}]);

I'm not sure of the right answer here -- it seems like allowing omit might make sense in this case. But I think we have to support any type of annotation, because for instance if someone defines a text-field to be ['number', 0] that should generation an expression parsing error, not an assertion failure.

I tried going back to the approach of using Coercion, but added explicit format-supporting code to the Coercion so that it wouldn't try to serialize as to-formatted.

@mollymerp since I've started modifying this PR, can I ask you to step in as reviewer?

I'm going to try to add some test cases for the different "type annotation" cases.

…tion time. - Add coercion after call to 'getValueAndResolveTokens`, since the non-expression pathway here skips the coercion logic in parsing_context - Remove 'string | Formatted' typing and 'text instanceof Formatted' checks - Add Coercion support for 'Formatted', along with dedicated serialization - Use Coercion when parsing expected.kind === 'formatted' instead of directly creating a FormatExpression. This is necessary to accommodate expressions such as 'coalesce' that introduce a typeAnnotation.

mollymerp

This looks good overall (to me, someone with limited expression parsing knowledge – thank you for the additional context in chat @ChrisLoer !)

mollymerp · 2018-09-18T22:13:34Z

test/integration/expression-tests/format/implicit-assert/test.json

+    "compiled": {
+        "result": "error",
+        "errors": [
+          {"key": "", "error": "Expected formatted but found number instead."}


This seems like a rather unhelpful error message, considering that most "text-field" values won't be actually using the "format" expression. Should we special case this type checking error for formatted or punt on this for a later error message improvement project?

🤔 I think defer? The way I kind of think about it is: it's confusing because people expect to be able to return a 'string', and have it implicitly-coerce to 'formatted' without them having to even know 'formatted' exists. So you'd kind of like it to say something like Expected formatted or string but found number instead. -- but that's still actually confusing because (1) things other than 'formatted' or 'string' can still successfully coerce to 'formatted', (2) even 'number' can coerce to 'formatted' - the problem is that the assertion disables the implicit coercion.

mollymerp · 2018-09-18T22:26:43Z

src/style-spec/expression/parsing_context.js

        if (expr === null || typeof expr === 'string' || typeof expr === 'boolean' || typeof expr === 'number') {
            expr = ['literal', expr];
        }

+        function annotate(parsed, type, typeAnnotation: 'assert' | 'coerce' | 'omit') {


I find typeAnnotation to be a very confusing name for this modifier (typeModifier 🤔 ) but we don't have to deal with that in this PR 😶

Auto-convert concat arguments to strings

f42af5b

jfirebaugh requested review from ChrisLoer, samanpwbb and mollymerp September 13, 2018 20:34

jfirebaugh force-pushed the concat-to-string branch from fcced57 to 1b5e28c Compare September 13, 2018 20:40

For string-valued properties, do coercion rather than assertion

4256281

jfirebaugh force-pushed the concat-to-string branch from 1b5e28c to 4256281 Compare September 13, 2018 20:41

samanpwbb reviewed Sep 13, 2018

View reviewed changes

Simplify how token strings are converted

1ff116c

No longer need to-string when using concat. However, it is still needed if the token string consists of a single token.

jfirebaugh force-pushed the concat-to-string branch from 0a5cacd to 1ff116c Compare September 13, 2018 22:31

Auto-convert format arguments to strings

3b480cd

ChrisLoer reviewed Sep 13, 2018

View reviewed changes

jfirebaugh added 2 commits September 13, 2018 17:08

Fix implicit conversions for "formatted" type

85fce01

Split Collator/Formatted types into separate files

11b6a19

This avoids a troublesome circular import.

jfirebaugh force-pushed the concat-to-string branch from 9680322 to 11b6a19 Compare September 14, 2018 00:08

jfirebaugh mentioned this pull request Sep 14, 2018

Auto-coerce top-level results and concat arguments to strings mapbox/mapbox-gl-native#12892

Merged

ChrisLoer self-assigned this Sep 17, 2018

ChrisLoer force-pushed the concat-to-string branch from f5cd8c3 to b0dffd2 Compare September 17, 2018 21:37

Add expression test cases for implicit format type annotations.

2474692

mollymerp approved these changes Sep 18, 2018

View reviewed changes

ChrisLoer merged commit a1233cb into master Sep 18, 2018

ChrisLoer deleted the concat-to-string branch September 18, 2018 23:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-coerce top-level results and concat arguments to strings #7280

Auto-coerce top-level results and concat arguments to strings #7280

jfirebaugh commented Sep 13, 2018

samanpwbb left a comment

samanpwbb Sep 13, 2018

jfirebaugh Sep 13, 2018

samanpwbb Sep 13, 2018

jfirebaugh commented Sep 13, 2018

ChrisLoer Sep 13, 2018

ChrisLoer commented Sep 17, 2018

ChrisLoer commented Sep 17, 2018

mollymerp left a comment

mollymerp Sep 18, 2018

ChrisLoer Sep 18, 2018

mollymerp Sep 18, 2018

Auto-coerce top-level results and concat arguments to strings #7280

Auto-coerce top-level results and concat arguments to strings #7280

Conversation

jfirebaugh commented Sep 13, 2018

Launch Checklist

samanpwbb left a comment

Choose a reason for hiding this comment

samanpwbb Sep 13, 2018

Choose a reason for hiding this comment

jfirebaugh Sep 13, 2018

Choose a reason for hiding this comment

samanpwbb Sep 13, 2018

Choose a reason for hiding this comment

jfirebaugh commented Sep 13, 2018

ChrisLoer Sep 13, 2018

Choose a reason for hiding this comment

ChrisLoer commented Sep 17, 2018

ChrisLoer commented Sep 17, 2018

mollymerp left a comment

Choose a reason for hiding this comment

mollymerp Sep 18, 2018

Choose a reason for hiding this comment

ChrisLoer Sep 18, 2018

Choose a reason for hiding this comment

mollymerp Sep 18, 2018

Choose a reason for hiding this comment