Skip to content
This repository has been archived by the owner on Aug 8, 2023. It is now read-only.

[core] Convert "legacy" filters directly into expressions #11610

Merged
merged 38 commits into from
May 10, 2018

Conversation

lucaswoj
Copy link
Contributor

@lucaswoj lucaswoj commented Apr 5, 2018

Follow up to #11429

Launch Checklist

  • Port the specialized filter-* expressions from GL JS, adding them to src/mbgl/style/expression/compound_expression.cpp.
  • Update src/mbgl/style/conversion/filter.cpp to always convert incoming JSON values to an ExpressionFilter (using the filter-* types as needed for legacy syntax)
  • Rewrite/simplify include/mbgl/style/filter.hpp and mbgl::style::FilterEvaluator, removing all the different *Filter types and the Filter variant and renaming ExpressionFilter to Filter.
  • Rebase on release-boba
  • update platform bindings to remove all uses of deprecated filter API
    • android-debug-arm-v7
    • android-release-all
    • clang-tidy
    • ios-debug
    • ios-sanitize
    • ios-sanitize-address
    • ios-static-analyzer
    • linux-clang-3.8-libcxx-debug
    • linux-clang4-sanitize-address
    • linux-clang4-sanitize-thread
    • linux-clang4-sanitize-undefined
    • linux-gcc4.9-debug
    • linux-gcc5-debug-coverage
    • linux-gcc5-release-qt4
    • linux-gcc5-release-qt5
    • macos-debug
    • macos-debug-qt5
    • macos-release-node4
    • macos-release-node6
    • nitpick
    • node4-clang39-release
    • node6-clang39-release
    • node6-gcc6-debug

@lucaswoj lucaswoj force-pushed the expression-filters-2 branch 2 times, most recently from 0b2cce0 to 4678133 Compare April 6, 2018 19:25
@lucaswoj lucaswoj force-pushed the expression-filters-2 branch 3 times, most recently from 4842bbd to ecf41db Compare April 17, 2018 23:01
@lucaswoj lucaswoj force-pushed the expression-filters-2 branch 2 times, most recently from 621a177 to daaefc1 Compare April 19, 2018 23:37
mbgl::Value expressionValue = filter.expression.get()->serialize();
return gson::JsonElement::New(env, expressionValue);
} else {
return null;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question for reviewers: Are we setting ourselves up for future NPEs by allowing filter.expression to be null? Is there a better solution?

args.push_back(std::make_unique<mbgl::style::expression::Literal>(mbgl::style::expression::Value(std::string("extrude"))));
args.push_back(std::make_unique<mbgl::style::expression::Literal>(mbgl::style::expression::Value(std::string("true"))));
mbgl::style::expression::ParsingContext parsingContext;
extrusionLayer->setFilter(mbgl::style::Filter { std::move(*mbgl::style::expression::createCompoundExpression("filter-==", std::move(args), parsingContext)) });
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question for future reviewer: should we create a more ergonomic syntax for instantiating expressions? Ideally something close to:

createExpression("filter-==", "extrude", "true")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely agree that creating filters (and, more generally, expressions) is very cumbersome in mbgl core. I'm just not sure how important it is to streamline this internal API, since its primary consumer is really ParsingContext.

Copy link
Contributor

@jfirebaugh jfirebaugh Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran into this while working on #11247 -- I wanted to completely remove the *Function constructors that take a Stops argument. But I gave up because it was sooo cumbersome to convert everything that used them to expressions. That included tests and a few other things that I can't recall at the moment.

So I think it would be worthwhile to provide a more convenient API at least for the subset of expression operators needed by legacy filters and tests. It will probably look similar to the convenience API that Android has.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically, this constructor. There’s a similar thing on iOS/macOS with +[NSExpression expressionWithMGLJSONObject:] and/or the MGL_FUNCTION() format string syntax.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some convenience overloads to createCompoundExpression and added createLiteral function. These reduces the boilerplate significantly:

ParsingContext ctx;
extrusionLayer->setFilter(Filter(createCompoundExpression("filter-==", createLiteral("extrude"), createLiteral("true"), ctx)));

@lucaswoj lucaswoj force-pushed the expression-filters-2 branch from 2c8d523 to d588405 Compare April 20, 2018 19:25
@1ec5 1ec5 added this to the ios-v4.0.1 milestone Apr 20, 2018
@lucaswoj lucaswoj force-pushed the expression-filters-2 branch from ab8cd73 to 4a4b44f Compare April 20, 2018 20:36
@lucaswoj lucaswoj requested a review from anandthakker April 20, 2018 20:50
@lucaswoj
Copy link
Contributor Author

@anandthakker This ready for an initial review 😄

Copy link
Contributor

@1ec5 1ec5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on the iOS/macOS side.

}
}
}

#define MGLAssertEqualFilters(actual, expected, ...) \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This macro is unused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️

Copy link
Contributor

@anandthakker anandthakker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucaswoj still in progress, but wanted to post these before I close my browser

class Filter : public FilterBase {
public:
using FilterBase::FilterBase;
Filter():expression({}) {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using nullptr to model the absence of a filter, should we have expression be optional<std::shared_ptr<Expression>>?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: surround the : with spaces (here and just below)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an advantage to using optional instead of a nullptr? For better or for worse, unique_ptr has "optional" functionality built in and every callsite ought to handle it gracefully.


namespace mbgl {
namespace style {
namespace conversion {

using GeometryValue = mapbox::geometry::value;
static bool isExpression(const Convertible& filter);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this static necessary?


namespace mbgl {
namespace style {
namespace conversion {

using GeometryValue = mapbox::geometry::value;
static bool isExpression(const Convertible& filter);
std::unique_ptr<expression::Expression> convertLegacyFilter(const Convertible& values, Error& error);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: unnecessary indentation

error = { parsingContext.getCombinedErrors() };
return {};
}
expression = std::move(*parseResult);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this just be return { Filter { std::move(*parseResult) } };, given the if(!parseResult) check above?


std::unique_ptr<expression::Expression> convertLiteral(const Convertible& convertible) {
optional<mapbox::geometry::value> value = toValue(convertible);
expression::Value expressionValue = value ? expression::toExpressionValue(*value) : expression::Null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this will effectively coerce array or object values to expression::Null, because toValue is supposed to return "none" if the value is not a boolean, number, or string.

I think this means that instead of getting a validation error for ["==", "key", {"x": 1}], we'd get the equivalent of ["==", "key", null].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Just seeing the Literal::parse method now. That looks like the right one to use.

*op == "none" ? createExpression("!", createExpression("any", convertLegacyFilterArray(values, error, 1), error), error) :
*op == "in" ? convertLegacyInFilter(values, error) :
*op == "!in" ? createExpression("!", convertLegacyInFilter(values, error), error) :
*op == "has" ? convertLegacyHasFilter(*toString(arrayMember(values, 1)), error) :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if there's an invalid filter of the form ["has", 1]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then everything will 💥 ! Will fix shortly.

@anandthakker
Copy link
Contributor

Is there an advantage to using optional instead of a nullptr? For better or for worse, unique_ptr has "optional" functionality built in and every callsite ought to handle it gracefully.

🤷‍♂️ I can see it both ways. IMO, the benefit of consistently modeling maybe-an-expression with optional<std::unique_ptr<Expression>> // pointer is never null is that we then have a way of saying "definitely an expression", using std::unique_ptr<Expression>. I guess what we really want is optional<non_nullable_unique_ptr<Expression>>

return { IdentifierFilterType {} };
std::unique_ptr<expression::Expression> convertLiteral(const Convertible& convertible, Error& error) {
expression::ParsingContext parsingContext;
expression::ParseResult parseResult = expression::Literal::parse(convertible, parsingContext);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this still does less type validation than was previously happening. For instance, Literal::parse() will successfully parse any string, number, boolean, or null value -- but previously, a legacy filter like ["==", "$type", null] would have caused the error, "value for $type filter must be Point, LineString, or Polygon"; and ones like ["==", "property_key", null] would have caused "filter expression value must be a boolean, number, or string".

define("filter-id->=", [](const EvaluationContext& params, double lhs) -> Result<bool> {
auto rhs = featureIdAsDouble(params);
return rhs ? rhs >= lhs : false;
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need string overloads of these comparison operators

ASSERT_TRUE(filter("[\"<=\", \"two\", \"2\"]", {{"two", std::string("2")}}));
ASSERT_FALSE(filter("[\"<\", \"two\", \"1\"]", {{"two", std::string("2")}}));
ASSERT_FALSE(filter("[\"==\", \"two\", 4]", {{"two", std::string("2")}}));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GL JS has tests for the expression version of any and all (to make sure that we're correctly parsing them as expressions as opposed to legacy filters): https://github.com/mapbox/mapbox-gl-js/blob/95bb4c0bcbf2da406da28795baf6a325c189cc04/test/unit/style-spec/feature_filter.test.js#L27-L37 -- would it make sense to add those?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -141,6 +141,18 @@ ParseResult createCompoundExpression(const CompoundExpressionRegistry::Definitio
ParseResult createCompoundExpression(const std::string& name,
std::vector<std::unique_ptr<Expression>> args,
ParsingContext& ctx);

ParseResult createCompoundExpression(const std::string& name, ParsingContext& ctx);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we stick a quick comment here indicating that these overloads are here to avoid inconvenience of having to create the args vector before calling?

return { IdentifierFilterType {} };
std::unique_ptr<Expression> convertLiteral(const Convertible& convertible, Error& error) {
ParsingContext parsingContext;
ParseResult parseResult = Literal::parse(convertible, parsingContext);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we've still got the issue described in #11610 (comment) : I think right now, this would allow ["==", "$type", null] or ["==", "property_key", null], whereas previously they would have caused errors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into the fix for $type, but the null property value case seems to be supported in gl-js: https://github.com/mapbox/mapbox-gl-js/blob/95bb4c0bcbf2da406da28795baf6a325c189cc04/test/unit/style-spec/feature_filter.test.js#L83-L84

Expression syntax also allows Null as a comparable type :

static bool isComparableType(const type::Type& type) {
return type == type::String ||
type == type::Number ||
type == type::Boolean ||
type == type::Null;
}

Copy link
Contributor

@asheemmamoowala asheemmamoowala May 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some test to verify this issue, and there is a change in the error returned, but there is still protection from invalid filter structure. Even when the Literal is parsed, the filter expression will not find a matching definition in the expression registry and complain about incorrect type or number of arguments.

["==", "$type"] //"Expected 1 arguments, but found 0 instead."
["==", "$type", null] //"[1]: Expected string but found null instead."
["==", "$type, "Point", 1] //"Expected 1 arguments, but found 2 instead."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, gotcha. I mistakenly thought that the filter-... expressions were accepting a Value argument, but yeah, as long as we're accepting/rejecting the same filters, I don't think it matters if the specific error message changed. 👍

define("!", [](bool e) -> Result<bool> { return !e; });

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: revert these whitespace changes -- I think we normally use Xcode's default, which to include indentation for blank lines.

Copy link
Contributor

@1ec5 1ec5 May 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, the preference in mbgl is still to avoid introducing indented blank lines, while the preference in platform/{darwin,ios,macos}/ and Objective-C code elsewhere is to follow the Xcode default of indenting blank lines. It isn’t really worth fighting over invisible characters, but that goes both ways: not really worth stripping them out, and not really worth putting them back in. 🤷‍♂️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn’t really worth fighting over invisible characters, but that goes both ways: not really worth stripping them out, and not really worth putting them back in

Yeah, that's basically my stance, too: the particular preference(s) matter less than avoiding whitespace-only changes (like these ^) in diffs.

Copy link
Contributor

@anandthakker anandthakker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit in the unit test, but otherwise LGTM!

ASSERT_TRUE(filter("[\"any\", true]"));
ASSERT_TRUE(filter("[\"any\",true, false]"));
ASSERT_TRUE(filter("[\"any\", true, true]"));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these will still end up testing the non-expression path -- how about just using the ones from the test immediately above, but replacing "foo" with ["get", "foo"]?

@asheemmamoowala asheemmamoowala force-pushed the expression-filters-2 branch from 2ffbb69 to 364e9e3 Compare May 9, 2018 21:24
@asheemmamoowala asheemmamoowala force-pushed the expression-filters-2 branch from 364e9e3 to 0359c4c Compare May 9, 2018 23:09
@asheemmamoowala asheemmamoowala merged commit a4e2c1a into release-boba May 10, 2018
@asheemmamoowala asheemmamoowala deleted the expression-filters-2 branch May 10, 2018 19:39
@friedbunny
Copy link
Contributor

Does this need changelog entries?

@1ec5
Copy link
Contributor

1ec5 commented May 11, 2018

Not sure. Is the change transparent to developers using the runtime styling API?

@lucaswoj
Copy link
Contributor Author

lucaswoj commented May 11, 2018 via email

@anandthakker
Copy link
Contributor

anandthakker commented May 11, 2018 via email

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants