Skip to content

Commit

Permalink
ast: New parser implementation
Browse files Browse the repository at this point in the history 
This commit replaces the existing PEG generated parser with a parser
implemented by hand. The new parser is more efficient (avoiding old
problems with pathological input cases like {{{{{{{{{}}}}}}}} and
deeply-nested composites in general) and offers better opportunities
for improved error reporting (which has been improved already but
there is still room to grow.)

During the test process of implementing the new parser, we identified
a few issues that were present in the old parser. Those issues are
fixed by this commit.

Fixes open-policy-agent#1251
Fixes open-policy-agent#501
Fixes open-policy-agent#2198
Fixes open-policy-agent#2199
Fixes open-policy-agent#2200
Fixes open-policy-agent#2201
Fixes open-policy-agent#2202
Fixes open-policy-agent#2203

Co-authored-by: Torin Sandall <torinsandall@gmail.com>
Co-authored-by: Patrick East <east.patrick@gmail.com>

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Signed-off-by: Patrick East <east.patrick@gmail.com>
  • Loading branch information
@tsandall
tsandall committed Mar 25, 2020
1 parent 5a5d2a4 commit 6d446f7
Show file tree
Hide file tree
Showing 53 changed files with 2,831 additions and 17,158 deletions.
91 changes: 91 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,97 @@ project adheres to [Semantic Versioning](http://semver.org/).

## Unreleased

### New Parser

The next minor release includes a new parser implementation that resolves a number
of existing issues with the old parser. As part of implementing the new parser
a small number of backwards incompatible changes have been made.

#### Backwards Compatibility

The new parser contains a small number of backwards incompatible changes that
correct questionable behaviour from the old parser. These changes affect
a very small number of actual policies and we feel confident in the decision to
break backwards compatibility here.

- Numbers no longer lose-precision [#501](https://github.com/open-policy-agent/opa/issues/501)
- Leading commas do not cause objects to lose values [#2198](https://github.com/open-policy-agent/opa/issues/2198)
- Rules wrapped with braces no longer parse [#2199](https://github.com/open-policy-agent/opa/issues/2199)
- Rule names can no longer contain dots/hyphens [#2200](https://github.com/open-policy-agent/opa/issues/2200)
- Object comprehensions now have priority over logical OR in all cases [#2201](https://github.com/open-policy-agent/opa/issues/2201)

In addition there are a few small changes backwards incompatible changes in APIs:

- The `message` field on `rego_parse_error` objects contains a human-readable description
of the parse error. The old parser would often report "no match found" to indicate
the input contained invalid syntax. The new parser has slightly more specific
errors. If you integrated with OPA and implemented error handling based on the
content of these human-readable error message strings, your integration may be affected.
- The `github.com/open-policy-agent/opa/format#Bytes` function has been removed (it was unused.)

#### Benchmark Results

The output below shows the Go `benchstat` result for master (5a5d2a42) compared to the new parser.

```
name old time/op new time/op delta
ParseModuleRulesBase/1-16 210µs ± 1% 4µs ± 1% -98.02% (p=0.008 n=5+5)
ParseModuleRulesBase/10-16 1.39ms ± 1% 0.03ms ± 0% -97.93% (p=0.008 n=5+5)
ParseModuleRulesBase/100-16 13.5ms ± 1% 0.3ms ± 1% -97.93% (p=0.008 n=5+5)
ParseModuleRulesBase/1000-16 148ms ± 5% 3ms ± 6% -97.77% (p=0.008 n=5+5)
ParseStatementBasicCall-16 141µs ± 5% 3µs ± 1% -97.92% (p=0.008 n=5+5)
ParseStatementMixedJSON-16 9.06ms ± 2% 0.07ms ± 1% -99.19% (p=0.008 n=5+5)
ParseStatementSimpleArray/1-16 131µs ± 6% 2µs ± 1% -98.10% (p=0.008 n=5+5)
ParseStatementSimpleArray/10-16 499µs ± 6% 7µs ± 2% -98.54% (p=0.008 n=5+5)
ParseStatementSimpleArray/100-16 4.00ms ± 2% 0.06ms ± 4% -98.58% (p=0.008 n=5+5)
ParseStatementSimpleArray/1000-16 42.0ms ± 3% 0.5ms ± 4% -98.70% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x1-16 233µs ± 6% 4µs ± 3% -98.49% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x1-16 514µs ± 0% 9µs ± 4% -98.33% (p=0.008 n=5+5)
ParseStatementNestedObjects/10x1-16 911µs ± 5% 14µs ± 5% -98.46% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x5-16 4.24ms ± 1% 0.01ms ± 1% -99.82% (p=0.016 n=4+5)
ParseStatementNestedObjects/1x10-16 138ms ± 1% 0ms ± 1% -99.99% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x5-16 714ms ± 0% 5ms ± 5% -99.26% (p=0.016 n=4+5)
ParseBasicABACModule-16 3.12ms ± 3% 0.04ms ± 4% -98.63% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
ParseModuleRulesBase/1-16 99.2kB ± 0% 5.7kB ± 0% -94.30% (p=0.008 n=5+5)
ParseModuleRulesBase/10-16 600kB ± 0% 29kB ± 0% -95.16% (p=0.008 n=5+5)
ParseModuleRulesBase/100-16 5.72MB ± 0% 0.27MB ± 0% -95.34% (p=0.008 n=5+5)
ParseModuleRulesBase/1000-16 58.0MB ± 0% 2.7MB ± 0% -95.42% (p=0.008 n=5+5)
ParseStatementBasicCall-16 70.2kB ± 0% 5.0kB ± 0% -92.82% (p=0.008 n=5+5)
ParseStatementMixedJSON-16 3.64MB ± 0% 0.06MB ± 0% -98.34% (p=0.008 n=5+5)
ParseStatementSimpleArray/1-16 63.7kB ± 0% 4.8kB ± 0% -92.42% (p=0.008 n=5+5)
ParseStatementSimpleArray/10-16 205kB ± 0% 8kB ± 0% -96.00% (p=0.008 n=5+5)
ParseStatementSimpleArray/100-16 1.64MB ± 0% 0.05MB ± 0% -97.19% (p=0.008 n=5+5)
ParseStatementSimpleArray/1000-16 16.5MB ± 0% 0.4MB ± 0% -97.50% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x1-16 98.6kB ± 0% 5.7kB ± 0% -94.22% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x1-16 224kB ± 0% 9kB ± 0% -96.05% (p=0.008 n=5+5)
ParseStatementNestedObjects/10x1-16 381kB ± 0% 13kB ± 0% -96.63% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x5-16 1.76MB ± 0% 0.01MB ± 0% -99.38% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x10-16 56.2MB ± 0% 0.0MB ± 0% -99.97% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x5-16 280MB ± 0% 4MB ± 0% -98.67% (p=0.008 n=5+5)
ParseBasicABACModule-16 1.27MB ± 0% 0.04MB ± 0% -97.08% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
ParseModuleRulesBase/1-16 2.28k ± 0% 0.07k ± 0% -96.75% (p=0.008 n=5+5)
ParseModuleRulesBase/10-16 16.1k ± 0% 0.5k ± 0% -96.59% (p=0.008 n=5+5)
ParseModuleRulesBase/100-16 159k ± 0% 5k ± 0% -96.64% (p=0.008 n=5+5)
ParseModuleRulesBase/1000-16 1.62M ± 0% 0.05M ± 0% -96.72% (p=0.008 n=5+5)
ParseStatementBasicCall-16 1.36k ± 0% 0.05k ± 0% -96.25% (p=0.008 n=5+5)
ParseStatementMixedJSON-16 105k ± 0% 1k ± 0% ~ (p=0.079 n=4+5)
ParseStatementSimpleArray/1-16 1.34k ± 0% 0.04k ± 0% -97.09% (p=0.008 n=5+5)
ParseStatementSimpleArray/10-16 5.49k ± 0% 0.12k ± 0% -97.90% (p=0.008 n=5+5)
ParseStatementSimpleArray/100-16 47.8k ± 0% 0.8k ± 0% ~ (p=0.079 n=4+5)
ParseStatementSimpleArray/1000-16 481k ± 0% 8k ± 0% -98.33% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x1-16 2.38k ± 0% 0.05k ± 0% -97.82% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x1-16 6.02k ± 0% 0.12k ± 0% -97.94% (p=0.008 n=5+5)
ParseStatementNestedObjects/10x1-16 10.6k ± 0% 0.2k ± 0% -98.01% (p=0.008 n=5+5)
ParseStatementNestedObjects/1x5-16 51.2k ± 0% 0.1k ± 0% ~ (p=0.079 n=4+5)
ParseStatementNestedObjects/1x10-16 1.66M ± 0% 0.00M ± 0% -99.99% (p=0.008 n=5+5)
ParseStatementNestedObjects/5x5-16 8.16M ± 0% 0.07M ± 0% -99.13% (p=0.008 n=5+5)
ParseBasicABACModule-16 36.5k ± 0% 0.7k ± 0% -98.09% (p=0.008 n=5+5)
```

## 0.18.0

### Features
Expand Down
28 changes: 14 additions & 14 deletions ast/check_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ func TestCheckInference(t *testing.T) {
obj[i] = v1;
arr[j] = v2;
set[v3];
obj = {"foo": "bar"}
obj = {"foo": "bar"};
arr = [1];
set = {1,2,3}
`, map[Var]types.Type{
Expand Down Expand Up @@ -574,19 +574,19 @@ func TestCheckMatchErrors(t *testing.T) {
note string
query string
}{
{"null", "{ null = true }"},
{"boolean", "{ true = null }"},
{"number", "{ 1 = null }"},
{"string", `{ "hello" = null }`},
{"array", "{[1,2,3] = null}"},
{"array-nested", `{[1,2,3] = [1,2,"3"]}`},
{"array-nested-2", `{[1,2] = [1,2,3]}`},
{"array-dynamic", `{ [ true | true ] = [x | a = [1, "foo"]; x = a[_]] }`},
{"object", `{{"a": 1, "b": 2} = null}`},
{"object-nested", `{ {"a": 1, "b": "2"} = {"a": 1, "b": 2} }`},
{"object-nested-2", `{ {"a": 1} = {"a": 1, "b": "2"} }`},
{"set", "{{1,2,3} = null}"},
{"any", `{x = ["str", 1]; x[_] = null}`},
{"null", "null = true"},
{"boolean", "true = null"},
{"number", "1 = null"},
{"string", `"hello" = null`},
{"array", "[1,2,3] = null"},
{"array-nested", `[1,2,3] = [1,2,"3"]`},
{"array-nested-2", `[1,2] = [1,2,3]`},
{"array-dynamic", `[ true | true ] = [x | a = [1, "foo"]; x = a[_]]`},
{"object", `{"a": 1, "b": 2} = null`},
{"object-nested", `{"a": 1, "b": "2"} = {"a": 1, "b": 2}`},
{"object-nested-2", `{"a": 1} = {"a": 1, "b": "2"}`},
{"set", "{1,2,3} = null"},
{"any", `x = ["str", 1]; x[_] = null`},
}
for _, tc := range tests {
test.Subtest(t, tc.note, func(t *testing.T) {
Expand Down
30 changes: 15 additions & 15 deletions ast/compile_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2118,27 +2118,27 @@ func TestCompilerRewriteDynamicTerms(t *testing.T) {
input string
expected string
}{
{`arr { [str] }`, `{__local0__ = data.test.str; [__local0__]}`},
{`arr2 { [[str]] }`, `{__local0__ = data.test.str; [[__local0__]]}`},
{`obj { {"x": str} }`, `{__local0__ = data.test.str; {"x": __local0__}}`},
{`obj2 { {"x": {"y": str}} }`, `{__local0__ = data.test.str; {"x": {"y": __local0__}}}`},
{`set { {str} }`, `{__local0__ = data.test.str; {__local0__}}`},
{`set2 { {{str}} }`, `{__local0__ = data.test.str; {{__local0__}}}`},
{`ref { str[str] }`, `{__local0__ = data.test.str; data.test.str[__local0__]}`},
{`ref2 { str[str[str]] }`, `{__local0__ = data.test.str; __local1__ = data.test.str[__local0__]; data.test.str[__local1__]}`},
{`arr { [str] }`, `__local0__ = data.test.str; [__local0__]`},
{`arr2 { [[str]] }`, `__local0__ = data.test.str; [[__local0__]]`},
{`obj { {"x": str} }`, `__local0__ = data.test.str; {"x": __local0__}`},
{`obj2 { {"x": {"y": str}} }`, `__local0__ = data.test.str; {"x": {"y": __local0__}}`},
{`set { {str} }`, `__local0__ = data.test.str; {__local0__}`},
{`set2 { {{str}} }`, `__local0__ = data.test.str; {{__local0__}}`},
{`ref { str[str] }`, `__local0__ = data.test.str; data.test.str[__local0__]`},
{`ref2 { str[str[str]] }`, `__local0__ = data.test.str; __local1__ = data.test.str[__local0__]; data.test.str[__local1__]`},
{`arr_compr { [1 | [str]] }`, `[1 | __local0__ = data.test.str; [__local0__]]`},
{`arr_compr2 { [1 | [1 | [str]]] }`, `[1 | [1 | __local0__ = data.test.str; [__local0__]]]`},
{`set_compr { {1 | [str]} }`, `{1 | __local0__ = data.test.str; [__local0__]}`},
{`set_compr2 { {1 | {1 | [str]}} }`, `{1 | {1 | __local0__ = data.test.str; [__local0__]}}`},
{`obj_compr { {"a": "b" | [str]} }`, `{"a": "b" | __local0__ = data.test.str; [__local0__]}`},
{`obj_compr2 { {"a": "b" | {"a": "b" | [str]}} }`, `{"a": "b" | {"a": "b" | __local0__ = data.test.str; [__local0__]}}`},
{`equality { str = str }`, `{data.test.str = data.test.str}`},
{`equality2 { [str] = [str] }`, `{__local0__ = data.test.str; __local1__ = data.test.str; [__local0__] = [__local1__]}`},
{`call { startswith(str, "") }`, `{__local0__ = data.test.str; startswith(__local0__, "")}`},
{`call2 { count([str], n) }`, `{__local0__ = data.test.str; count([__local0__], n)}`},
{`eq_with { [str] = [1] with input as 1 }`, `{__local0__ = data.test.str with input as 1; [__local0__] = [1] with input as 1}`},
{`term_with { [[str]] with input as 1 }`, `{__local0__ = data.test.str with input as 1; [[__local0__]] with input as 1}`},
{`call_with { count(str) with input as 1 }`, `{__local0__ = data.test.str with input as 1; count(__local0__) with input as 1}`},
{`equality { str = str }`, `data.test.str = data.test.str`},
{`equality2 { [str] = [str] }`, `__local0__ = data.test.str; __local1__ = data.test.str; [__local0__] = [__local1__]`},
{`call { startswith(str, "") }`, `__local0__ = data.test.str; startswith(__local0__, "")`},
{`call2 { count([str], n) }`, `__local0__ = data.test.str; count([__local0__], n)`},
{`eq_with { [str] = [1] with input as 1 }`, `__local0__ = data.test.str with input as 1; [__local0__] = [1] with input as 1`},
{`term_with { [[str]] with input as 1 }`, `__local0__ = data.test.str with input as 1; [[__local0__]] with input as 1`},
{`call_with { count(str) with input as 1 }`, `__local0__ = data.test.str with input as 1; count(__local0__) with input as 1`},
}

for _, tc := range tests {
Expand Down
Loading

0 comments on commit 6d446f7

Please sign in to comment.