Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Parser #262

Merged
merged 65 commits into from
Jan 23, 2023
Merged
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
483de42
[parser] Initial design concept for a new parser that supports BNF-st…
AntonLydike Dec 8, 2022
8f39dbd
[parser] fixes for some correctnes issues with the parsing and error …
AntonLydike Dec 9, 2022
86fb17c
[parser] basic parsing of operation working
AntonLydike Dec 9, 2022
7de6326
[parser] cleanup of backtracking error reporting
AntonLydike Dec 10, 2022
1f0cecb
[parser] improved region and attribute parsing, building
AntonLydike Dec 10, 2022
0560fde
[parser] fixes true/false switcheroo
AntonLydike Dec 10, 2022
3bdf8aa
[parser] minor fixes in integer and float type parsing
AntonLydike Dec 10, 2022
b5123e8
[parser] yapf formatted to make CI happy
AntonLydike Dec 10, 2022
57091bc
[parser] improve error logging and add docstrings to methods
AntonLydike Dec 12, 2022
d416eff
[parser] Add basics for parsing xDSL
AntonLydike Jan 6, 2023
48b92a1
[parser] separate MLIR and xDSL parsing into two separate classes
AntonLydike Jan 9, 2023
2940b98
[parser] fixed bug in EOL handling for span context printing + order …
AntonLydike Jan 9, 2023
33d59fa
[parser] now able to parse some MLIR and xDSL code successfully
AntonLydike Jan 10, 2023
881f8d0
[parser] clean up lots of unused or unnecessary parts
AntonLydike Jan 10, 2023
280d7ee
[parser] clean up lots of unused or unnecessary parts
AntonLydike Jan 10, 2023
3073039
parser: remove BNF stuff for now
AntonLydike Jan 10, 2023
4464c32
parser: unify code style and apply yapf
AntonLydike Jan 10, 2023
cfd5c62
parser: a lot more cleanup in various places and correctnes fixes
AntonLydike Jan 11, 2023
7f41009
parser: add proper handling of forward references to blocks
AntonLydike Jan 11, 2023
4404ffc
parser: improved compatibility layer to support parsing of cmath.xdsl
AntonLydike Jan 11, 2023
3a06393
parser: fixed a bug with expect_brackets in parse_paramattr_parameters
AntonLydike Jan 12, 2023
7b94084
xdsl: disallow invalid ssa value name hints in the class
AntonLydike Jan 12, 2023
ed3a997
parser: fixing tests to work with the new parser
AntonLydike Jan 13, 2023
34157f5
parser: fixed all remaining tests
AntonLydike Jan 13, 2023
649e43a
parser: yapf formatting run
AntonLydike Jan 13, 2023
a4bb049
parser: fixed xdsl_opt tests
AntonLydike Jan 13, 2023
af8cebf
parser: fix all tests failing after rebase
AntonLydike Jan 17, 2023
64ac94e
parser: run yapf one last time before PR is done
AntonLydike Jan 17, 2023
c51e7a9
tests: fix broken float parsing filecheck
AntonLydike Jan 17, 2023
7782654
parser: fix xdsl-opt command to properly use new parser class
AntonLydike Jan 17, 2023
d77fa27
parser: fix UnregisteredOp interaction with new parser
AntonLydike Jan 17, 2023
b415f6a
parser: add function_type as possible value for parse_attribute
AntonLydike Jan 17, 2023
ba39207
parser: fix parsing of symbol reference
AntonLydike Jan 17, 2023
1e9ccc8
printer: fix printing of strings containing special characters
AntonLydike Jan 17, 2023
e21d7a0
parser: fix some more failing tests
AntonLydike Jan 17, 2023
9365029
parser: add opaque attribute and fix lots of small issues
AntonLydike Jan 18, 2023
684c809
parser+tests: fix module-op at top level error
AntonLydike Jan 18, 2023
c49fc29
parser: added memref support
AntonLydike Jan 18, 2023
0420b56
parser: change type annotations to make python3.11 happy
AntonLydike Jan 18, 2023
b58143a
parser: fixed special attribute-entry parsing for UnitAttr
AntonLydike Jan 18, 2023
58339bc
xdsl: fix how the parser is used in tests
AntonLydike Jan 20, 2023
d0b5a5b
xdsl: fix typo in Block.delcared_at
AntonLydike Jan 20, 2023
41049fe
xdsl: fixed errorneous type hints on DictionaryAttr
AntonLydike Jan 20, 2023
7fda7e0
tests: fix tests that don't wrap their input in a builtin.module
AntonLydike Jan 20, 2023
70ecada
parser: fix a typo
AntonLydike Jan 20, 2023
8e5ce88
parser: add docstring to tokenizer
AntonLydike Jan 20, 2023
6968619
parser: remove BacktrackingAbort
AntonLydike Jan 20, 2023
ebbc7bc
parser: add docstring for BacktrackingHistory
AntonLydike Jan 20, 2023
56490fd
parser: renamed begin_parse to parse_module
AntonLydike Jan 20, 2023
932e403
parser: added return type to get_block_from_name
AntonLydike Jan 20, 2023
4e985cd
parser: fix typos and alignement issues
AntonLydike Jan 23, 2023
ea9a75b
parser: removed get_nth_line_bounds - unused function
AntonLydike Jan 23, 2023
db7c4cb
parser: fix minor nitpicks in tests
AntonLydike Jan 23, 2023
4e27ede
xdsl: revert back to a callable interface for xdsl-opt frontends
AntonLydike Jan 23, 2023
7a40bf2
tests: removed a bunch of unneeded arguments for the parser
AntonLydike Jan 23, 2023
46fee68
parser: move stuff around, fix formatting
AntonLydike Jan 23, 2023
2f09883
parser: uppercase comments
AntonLydike Jan 23, 2023
91a8d12
parser: make a bunch of methods private
AntonLydike Jan 23, 2023
85810fc
tests: clean up imports and unused vars in test_parse_error
AntonLydike Jan 23, 2023
54a7788
tools: removed assertion text - off topic
AntonLydike Jan 23, 2023
9db3804
tests: fix docstring
AntonLydike Jan 23, 2023
868e509
xdsl: removed must_ prefix from parser methods
AntonLydike Jan 23, 2023
de5fa12
formatting: run yapf
AntonLydike Jan 23, 2023
cc67615
formatting: fixed typos and other minor issues
AntonLydike Jan 23, 2023
3c4349a
formatting: yapf run
AntonLydike Jan 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
parser: fixed a bug with expect_brackets in parse_paramattr_parameters
  • Loading branch information
AntonLydike committed Jan 23, 2023
commit 3a06393df3bd18264d3fab08e99b2d6ee0ec68fe
66 changes: 39 additions & 27 deletions xdsl/parser_ng.py
Original file line number Diff line number Diff line change
@@ -14,10 +14,10 @@
from typing import TypeVar, Iterable

from xdsl.dialects.builtin import (
AnyTensorType, AnyVectorType,
Float16Type, Float32Type, Float64Type, FloatAttr,
FunctionType, IndexType, IntegerType, Signedness, StringAttr,
IntegerAttr, ArrayAttr, TensorType, UnrankedTensorType, VectorType, DefaultIntegerAttrType, FlatSymbolRefAttr)
AnyTensorType, AnyVectorType, Float16Type, Float32Type, Float64Type,
FloatAttr, FunctionType, IndexType, IntegerType, Signedness, StringAttr,
IntegerAttr, ArrayAttr, TensorType, UnrankedTensorType, VectorType,
DefaultIntegerAttrType, FlatSymbolRefAttr)
from xdsl.ir import (SSAValue, Block, Callable, Attribute, Operation, Region,
BlockArgument, MLContext, ParametrizedAttribute)
from .printer import Printer
@@ -48,7 +48,13 @@ def print_with_history(self):
class MultipleSpansParseError(ParseError):
ref_text: str | None
refs: list[tuple[Span, str]]
def __init__(self, span: Span, msg: str, ref_text: str, refs: list[tuple[Span, str | None]], history: BacktrackingHistory | None = None):

def __init__(self,
span: Span,
msg: str,
ref_text: str,
refs: list[tuple[Span, str | None]],
history: BacktrackingHistory | None = None):
super(MultipleSpansParseError, self).__init__(span, msg, history)
self.refs = refs
self.ref_text = ref_text
@@ -161,7 +167,8 @@ def print_with_context(self, msg: str | None = None) -> str:
return capture.getvalue()

def __repr__(self):
return "{}[{}:{}](text='{}')".format(self.__class__.__name__, self.start, self.end, self.text)
return "{}[{}:{}](text='{}')".format(self.__class__.__name__,
self.start, self.end, self.text)


@dataclass(frozen=True, repr=False)
@@ -529,7 +536,8 @@ class ParserCommons:
# TODO: add all the Float8E4M3FNType, Float8E5M2Type, and BFloat16Type
)
builtin_type = re.compile('(({}))'.format(')|('.join(_builtin_type_names)))
builtin_type_xdsl = re.compile('!(({}))'.format(')|('.join(_builtin_type_names)))
builtin_type_xdsl = re.compile('!(({}))'.format(
')|('.join(_builtin_type_names)))
double_colon = re.compile('::')
comma = re.compile(',')

@@ -608,7 +616,8 @@ def must_parse_block(self) -> Block:

if block_id is None:
block = Block(self.tokenizer.last_token)
elif self.forward_block_references.pop(block_id.text, None) is not None:
elif self.forward_block_references.pop(block_id.text,
None) is not None:
block = self.blocks[block_id.text]
block.delcared_at = block_id
else:
@@ -618,8 +627,7 @@ def must_parse_block(self) -> Block:
"Re-declaration of block {}".format(block_id.text),
'Originally declared here:',
[(self.blocks[block_id.text].delcared_at, None)],
self.tokenizer.history
)
self.tokenizer.history)
block = Block(block_id)
self.blocks[block_id.text] = block

@@ -1083,17 +1091,19 @@ def must_parse_region(self) -> Region:
while self.tokenizer.starts_with('^'):
region.add_block(self.must_parse_block())

end = self.must_parse_characters('}',
'Reached end of region, expected `}`!')
end = self.must_parse_characters(
'}', 'Reached end of region, expected `}`!')

if len(self.forward_block_references) > 0:
raise MultipleSpansParseError(
end,
"Region ends with missing block declarations for block(s) {}!".format(', '.join(self.forward_block_references.keys())),
"Region ends with missing block declarations for block(s) {}!"
.format(', '.join(self.forward_block_references.keys())),
'The following block references are dangling:',
[(span, "Reference to block \"{}\" without implementation!".format(span.text)) for span in itertools.chain(*self.forward_block_references.values())],
self.tokenizer.history
)
[(span, "Reference to block \"{}\" without implementation!"
.format(span.text)) for span in itertools.chain(
*self.forward_block_references.values())],
self.tokenizer.history)

return region
finally:
@@ -1157,8 +1167,7 @@ def try_parse_builtin_attr(self) -> Attribute:
attrs = (self.try_parse_builtin_float_attr,
self.try_parse_builtin_int_attr,
self.try_parse_builtin_str_attr,
self.try_parse_builtin_arr_attr,
self.try_parse_function_type,
self.try_parse_builtin_arr_attr, self.try_parse_function_type,
self.try_parse_ref_attr)

for attr_parser in attrs:
@@ -1232,7 +1241,7 @@ def try_parse_builtin_arr_attr(self) -> list[Attribute] | None:
attrs = self.must_parse_list_of(self.try_parse_attribute,
'Expected array entry!')
self.must_parse_characters(
']', 'Array literals must be enclosed by square brackets!')
']', 'Malformed array contents (expected end of array here!')
return ArrayAttr.from_list(attrs)

@abstractmethod
@@ -1402,7 +1411,8 @@ def parse_op_with_default_format(
"""
# TODO: remove this function and restructure custom op / irdl parsing
assert isinstance(self, XDSLParser)
args, successors, attributes, regions, _ = self.must_parse_operation_details()
args, successors, attributes, regions, _ = self.must_parse_operation_details(
)

for x in args:
if x.text not in self.ssaValues:
@@ -1421,30 +1431,32 @@ def parse_paramattr_parameters(
self,
expect_brackets: bool = False,
skip_white_space: bool = True) -> list[Attribute]:
if self.tokenizer.next_token_of_pattern(
'<') is None and expect_brackets:
opening_brackets = self.tokenizer.next_token_of_pattern('<')
if expect_brackets and opening_brackets is None:
self.raise_error("Expected start attribute parameters here (`<`)!")

res = self.must_parse_list_of(self.try_parse_attribute,
'Expected another attribute here!')

if self.tokenizer.next_token_of_pattern(
'>') is None and expect_brackets:
if opening_brackets is not None and self.tokenizer.next_token_of_pattern(
'>') is None:
self.raise_error(
"Malformed parameter list, expected either another parameter or `>`!"
)

return res

def parse_char(self, text: str) -> Span:
self.must_parse_characters(text, "Expected {} here!".format(text))
def parse_char(self, text: str):
self.must_parse_characters(text, "Expected '{}' here!".format(text))

def parse_str_literal(self) -> str:
return self.expect(self.try_parse_string_literal, 'Malformed string literal!').string_contents
return self.expect(self.try_parse_string_literal,
'Malformed string literal!').string_contents

def parse_attribute(self) -> Attribute:
return self.must_parse_attribute()


class MLIRParser(BaseParser):

def try_parse_builtin_type(self) -> Attribute | None: