First PR for new annotations and new model for DFG nodes #583

festutz · 2022-06-23T19:19:14Z

We remodeled DFG nodes which became a CommandInvocation (i.e., a well structured parsed command, separated options etc). We use symbolic variables for filenames, paths, and file descriptors. These represent edges in the graph. This separation simplifies graph transformations. The new structure also allows to trivially convert DFG nodes back to string representations for command invocations.

Parsing into a graph structure and back should work for any command with specified annotations in the new annotations repository. Regarding parallelization, this first version supports "consecutive chunks" parallelization (even though the information is taken from a round robin parallelizer which is why the --r_split option is given in the command below).

Working example:

script: pash/evaluation/benchmarks/oneliners/test_new_annotation/wf.sh
command: pa.sh -d 1 --r_split --no_eager --log_file log.txt --assert_compiler_success wf.sh

Note that, for now, one needs to clone the repository for new annotations and specify the path as described in the ReadMe.

…ions

…ely remove use of old aggregator com_aggregator

…handle special case of intermediate file as last operand and fixed parsing issue for newline

angelhof

Left a bunch of comments, let me know which of those make sense or not.

README.md

compiler/definitions/definition_path_for_annotation_repo.py

compiler/definitions/ir/aggregator_node.py

compiler/definitions/ir/dfg_node.py

compiler/definitions/remodel/IOVar.py

compiler/pash_runtime.py

compiler/util_new_cmd_invocations.py

…otations # Conflicts: # compiler/ir.py # compiler/pash_runtime.py

github-actions · 2022-06-27T17:48:04Z

OS:ubuntu-18.04
Mon Jun 27 17:48:04 UTC 2022
intro: 0/2 tests passed.
interface: 6/38 tests passed.
compiler: 0/54 tests passed.
agg: 10/109 tests passed.
demo-spell.sh are not identical
hello-world.sh are not identical
test1 are not identical
test2 are not identical
test3 are not identical
test4 are not identical
test5 are not identical
test6 are not identical
test8 are not identical
test9 are not identical
test10 are not identical
test12 are not identical
test13 are not identical
test14 are not identical
test15 are not identical
test16 are not identical
test17 are not identical
test18 are not identical
test_set are not identical
test_set_e are not identical
test_redirect are not identical
test_unparsing are not identical
test_set_e_2 are not identical
test_set_e_3 are not identical
test_new_line_in_var are not identical
test_cmd_sbst are not identical
test_cmd_sbst2 are not identical
test_cat_hyphen are not identical
test_trap are not identical
test_umask are not identical
test_quoting are not identical
test_var_assgn_default are not identical
test_exclam are not identical
test_redir_var_test are not identical
diff.sh are not identical
diff.sh are not identical
set-diff.sh are not identical
set-diff.sh are not identical
export_var_script.sh are not identical
export_var_script.sh are not identical
comm-par-test.sh are not identical
comm-par-test.sh are not identical
comm-par-test2.sh are not identical
comm-par-test2.sh are not identical
tee_web_index_bug.sh are not identical
tee_web_index_bug.sh are not identical
fun-def.sh are not identical
fun-def.sh are not identical
bigrams.sh are not identical
bigrams.sh are not identical
spell-grep.sh are not identical
spell-grep.sh are not identical
grep.sh are not identical
grep.sh are not identical
minimal_sort.sh are not identical
minimal_sort.sh are not identical
minimal_grep.sh are not identical
minimal_grep.sh are not identical
topn.sh are not identical
topn.sh are not identical
wf.sh are not identical
wf.sh are not identical
spell.sh are not identical
spell.sh are not identical
shortest_scripts.sh are not identical
shortest_scripts.sh are not identical
alt_bigrams.sh are not identical
alt_bigrams.sh are not identical
deadlock_test.sh are not identical
deadlock_test.sh are not identical
double_sort.sh are not identical
double_sort.sh are not identical
no_in_script.sh are not identical
no_in_script.sh are not identical
for_loop_simple.sh are not identical
for_loop_simple.sh are not identical
minimal_grep_stdin.sh are not identical
minimal_grep_stdin.sh are not identical
micro_10.sh are not identical
micro_10.sh are not identical
sed-test.sh are not identical
sed-test.sh are not identical
tr-test.sh are not identical
tr-test.sh are not identical
grep-test.sh are not identical
grep-test.sh are not identical
ann-agg.sh are not identical
ann-agg.sh are not identical
test-1 are not identical
test-2 are not identical
test-3 are not identical
test-4 are not identical
test-5 are not identical
test-6 are not identical
test-7 are not identical
test-10 are not identical
test-11 are not identical
test-12 are not identical
test-13 are not identical
test-14 are not identical
test-17 are not identical
test-18 are not identical
test-21 are not identical
test-24 are not identical
test-25 are not identical
test-28 are not identical
test-30 are not identical
test-31 are not identical
test-32 are not identical
test-34 are not identical
test-35 are not identical
test-36 are not identical
test-38 are not identical
test-40 are not identical
test-41 are not identical
test-42 are not identical
test-43 are not identical
test-45 are not identical
test-46 are not identical
test-47 are not identical
test-48 are not identical
test-50 are not identical
test-51 are not identical
test-52 are not identical
test-53 are not identical
test-55 are not identical
test-56 are not identical
test-58 are not identical
test-62 are not identical
test-63 are not identical
test-70 are not identical
test-73 are not identical
test-82 are not identical
test-84 are not identical
test-89 are not identical
test-90 are not identical
test-93 are not identical
test-95 are not identical
test-96 are not identical
test-97 are not identical
test-98 are not identical
test-99 are not identical
test-100 are not identical
test-101 are not identical
test-102 are not identical
test-103 are not identical
test-104 are not identical
test-105 are not identical
test-106 are not identical
test-107 are not identical
test-108 are not identical
test-109 are not identical
test-110 are not identical
test-111 are not identical
test-113 are not identical
test-114 are not identical
test-115 are not identical
test-116 are not identical
test-117 are not identical
test-118 are not identical
test-120 are not identical
test-121 are not identical
test-122 are not identical
test-123 are not identical
test-125 are not identical
test-129 are not identical
test-130 are not identical
test-131 are not identical
test-140 are not identical
test-149 are not identical
test-150 are not identical
test-152 are not identical
test-153 are not identical
test-160 are not identical
test-163 are not identical
test-165 are not identical
test-170 are not identical
test-175 are not identical
test-176 are not identical
test-177 are not identical
test-178 are not identical
test-179 are not identical
test-180 are not identical
test-181 are not identical
test-182 are not identical
test-187 are not identical
test-192 are not identical

festutz added 15 commits April 14, 2022 16:57

WIP: first changes for new annotations

af308e6

Make parallelization use mapper info from new annotations

a3fd11d

Make parallelization use mapper info from new annotations

20bc389

Delete com_mapper field in dfg_node and use the info from new annotat…

c22ec79

…ions

Use input info from new annotations

65d2580

Use output info from new annotations

95de306

Make dfg_options use the information from new annotations and complet…

a1d7e88

…ely remove use of old aggregator com_aggregator

Make to_ast work for wf.sh with some hacks, e.g. to_ast for eager to …

7627d4d

…handle special case of intermediate file as last operand and fixed parsing issue for newline

Remove com_mapper and com_aggregator from DFGNode

296bdc9

WIP: incorporating remodelled command invocations

cf9438b

Parsing with new dataflow node model works

2f83af0

WIP: parallelization

00462e6

Rudimentary parallelization with new annotations works

a1ec6bd

Add way to specify where to find repository for annotations repository

02b6031

Do not require flag r_split since we do consecutive chunks for now

26bab26

festutz requested a review from angelhof June 23, 2022 19:19

angelhof approved these changes Jun 23, 2022

View reviewed changes

festutz added 4 commits June 24, 2022 12:25

Merge remote-tracking branch 'origin/future_annotations' into new_ann…

6f8ed01

…otations # Conflicts: # compiler/ir.py # compiler/pash_runtime.py

1st part of changes due to comments for PR

8d9fbf7

minor fix

ed2f411

2nd part of changes due to comments for PR

f1dade0

angelhof marked this pull request as ready for review June 27, 2022 17:38

angelhof merged commit 613d090 into future_annotations Jun 27, 2022

angelhof deleted the new_annotations branch June 27, 2022 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First PR for new annotations and new model for DFG nodes #583

First PR for new annotations and new model for DFG nodes #583

festutz commented Jun 23, 2022

angelhof left a comment

github-actions bot commented Jun 27, 2022

First PR for new annotations and new model for DFG nodes #583

First PR for new annotations and new model for DFG nodes #583

Conversation

festutz commented Jun 23, 2022

angelhof left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 27, 2022