Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
"1:20pm. Came back for a bit so I can have breakfast. I did figure ou…
…t a bunch of thing. Let me continue reading vol 4 of the GGO novel for a bit and I will get back to chaos meditating.
1:35pm. Ok, let me get back to bed. I do not feel like writing about what comes next just yet. I need to focus on raising my tension internally.
4pm. The weather was bad recently so my timing to get away from the screen for a while was good.
4:05pm. Ok, I've realized various things. Let me put my thoughts together.
1) Type wildcard.
2) Paired symbol pattern.
3) Semantic group enum.
4) Reference dictionary extension.
These 4 things should not mean anything to anybody just dropping in, but let me go at them in turn. Let me do some programming here. Apart from the last one, I can do the first 3 items quickly enough now.
Let me do the second one first.
```fs
let pat_symbol_paired =
(many1 (read_symbol_paired .>>. opt root_pattern_pair) |>> fun l ->
match l |> List.map (function a, b -> a, defaultArg b (PatVar a)) |> List.unzip with
| (k :: k' as keys), v ->
let body keys =
let b = Text.StringBuilder()
List.iter (fun (x : string) -> b.Append(x).Append('_') |> ignore) keys
List.reduceBack (fun a b -> PatPair(a,b)) (PatSymbol (b.ToString()) :: v)
if Char.IsLower(k,0) then body keys else body (to_lower k :: k') |> PatUnbox
| _ -> failwith "Compiler error: Should be at least one key in symbol_paired_process_pattern"
)
<|> pat_pair
```
Ah, I did not fuck this up. Ok, good. I was afraid for a bit there.
You know, I should do the same on the value level.
Having to write `(Some: x)` with a parenth instead of normally does not sit right with me. It should be right after statements.
I'll have to do an indentation based check. I'll do the same thing in types and get rid of the flag.
4:15pm. I think my problem with paired symbol is that I am obsessesed with opening a new scope every time I enter one, but I can just restrain myself a little when it comes to this.
Yeah.
Ok...let me work on that then for the rest of the day. I should be able to manage this.
```fs
and root_type allow_metavars allow_value_exprs allow_forall allow_symbol_paired d =
```
Agh, this flag.
```fs
let symbol_paired d =
let i = col d
(pipe2 (read_symbol_paired_big .>>. next) (many (indent i (<=) read_symbol_paired .>>. next)) (fun a b ->
let k,v = List.unzip ((to_lower (fst a), snd a) :: b)
let b = Text.StringBuilder()
List.iter (fun (x : string) -> b.Append(x).Append('_') |> ignore) k
List.reduceBack (fun a b -> RawTPair(a,b)) (RawTSymbol (b.ToString()) :: v))) d
```
Let me do it like this.
...Hmmm, just like with operators, let me relax the rule just a bit.
```fs
(pipe2 (read_symbol_paired_big .>>. next) (many (indent (i-1) (<=) read_symbol_paired .>>. next)) (fun a b ->
```
This will allow me to write things like...
```
(a: 1
b: 2)
```
Now let me do the same thing...
No actually, let me make it the same the pattern.
```fs
let symbol_paired d =
let i = col d
let next = forall
(pipe2 (read_symbol_paired_big .>>. next) (many (indent (i-1) (<=) read_symbol_paired .>>. next)) (fun a b ->
let k,v = List.unzip ((to_lower (fst a), snd a) :: b)
let b = Text.StringBuilder()
List.iter (fun (x : string) -> b.Append(x).Append('_') |> ignore) k
List.reduceBack (fun a b -> RawTPair(a,b)) (RawTSymbol (b.ToString()) :: v))
<|> next) d
symbol_paired d
```
This is kind of the rough work of language design. I have to optimize for the commonest use case.
```fs
let symbol_paired d =
let next = operators
let i = col d
let pat = indent (i-1) (<=) read_symbol_paired .>>. opt next
(many1 pat |>> fun l ->
let a,b = List.unzip l
let x,x' = match a with x::x' -> x,x' | _ -> failwith "Compiler error: Expected `many1 pat` to produce an element."
let is_upper = Char.IsUpper(x, 0)
let a = if is_upper then to_lower x :: x' else a
let b = List.map2 (fun a b -> defaultArg b (v a)) a b
let a =
let sb = Text.StringBuilder()
a |> List.iter (fun x -> sb.Append(x).Append('_') |> ignore)
sb.ToString()
if is_upper then ap (v a) (List.reduceBack (binop TupleCreate) b)
else List.reduceBack (binop TupleCreate) (RawSymbolCreate a :: b)) d
let statements d =
let next = symbol_paired
```
This should be good.
Ah, no wait, I forgot the alternative.
```fs
let symbol_paired d =
let next = operators
let i = col d
let pat = indent (i-1) (<=) read_symbol_paired .>>. opt next
((many1 pat |>> fun l ->
let a,b = List.unzip l
let x,x' = match a with x::x' -> x,x' | _ -> failwith "Compiler error: Expected `many1 pat` to produce an element."
let is_upper = Char.IsUpper(x, 0)
let a = if is_upper then to_lower x :: x' else a
let b = List.map2 (fun a b -> defaultArg b (v a)) a b
let a =
let sb = Text.StringBuilder()
a |> List.iter (fun x -> sb.Append(x).Append('_') |> ignore)
sb.ToString()
if is_upper then ap (v a) (List.reduceBack (binop TupleCreate) b)
else List.reduceBack (binop TupleCreate) (RawSymbolCreate a :: b))
<|> next) d
```
I meant to do this.
Now let me move to testing this.
4:35pm.
```
instance bind option a : forall b. x f =
match x with
| Some: x => Some: f x : Some: i32
a: q
| None => None
```
Hmmm, why does this parse?
Ah, it probably gets parsed as two statements.
```
instance bind option a : forall b. x f =
match x with
| Some: x => (Some: f x : Some: i32)
a: q
| None => None
```
Like this.
Ok, it makes sense.
```
let g x =
a: 1
inl x = 1
a: 2
```
That last token being dedented gives the compiler error message, even though it is really not a compiler error.
Forget that.
```fs
else Error [fst s.tokens.[s.Index], ExpectedEob]
```
Let me replace the unknown error with this.
Ok, let me go with this. It is not ideal, but I do not know of a better way.
That is paired symbols done again. Now their parsing is consistent everywhere.
4:50pm. Next.
1) Type wildcard.
I forgot about wildcards in types.
```fs
and RawTExpr =
| RawTWildcard
```
Let me add this to the union type.
```fs
and root_type allow_metavars allow_value_exprs allow_forall d =
let next = root_type allow_metavars allow_value_exprs allow_forall
let allow_wildcard = allow_metavars || allow_forall
let cases d =
let wildcard d = if allow_wildcard then (skip_keyword SpecWildcard >>% RawTWildcard) d else Error []
```
Plugged this in.
Let me try it out.
```
let g x =
typecase i32 with
| _ => 1
```
Yeah, it works. Nice.
Let me get to item number 3.
3) Semantic group enum.
```fs
let token_groups = function
| TokVar _ -> 0 // variable
| TokSymbol _ | TokSymbolPaired _ -> 1 // symbol
| TokValue(LitString _) -> 2 // string
| TokValue _ | TokDefaultValue -> 3 // number
| TokOperator _ -> 4 // operator
| TokUnaryOperator _ -> 5 // unary operator
| TokComment _ -> 6 // comment
| TokKeyword _ -> 7 // keyword
| TokParenthesis _ -> 8 // parenthesis
```
Finally found this.
This should be an enum.
```fs
type SemanticTokenLegend =
| variable = 0
| symbol = 1
| string = 2
| number = 3
| operator = 4
| unaryOperator = 5
| comment = 6
| keyword = 7
| parenthesis = 8
| typeVariable = 9
```
Let me plug this in.
...Hmmm, I was sure there was a type variable in there somewhere.
```fs
type SemanticTokenLegend =
| variable = 0
| symbol = 1
| string = 2
| number = 3
| operator = 4
| unary_operator = 5
| comment = 6
| keyword = 7
| parenthesis = 8
| type_variable = 9
```
It is also strange, I thought that `_` was forbidden in ids, but I see that it is now.
```
typeParameter
```
Ah, there was this. But is not in the other table, so nevermind it.
I got this out of the way, and I even added to the legend.
5:05pm.
4) Reference dictionary extension.
The next thing is this. What I am going to do here is finish the semantic tokenization properly.
I thought of doing this work in different places, but I want parsing to be consistent.
I thought it might be more elegant to do a proper pass to get variable information based off ranges, but in the end, the easisest way to get proper grouping is to just hack the parser and be done with it.
Let me do it.
```fs
let token_groups (overrides : Collections.Generic.Dictionary<string,SemanticTokenLegend>) =
let f def x = match overrides.TryGetValue(x) with true,v -> v | _ -> def
function
| TokVar x -> f SemanticTokenLegend.variable x
| TokSymbol x | TokSymbolPaired x -> f SemanticTokenLegend.symbol x
| TokValue(LitString _) -> SemanticTokenLegend.string
| TokValue _ | TokDefaultValue -> SemanticTokenLegend.number
| TokOperator _ -> SemanticTokenLegend.operator
| TokUnaryOperator _ -> SemanticTokenLegend.unary_operator
| TokComment _ -> SemanticTokenLegend.comment
| TokKeyword _ -> SemanticTokenLegend.keyword
| TokParenthesis _ -> SemanticTokenLegend.parenthesis
```
Here is the plan.
5:10pm.
```fs
| GetRange((a,b),res) -> // It is assumed that a.character = 0 and b.character = length of the line
let from, near_to = a.line, b.line+1
vscode_tokens from (lines.GetRange(from,near_to-from).ToArray()) |> IVar.fill res
```
Agh, this is not good. How do I get the dictionary here?
```fs
let token_groups = function
| TokVar (_,r) | TokSymbol (_,r) | TokSymbolPaired (_,r) -> !r
```
Let me do this.
5:25pm.
```fs
let var = read_var' |>> fun (x,r) ->
r := SemanticTokenLegend.type_variable
match x with
```
This is hacky as shit, but it will work. Let me test this first extension.
```fs
let g (x : qwe) =
1 : i32
```
Great, the type variable is in green here.
Now I just need to take care of the paired symbol patterns.
```fs
let g (x : qwe) =
inl f = function
| a:b:c: => 1
()
```
Let me start with the stuff in root pattern.
```fs
let pat_symbol_paired =
(many1 (read_symbol_paired' .>>. opt root_pattern_pair) |>> fun l ->
let f ((a,r),b) = a, Option.defaultWith (fun () -> r := SemanticTokenLegend.variable; PatVar a) b
match l |> List.map f |> List.unzip with
```
This should do it here.
```fs
let g (x:e) =
inl f = function
| a:x b:q c: => 1
()
```
Yeah, it works correctly. Let me also do it in `root_term` and I am done.
5:35pm.
```
let g (x:e) =
inl f = function
| a: b:q c: => 1
q: 1 w: e: r: x
```
This is great. Everything here is highlighted correct. `w:` is in blue and `r:` is in yellow.
5:40pm. Now my beautiful and clean token array which was functionaly pure before has the stentch of mutation on it, and code is coupled in ways it should not be, but nevermind this.
I am not getting graded on functional purity, but on real life functionality.
I picked the minimum effort path to full semantic highlighting and I'll be proud of it. Doing it this way is probably the most efficient way as well.
5:45pm. This mutation I've inserted into the AST does not affect the actual functionality of the compiler so it should be fine.
It would just be too much effort to write extra code and triangulate the overrides by ranges.
Let me stop here for the day. It is lunch time."- Loading branch information
