Syntactical feasibility/implementation advice

[samuelgoto]

Per discussion on this issue, kicking off a separate one to ask advice on syntax feasibility and implementation strategies.

I started exploring a DST for javascript along the lines of kotlin builders, JFX and JSX. I've only written some high level ideas and started kicking off a sweet.js macro implementation prototype.

Can you help me sanity check (a) whether the syntax here is feasible or not (i.e. does it create ambiguity with the JS grammar (you can make comments here directly if you'd like)? and if so, what are the most common strategies to avoid them) and (b) once we settle on a feasible design what's the best way to go about it wrt sweet.js?

[disnet]

I'm excited you're looking into this! Definitely a perfect area to explore with macros.

Unless you want to go down exactly the JSX route I don't think you necessarily need readtables. My understanding is the only reason @jlongster used readtables back when he did jsx-reader is because JSX introduces some specific lexical problems like </div> confusing the regular expression tokenizer.

Does {} conflict with object literals or with for-if-switch-{}-like statements? Would it be impossible to disambiguate?

Yep:

let x = { a };

is that a doctree or an object literal shorthand?

Bare delimiters are tricky, both from a macro implementation perspective and an ambiguity perspective.

I like your:

doc {
  div {
    span {
      "hello world"
    }
  }
}

syntax best. The doc identifier makes it pretty straightforward to implement as a macro. The doc macro can just consume everything inside its delimiters and parse appropriately.

If you go with this syntax you might have some fun issues with ASI inside a doctree to figure out:

doc {
  div 
  {
    "hello"
  }
}
// should you interpret as
doc {
  div;
  {
    "hello"
  }
}
// or
doc {
  div {
    "hello"
  }
}

Otherwise, if I'm understanding your thinking correctly something like that syntax should work out.

[samuelgoto]

The doc identifier makes it pretty straightforward to implement as a macro.

I generally the doc {}-like syntax too, but I wasn't sure how to go to code and come back to trees. For example:

doc {
  div {
    for (user in users) {
      // is something as short as this possible?
     span { `${user.name}` }

      // or do i need to use the doc {} be recursively?
      doc { span { `${user.name}` } }
    }
  }
}

[samuelgoto]

Yep:
let x = { a };
is that a doctree or an object literal shorthand?

Yep, I was a little worried about that. Note that there is a token that goes before {} that could disambiguate, i.e. I'd think one can make a distinction between:

let x = {a}; // object literal
let y = span { a }; // doctree

A couple of questions:

(a) Wouldn't the span token before the {} give me the ability to differentiate between the two?
(b) How does kotlin/groovy get away being able to parse builders?

fun result(args: Array<String>) =
    html {
        head {
            title {+"XML encoding with Kotlin"}
        }
        body {
            h1 {+"XML encoding with Kotlin"}
            p  {+"this format can be used as an alternative markup to XML"}

            // an element with attributes and text content
            a(href = "http://kotlinlang.org") {+"Kotlin"}

            // mixed content
            p {
                +"This is some"
                b {+"mixed"}
                +"text. For more see the"
                a(href = "http://kotlinlang.org") {+"Kotlin"}
                +"project"
            }
            p {+"some text"}

            // content generated by
            p {
                for (arg in args)
                    +arg
            }
        }
    }

[samuelgoto]

(b) How does kotlin/groovy get away being able to parse builders?

Wow, just reading more closely what Kotlin seems to be doing, it seems like the trick here is a syntactic sugar that allows some higher-order functions (specifically, high-order functions that takes functions as the last parameter) to be called in the form {}. Kinda of like if this was typescript, something along the lines:

// typescript
function a(init: () => void) {
}

// traditional function call with arrow functions:
a(() => {
  // hello world
});

// but kotlin adds this syntactical extension to also allow:
a {
  // hello world
}

// Both of these calls are semantically equivalent.

Since the last parameter is a function, you add nesting by biding the parent to "this". For example:

function a(init: () => void) {
  var result = new A();

  var result = new A();
  // calls the lambda function that was passed as a parameter to an instance
  // of the class A, which has a b() method.
  init.call(a);
  return result;
}

class A {
  b(init: () => void) {
  }
}

// such that

a {
  // implicitly, this is this.b(() => {})
  b {
  }
}

// is equivalent to

a(() => {
  this.b(() => {
  });
});

Because the lambdas are functions, you can embed whichever statements you'd like (e.g. for loops).

Kind of a neat trick. Wondering: is this something that can be prototyped with sweet.js?

[gabejohnson]

is this something that can be prototyped with sweet.js?

Absolutely. Seems like you could just define a bunch of macros and dispatch to functions based on MacroContext#name. However, I don't know that I would create a bunch of classes. Functions alone should do what you want.

[samuelgoto]

Absolutely. Seems like you could just define a bunch of macros and dispatch to functions based on MacroContext#name. However, I don't know that I would create a bunch of classes. Functions alone should do what you want.

hummm unclear to me how I'd accomplish that with macros ...

Are you saying that the syntactical simplification provided by this:

https://kotlinlang.org/docs/reference/lambdas.html#higher-order-functions

Would be possible to implement with sweet.js?

Specifically, are you saying that it would be possible for me to, with sweet.js, enable the following syntax?

a {
  // foobar
}

to be transpiled into

a(() => {
  // foobar
})

For any a (i.e. without hard-coding a)?

Would it be possible to make a distinction too when parameters are available? E.g.

a(b) {
  // foobar
}
// to be transpiled as
a(b, () => {
  // foobar
});

Can you be more specific by what you mean by MacroContext#name?

[samuelgoto]

from the little that i'm playing with the editor and the reference documentation, it doesn't seem like i'd be able to accomplish this with the current affordances. it seems sweet.js offers me two matching opportunities:

• keywords (with macros/syntax)
• operators

I'd think I'd need something along the lines of

syntax (IdentifierExpression, BlockStatement) =>
  // transform the AST

to add the ability to open a BlockStatement after an IdentifierExpression to legitimize:

a {
}

right?

On the other hand, JSX managed to get access to the internals of sweet and get into somewhat what i think would make things possible.

let _DOM = macro {
  rule { { $a . $b $expr ... } } => {
    _DOM_member { $a . $b $expr ... }
  }

  rule { { $el $attrs } } => {
    $el($attrs)
  }

  rule { { $el $attrs , } } => {
    $el($attrs)
  }

  rule { { $elStart $attrs $($children:expr,) ... } } => {
    $elStart($attrs, $children (,) ...)
  }

  rule { } => { _DOM }
}

is this the macro that was referred to earlier? and if so, (a) is there documentation that i could use to follow how this works (doesn't seem supported here?) and (b) does that mean that I won't be able to use the online editor?

[gabejohnson]

@samuelgoto what you need is something along the lines of

import {unwrap, fromStringLiteral} from 'sweet.js/helpers' for syntax;

syntax a = ctx => {
  const name = unwrap(ctx.name()).value;
  const dummy = #`d`.get(0);
  const nameStr = fromStringLiteral(dummy, name);
  const body = ctx.next().value;
  return #`
  tags[${nameStr}](() => ${body});
  `;
}

a { console.log('in a') }

You can also check to see whether the first value returned from a call to ctx.next is an argument list or a body and build the template conditionally

return #`tags[${nameStr}](${args}, () => ${body})`;

If you haven't already done so, go through the tutorial https://www.sweetjs.org/doc/tutorial

And feel free to ask questions in the gitter room https://gitter.im/sweet-js/sweet.js

[gabejohnson]

There's also nothing in the a macro specific to a so you could make this more generic with a helper imported from another file.

// tagHelper.js
'lang sweet.js';

export const tagMacro = ctx => {
  const name = unwrap(ctx.name()).value;
  const dummy = #`d`.get(0);
  const nameStr = fromStringLiteral(dummy, name);
  const body = ctx.next().value;
  return #`
  tags[${nameStr}](() => ${body});
  `;
}

export const tags = {
  a: (fn, ...props) => ...,
  div: (fn, ...props) => ...,
  ...
};

// tags.js
import { tagMacro } from './tagHelper' for syntax;

syntax a = tagMacro;
syntax div = tagMacro;
...
export { a, div, ... };

@disnet please correct me if I've missed something.

[samuelgoto]

There's also nothing in the a macro specific to a so you could make this more generic with a helper imported from another file.

I think this is the challenging part: I think this does make it specific to a (even if you refactor the code to loop through tags). This approach still requires me to enumerate all of the possible tags, right?

syntax a = tagMacro;
syntax div = tagMacro;
...

But what if one wants to invent new tags? Or compose them? For example, both JSX as well as Kotlin enables you to write:

var doc = MyOwnTag {
  AnotherTagThatDidNotExistPreviously {
    ThisOneIsntAvailableAtCompilineTime {
    }
  }
}

Along the lines of web-components:

 <X-MyOwnTag>
  <X-AnotherTagThatDidNotExistPreviously>
    <X-ThisOneIsntAvailableAtCompilineTime/>
    </X-AnotherTagThatDidNotExistPreviously>
 </X-MyOwnTag>

[gabejohnson]

You'd either have to create a macro

syntax MyOwnTag = tagMacro;
tags.MyOwnTag = ...

or create a new language once readtables are exposed as discussed in #687

[disnet]

For custom tags (tags that aren't pre-defined) you could have a parameterized macro:

import X_ from 'tag-macros';

var doc = X_(MyOwnTag) {
  X_(AnotherTagThatDidNotExistPreviously) {
    X_(ThisOneIsntAvailableAtCompilineTime) {
    }
  }
}
``