[inheritance.yml] Override parent with newline spec. Are block tags standalone?

[agentgt]

I cannot figure out where the spec says it is OK for the parent to remove new lines from parameters.

spec/specs/~inheritance.yml

Lines 146 to 152 in 5d3b58e

	- name: Override parent with newlines
	desc: Override parent with newlines
	data: { }
	template: "{{<parent}}{{$ballmer}}\npeaked\n\n:(\n{{/ballmer}}{{/parent}}"
	partials:
	parent: "{{$ballmer}}peaking{{/ballmer}}"
	expected: "peaked\n\n:(\n"

I expect (notice the starting newline char):

"\npeaked\n\n:(\n"

Are block tags standalone (and even then the above still wouldn't make since)?

In addition, when used inside of a Parent tag, the template text between a
Block tag and its matching End Section tag defines content that replaces the
default defined in the Parent template. This content is the argument passed
to the Parent template.

Where does it say chop off white space? Where does it say that BLOCK tags are standalone?

What probably needs to be explicitly said is:

"Parent and Block tags SHOULD be treated as standalone when appropriate."

[agentgt]

However if the block tags are standalone wouldn't the expected output be:

peaked\n\n:(?

Like I can't figure out why it removes the first newline only.

[jgonggrijp]

Very good, justified questions, @agentgt. You are right to point out that all of this is very confusing.

What probably needs to be explicitly said is:

"Parent and Block tags SHOULD be treated as standalone when appropriate."

Yes, I think that is what was intended. Although a block within a parent tag pair is a bit of a special case, anyway, where the usual concept of "standalone" does not fit entirely.

Essentially, this spec (and several others, for that matter) makes an implicit statement about what whitespace goes with the tag and what whitespace goes with the content. The first newline is removed because it belongs with the opening {{$ballmer}} tag and the last newline is retained because it belongs to the content. I think the original author of this spec had an intuition about why it should behave like this, but found it too difficult to formulate explicitly.

I have formulated explicit, generic rules for whitespace in parents and blocks in #131. Following those rules does indeed lead to an implementation that passes this particular spec (and all others in the inheritance module). Those rules also address what to do with reindentation when blocks are passed between templates. Please let me know whether you feel those rules would solve your question.

[jgonggrijp]

I'll expand a bit on the whitespace semantics of Mustache, as @agentgt indicated that he was still unclear on the matter in #131 (comment).

If we render the newlines in template as such, it looks like this:

{{<parent}}{{$ballmer}}
peaked

:(
{{/ballmer}}{{/parent}}

The test is essentially stating that every newline in this particular template belongs to whatever came immediately before it. So the newline directly after {{$ballmer}} belongs to that tag, while all other newlines belong to the literal content of the ballmer block. When you strip the tags, that means that the first newline disappears while all others remain:

peaked

:(

"Stripping the tags" also happens to be the net effect of a template that only passes a block to a parent that only expands said block.

To understand why every newline belongs to what precedes it in this template, we need to observe two things:

1. In general, we consider a newline character to belong to the preceding line, rather than the following line.
2. A block is an inline template. It is as we define a template file within a larger template file.

In both examples below, we intuitively understand Here to be the start of the nested template:

{{<parent}}
{{$block}}Here goes a template.{{/block}}
{{/parent}}

{{<parent}}{{$block}}
Here goes a template.
{{/block}}{{/parent}}

That is, in both cases we interpret the block as equivalent to an external template that does not start with a blank line:

block.mustache

Here goes a template.

In other words, we do not care whether there is a newline between the tag and Here. Here is always the start of the template, so if there is a newline before it, it must belong to the tag. Only when there is a second newline, we start to interpret the situation as if we're dealing with a template that actually starts with a blank line:

{{<parent}}{{$block}}

Here goes a template.
{{/block}}{{/parent}}

This is why we strip the first newline after the block open tag (if it's standalone or inside a parent tag pair), but retain any subsequent newlines if present.

We can apply similar reasoning to the end of the block. Suppose that parent.mustache repeats block for every element of a list.

{{#list}}{{$block}}Expecting a template here.{{/block}}{{/list}}

If we now look back at the two block examples from before,

{{<parent}}
{{$block}}Here goes a template.{{/block}}
{{/parent}}

{{<parent}}{{$block}}
Here goes a template.
{{/block}}{{/parent}}

we realize that we expect them to behave differently in the context of that repetition. The first variant of block does not contain any line breaks, so we expect every repetition to start directly after the previous:

Here goes a template.Here goes a template.Here goes a template.

The second variant does have a linebreak, so we also expect every repetition to be on a separate line:

Here goes a template.
Here goes a template.
Here goes a template.

The first is equivalent to an external template without a final linebreak, while the second is equivalent to an external template with a final linebreak. This is why we always retain linebreaks before the block end tag.

Does this answer your question?

[agentgt]

Yes it makes since and I feel like an idiot yesterday because I kept mentally reading:

\npeaked\n\n:(\n

as

\npeaked\n:(\n\n

You had ask in another thread (#131) why I would need to buffer till newline. Well the fundamental problem with the parser I inherited which I will eventually rewrite is that it is a single pass. Because blocks can be anywhere and the matching pair rule I have to read all of them till new line for example:

Data:

const data = {
    people: [
        {name: 'Alice'},
        {name: 'Bob'},
    ],
};

Template

{{#people}}{{$stuff}}{{$blah}}\n{{name}}\n{{/blah}}{{/stuff}}{{/people}}

Is entirely standalone correct? (that is apparently how wontache does it as I tested it using your lib)

Now using wontache again with this which I assumed was not standalone:

{{#people}}This should be on its own line {{$stuff}}{{$blah}}\n{{name}}\n{{/blah}}{{/stuff}}{{/people}}

We get (I added quotes to show start and end of output):

"This should be on its own line Alice
This should be on its own line Bob
"

I expected:

"This should be on its own line
Alice
This should be on its own line
Bob
"

Yeah so I'm still confused. I would consider the above a bug but I have to look more at #131.

In my current implementation I consider tags only standalone if a single tag is on a line with possible non newline whitespace before and after the terminating newline. Consequently I only need to buffer ~4 tokens max ( [text] [tag] [text] [newline] )

If I'm going to start matching block pair then I need to take all the tokens till newline or what I really should do is make it a multipass compiler (which given the block scoping rules of parent I will probably have to do).

[jgonggrijp]

Yes it makes since

Glad to hear that!

{{#people}}{{$stuff}}{{$blah}}\n{{name}}\n{{/blah}}{{/stuff}}{{/people}}

Is entirely standalone correct?

Actually, I don't think so. I think the current spec considers all these tags not standalone. It's a bit confusing, though; the block tags do nothing if you don't override them, and if they wouldn't be there, the section tags would be standalone. As it currently stands, just the fact that these tags are on the same line prevents them from being standalone. This is also something that I have previously discussed with @gasche and which we agreed should probably be approached differently in a hypothetical version 2.0 of the spec.

(that is apparently how wontache does it as I tested it using your lib)

This surprises me!

Now using wontache again with this which I assumed was not standalone:

{{#people}}This should be on its own line {{$stuff}}{{$blah}}\n{{name}}\n{{/blah}}{{/stuff}}{{/people}}

We get (I added quotes to show start and end of output):

"This should be on its own line Alice
This should be on its own line Bob
"

I expected:

"This should be on its own line
Alice
This should be on its own line
Bob
"

Yeah so I'm still confused. I would consider the above a bug but I have to look more at #131.

Strange! I can confirm Wontache is giving the first output, and I think that is a bug. Thanks for pointing it out to me. I'll look into what's going on and report back here when I've figured it out.

In my current implementation I consider tags only standalone if a single tag is on a line with possible non newline whitespace before and after the terminating newline.

That seems correct to me, I think this is what the spec dictates (other than the implicit special cases for parents and blocks, which I think I made explicit in #130/#131).

Consequently I only need to buffer ~4 tokens max ( [text] [tag] [text] [newline] )

Oh, by "buffer" you just mean "don't discard the tokens immediately"? In that case, I have to admit Wontache does a ton of buffering, since I keep the entire token stream in memory until I'm done parsing (even though I don't think I strictly need all of it until the end).

If I'm going to start matching block pair then I need to take all the tokens till newline or what I really should do is make it a multipass compiler (which given the block scoping rules of parent I will probably have to do).

You might need some form of multipass in order to implement reindentation. Arguably, that is what Wontache does (but cached). I don't think you need multiple passes to correctly implement block scoping rules; a single pass with a recursive parser should be enough. Determining whether a parent or block pair is standalone, per the rules proposed in #131, only requires that you collect sufficient information in the parse tree during that single pass.

[jgonggrijp]

Update: I found out what is wrong with Wontache and documented it here. Will be fixed in the next Wontache release.

[jgonggrijp]

Fix here: https://gitlab.com/jgonggrijp/wontache/-/merge_requests/4

Thank you for helping me to improve Wontache, @agentgt!

[agentgt]

@jgonggrijp I should be thanking you way more for your patience, guidance and expertise! You are doing a fantastic job of stewarding this spec and I'm greatly appreciative.

I still might have some questions and concerns about block scoping (#129). Ignoring the whitespace handling of blocks (which I presume is still mostly unofficial) my implementation passes all the test in the baseline (well ignoring dynamic things) with the giant exception of block scoping.

I presume #129 is the best place to put my questions or thoughts on block scoping?

[jgonggrijp]

Yes, #129 is the right place for block scoping questions.

Shall we close the current ticket?

[jgonggrijp]

Question seems answered, so closing. Comments still welcome, though.