✨ Brand New Antlers Engine ✨

[JohnathonKoster]

This PR provides a complete rewrite of the Antlers parser to separate it into two pieces:

• An Antlers parser: parses input strings into a list of nodes that can be evaluated to produce a final string
• An Antlers Runtime: a virtual execution environment to evaluate parser nodes

Important Note: While some new Antlers features look like PHP, Antlers is still not PHP :)

Issues Accounted For

This PR resolves, or makes improvements for, the following issues:

• Closes ensure_right doesn't work with |  #4249 - ensure_right doesn't work with |

Developers may use shorthand syntax to supply the vertical pipe as an argument: {{ title | ensure_right:"|" }}

Developers utilizing parameter-style modifiers will still need to use the HTML entity version: {{ title ensure_right=" |" }}

• Closes Logical operator '||' seen as modifier in some cases #4099 - Logical operator || seen as modifier in some case
• Closes where doesn't work on terms #3636 - where doesn't work on terms
• Closes in_array or contains not working as expected #4066 - in_array or contains not working as expected
• Closes Passing data through a partial to a grandchild partial results in "NULL" #2671 - Passing data through a partial to a grandchild partial results in "NULL"
• Closes {{ select_var:label }} is not working #2066 - {{ select_var:label }} is not working
• Closes Can't render Antlers in content #1798 - Can't render Antlers in content
• Closes Yield and section tags are affected by order #3286 - Yield and section tags are affected by order
• Closes Modifier doesn't accept parameter #2505 - Modifier doesn't accept parameter

Any tag/expression can be returned from an Antlers sub-expression. The {{ title translate="{site:short_locale}" }} syntax will work, but the variable reference will not.

• Closes multi condition in if statement does not work #3624
• Closes starts_with doesn't accept £ param strings #5128
• Closes Shorthand if/else doesn't work with pagination #3980 - Shorthand if/else doesn't work with pagination
• Closes Antlers tag is rendered in bard code block #3631 - Antlers tag is rendered in bard code block
• Closes Noparse issue with markdown fields #2161 - Noparse issue with markdown fields
• Closes modifier & logical operators in conditions #2655 - modifier & logical operators in conditions
• Closes Some Antlers logic not working combining modifiers #2722 - Some Antlers logic not working combining modifiers
• Closes {{ if {collection:count in="collection_name" taxonomy:year="2019"} > 0}} throws an error when taxonomy condition is added #1945 - {{ if {collection:count in="collection_name" taxonomy:year="2019"} > 0}} throws an error when taxonomy condition is added
• Closes Antlers if-statements do not evaluate tags correctly when using more than one #3381 - Antlers if-statements do not evaluate tags correctly when using more than one
• Closes Using double quoted parameters for modifiers in Antlers if condition throws error about unexpected double-quote #3610 - Using double quoted parameters for modifiers in Antlers if condition throws error about unexpected double-quote
• Closes {{ if (collection == "pages") && (parent:slug != "home") }} causes error #2001 - {{ if (collection == "pages") && (parent:slug != "home") }} causes error
• Closes simple OR/AND condition does not work  #3685 - simple OR/AND condition does not work
• Closes Using ?? instead of or causes form tag parser weirdness #3835 - Using ?? instead of or causes form tag parser weirdness

These two are now equivalent: { foo || 'contact' } and { foo or 'contact' }

• Closes Can't access view's array variable with dynamic key #3790 - Can't access view's array variable with dynamic key
• Closes Can't use old variable with dynamic key #3514 - Can't use old variable with dynamic key
• Closes Collection tag flawed. Maybe deeper Antlers/Syntax issues? #3097 - Collection tag flawed. Maybe deeper Antlers/Syntax issues?

For 3097, the syntax should preferentially use single-braces inside the tag, but double braces will technically work here:

{{ collection from="faq" limit="3"
  faq_categories:contains="{faq_categories:0:id}"
  :id:isnt="id"
  sort="{ randomizer ? 'random' : 'order' }"
}}

Adds test coverage for #2529 (Can't use modifiers in array syntax). Cannot repro in current 3.2 branch, however.

Cannot reproduce #4993 with Runtime parser. Partial tag receives expected parameters.

Ideas Implemented

• Fixes Named slots for partials ideas#492 - Named slots for partials

Partial: partials/_card.antlers.html:

<div class="max-w-sm rounded overflow-hidden shadow-lg">
    <img class="w-full" src="image/path.jpg" alt="{{ title }}">
    <div class="px-6 py-4">
        <div class="font-bold text-xl mb-2">{{ title }}</div>
        <p class="text-gray-700 text-base">{{ slot }}</p>
    </div>
    <div class="px-6 pt-4 pb-2">
        {{ if !slot:bottom }}
        <span class="inline-block bg-gray-200 rounded-full px-3 py-1 text-sm font-semibold text-gray-700 mr-2 mb-2">#tag1</span>
        <span class="inline-block bg-gray-200 rounded-full px-3 py-1 text-sm font-semibold text-gray-700 mr-2 mb-2">#tag2</span>
        <span class="inline-block bg-gray-200 rounded-full px-3 py-1 text-sm font-semibold text-gray-700 mr-2 mb-2">#tag3</span>
        {{ else }}
            {{ slot:bottom }}
        {{ /if }}
    </div>
</div>

Other code:

{{ partial:card }}

{{ slot:bottom }}
    <span>I'll appear in the bottom section.</span>
{{ /slot:bottom }}

Default slot content.
{{ /partial:card }}

• Fixes user:can should be a conditional ideas#479 - user:can should be a conditional

Any Antlers tag/expression can now be used in a condition (even if has spaces, or HTML-like parameters):

{{ if {user:cant do="edit projects collection"} }}

{{ else }}

{{ /if }}

• Fixes Output array from custom tag as var rather than iterable tag pair ideas#364 - Output array from custom tag as var rather than iterable tag pair
  • Results of tags (or any Antlers) can be assigned to variables if from an interpolation region:
    • {{ articles = {collection:articles} }}
    • {{ data = {my_tag_pair_data} }}
• Fixes YAML front matter with Antlers ideas#200 - YAML front matter with Antlers
  • Custom variables now takes its place
• Fixes Temporary Variables in Antlers ideas#41 - Temporary Variables in Antlers

Backwards Compatibility

• All existing Antlers tests pass when targeting the updated runtime

There are some instances where backwards compatibility has been broken (Antlers tags nested inside other tags, and anything that is currently taking advantage of parser bugs).

Great care has been taken to break as little as possible, but accounting for every possible scenario in the wild is not possible. Because of this, existing Antlers has not been removed, and can be used on existing projects.

Creating Variables

This PR allows developers to create variables within Antlers:

{{ total = 0 }}

{{ loop from="1" to="10" }}
    {{ total += 1 }}
{{ /loop }}

<p>The total is: {{ total }}</p>

Sub-expression/interpolated regions can also be assigned to variables:

{{ pages = {collection:pages limit="5"} }}

<p>Pages:</p>

{{# Use pages as if it were the actual tag #}}
{{ pages }}
    {{ if no_results }}

    {{ /if }}
{{ /pages }}

Creating Arrays

Developers may create arrays within Antlers if they want to (although getting them from a view model, view composer, etc. is still preferred):

{{ myarray = arr('one', 'two', 'three') }}

Associative arrays are also supported:

{{
    my_array = arr(
                    "one" => 1,
                    "two" => 2,
                    "three" => arr(
                        1,
                        2,
                        3,
                        4 => arr(
                            1,
                            2
                        )
                    ))
}}

Calling Methods

Developers may now call methods on objects using the {{ VARNAME:METHOD_NAME() }} syntax:

{{ object:method() }}

Developers may also pass arguments to methods:

{{ object:method('arg1', 'arg2', 'etc') }}

Additional Improvements

• Can now use nested parenthesis to get as complex as you want
• Behavior of {} single brace sub-expressions (interpolations) as well as nested parenthesis is now deterministic
• Predictable/deterministic tag pairing algorithm
• Developers can now use self-closing Antlers tags: {{ myarray | length /}}
• Better parser error messages
• Internal runtime caching for augmentation (only persists for the current request)
• Large and extensive test suite
• Introduces a once tag-like to ensure that a part of a template only executes once
• Adds support for template stacks and queues
• Adds ways to tell the runtime how to resolve variable and tag name collisions
• Improves escaped content handling
• A low-level runtime "tracing" API that allows developers to create very complex integrations with third party tools
• Configurable block-list to prevent variable name patterns, tags, or modifiers from being executed
• And a whole lot more

Upgrading/Downgrading

By default, all existing sites and all new sites will continue to use the regex Antlers parser.

To upgrade, set the statamic.antlers.version configuration option to runtime.

In config/statamic/antlers.php:

<?php

return [


    'version' => 'runtime',

    // ...
];

To downgrade to "legacy" Antlers, set the value to regex (new default):

In config/statamic/antlers.php:

<?php

return [


    'version' => 'regex',

    // ...
];

[jasonvarga]

Epic.

[robdekort]

Insane but so welcome. ❤️

[jelleroorda]

You’re my personal hero 🦸‍♂️ , this is epic!

[FrittenKeeZ]

How will this affect performance?

[jasonvarga]

I personally haven't benchmarked it yet but apparently there's a huge improvement.

[JohnathonKoster]

How will this affect performance?

For the vast majority of situations the parsing time is negligible now. As Jason mentioned, there will need to be benchmarks done :)

There are numerous things going on under the hood to help optimize performance:

1. Internal parser cache: remembers details about previously seen items (i.e., it is not uncommon to see {{ url }} and {{ title }} many times. The details of these are cached, and only the new positions are updated, for example). This technique is also utilized heavily in loops where the body of the loop is always only ever parsed once, and reused across all elements of the array.
2. Runtime augmentation cache: asking for repeated information on an object that required a toAugmentedArray() call will have that initial call cached in a smart way (regardless of original tag pairs, etc.) - this alone has positive results for asset and entry heavy sites
3. Dynamic parser and analysis strategies (an example of this would be in the conditional pairing logic where it will shift its strategy depending on what it thinks the final result may look like [i.e., indexes are slower on small arrays, but are worth it on very complicated/large nested if statements])
4. Dynamic multi-byte support: analyzes upcoming input to see if str_split would be fine to use over mb_str_split (and related functions) for performance and memory improvements

Of course, the same advice holds as always to help improve performance:

1. Keep computationally expensive work out of Antlers: do complicated things in PHP, and use a view model, view composer, etc.
2. Deeply nested tag structures will still cause internal context switching, which will cause a lot of internal stack creation, and hurt performance over the long run

Hard numbers are difficult to throw around since it also depends on the execution environment, but on the limited number of complex sites that have been tested during development, a positive improvement has been seen. Undoubtedly there will be room for improvement :)

tl;dr: The compiled PHP from Blade will still win, but new Antlers gets very close to those speeds when sticking to the language features. The performance cost of actually getting your page's data and assets will be the bottleneck in the vast majority of cases.

There will also be new tooling available within the runtime itself, and the VS Code extension that will allow developers to see how much time each block of code is contributing to the total runtime :)

[FrittenKeeZ]

That sounds very promising, as we have a quite entry and asset heavy site - I'm still in the process of migrating from v2, but any performance improvements are welcome :)

[ebeauchamps]

Wow. Just wow. Is this 4.0 time already?

[hmalaud]

Exciting 🤩
Really.

[jasonvarga]

@JohnathonKoster Ignore the test failures for now - I'm working on those still.

[jasonvarga]

Alright the tests are updated.

Now our old ParserTest and your new TemplateTest will both use the exact same tests, they are shared through a trait.

Now there are a handful of failures throughout the entire suite, and they're all for the same reason: #5238

The parser should be able to handle QueryBuilder instances, and automatically call ->get() on them to convert them from a query to a collection of items.

There's a few new tests in TemplateTest.php that fail regarding query builders. If you can fix those, the other tests throughout the whole suite should resolve themselves.

[JohnathonKoster]

Sounds great! Thanks for getting the tests consolidated. I will take care of the failing items later this evening my time :)

[jasonvarga]

🥁 ...

[RafaelKr]

YES! 🚀 Been following this for a long time now. How will you celebrate this epic moment?
Awesome work guys! Didn't test it yet, but I'm sure it was absolutely worth the effort!

[RafaelKr]

@michaelr0 here's a pretty good explanation of why Antlers is so powerful and important: statamic-3-dot-3.netlify.app/new-antlers-parser/#about. Many of things things wouldn't be possible with flat, compiled PHP.

@jackmcdade the new docs state A smarter, more forgiving matching engine so more things Just Work™".

Could someone elaborate what exactly this means? I'm a big fan of it myself, for example, when syntax-related things are more opinionated. So it doesn't matter whoever writes the Code in the end it will be very similiar - this leads to better readability and therefore better maintainability. This is one of the reasons the programming language Go was designed with very few keywords, so it's as simple as possible but of course still flexible enough to accomplish any task.

If "more things Just Work" means there are less bugs and discrepancies it's a good thing, of course!

[JohnathonKoster]

The easiest way I can think to explain the "more forgiving matching engine" is something like this:

{{ variable raw="true" }}

{{ /variable }}

{{ variable | upper }}
    {{ variable /}}
{{ /variable | upper }}

The updated engine would match those two pairs correctly, while the current engine may produce output similar to {{ /variable | upper }} (note that the modifier usage on the pair is not technically correct - but does reproduce the issue :) ).