feat: static variable analysis #770

jg-rp · 2024-11-16T12:45:12Z

Statically analyze templates and report variable usage.

Usage

Retrieve the names of variables used in a template with Liquid.variables(template). It returns an array of strings, one string for each distinct variable, without its properties.

import { Liquid } from 'liquidjs'

const engine = new Liquid()

const template = engine.parse(`\
<p>
  {% assign title = user.title | capitalize %}
  {{ title }} {{ user.first_name | default: user.name }} {{ user.last_name }}
  {% if user.address %}
    {{ user.address.line1 }}
  {% else %}
    {{ user.email_addresses[0] }}
    {% for email in user.email_addresses %}
       - {{ email }}
    {% endfor %}
  {% endif %}
<p>
`)

console.log(engine.variablesSync(template))

Output

[ 'user', 'title', 'email' ]

Alternatively, use Liquid.fullVariables(template) to get a list of variables including their properties. Notice that variables from tag and filter arguments are included too.

// continued from above
engine.fullVariables(template).then(console.log)

Output

[
  'user.title',
  'user.first_name',
  'user.name',
  'user.last_name',
  'user.address',
  'user.address.line1',
  'user.email_addresses[0]',
  'user.email_addresses',
  'title',
  'email'
]

Or use Liquid.variableSegments(template) to get an array of strings and numbers that make up each variable's path.

// continued from above
engine.variableSegments(template).then(console.log)

Output

[
  [ 'user', 'title' ],
  [ 'user', 'first_name' ],
  [ 'user', 'name' ],
  [ 'user', 'last_name' ],
  [ 'user', 'address' ],
  [ 'user', 'address', 'line1' ],
  [ 'user', 'email_addresses', 0 ],
  [ 'user', 'email_addresses' ],
  [ 'title' ],
  [ 'email' ]
]

Global Variables

Notice, in the examples above, that title and email are included in the results. Often you'll want to exclude names that are in scope from {% assign %} tags, and temporary variables like those introduced by a {% for %} tag.

To get names that are expected to be global, that is, provided by application developers rather than template authors, use the globalVariables, globalFullVariables or globalVariableSegments methods (or their synchronous equivalents) of a Liquid class instance.

// continued from above
engine.globalVariableSegments(template).then(console.log)

Output

[
  [ 'user', 'title' ],
  [ 'user', 'first_name' ],
  [ 'user', 'name' ],
  [ 'user', 'last_name' ],
  [ 'user', 'address' ],
  [ 'user', 'address', 'line1' ],
  [ 'user', 'email_addresses', 0 ],
  [ 'user', 'email_addresses' ]
]

Partial Templates

By default, LiquidJS will try to load and analyze any included and rendered templates too.

import { Liquid } from 'liquidjs'

const footer = `\
<footer>
  <p>&copy; {{ "now" | date: "%Y" }} {{ site_name }}</p>
  <p>{{ site_description }}</p>
</footer>`

const engine = new Liquid({ templates: { footer } })

const template = engine.parse(`\
<body>
  <h1>Hi, {{ you | default: 'World' }}!</h1>
  {% assign some = 'thing' %}
  {% include 'footer' %}
</body>
`)

engine.globalVariables(template).then(console.log)

Output

[ 'you', 'site_name', 'site_description' ]

You can disable analysis of partial templates by setting the partials options to false.

// continue from above
engine.globalVariables(template, { partials: false }).then(console.log)

Output

[ 'you' ]

If an {% include %} tag uses a dynamic template name (one that can't be determined without rendering the template) it will be ignored, even if partials is set to true.

coveralls · 2024-11-16T12:47:49Z

Pull Request Test Coverage Report for Build 12525259260

Details

325 of 325 (100.0%) changed or added relevant lines in 24 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.07%) to 99.848%

Files with Coverage Reduction	New Missed Lines	%
src/parser/tokenizer.ts	1	99.75%

Totals
Change from base Build 12525024184:	-0.07%
Covered Lines:	2853
Relevant Lines:	2855

💛 - Coveralls

src/tags/capture.ts

src/tags/cycle.ts

test/integration/static_analysis/variables.spec.ts

jg-rp · 2024-11-18T15:07:00Z

With the latest commit, I've changed several of the built-in tags to fix their row and column numbers reported from Token.getPosition().

I guess this is a good point to decide if having correct row and column is important enough to warrant these changes.

Notice that when parsing if, elsif and unless tags, we're now avoiding some string slicing. Instead, when new Value() is given a TagToken, we pass the entire input string and a range to Tokenizer, which I'm hoping will result in a performance boost rather than a performance penalty.

src/template/value.ts

test/integration/util/error.spec.ts

jg-rp · 2024-11-23T09:02:26Z

I've provisionally implemented some convenience analysis methods on the Liquid class. Please consider these (along with any other features) as ideas that can be removed before merging.

Other ideas that I've yet to implement:

Report names of Liquid filters and their locations in analysis results (I've seen people ask for this before).
Report names of tags and their locations in analysis results.
Add options to control partial template analysis. At the moment we throw an error if a partial template can't be loaded, and silently ignore partial templates included/rendered with a dynamic name.

jg-rp · 2024-12-04T11:47:39Z

@harttle 👋 , I've not forgotten about this or your previous comments about keeping track of aliases. I'll get back to it soon.

docs/source/tutorials/static-analysis.md

src/liquid.ts

jg-rp · 2024-12-22T16:47:24Z

Can you think of better names for fullVariables, globals, locals, and/or segments?

At the moment:

The name "global" or "global variables" is inspired by Python's built-in globals() function. In our case it means names added to a template's scope by application developers at render time. Global variables are available to the root template and any templates that it includes, including those rendered with the {% render %} tag.
The name "locals" or "template local variables" is inspired by Python's built-in locals() function. Here it means names that have been added to a template's scope from an {% assign %}, {% capture %}, {% increment %}, etc. tag.
"Segments" is a term used by RFC 9535. One can think of Liquid variables as paths, where each path is made up of one or more segments.
A "fullVariable" is roughly equivalent to a "normalized path" in RFC 9535.

test/integration/static_analysis/variables.spec.ts

src/liquid.ts

docs/source/tutorials/static-analysis.md

harttle · 2024-12-23T02:22:59Z

your current naming is OK for me. Sorry some comments are not submitted yesterday, causing confusion.

harttle · 2024-12-28T08:12:08Z

I can help update the docs afterwards. Are we good to merge this PR now?

jg-rp · 2024-12-28T08:53:03Z

I can help update the docs afterwards. Are we good to merge this PR now?

There's still a couple of corner cases to be addressed when tracking aliased variables in deeply nested rendered templates and handling Jekyll-style includes. Otherwise, if you're happy, it's good to go.

harttle · 2024-12-28T13:51:48Z

Merged and ddded an experimental notice in documents so we people know we're improving it.

# [10.20.0](v10.19.1...v10.20.0) (2024-12-28) ### Features * `size`, `first`, `last` support arraylike objects, [#781](#781) ([35a8442](35a8442)) * static variable analysis ([#770](#770)) ([3492ff6](3492ff6))

github-actions · 2024-12-28T13:53:46Z

🎉 This PR is included in version 10.20.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

feat: static variable analysis

4f0c2c4

harttle reviewed Nov 16, 2024

View reviewed changes

src/tags/capture.ts Show resolved Hide resolved

harttle reviewed Nov 16, 2024

View reviewed changes

src/tags/cycle.ts Outdated Show resolved Hide resolved

harttle reviewed Nov 16, 2024

View reviewed changes

test/integration/static_analysis/variables.spec.ts Outdated Show resolved Hide resolved

Accept any iterable from children, arguments, etc.

e7b8559

jg-rp mentioned this pull request Nov 17, 2024

feat: static analysis #767

Closed

2 tasks

Test analysis of standard tags

a69f33c

harttle reviewed Nov 18, 2024

View reviewed changes

src/template/value.ts Outdated Show resolved Hide resolved

harttle reviewed Nov 18, 2024

View reviewed changes

src/template/value.ts Outdated Show resolved Hide resolved

harttle reviewed Nov 18, 2024

View reviewed changes

test/integration/util/error.spec.ts Show resolved Hide resolved

jg-rp and others added 15 commits November 19, 2024 09:34

Merge branch 'harttle:master' into static-analysis-alternate

e5163ba

Use TagToken.tokenizer instead of creating a new one

5705bc6

Test analysis of netsted tags

2081083

Group variables by their root value

502a80d

Test analysis of nested globals and locals

5a9d192

Analyze included and rendered templates WIP

2cb9a4f

Use existing tokenizer when constructing Hash

bc6be99

Improve test coverage

7f63cef

Analyze variables from layout and block tags

0d1393b

Test analysis of Jekyll style includes

a1972ab

Handle variables that start with a nested variable

730ab19

Async analysis

c0a19e3

Test non-standard tag end to end

1a79437

Implement convenience analysis methods on the Liquid class

d9f47d6

More analysis convenience methods

67fdbe5

jg-rp added 2 commits November 23, 2024 15:28

Accept string or template array

cde3b5d

Draft static analysis docs

a3a93cc

Deduplicate variables names

2bf55db

jg-rp marked this pull request as ready for review November 24, 2024 08:12

jg-rp mentioned this pull request Dec 4, 2024

Extract Liquid.js valid and invalid variables #777

Closed

jg-rp added 5 commits December 5, 2024 12:20

Fix isolated scope global variable map

3ff787d

Coerce variables to strings instead of extending String

5c76035

Private map instead of extending Map

9770ff3

Fix e2e test

ad2333e

Tentatively implement analysis of aliased variables

f73f0d1

harttle approved these changes Dec 22, 2024

View reviewed changes

docs/source/tutorials/static-analysis.md Show resolved Hide resolved

src/liquid.ts Show resolved Hide resolved

src/liquid.ts Show resolved Hide resolved

Fix nested variable segments array

e9b11f4

harttle reviewed Dec 23, 2024

View reviewed changes

test/integration/static_analysis/variables.spec.ts Outdated Show resolved Hide resolved

src/liquid.ts Show resolved Hide resolved

docs/source/tutorials/static-analysis.md Show resolved Hide resolved

docs/source/tutorials/static-analysis.md Show resolved Hide resolved

jg-rp and others added 2 commits December 28, 2024 08:33

Merge branch 'harttle:master' into static-analysis-alternate

d1f7bea

Update docs sidebar

e241ef9

harttle merged commit 3492ff6 into harttle:master Dec 28, 2024
13 checks passed

github-actions bot added the released label Dec 28, 2024

bzerangue mentioned this pull request Jan 7, 2025

updated liquid to latest, patched version, 10.20.1 MinistryPlatform-Community/MPCustomWidgets#9

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: static variable analysis #770

feat: static variable analysis #770

jg-rp commented Nov 16, 2024 •

edited

Loading

coveralls commented Nov 16, 2024 •

edited

Loading

jg-rp commented Nov 18, 2024 •

edited

Loading

jg-rp commented Nov 23, 2024

jg-rp commented Dec 4, 2024 •

edited

Loading

jg-rp commented Dec 22, 2024

harttle commented Dec 23, 2024

harttle commented Dec 28, 2024

jg-rp commented Dec 28, 2024

harttle commented Dec 28, 2024

github-actions bot commented Dec 28, 2024

feat: static variable analysis #770

feat: static variable analysis #770

Conversation

jg-rp commented Nov 16, 2024 • edited Loading

Global Variables

Partial Templates

coveralls commented Nov 16, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12525259260

Details

💛 - Coveralls

jg-rp commented Nov 18, 2024 • edited Loading

jg-rp commented Nov 23, 2024

jg-rp commented Dec 4, 2024 • edited Loading

jg-rp commented Dec 22, 2024

harttle commented Dec 23, 2024

harttle commented Dec 28, 2024

jg-rp commented Dec 28, 2024

harttle commented Dec 28, 2024

github-actions bot commented Dec 28, 2024

jg-rp commented Nov 16, 2024 •

edited

Loading

coveralls commented Nov 16, 2024 •

edited

Loading

jg-rp commented Nov 18, 2024 •

edited

Loading

jg-rp commented Dec 4, 2024 •

edited

Loading