Skip to content

Commit

Permalink
Add Jsonnet Lexer (#1011)
Browse files Browse the repository at this point in the history 
No open issue, but I wanted this for myself so I figured I'd open a PR
as well. 🙂

I've used the third example from the jsonnet site as the test:
https://jsonnet.org/

Converted uses the pygments converter, and here's a quick screenshot of
the playground:

![Screenshot 2024-10-15
115533](https://github.com/user-attachments/assets/d0b6d054-4e42-4911-ba67-670bdc55bf3e)

It seems to have a small problem with strings as keys, but otherwise
looks close.

Signed-off-by: jolheiser <git@jolheiser.com>
  • Loading branch information
@jolheiser
jolheiser authored Oct 15, 2024
1 parent 114c584 commit 2e669a2
Show file tree
Hide file tree
Showing 4 changed files with 342 additions and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ translators for Pygments lexers and styles.
| G | GAS, GDScript, Genshi, Genshi HTML, Genshi Text, Gherkin, Gleam, GLSL, Gnuplot, Go, Go HTML Template, Go Text Template, GraphQL, Groff, Groovy |
| H | Handlebars, Hare, Haskell, Haxe, HCL, Hexdump, HLB, HLSL, HolyC, HTML, HTTP, Hy |
| I | Idris, Igor, INI, Io, ISCdhcpd |
| J | J, Java, JavaScript, JSON, Julia, Jungle |
| J | J, Java, JavaScript, JSON, Jsonnet, Julia, Jungle |
| K | Kotlin |
| L | Lighttpd configuration file, LLVM, Lua |
| M | Makefile, Mako, markdown, Mason, Materialize SQL dialect, Mathematica, Matlab, MCFunction, Meson, Metal, MiniZinc, MLIR, Modula-2, MonkeyC, MorrowindScript, Myghty, MySQL |
Expand Down
138 changes: 138 additions & 0 deletions lexers/embedded/jsonnet.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@

<lexer>
<config>
<name>Jsonnet</name>
<alias>jsonnet</alias>
<filename>*.jsonnet</filename>
<filename>*.libsonnet</filename>
</config>
<rules>
<state name="_comments">
<rule pattern="(//|#).*\n"><token type="CommentSingle"/></rule>
<rule pattern="/\*\*([^/]|/(?!\*))*\*/"><token type="LiteralStringDoc"/></rule>
<rule pattern="/\*([^/]|/(?!\*))*\*/"><token type="Comment"/></rule>
</state>
<state name="root">
<rule><include state="_comments"/></rule>
<rule pattern="@&#x27;.*&#x27;"><token type="LiteralString"/></rule>
<rule pattern="@&quot;.*&quot;"><token type="LiteralString"/></rule>
<rule pattern="&#x27;"><token type="LiteralString"/><push state="singlestring"/></rule>
<rule pattern="&quot;"><token type="LiteralString"/><push state="doublestring"/></rule>
<rule pattern="\|\|\|(.|\n)*\|\|\|"><token type="LiteralString"/></rule>
<rule pattern="[+-]?[0-9]+(.[0-9])?"><token type="LiteralNumberFloat"/></rule>
<rule pattern="[!$~+\-&amp;|^=&lt;&gt;*/%]"><token type="Operator"/></rule>
<rule pattern="\{"><token type="Punctuation"/><push state="object"/></rule>
<rule pattern="\["><token type="Punctuation"/><push state="array"/></rule>
<rule pattern="local\b"><token type="Keyword"/><push state="local_name"/></rule>
<rule pattern="assert\b"><token type="Keyword"/><push state="assert"/></rule>
<rule pattern="(assert|else|error|false|for|if|import|importstr|in|null|tailstrict|then|self|super|true)\b"><token type="Keyword"/></rule>
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule pattern="function(?=\()"><token type="Keyword"/><push state="function_params"/></rule>
<rule pattern="std\.[^\W\d]\w*(?=\()"><token type="NameBuiltin"/><push state="function_args"/></rule>
<rule pattern="[^\W\d]\w*(?=\()"><token type="NameFunction"/><push state="function_args"/></rule>
<rule pattern="[^\W\d]\w*"><token type="NameVariable"/></rule>
<rule pattern="[\.()]"><token type="Punctuation"/></rule>
</state>
<state name="singlestring">
<rule pattern="[^&#x27;\\]"><token type="LiteralString"/></rule>
<rule pattern="\\."><token type="LiteralStringEscape"/></rule>
<rule pattern="&#x27;"><token type="LiteralString"/><pop depth="1"/></rule>
</state>
<state name="doublestring">
<rule pattern="[^&quot;\\]"><token type="LiteralString"/></rule>
<rule pattern="\\."><token type="LiteralStringEscape"/></rule>
<rule pattern="&quot;"><token type="LiteralString"/><pop depth="1"/></rule>
</state>
<state name="array">
<rule pattern=","><token type="Punctuation"/></rule>
<rule pattern="\]"><token type="Punctuation"/><pop depth="1"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="local_name">
<rule pattern="[^\W\d]\w*(?=\()"><token type="NameFunction"/><push state="function_params"/></rule>
<rule pattern="[^\W\d]\w*"><token type="NameVariable"/></rule>
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule pattern="(?==)"><token type="TextWhitespace"/><push state="#pop" state="local_value"/></rule>
</state>
<state name="local_value">
<rule pattern="="><token type="Operator"/></rule>
<rule pattern=";"><token type="Punctuation"/><pop depth="1"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="assert">
<rule pattern=":"><token type="Punctuation"/></rule>
<rule pattern=";"><token type="Punctuation"/><pop depth="1"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="function_params">
<rule pattern="[^\W\d]\w*"><token type="NameVariable"/></rule>
<rule pattern="\("><token type="Punctuation"/></rule>
<rule pattern="\)"><token type="Punctuation"/><pop depth="1"/></rule>
<rule pattern=","><token type="Punctuation"/></rule>
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule pattern="="><token type="Operator"/><push state="function_param_default"/></rule>
</state>
<state name="function_args">
<rule pattern="\("><token type="Punctuation"/></rule>
<rule pattern="\)"><token type="Punctuation"/><pop depth="1"/></rule>
<rule pattern=","><token type="Punctuation"/></rule>
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="object">
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule pattern="local\b"><token type="Keyword"/><push state="object_local_name"/></rule>
<rule pattern="assert\b"><token type="Keyword"/><push state="object_assert"/></rule>
<rule pattern="\["><token type="Operator"/><push state="field_name_expr"/></rule>
<rule pattern="(?=[^\W\d]\w*)"><token type="Text"/><push state="field_name"/></rule>
<rule pattern="\}"><token type="Punctuation"/><pop depth="1"/></rule>
<rule pattern="&quot;"><token type="NameVariable"/><push state="double_field_name"/></rule>
<rule pattern="&#x27;"><token type="NameVariable"/><push state="single_field_name"/></rule>
<rule><include state="_comments"/></rule>
</state>
<state name="field_name">
<rule pattern="[^\W\d]\w*(?=\()"><token type="NameFunction"/><push state="field_separator" state="function_params"/></rule>
<rule pattern="[^\W\d]\w*"><token type="NameVariable"/><push state="field_separator"/></rule>
</state>
<state name="double_field_name">
<rule pattern="([^&quot;\\]|\\.)*&quot;"><token type="NameVariable"/><push state="field_separator"/></rule>
</state>
<state name="single_field_name">
<rule pattern="([^&#x27;\\]|\\.)*&#x27;"><token type="NameVariable"/><push state="field_separator"/></rule>
</state>
<state name="field_name_expr">
<rule pattern="\]"><token type="Operator"/><push state="field_separator"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="function_param_default">
<rule pattern="(?=[,\)])"><token type="TextWhitespace"/><pop depth="1"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="field_separator">
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
<rule pattern="\+?::?:?"><token type="Punctuation"/><push state="#pop" state="#pop" state="field_value"/></rule>
<rule><include state="_comments"/></rule>
</state>
<state name="field_value">
<rule pattern=","><token type="Punctuation"/><pop depth="1"/></rule>
<rule pattern="\}"><token type="Punctuation"/><pop depth="2"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="object_assert">
<rule pattern=":"><token type="Punctuation"/></rule>
<rule pattern=","><token type="Punctuation"/><pop depth="1"/></rule>
<rule><include state="root"/></rule>
</state>
<state name="object_local_name">
<rule pattern="[^\W\d]\w*"><token type="NameVariable"/><push state="#pop" state="object_local_value"/></rule>
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
</state>
<state name="object_local_value">
<rule pattern="="><token type="Operator"/></rule>
<rule pattern=","><token type="Punctuation"/><pop depth="1"/></rule>
<rule pattern="\}"><token type="Punctuation"/><pop depth="2"/></rule>
<rule><include state="root"/></rule>
</state>
</rules>
</lexer>

36 changes: 36 additions & 0 deletions lexers/testdata/jsonnet.actual
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
local application = 'my-app';
local module = 'uwsgi_module';
local dir = '/var/www';
local permission = 644;

{
'uwsgi.ini': std.manifestIni({
sections: {
uwsgi: {
module: module,
pythonpath: dir,
socket: dir + '/uwsgi.sock',
'chmod-socket': permission,
callable: application,
logto: '/var/log/uwsgi/uwsgi.log',
},
},
}),

'init.sh': |||
#!/usr/bin/env bash
mkdir -p %(dir)s
touch %(dir)s/initialized
chmod %(perm)d %(dir)s/initialized
||| % {dir: dir, perm: permission},

'cassandra.conf': std.manifestYamlDoc({
cluster_name: application,
seed_provider: [
{
class_name: 'SimpleSeedProvider',
parameters: [{ seeds: '127.0.0.1' }],
},
],
}),
}
167 changes: 167 additions & 0 deletions lexers/testdata/jsonnet.expected
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
[
{"type":"Keyword","value":"local"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"application"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"="},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'my-app'"},
{"type":"Punctuation","value":";"},
{"type":"TextWhitespace","value":"\n"},
{"type":"Keyword","value":"local"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"module"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"="},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'uwsgi_module'"},
{"type":"Punctuation","value":";"},
{"type":"TextWhitespace","value":"\n"},
{"type":"Keyword","value":"local"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"dir"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"="},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'/var/www'"},
{"type":"Punctuation","value":";"},
{"type":"TextWhitespace","value":"\n"},
{"type":"Keyword","value":"local"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"permission"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"="},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralNumberFloat","value":"644"},
{"type":"Punctuation","value":";"},
{"type":"TextWhitespace","value":"\n\n"},
{"type":"Punctuation","value":"{"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"'uwsgi.ini'"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameBuiltin","value":"std.manifestIni"},
{"type":"Punctuation","value":"({"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"sections"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"{"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"uwsgi"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"{"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"module"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"module"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"pythonpath"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"dir"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"socket"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"dir"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"+"},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'/uwsgi.sock'"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"'chmod-socket'"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"permission"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"callable"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"application"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"logto"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'/var/log/uwsgi/uwsgi.log'"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"},"},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"},"},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"}),"},
{"type":"TextWhitespace","value":"\n\n "},
{"type":"NameVariable","value":"'init.sh'"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"|||\n #!/usr/bin/env bash\n mkdir -p %(dir)s\n touch %(dir)s/initialized\n chmod %(perm)d %(dir)s/initialized\n |||"},
{"type":"TextWhitespace","value":" "},
{"type":"Operator","value":"%"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"{"},
{"type":"NameVariable","value":"dir"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"dir"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"perm"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"permission"},
{"type":"Punctuation","value":"},"},
{"type":"TextWhitespace","value":"\n\n "},
{"type":"NameVariable","value":"'cassandra.conf'"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameBuiltin","value":"std.manifestYamlDoc"},
{"type":"Punctuation","value":"({"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"cluster_name"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"application"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"seed_provider"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"["},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"{"},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"class_name"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'SimpleSeedProvider'"},
{"type":"Punctuation","value":","},
{"type":"TextWhitespace","value":"\n "},
{"type":"NameVariable","value":"parameters"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"[{"},
{"type":"TextWhitespace","value":" "},
{"type":"NameVariable","value":"seeds"},
{"type":"Punctuation","value":":"},
{"type":"TextWhitespace","value":" "},
{"type":"LiteralString","value":"'127.0.0.1'"},
{"type":"TextWhitespace","value":" "},
{"type":"Punctuation","value":"}],"},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"},"},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"],"},
{"type":"TextWhitespace","value":"\n "},
{"type":"Punctuation","value":"}),"},
{"type":"TextWhitespace","value":"\n"},
{"type":"Punctuation","value":"}"},
{"type":"TextWhitespace","value":"\n"}
]

0 comments on commit 2e669a2

Please sign in to comment.