Skip to content

Commit

Permalink
Add option to aggressively inline all literals and consts
Browse files Browse the repository at this point in the history
This new inlining option does two things:
- Inlines any variable which looks like a literal constant
  (`hasInitDeps` is false) and which does not appear in an lvalue
  position
- Inlines any variable marked as `const` whatsoever.

This can increase program size, if a large literal is inlined over and
over. However, the idea is exactly why the Google Closure Compiler also
always inlines constants: a (the?) major use-case of this minifier is
for demoscene shaders, which are almost certainly going to be put
through compression later anyway -- so what matters is the size after
compression, not the actual output size. And reducing the redudancy that
this inlining creates can be done by the compressor in way fewer bytes
than declaring a new variable in the program text.

https://github.com/google/closure-compiler/wiki/FAQ#closure-compiler-inlined-all-my-strings-which-made-my-code-size-bigger-why-did-it-do-that

This option isn't the default; we likely want to do more testing to see
if it actually works out that way in practise (and to make sure the
output doesn't break things!) However, when I tested it on the two
largest shaders in `tests/real` (`the_real_party...` and `lunaquatic`),
it not only reduced gzip'd size but also raw output size too. So it may
just be a win all-around anyway?

- `the_real_party...`: 17708 -> 17699 (gzipped 5181 -> 5173)
- `lunaquatic`: 7356 -> 7347 (gzipped 2745 -> 2739)
  • Loading branch information
jwatzman committed May 10, 2022
1 parent eee9187 commit bcfd927
Show file tree
Hide file tree
Showing 11 changed files with 505 additions and 26 deletions.
38 changes: 31 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,13 @@ $ mono shader_minifier.exe # Linux, Mac...
```

```
USAGE: shader_minifier.exe [--help] [-o <string>] [-v] [--hlsl] [--format <text|indented|c-variables|c-array|js|nasm>]
[--field-names <rgba|xyzw|stpq>] [--preserve-externals] [--preserve-all-globals]
[--no-inlining] [--no-renaming] [--no-renaming-list <string>] [--no-sequence] [--smoothstep]
[<filename>...]
USAGE: shader_minifier [--help] [-o <string>] [-v] [--hlsl]
[--format <text|indented|c-variables|c-array|js|nasm>]
[--field-names <rgba|xyzw|stpq>] [--preserve-externals]
[--preserve-all-globals] [--no-inlining]
[--aggressive-inlining] [--no-renaming]
[--no-renaming-list <string>] [--no-sequence]
[--smoothstep] [<filename>...]
FILENAMES:
Expand All @@ -97,13 +100,22 @@ OPTIONS:
-v Verbose, display additional information
--hlsl Use HLSL (default is GLSL)
--format <text|indented|c-variables|c-array|js|nasm>
Choose to format the output (use none if you want just the shader)
Choose to format the output (use 'text' if you want
just the shader)
--field-names <rgba|xyzw|stpq>
Choose the field names for vectors: 'rgba', 'xyzw', or 'stpq'
Choose the field names for vectors: 'rgba', 'xyzw',
or 'stpq'
--preserve-externals Do not rename external values (e.g. uniform)
--preserve-all-globals
Do not rename functions and global variables
--no-inlining Do not automatically inline variables
--aggressive-inlining Aggressively inline constants. This can reduce output
size due to better constant folding. It can also
increase output size due to repeated inlined
constants, but this increased redundancy can be
beneficial to gzip leading to a smaller final
compressed size anyway. Does nothing if inlining is
disabled.
--no-renaming Do not rename anything
--no-renaming-list <string>
Comma-separated list of functions to preserve
Expand Down Expand Up @@ -213,7 +225,17 @@ This happens when:
If inlining causes a bug in your code, you can disable it with `--no-inlining`
and please report a bug.
### Explicit Inlining
### Aggressive inlining
Shader Minifier can optionally/experimentally inline even more aggressively.
Along with the above cases, it will inline *any* variable marked `const`, and
also when:
- the variable is never written to after initalization
- and the init value is trivial (doesn't depend on a variable).
This is enabled with `--aggressive-inlining`.
### Explicit inlining
Shader Minifier will always inline variables that starts with `i_`. Inlining can allow
the Minifier to simplify the code further.
Expand Down Expand Up @@ -246,6 +268,8 @@ If you want to aggressively reduce the size of your shader, try inlining more
variables. Inlining can have performance implications though (if the variable
stored the result of a computation), so be careful with it.
### Compression
Inlining can lead to repetition in the shader code, which may make the shader
longer (but the output may be more compression-friendly).
Expand Down
10 changes: 8 additions & 2 deletions src/ast.fs
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,15 @@ open Options.Globals
type Ident(name: string) =
let mutable newName = name
let mutable inlined = newName.StartsWith("i_")
let mutable lValue = false

member this.Name = newName
member this.OldName = name
member this.Rename(n) = newName <- n
member this.ToBeInlined = inlined
member this.Inline() = inlined <- true
member this.IsLValue = lValue
member this.MarkLValue() = lValue <- true

// Real identifiers cannot start with a digit, but the temporary ids of the rename pass are numbers.
member this.IsUniqueId = System.Char.IsDigit this.Name.[0]
Expand Down Expand Up @@ -128,9 +131,10 @@ type MapEnv = {
fExpr: MapEnv -> Expr -> Expr
fStmt: Stmt -> Stmt
vars: Map<string, Type * DeclElt>
fns: Map<string, FunctionType>
}

let mapEnv fe fi = {fExpr = fe; fStmt = fi; vars = Map.empty}
let mapEnv fe fi = {fExpr = fe; fStmt = fi; vars = Map.empty; fns = Map.empty}

let foldList env fct li =
let mutable env = env
Expand Down Expand Up @@ -206,6 +210,8 @@ let mapTopLevel env li =
| TLDecl t ->
let env, res = mapDecl env t
env, TLDecl res
| Function(fct, body) -> env, Function(fct, snd (mapStmt env body))
| Function(fct, body) ->
let env = {env with fns = env.fns.Add(fct.fName.Name, fct)}
env, Function(fct, snd (mapStmt env body))
| e -> env, e)
res
7 changes: 6 additions & 1 deletion src/options.fs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ type CliArguments =
| [<CustomCommandLine("--preserve-externals")>] PreserveExternals
| [<CustomCommandLine("--preserve-all-globals")>] PreserveAllGlobals
| [<CustomCommandLine("--no-inlining")>] NoInlining
| [<CustomCommandLine("--aggressive-inlining")>] AggroInlining
| [<CustomCommandLine("--no-renaming")>] NoRenaming
| [<CustomCommandLine("--no-renaming-list")>] NoRenamingList of string
| [<CustomCommandLine("--no-sequence")>] NoSequence
Expand All @@ -42,11 +43,12 @@ type CliArguments =
| OutputName _ -> "Set the output filename (default is shader_code.h)"
| Verbose -> "Verbose, display additional information"
| Hlsl -> "Use HLSL (default is GLSL)"
| FormatArg _ -> "Choose to format the output (use none if you want just the shader)"
| FormatArg _ -> "Choose to format the output (use 'text' if you want just the shader)"
| FieldNames _ -> "Choose the field names for vectors: 'rgba', 'xyzw', or 'stpq'"
| PreserveExternals _ -> "Do not rename external values (e.g. uniform)"
| PreserveAllGlobals _ -> "Do not rename functions and global variables"
| NoInlining -> "Do not automatically inline variables"
| AggroInlining -> "Aggressively inline constants. This can reduce output size due to better constant folding. It can also increase output size due to repeated inlined constants, but this increased redundancy can be beneficial to gzip leading to a smaller final compressed size anyway. Does nothing if inlining is disabled."
| NoRenaming -> "Do not rename anything"
| NoRenamingList _ -> "Comma-separated list of functions to preserve"
| NoSequence -> "Do not use the comma operator trick"
Expand Down Expand Up @@ -91,6 +93,9 @@ type Options() =
member this.noInlining =
args.Contains(NoInlining)

member this.aggroInlining =
args.Contains(AggroInlining) && not this.noInlining

member this.noSequence =
args.Contains(NoSequence)

Expand Down
75 changes: 74 additions & 1 deletion src/rewriter.fs
Original file line number Diff line number Diff line change
Expand Up @@ -186,10 +186,14 @@ let collectReferences stmtList =
count

// Mark variables as inlinable when possible.
// For now, only mark a variable when:
// Variables are always safe to inline when all of:
// - the variable is used only once in the current block
// - the variable is not used in a sub-block (e.g. inside a loop)
// - the init value is trivial (doesn't depend on a variable)
// When aggressive inlining is enabled, additionally inline when all of:
// - the variable never appears in an lvalue position (is never written to
// after initalization)
// - the init value is trivial
let findInlinable foundInlinable block =
// Variables that are defined in this scope.
// The boolean indicates if the variable initialization has dependencies.
Expand Down Expand Up @@ -218,6 +222,8 @@ let findInlinable foundInlinable block =
for def in localDefs do
let ident, hasInitDeps = def.Value
if not ident.ToBeInlined then
if options.aggroInlining && not hasInitDeps && not ident.IsLValue then
ident.Inline(); foundInlinable := true
match localReferences.TryGetValue(def.Key), allReferences.TryGetValue(def.Key) with
| (true, 1), (true, 1) when not hasInitDeps -> ident.Inline(); foundInlinable := true
| (false, _), (false, _) -> ident.Inline(); foundInlinable := true
Expand Down Expand Up @@ -285,8 +291,75 @@ let rec iterateSimplifyAndInline li =
let simplified = mapTopLevel (mapEnv simplifyExpr simplifyStmt) li
if foundInlinable then iterateSimplifyAndInline simplified else simplified

let inlineAllConsts li =
let mapInnerDecl = function
// Unconditional inlining of anything marked "const" -- trust that the
// compiler would have yelled if it weren't really really const, so we
// can brutishly just inline it.
| ({typeQ = tyQ}, defs) as d when List.contains "const" tyQ ->
for (def:DeclElt) in defs do def.name.Inline()
d
| d -> d
let mapStmt = function
| Decl d -> Decl (mapInnerDecl d)
| s -> s
let mapExpr _ e = e
let mapTLDecl = function
| TLDecl d -> TLDecl (mapInnerDecl d)
| d -> d
li
|> mapTopLevel (mapEnv mapExpr mapStmt)
|> List.map mapTLDecl

let markLValues li =
// Helpers for the bodies of functions: find any expression of the form
// "foo = ..." or "foo += ..." etc, then scan through all of "foo" marking
// any variable seen there as potentially in an lvalue position. This will
// over-mark things, e.g., "x[i]" both "x" and "i" will get marked even
// though the latter does not need to be, but it is simple.
let markVars env = function
| Var v as e ->
match env.vars.TryFind v.Name with
| Some (_, {name = vv}) -> vv.MarkLValue(); e
| _ -> e
| e -> e
let assignOps = Set.ofList ["="; "+="; "-="; "*="; "/="; "%=";
"<<="; ">>="; "&="; "^="; "|="; "_++"; "_--"; "$++"; "$--"]
let findWrites env = function
| FunCall(Op o, e::args) when Set.contains o assignOps ->
let newEnv = {env with fExpr = markVars}
FunCall(Op o, (mapExpr newEnv e)::args)
| FunCall(Var v, _) as e ->
match env.fns.TryFind v.Name with
| Some fct when fct.fName.IsLValue ->
let newEnv = {env with fExpr = markVars}
mapExpr newEnv e
| _ -> e
| e -> e
// Helpers for function declarations: if any parameter to the function
// could be written to (e.g., "out"), mark the entire function. We don't
// attempt to match up param-for-param but just mark everything if anything
// could write, for simplicity.
let maybeMarkFct {fName = id; args = args} =
let assignQuals = Set.ofList ["out"; "inout"]
let argAssigns (ty, _) =
List.exists (fun tyQ -> Set.contains tyQ assignQuals) ty.typeQ
if List.exists argAssigns args then id.MarkLValue()
let processTl = function
| Function(fct, _) -> maybeMarkFct fct
| _ -> ()
// Mark functions then bodies; order is important since, when scanning
// bodies, we need to look up which functions might write via "out"
// parameters.
List.iter processTl li
mapTopLevel (mapEnv findWrites id) li

let simplify li =
li
// markLValues doesn't change the AST so we could do it unconditionally,
// but we only need the information for aggroInlining so don't bother if
// it's off.
|> if options.aggroInlining then markLValues >> inlineAllConsts else id
|> reorderTopLevel
|> iterateSimplifyAndInline
|> List.map (function
Expand Down
3 changes: 3 additions & 0 deletions tests/commands.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,10 @@
--no-renaming --no-inlining --format c-array -o tests/unit/operators.expected tests/unit/operators.frag
--no-renaming --no-inlining --format c-array -o tests/unit/minus-zero.expected tests/unit/minus-zero.frag
--no-renaming --format indented -o tests/unit/inline.expected tests/unit/inline.frag
--no-renaming --format indented --aggressive-inlining -o tests/unit/inline.aggro.expected tests/unit/inline.frag
--no-renaming --format indented --no-inlining -o tests/unit/inline.no.expected tests/unit/inline.frag
--no-renaming --format indented -o tests/unit/inline-aggro.expected tests/unit/inline-aggro.frag
--no-renaming --format indented --aggressive-inlining -o tests/unit/inline-aggro.aggro.expected tests/unit/inline-aggro.frag
--no-renaming --format c-array --no-inlining -o tests/unit/float.frag.expected tests/unit/float.frag

# Partial renaming tests
Expand Down
112 changes: 112 additions & 0 deletions tests/unit/inline-aggro.aggro.expected
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
float inl1()
{
return 126.;
}
float inl2(float x)
{
if(x>246913578.)
return 100.;
if(x>123456789.)
return 101.;
return 102.;
}
float inl3()
{
return.75*(2.*acos(-1.));
}
float inl4()
{
return.75*(2.*acos(-1.));
}
float inl5()
{
return 579.;
}
float inl6()
{
return 2.*(2.*acos(-1.));
}
void notevil(float x)
{
x=42.;
}
float inl7()
{
return notevil(101.),101.;
}
int inl8(ivec3 x)
{
int i=1;
x[i]+=1;
return x[i]+i;
}
float noinl1()
{
float f=42.;
f=101.;
return f;
}
float noinl2()
{
float f=42.;
f++;
return f;
}
float noinl3()
{
float f=42.;
if(acos(-1.)<3.)
f=101.;
return f;
}
float noinl4()
{
float f=42.;
for(f=0.;false;)
;
return f;
}
float noinl5()
{
float f=42.;
for(;false;f++)
;
return f;
}
float noinl6(float x)
{
float f=x+1.;
x=100.;
return f+2.;
}
float noinl7(const float x)
{
return x+1.2;
}
float quux=1.;
float noinl8()
{
return quux+2.;
}
float noinl9(float x)
{
float bar=x+1.;
x=100.;
return bar+2.;
}
void evil(inout float x)
{
x=42.;
}
float noinl10()
{
float f=101.;
evil(f);
return f;
}
int noinl11(ivec3 x)
{
int i=1;
x[i++]+=1;
return x[i]+i;
}
Loading

0 comments on commit bcfd927

Please sign in to comment.