-
Notifications
You must be signed in to change notification settings - Fork 12
Preparation
To make the best use out of Roadroller your demo should be prepared accordingly.
Roadroller algorithm is much stronger than DEFLATE (what the ZIP file internally uses) so everything has to be run through Roadroller for the ideal compression. It is as easy as the following:
document.write(`
<style>
#something { ... }
...
</style>
<div id="something">
...
</div>
`);Note that CSS and HTML inserted in this way cannot be further optimized, so you would want an automated pipeline like this:
+----------+ +---------+ +----------+
| HTML | | CSS | |JavaScript|
+----------+ +---------+ +----------+
| HTML | CSSO, | Terser,
| minifier | Cssnano | Closure Compiler
| etc. | etc. | etc.
v v v
+----------+ +---------+ +----------+
| minified | |optimized| | minified |
| HTML | | CSS | |JavaScript|
+----------+ +---------+ +----------+
| | |
v v v
+-+--------------------------+------------+
| | a call to document.write | |
| +--------------------------+ |
| combined JavaScript |
+-----------------------------------------+
|
Roadroller |
v
+-----------------------------+
| JavaScript after Roadroller |
+-----------------------------+
Once you've put CSS and most HTML to JavaScript, your index.html template should be ideally as simple as follows:
<script>/* compressed script here */</script>Not everything can be put into JavaScript in this way though, for example dynamically inserting <meta name="viewport"> element might not work in some browsers, and if you are into the search engine optimization <meta name="description"> etc. should be also retained. But virtually everything that would be in <body> can go into JavaScript.
js13kGames lets you to submit a ZIP file containing index.html and assets, presumably to save you from doing crazy things like zpng. You still should have the minimal number (optimally, 1) of files in your ZIP files though because:
- An additional file to the ZIP file costs at least 88 + 2n additional bytes, where n is the length of file name. This is pretty significant for js13kGames.
- You cannot run Roadroller for multiple files (yet), so a possible gain out of Roadroller would be limited.
You are still allowed to put images to your ZIP file because Roadroller (currently) can't recompress images and an inlined image with base64 can possibly negate the benefit. But even in that case:
- You should minimize the number of images, for example by generating them on the fly or using a sprite atlas or similar.
- You should also try using lossless WebP which can be substantially smaller than a typical PNG even after optimization.
- File names of additional assets should be also minimized.
Once you've got index.html and assets you would like to put it into a ZIP file, but ordinary tools are known to be suboptimal:
- Roadroller heavily exploits DEFLATE and requires a special treatment but stock tools are oblivious of that.
- Some tools are unable to set the compression level, so they are even more inferior to some other stock tools. For example yazl always compresses in the default level.
Ordinary tools are however easier to integrate, so we recommend to make a ZIP file as you wish and then recompress it with external tools. We specifically recommend:
- ADVZIP from AdvanceCOMP. Quick usage:
advzip -z -4 -i 99 submission.zip -
Efficient Compression Tool (ECT). Quick usage:
ect -z -9 submission.zip
They are known to be about equally efficient for typical js13kGames submissions, so you can use any of them. If you have any additional assets ECT might be better because it apparently recompresses files in the ZIP archive as well. You can of course run both for the maximal recompression (can save 10 more bytes or so).
As such, we recommend the following automated pipeline for js13kGames submissions:
+----------+ +---------+ +----------+
| HTML | | CSS | |JavaScript|
+----------+ +---------+ +----------+
| HTML | CSSO, | Terser,
+--------------| minifier | Cssnano | Closure Compiler
| | etc. | etc. | etc.
v v v v
+----------+ +----------+ +---------+ +----------+
| minimal | | minified | |optimized| | minified |
| HTML | | HTML | | CSS | |JavaScript|
| template | +----------+ +---------+ +----------+
+----------+ | | |
| v v v
| +-+--------------------------+------------+
| | | a call to document.write | |
| | +--------------------------+ |
| | combined JavaScript |
| +-----------------------------------------+
| |
| | +------------+
| Roadroller |<---| parameters |<----+
| | +------------+ |
| v | optimize
| +-----------------------------+ |
| | JavaScript after Roadroller |--------+
| +-----------------------------+
| |
+--------------------------+ |
| |
+----------------+ v v
| uncompressible | +-----------------------------+
| additional | | standalone index.html |
| assets | +-----------------------------+
+----------------+ |
| |
+----------------------+ | stock ZIP library or application
| |
v v
+-----------------------------+
| submission.zip |
+-----------------------------+
|
| ADVZIP, ECT etc.
v
+-----------------------------+
| optimized submission.zip |
+-----------------------------+
Roadroller has a large number of compression parameters that solely affect the compression ratio and don't affect the decompression speed or memory usage. The default parameters are known to work moderately well, but for the best compression you can search through other parameters.
Fortunately you don't have to manually tune them; Roadroller provides an interface for the search. Depending on the input this can shave up to 5% of your precious bytes, so you should absolutely try the search if you can. The search is randomized, so it is also a good idea to try the search multiple times.
If you are using the CLI, you have already seen the search process like this:
$ roadroller index.js -o out.js
(initial) -Sx12: 10663 <-
(modelRecipBaseCount 0.0%) -Zmd10 -Sx12: 10656 <-
(modelRecipBaseCount 25.0%) -Zmd20 -Sx12: 10663 x
(modelRecipBaseCount 50.0%) -Zmd50 -Sx12: 10698 x
(modelRecipBaseCount 75.0%) -Zmd100 -Sx12: 10738 x
(modelMaxCount 0.0%) -Zmc4 -Zmd10 -Sx12: 10635 <-
(modelMaxCount 33.3%) -Zmc5 -Zmd10 -Sx12: 10656 x
(modelMaxCount 66.7%) -Zmc6 -Zmd10 -Sx12: 10684 x
(numAbbreviations 0.0%) -Zab0 -Zmc4 -Zmd10 -Sx12: 10841 x
(numAbbreviations 25.0%) -Zab16 -Zmc4 -Zmd10 -Sx12: 10683 x
(numAbbreviations 50.0%) -Zab32 -Zmc4 -Zmd10 -Sx12: 10628 <-
(numAbbreviations 75.0%) -Zab64 -Zmc4 -Zmd10 -Sx12: 10635 x
(preferTextOverJS) -t text -Zab32 -Zmc4 -Zmd10 -Sx12: 10949 x
(sparseSelectors 0.0%) -Zab32 -Zmc4 -Zmd10 -S0,1,2,3,6,7,12,21,25,42,50,57: 10652 x
(sparseSelectors 9.5%) -Zab32 -Zmc4 -Zmd10 -S0,1,2,3,6,7,13,21,25,42,50,331: 10626 <-
[... SNIP ...]
(recipLearningRate 60.0%) -Zab32 -Zlr1250 -Zmc4 -Zmd10 -Zpr14 -S0,1,2,3,6,7,13,21,42,50,393,489: 10612 x
(recipLearningRate 80.0%) -Zab32 -Zlr1500 -Zmc4 -Zmd10 -Zpr14 -S0,1,2,3,6,7,13,21,42,50,393,489: 10614 x
search done in 10.5s, use `-Zab32 -Zlr750 -Zmc4 -Zmd10 -Zpr14 -S0,1,2,3,6,7,13,21,42,50,393,489` to replicate: 10611 (estimated, 67.63% smaller)
If you are willing to spend more time (read: a few minutes), you can give the -O2 option. The default is, by the way, -O1 and you can explicitly disable the search with -O0 (which would result in 10,663 bytes in this case).
$ roadroller index.js -o out.js -O2
(initial) -Sx12: 10663 <-
(modelRecipBaseCount 0.0%) -Zmd1 -Sx12: 10911 x
(modelRecipBaseCount 0.0%) -Zmd6 -Sx12: 10672 x
(modelRecipBaseCount 0.0%) -Zmd32 -Sx12: 10677 x
(modelRecipBaseCount 0.0%) -Zmd179 -Sx12: 10779 x
(modelRecipBaseCount 0.0%) -Zmd1000 -Sx12: 10936 x
[... SNIP ...]
(recipLearningRate 99.7%) -Zab33 -Zlr848 -Zmc4 -Zmd10 -S0,1,2,3,6,13,21,49,101,106,410,417: 10606 x
(recipLearningRate 99.7%) -Zab33 -Zlr851 -Zmc4 -Zmd10 -S0,1,2,3,6,13,21,49,101,106,410,417: 10606 x
(recipLearningRate 99.8%) -Zab33 -Zlr849 -Zmc4 -Zmd10 -S0,1,2,3,6,13,21,49,101,106,410,417: 10606 x
search done in 93.5s, use `-Zab33 -Zlr930 -Zmc4 -Zmd10 -S0,1,2,3,6,13,21,49,101,106,410,417` to replicate: 10605 (estimated, 67.65% smaller)
Each search gives you the best parameters found, so you can directly put them into the command line to avoid further search. You can also run another search on top of them with an explicit -O2 option, or continuously search through better parameters with -OO. (You can stop the search at any point by pressing Ctrl-C.)
$ roadroller index.js -o out.js -Zab33 -Zlr930 -Zmc4 -Zmd10 -S0,1,2,3,6,13,21,49,101,106,410,417
compressed 32782B into 10605B (estimated, 67.65% smaller).
For the API it is a bit more complicated. First you should adjust your integration code as follows:
const inputs = [
{ data, type: 'js', action: 'eval' },
];
const options = {
maxMemoryMB: 150,
// use the default for now
};
const packer = new Packer(inputs, options);
console.log(await packer.optimize()); // <-- insert this line
//console.log(await packer.optimize(2)); // <-- or this line, if you want -O2
const { firstLine, secondLine } = packer.makeDecoder();
const compressedData = firstLine + secondLine;Packer.optimize method runs the same search as above, but with a pluggable logger (it doesn't print anything if no logger is given). The method returns the best parameters like this:
{
elapsedMsecs: 10518,
best: {
numAbbreviations: 32,
recipLearningRate: 1500,
modelMaxCount: 4,
modelRecipBaseCount: 10,
precision: 14,
sparseSelectors: [ 0, 1, 2, 3, 6, 7, 13, 21, 42, 50, 393, 489 ]
},
bestSize: 10611
}
You can now update the options object to make use of these parameters:
const options = {
maxMemoryMB: 150,
// search result
numAbbreviations: 32,
recipLearningRate: 1500,
modelMaxCount: 4,
modelRecipBaseCount: 10,
precision: 14,
sparseSelectors: [ 0, 1, 2, 3, 6, 7, 13, 21, 42, 50, 393, 489 ]
};As you might have guessed, CLI arguments and API parameters are compatible to each other within the same version. You can for example search from the CLI and translate them into the API or vice versa.
Note: You can do the same thing in the online demo, but we don't guarantee that the online demo is using the same version of Roadroller as you have (it might even be using an unreleased future version).
Roadroller specifically models string literals for the JavaScript input, which improves the compression because characters in the typical code and in string literals have substantially different distributions. This is also why putting CSS and HTML into JS works; they don't adversely affect the compression of the JS code itself that much.
For this reason, string packing can be used to guide Roadroller to assume a different distribution for quoted characters. Roadroller can compress a large data array a lot, but by doing so it affects the compression of neighboring JS code. String literals will instead affect the compression of neighboring string literals, which are relatively okay to do so.
Case study: Zzfx Music uses a large amount of pattern data stored as nested JavaScript arrays. They can be converted to an array of packed strings for longer music. For example the original pattern data for Sanxion by Rob Hubbard is 11,445 bytes originally or 1,651 bytes after Roadroller (with the default parameters):
[[
[,-1,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,8,,,,,,8,,],
[2,-1,,20,8,8,20,8,,8,,20,,8,20,8,,20,,20,8,8,20,8,,8,,20,,8,20,8,,8,,20,8,8,20,8,,8,,20,,8,20,8,,8,,20,8,8,20,8,,8,,20,,8,20,8,,20],
[,1,32,22,22,32,32,22,32,27,32,22,22,32,32,22,32,32,32,22,22,32,32,22,32,27,32,22,22,32,32,22,32,32,32,22,22,32,32,22,32,27,32,22,22,32,32,22,32,32,32,22,22,32,32,22,32,22,32,22,22,32,32,22,32,32],
...,
[,-1,,22,22,,,22,,,,22,22,,,22,,,,22,22,,,22,,22,,22,22,,,22,,,,22,22,,,22,,22,,22,22,,,22,,,,22,22,,,22,,22,,22,22,,,22,,,],
[3,-1,,,,,32,,,,,,,,32,,,,,,,,32,,,,,,,,32,,,,,,,,32,,,,,,,,32,,,,,,,,32,,,,,,,,32,,,,]
]]A simple tool can be used to make it more Roadroller-friendly:
[[
[,-1,".&&&&&.&.&&&&&.&.&&&&&.&.&&&&&.&.&&&&&.&.&&&&&.&.&&&&&.&.&&&&&.&"],
[2,-1,"&:..:.&.&:&.:.&:&:..:.&.&:&.:.&.&:..:.&.&:&.:.&.&:..:.&.&:&.:.&:"],
[,1,"F<<FF<FAF<<FF<FFF<<FF<FAF<<FF<FFF<<FF<FAF<<FF<FFF<<FF<F<F<<FF<FF"],
...,
[,-1,"&<<&&<&&&<<&&<&&&<<&&<&<&<<&&<&&&<<&&<&<&<<&&<&&&<<&&<&<&<<&&<&&"],
[3,-1,"&&&&F&&&&&&&F&&&&&&&F&&&&&&&F&&&&&&&F&&&&&&&F&&&&&&&F&&&&&&&F&&&"]
]].map(p=>p.map(p=>[...p.pop()].map(b=>p.push((b=b.charCodeAt())<74?b-38:p.pop()+(b-72>>1)/(b&1?10:100)))&&p))This compresses into 1,383 bytes after Roadroller with the same parameters. These parameters are by the way suboptimal for those inputs, but the latter is still about 1--200 bytes smaller even after optimization. Note that Roadroller is still doing a hard job to infer JavaScript array syntax and this transformation is only worthy for a large pattern data.