Skip to content

svenkreiss/localcrawl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

localcrawl

Command line tool that crawls a local webserver with a headless browser and outputs static html files. Works well with html5validator to validate HTML5 from dynamic content with Javascript.
https://travis-ci.org/svenkreiss/localcrawl.svg?branch=master

Run localcrawl --help:

https://raw.githubusercontent.com/svenkreiss/localcrawl/master/docs/help.png

PhantomJS is required. It is pre-installed on TravisCI. On a Mac run brew install PhantomJS.

Example Command

localcrawl --start _build/html/index.html --out _crawled/ --depth 3

Mustache Example

This can be used to convert templated files to HTML files (e.g. for validation with html5validator).

Input:

<html>
<head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output"></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>
</body>
</html>

The crawled output includes the output from processing the template (Fork: <b>$1.19</b>):

<html><head>
  <title>Mustache Test</title>
</head>
<body>
  <div id="output">Fork: <b>$1.19</b></div>

  <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mustache.js/2.2.1/mustache.min.js"></script>
  <script>
    var data = {
      item: 'Fork',
      price: function() { return (1.10 * 1.08).toFixed(2); },
    };
    var html = Mustache.render('{{item}}: <b>${{price}}</b>', data);
    document.getElementById('output').innerHTML = html;
  </script>


</body></html>

Should play nice with:

JavaScript template engines / JS frameworks:

Static site generators:

About

Crawl and render JavaScript templates.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published