tool-v0.18.0

[tool-v0.18.0] - 2024-11-20: Incremental improvements

Added

• export mirror:

  • Implemented the --boring option, which allows you to load some input PATHs without adding them as roots, even when no --root-* options are specified.

    This make CLI a bit more convenient to use.
    The README.md has a new example showcasing it.

• export mirror, scrub:

  • Implemented support for @import CSS rules using a string token in place of a URL.

    As far as I can see, this syntax is rarely used in practice.
    But the spec allows this, so.

  • Implemented interpret_noscript option, which enables inlining of noscript tags when scrub is running with -scripts.

    That is, export mirror will now use this feature by default.

    This is needed because some websites put link tags with CSS under noscript, thus making such pages look broken when scrubbed with -scripts (which is the default) and then opened in a browser with scripts enabled.

Changed

• *: Refactored/reworked a large chunk of internals, as a result:

  • organize can now take WRR bundles as inputs too,
  • export mirror became much faster at indexing inputs that contain archives of the same URLs, repeatedly.

  In general, these changes are aimed towards making hoardy-web completely input-agnostic.
  That is, wouldn't it be nice if you could feed mitmproxy files to export mirror directly, instead of going through import mitmproxy first?

• export mirror, scrub:

  • From now on, it will stop generating link tags with void URLs, it will simply censor them out instead.

  • scrub with +verbose set will now also show original rel attr values for censored out tags.

  • Also, in general, the outputs of scrub with +verbose set are much prettier now.

• Improved documentation.