A cross-platform command line tool for parallelized, distributed content-extraction. Built on top of Apache Tika and an essential part of the engineering behind the Panama Papers, Swiss Leaks and Luxembourg Leaks investigations.
It supports Redis-backed queueing for distributed, parallel extraction and will write to Solr, plain text files or standard output.
For guidance and instructions, please see the wiki.
Initialy developed by Matthew Caruana Galizia at ICIJ.
We welcome contributions! Please submit pull requests or contact us directly.
Copyright (c) 2018 International Consortium of Investigative Journalists. See LICENSE.