multithread the O in I/O

[odanoburu]

multithread the input in this library is a complex task, because we'd have only one encode_dictionary to be kept consistent across each thread. but the output is another matter: we could have each document to be written to disk as soon as it is hashed. as I/O Is our main bottleneck, this could improve performance up to a point. (but how many threads we could have until we are bounded by the machine's maximum I/O capability?)

[fccoelho]

If its more than one we. It will be a gain.

…

On Fri, May 19, 2017 at 4:29 PM, bruno cuconato ***@***.***> wrote: multithread the input in this library is a complex task, because we'd have only one encode_dictionary to be kept consistent across each thread. but the output is another matter: we could have each document to be written to disk as soon as it is hashed. as I/O Is our main bottleneck, this could improve performance up to a point. (but how many threads we could have until we are bounded by the machine's maximum I/O capability?) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIjW10_8M6DqIOmN5c-QpPD3zyaTRPNks5r7e2qgaJpZM4Ng7Xw> .

-- Flávio Codeço Coelho ================ +55(21) 3799-5551 Professor Escola de Matemática Aplicada Fundação Getulio Vargas Praia de Botafogo, 190 sala 312 Rio de Janeiro - RJ 22250-900 Brasil