Streams ROADMAP

Done

1. Pre crafted collection of objects (in RAM) =>
   1. Transform =>
   2. store in file [end]
2. Async Iterator crafted objects (simulated external stream) =>
   1. Transform =>
   2. store in file [end]
3. Async Iterator crafted objects (simulated external stream) =>
   1. Transform data (prepare for CSV format) =>
   2. Transform to CSV =>
   3. store in file [end]
4. Sync Iterator crafted objects (Speed demo up) =>
   1. Transform data (prepare for CSV format) =>
   2. Transform to CSV =>
   3. store in file [end]
5. Sync Iterator crafted objects (Speed demo up) =>
   1. Transform data (prepare for CSV format) =>
   2. Transform to CSV =>
   3. Compress GZip =>
   4. store in file [end]

TODO ?

1. Real basics of Streams

   1. listeners
   2. back pressure thingies
   3. memory / cpu etc...

2. What not to do!

   1. buffer files completely in RAM, then store!
   2. write to local temp file (often slow, and solvable by implementing streams api)
   3. Prepare big chunks of collections in ram (use iterators pl0s)

3. From MongoDB Stream (simple find) => stuff

4. From MongoDB Stream (aggregations) => stuff

5. From SQL ;) => stuff

6. Something with encryption

7. Arnold Generator

8. Imagary/Video streaming to front end

9. UDP stream (imagery / data)

10. Maybe incorporating socket.io

How to: Unit test streams

1. Intercept traffic in a stream
2. Simulate/TEST an error somewhere inside the stream@example``````

Profile

(See: https://clinicjs.org/)

1. npm i -g clinic`
2. clinic doctor -- node ./dist/index.js

Logging

This project does logging to a local file : output.log

If you want to see what's happening: tail -f output.log

Sources

• https://blog.risingstack.com/the-definitive-guide-to-object-streams-in-node-js/
• https://nodejs.dev/en/learn/nodejs-streams/#how-to-create-a-readable-stream
• https://jscomplete.com/learn/node-beyond-basics/node-streams
• https://nodejs.org/api/process.html#processstdout (about blocking the event loop)
• https://thenewstack.io/node-js-readable-streams-explained/
• https://www.knowledgehut.com/blog/web-development/compression-decompression-of-data-using-zlib-in-Nodejs
• https://medium.com/@hemitrana1409/node-js-zlib-module-a6c2d8b9c195
• https://clinicjs.org/