split the streamly package into multiple packages

[harendra-kumar]

The streamly package is approaching 200 files and takes significant time to compile with more than 70 source modules, we may want to look at the possibility of splitting the package into smaller packages to make the ecosystem modular. Here is one approach:

1. streamly: remains as it is, a batteries included master package, including everything
2. streamly-core: Unfold/Stream/Pipe/Fold/Parser/Array/Time modules
3. streamly-network: the networking stuff
4. streamly-filesystem: the filesystem stuff
5. streamly-unicode: the unicode text processing stuff
6. streamly-json: the json parsing, serialization stuff
7. ... many other streamly packages like streamly-zlib, streamly-statistics

Programmers can mix and match the packages they want to include. This can possibly reduce the amount of time take to build, though streamly-core would still remain quite heavy.

We can continue to have all the packages in the same mega repository.

[Bodigrim]

Would it be theoretically possible to have a hypothetical streamly-core depending on GHC boot libraries only? Or does it fundamentally entail something else?

[harendra-kumar]

If you look at the dependencies of streamly, other than boot libs this is what we have:

• atomic-primops (for concurrent streams)
• monad-control (for concurrent streams)
• heaps (for concurrent streams)
• lockfree-queue (for concurrent streams)
• network

So if we just take out the concurrent streams and network stuff in separate packages, which is very much possible, then we will depend only on boot libraries.

[harendra-kumar]

We can try a feasibility prototype after the 0.8.0 release in next few weeks.

[harendra-kumar]

I am working on separating a streamly-core package. There is a decision to be made about the safe resource management combinators bracket and finally. Those are important operations that would be useful in a core package. On the other hand they require a dependency on monad-control. Another class of operations that require a dependency on monad-control is time related combinators like takeInterval or intervalsOf. If we move these operations in another package then we could depend only on boot packages.

@Bodigrim do you have any suggestions or opinions on this?

[Bodigrim]

Take my words with a grain of salt, because I am only a potential user and not well acquainted with the library, but I'd slightly prefer only boot libraries in dependencies of streamly-core. The reason is that monad-control entails several more transitive dependencies, which are all well-maintained, but nevertheless elongate the critical path.

@hasufell what do you think?

[harendra-kumar]

Though I would like to avoid any additional deps, but we may have to do some balancing here.

monad-control only brings transformers-base and base-orphans additional. These are fairly small packages, the former is a single type class, and the latter is some instances.

[harendra-kumar]

Since this might possibly break things we will do this in the upcoming major 0.9.0 release.

[adithyaov]

streamly is currently split to streamly and streamly-core in 0.9.0.
We can target the rest to 0.10.0.

[georgefst]

Is there any reason the decision wasn't taken to have streamly re-export everything in streamly-core, as e.g. optics does with optics-core? This would have made migration simpler. And more importantly, it would make it much easier to search for definitions on Hackage.

[harendra-kumar]

We did re-export everything from streamly initially. I think the problem was that if we were to re-export than we had to re-export everything including all the internal modules, otherwise there was some issue. I do not remember the exact issue, need to find out what it was. Maybe because we have to depend on both streamly and streamly-core to use internal modules from streamly-core and that caused some ambiguity in importing modules.

[harendra-kumar]

If it works we can think of doing it in 0.9.1 or would that cause more breakage?

[harendra-kumar]

I think if we re-export then we cannot depend on both streamly, and streamly-core becuase GHC cannot distinguish if a module is re-exported, and would consider it as another module with the same name.

[harendra-kumar]

For the documentation issue we can present a unified documentation/search on the streamly website https://streamly.composewell.com/ and a hoogle search as well. But searching on hackage will still remain problematic.

[georgefst]

I think if we re-export then we cannot depend on both streamly, and streamly-core becuase GHC cannot distinguish if a module is re-exported, and would consider it as another module with the same name.

Ah, it's annoying that that would be another breaking change for anyone who has already adapted their codebase to 0.9. I'd personally consider it worth it in 0.10, but others may disagree.

[harendra-kumar]

Current division in streamly and streamly-core is good enough for now. Raised #2690 for the re-export issue.