feat(virtual-stream): add POC for discussion

[briancavalier]

Add virtual-stream, a package for creating "virtual streams", which are in-memory, "fake" representations of event streams where all the events (and their occurrence times) are known ahead of time. They are easy to visualize (i.e. render), and useful for testing. It'd be easy to write constructors to create virtual streams from ascii marble diagrams, arrays, etc to facilitate testing. It should also be possible to implement functions that convert to/from real most.js event streams, again for testing and visualization.

Motivations

Testing

Given a way to convert between real event streams, virtual streams could provide a simple way of testing event streams. A few examples, with the help of some imaginary helper functions:

it('...', () => {
  // Define a virtual stream as a starting point to test the
  // map() operator using a template string
  const vs = vstreamFromString`-${1}-${2}-${3}`

  // Turn it into a real stream 
  const s = vstreamToStream(vs)

  // Define the expected result as a virtual stream
  const expected = vstreamFromString `-${2}-${3}-${4}`

  // Map the real stream
  const actual = map(add1, s1)

  // Turn the newly mapped actual (a real stream) back into a
  // virtual stream and compare it to the expected virtual stream
  return vstreamFromStream(actual).then(eq(expected))
})

Visualization

Envision a virtual stream as a finite Array of { time, value } objects. Given that, rendering it into various formats is relatively simple:

// Imaginary vstreamFromArray() that turns an array into a virtual stream
const vs = vstreamFromArray([{ time: 1, value: 'a' }, { time: 3, value: 'b' }, { time: 5, value: 'c' }])

// Render the virtual stream to a string
const rendered = vstreamToString(vs)

console.log(rendered) //> -(a)-(b)-(c)|

And given a way to convert from a real stream to a virtual stream, real streams can be rendered easily:

// Turn an event stream into a virtual stream
const vs = vstreamFromStream(take(3, constant('a', periodic(2))))

// Render the virtual stream to a string
const rendered = vstreamToString(vs)

console.log(rendered) //> (a)-(a)-(a)|

Details

This implementation uses an array of "virtual events", like Array<{ time: number, value: A }>, which seems like a solid, explicit way to represent events. On top of that it adds a VirtualStream sum type that explicitly captures the notions of a:

• finite stream - a stream known to end at a particular time
• infinite stream (in reality, potentially infinite) - a stream where we know all the events up to a particular time, but don't know anything about the stream after that
• errored stream - a stream known to have failed at a particular time with a particular Error value

These variants make it relatively easy to do things like:

• explicitly compare a virtual stream to a real event stream for testing
• parse a virtual stream from an ascii marble diagram
• render to an ascii marble diagram or html

Open questions

• Is Array really the right underlying type? I like the simplicity of it, however:
  • A lazy structure, such as a function that produces an array, might be more useful in some cases. For example, a lazy structure allows turning infinite real streams into virtual streams, whereas requiring a synchronous Array would not. See the periodic Visualization example above, which had to use take before calling vstreamFromStream.
• Does this belong in the @most/core monorepo or maybe another (mono)repo dedicated to virtual streams or testing?
• Should we move VirtualTimer here, too?
• Should test-specific utils, like assertions about streams, live here or in a separate package?

Todo

• to/fromStream
• types
• tests
• docs

[davidchase]

Does this belong in the @most/core monorepo or maybe another (mono)repo dedicated to virtual streams or testing?

Have we decide for sure about the monorepo ? If so do we know what should go in and what shouldnt? like couldnt we add dom-event, sample, etc to the monorepo as well?

Maybe once we figure what goes in and what doesnt it would help answer about VirtualTimer and specific utils.

I can see @most/virtual-stream that would house all of the aforementioned stuff instead of putting into @most/core monorepo

On the other hand couldnt move all of the packages under github.com/mostjs to the @most/core monorepo? other than package-starter, eslint and examples.

If that was the case then it would just be another package under the core monorepo.

[briancavalier]

Have we decide for sure about the monorepo

I think so. I mean, this repo already is a monorepo :). We still have manyrepos since we started this one later that the others, of course. As for what goes in and what doesn't, lately I feel like we should just move everything from mostjs that we plan to continue developing into this repo. Well, ok maybe experimental stuff that we're just incubating should stay out.

If that was the case then it would just be another package under the core monorepo.

Yeah, that's now I feel right now, but I'm also open to separating it into a second monorepo if there's a compelling reason.

p.s. monorepo feels like the wrong word now, lol. it's more like a multirepo, as in "multiple packages per repo". 🤷‍♂️

[briancavalier]

Just pushed an update that includes toStream/fromStream and VirtualTimer.

[briancavalier]

Hmmm, there's a potential circular dependency: eventually, we'd probably want to rewrite the @most/core package's tests using virtual-stream, but virtual-stream's fromStream needs runEffects. I can think of 2 ways to deal with it--maybe there are others:

1. Duplicate some part of runEffects in virtual-stream. Not crazy about this, although the current runEffects does more than fromStream actually needs.
2. Move runEffects to its own package, e.g. @most/run or whatever. I don't love this either, although maybe it's not terrible. For example, it's not hard to imagine other external "run" functions that do things like automatically providing a customized scheduler to integrate with systems that already schedule tasks, etc.

[briancavalier]

Rebased on latest master ... also, simplified fromStream a bit more. It's quite nice now, imho.

[TylorS]

Anything happening here still?

[briancavalier]

Yes, definitely. I've been focusing on docs and getting @most/core 1.0 ready to publish. Once that's done, I'm def planning to come back here.

[gabejohnson]

Is there still a desire to move on this?

[briancavalier]

Hey @gabejohnson. Yeah, I'm still keen on something like this, and I need to make time to revisit.

Lately, I think it can all be much simpler, though. For example, it's actually possible to do most/all of this stuff by just having a virtual time scheduler, and using at and merge or at and continueWith. That is, I'm not as convinced we really need to create a bunch of new code to achieve the goals of this PR.

There are definitely still things that would be useful, such as rendering a stream into an ascii diagram.