🎯 Understanding MVP Requirements for Project Motion

[masih]

Hello everyone, 👋

I'm initiating this discussion to gain a clearer understanding of the Minimum Viable Product (MVP) requirements for Project Motion. I have some specific areas where I'm seeking clarity, and I hope we can leverage this discussion to gather insights, answer questions, and ensure everyone is on the same page. 🤝

Here are some of the points that need defining in terms of scale and operation:

1. Number of store requests per second: 📈 What's the scale we're aiming for in terms of store requests per second? Are there benchmarks we're striving towards, or any existing standards or data we should keep in mind?

2. Number of retrieve requests per second: 📉 Similarly, what are our goals for the number of retrieve requests per second? How does this compare to the number of store requests, and what are the considerations we should take into account?

3. Data size (median, p95): 📊 Could we get some insights on the expected median and p95 data size for our operations? Understanding this could help us optimize data handling and management.

4. Read pattern (e.g. random/sequential relative to order of storage): 🔀 What are the expected patterns for data read? Will it be random or sequential relative to the order of storage? This could significantly impact our design and optimization strategies.

5. TTFB (Time To First Byte): ⏱️ What are our expectations for TTFB? This could be particularly important for understanding our network and data transmission needs.

6. Maximum content length accepted by POST /v0/data: 🏋️‍♀️ Lastly, what is the maximum content length accepted by the storage API.

I believe addressing these points will help us carve out a clear roadmap for the MVP and beyond. I'd appreciate if those who have information, thoughts, or insights could contribute to this discussion. 🙏

[willscott]

I see the first 3 points as all being part a question of how motion should handle horizontal scale-out.

As data size increases, we'll presumably want some form of data parallelism - multiple instances to handle gets/puts, and more than one 'cache volume' for temporary storage.

If we first get everything set up with docker compose, it seems like it should be then in a subsequent milestone to better understand how we can scale out the 'motion' service pods and storage separately from a master database containing the state / mapping / metadata

[masih]

Is there a sense of the volume Motion MVP should be able to handle?

[xinaxu]

Data size (median, p95)

Should we also define a max size of file we should support for MVP? i.e. AWS S3 supports up to 5TiB for each file. It also recommends to use multipart upload when the size of file is more than 100MB.
We may want to define a max of 100GB, or something even less like 10GB for MVP then figure out the multipart upload beyond MVP. Having a large file streamed via a single connection over the wire is not very practical.

[masih]

That's a great point. For MVP, considering the FileCoin sector sizes, maybe the easiest thing to do is to pick either 32GiB or 64GiB as the maximum size of bytes accepted by POST /v0/data.

I'll add this to the discussion description. Thanks

[TorfinnOlsen]

@masih just want to confirm here in case anyone reads back through this we are locking this at 31GiB correct?

[masih]

Yes

[marinakosti]

From @Angelo-gh3990

1. Number of store requests per second: 1 (assumes requests can be done in parallel)
2. Number of retrieve requests per second: 1 (assumes requests can be done in parallel)
3. Data size (median, p95): 1 GiB - 10 TiB
4. Read pattern: random
5. TTFB (Time To First Byte): 1s
6. Maximum content length accepted by POST /v0/data:

• No limits if we are streaming data
• Need to be able to resume if there is an interruption

[masih]

No limits if we are streaming data
Need to be able to resume if there is an interruption

What is the rationale for the MVP to allow limitless streaming and support resumption?
Instead, does it make sense to reduce the scope of MVP and postpone these requirements for later?
For the MVP we would then use the limit of 32GiB max upload size and no support for resumption.

What are your thoughts @Angelo-gh3990?

[marinakosti]

@masih can consider this potentially as a "nice to have" but needs to be supported in subsequent releases

[masih]

Further to discussion with @marinakosti, for the MVP maximum size of 32GiB upload with no support for resumption is acceptable.
Beyond the MVP, limitless streaming with support for resumption is needed.

Therefore, the requirements rot the MVP are:

1. 1 rps
2. 1 rps
3. Median: 1GiB, p95: 10TiB
4. Read pattern: random
5. 1s
6. 32GiB, with no support for resumption

[xinaxu]

@masih The theoretic max size to fit one item into a sector is around 31.74GiB

• 127/128 for padding
• account for CAR headers and Dag nodes other than the leaf

[masih]

Let's go with a round 31Gi and increase as needed. It will be a one liner change.

[marinakosti]

For Cloud Forest:

1. 1 for storage
2. no requirement (retrieval only needed in case of a diaster)
3. ongoing storage: 2-3TiBs / day -> 1PiB / year
   Plus existing data clients already have (e.g., some companies have tape / cloud backup - will be more of a cold storage, sometimes)
4. Will need to test, TBD
5. 1 second is fine. Max delay = 1 hour
6. largest files 160-180 GIB, 2-3 of those/ day per client