Review the MFS interface

[schomatis]

As suggested by @Stebalien we need to review (and redesign) the MFS interface. I need to import some PRs left in go-ipfs (created before the MFS package extraction) here and in the course of that try to think what could be improved here.

(@Stebalien Any pointers of what you had in mind regarding changes to the MFS interface will be very useful.)

[Stebalien]

In terms of the code itself, we need to do a through audit of performance and correctness. In terms of the interfaces, I'd like to look into adding transactions and using them. Currently, many of our operations that should be atomic are most definitely not.

[schomatis]

In terms of the code itself, we need to do a through audit of performance and correctness.

Agreed!

Currently, many of our operations that should be atomic are most definitely not.

Could you provide an example of that? Are you referring to the write-flush issues of ipfs/kubo#4758?

[Stebalien]

I'm referring to ipfs commands like ipfs files create ..., ipfs files mv .... Ideally, each MFS command would open a transaction, perform some operation, and then commit the transaction. The IPFS interfaces need to support these kinds of transactions (IMO).

But we also need to fix the flushing issues.

[schomatis]

The IPFS interfaces need to support these kinds of transactions (IMO).

@Stebalien What would be the difference between a commit and a flush? Could you give me some concrete example of the API you would expect from it to start thinking how that would look like in the code? (I mean the series of calls an ipfs files mv command would do to the MFS API.)

[Stebalien]

What would be the difference between a commit and a flush?

A flush just writes the current state to disk. When I say "commit", I'm talking in the transactional sense. That is, filesystem operations should be atomic. Ideally, if two clients rename a file at the same time, we should end up with one file not two. If one client renames a file while the other renames the parent directory, the file should end up renamed in the new parent directory (currently, the file could get deleted).

[schomatis]

Ideally, if two clients rename a file at the same time, we should end up with one file not two.

That would be for example two CLI users sending an ipfs files mv command, right? This would trigger two MFS Mv operations over the same FilesRoot, could you describe in more detail how would you think the MFS would implement this transaction model? The first solution that comes to mind would be to just use a lock mechanism over directories to not allow two moves to take place (or a simpler first approach, just locking the entire MFS root on the server side, which may be too much).

[Stebalien]

Locking everything may be the simplest solution for now.

We could also introduce a concept of opening a directory for writing (locking on open). However, if we do that, we have to be careful to either:

1. Take locks in the right order.
2. Not block if the lock is taken and, instead, enter a re-try loop.

[aschmahmann]

This repository is no longer maintained and has been copied over to Boxo. In an effort to avoid noise and crippling in the Boxo repo from the weight of issues of the past, we are closing most issues and PRs in this repo. Please feel free to open a new issue in Boxo (and reference this issue) if resolving this issue is still critical for unblocking or improving your usecase.

You can learn more in the FAQs for the Boxo repo copying/consolidation effort.