PseudoFS with extra contracts

[jkawamoto]

Currently, PseudoFS needs that all underlying hosts are accessible to write a file and close the filesystem. However, it hardly expects it, especially if you increase the number of hosts. Since this is a network application, it might not be a good idea to assume all remote hosts are always working.

I'd propose that PseudoFS should have some extra hosts so that it can select live hosts for each operation (flushSectors). For example, if we need to upload files to 30 hosts and create a PseudoFS with 35 contracts (hosts), PseudoFS.flushSectors tries to acquire 30 hosts and can skip some hosts that don't respond. Also, if h.Append(sector) fails with a host, it can replace the host with another extra host. Then, we can keep using the PseudoFS even if some hosts get temporarily unreachable. As a result, each sector would be stored in a different host set, but it won't be a problem because metafile keeps which hosts have each sector.

This seems like it doesn't require a significant change but will increase reliability.

It is related to #69.

[lukechampine]

As a result, each sector would be stored in a different host set, but it won't be a problem because metafile keeps which hosts have each sector.

The metafile format assumes that all sectors in a file are stored on the same set of hosts, and that every host has all of the shards. So the format would need to be changed to accommodate this functionality.

This definitely seems in the same vein as #69, which would also likely require a change to the metafile format.

Perhaps this is the right catalyst to pursue PseudoS3 (#67). PseudoS3 has different needs from PseudoFS and will require a different format anyway. Much of the session-related code of PseudoFS can be reused, but overall I expect that PseudoS3 will be much simpler, because it doesn't need to support file attributes and values are immutable (i.e. you don't need to support random-access writes, which are a big source of pain in PseudoFS). I will take a stab at this over the weekend.

[jkawamoto]

Since the number of files we have is not huge, we wouldn’t mind if metafile format changes. But, migrating from PseudoFS to PseudoS3 needs a lot of work, and we prefer to keep using PseudoFS. Actually, we need random reads although we don't use random writes.

[lukechampine]

The desired change -- maintaining a "buffer" of extra hosts, using only e.g. 30 of 35 hosts for each chunk, and allowing each chunk of a file to be stored on a different host set ("heterogenous chunks") -- is significant. I'm not sure it can be achieved in a short timeframe. Here are some alternatives:

1. A "host buffer" could be used for just the first sector of a file. The first 30 hosts that work become the single host set for that file. This would not require changing the metafile format, and only small changes to PseudoFS. It should definitely reduce the number of upload failures, but it's hard to say how effective it would be.
2. StoreWise could create their own file format, where each chunk of a file is stored as a separate metafile. This would enable storing each chunk on a different host set, but would probably entail a lot of work.
3. StoreWise could use smaller host sets. Instead of 30 hosts, 15 or 24 hosts is probably fine. This should reduce failures by weeding out less-reliable hosts.
4. I have made good progress towards a PseudoKV abstraction. It will support heterogenous chunks. Ultimately I think this is what StoreWise will want to use. However, it's unclear how much more time/energy is required for PseudoKV to be production-ready.

I will take a stab at a new metafile format that supports heterogenous chunks. If I feel it can be done within a week or two, I'll go for it. If not, we should choose another option; option 1 seems to be the most viable.

[MeijeSibbel]

Thanks for the extra feedback Luke, much appreciated. As a imminent solution, especially for the Clear Center Integration, decreasing the host-set size will relieve some stress on the system. That said it's not a permanent nor comfortable solution but we can do it temporarily while we work on (4). I think (1) could work as well, however, it does not tolerate anymore host failures after the working contract set is formed so it won't be as resilient as (4), which if I'm correct, would allow > 1 hosts to fail mid-upload stream. That said it might get us halfway there. Will ask Junpei provide some more feedback as well.

Please keep us posted and let us know if you need any help. Thanks!

[jkawamoto]

To my understanding, (1) means we create a PseudoFS with say 35 hosts, and if we store a file, PseudoFS selects 30 live hosts to upload the first sector. Then, it uses the same hosts to upload the rest of the sectors. If we store another file, it re-selects 30 hosts that are alive at that moment. Is that correct?

If so, I think (1) is a good option for now. We could migrate to PseudoKVS at some point. However, we wouldn't be able to make it by the next release schedule.

[lukechampine]

Hmm, unfortunately I think (1) is less viable than I originally thought. PseudoFS buffers writes across all files, so the "first sector" of a file may also include the "third sector" of another file -- meaning you can't select the hosts for the first file arbitrarily, since they need to match the hosts for the other file as well.

[jkawamoto]

I think that is not a problem.

the "first sector" of a file may also include the "third sector" of another file

This means we could upload at least two sectors to the host set, and then we might be able to upload another sector to them.

If the upload of the previous two sectored is done say one hour ago, we should consider some host gets offline. However, we call PseudoFile.Sync and PseudoFS.Close frequently, we can expect the host set is still working.