Design: Support truncate discussion

[CamStan]

I believe UnifyFS's current implementation for truncate() is left over from CRUISE, which didn't worry about shared files. Implementing this for UnifyFS could get complicated as, for one, any process could call truncate and it would need to reflect globally.

This issue is for discussing whether or not we want to support file truncation, and if so, how to go about doing so.

Current Implementation

At the moment, when unifyfs_fid_open() gets called on a file that already exists(but is not laminated) with the O_TRUNC flag, unifyfs_fid_truncate() gets called:

UnifyFS/client/src/unifyfs.c

Lines 1359 to 1360 in 9b9a831

	if ((flags & O_TRUNC) && (flags & (O_RDWR | O_WRONLY))) {
	unifyfs_fid_truncate(fid, 0);

which calls unifyfs_fid_shrink()and returns an error:

UnifyFS/client/src/unifyfs.c

Lines 1182 to 1186 in 9b9a831

	/* unknown storage type */
	rc = (int)UNIFYFS_ERROR_IO;
	}
	return rc;

However, when this error is returned back to unifyfs_fid_open(), it simply gets ignored. All this only happens on the process/processes that open the file with the O_TRUNC flag.

Note

It looks like we use truncate() in unlink() as well (#197)

UnifyFS/client/src/unifyfs.c

Lines 1427 to 1430 in 9b9a831

	int unifyfs_fid_unlink(int fid)
	{
	/* return data to free pools */
	int rc = unifyfs_fid_truncate(fid, 0);

Reproduce

Reopen any file with the O_TRUNC flag that has already been created by UnifyFS.

Ran into this while integrating UnifyFS with VeloC. This gets called in the VeloC heatdis_file.c example which gets intercepted and calls:

UnifyFS/client/src/unifyfs-stdio.c

Lines 309 to 313 in 6368cc2

	/* w+ ==> truncate to zero length or create file for update
	* (read/write)
	*/
	open_rc = unifyfs_fid_open(path, O_RDWR | O_CREAT | O_TRUNC,
	perms, &fid, &pos);

[adammoody]

The veloc case comes up via the fopen("w") wrapper, which adds O_TRUNC when calling unifyfs_fid_open(). Similarly, one might use the O_TRUNC flag directly in open(). The semantics in this case are to open the file and, if the file already exists, set its file size to 0.

Other wrappers of interest are ftruncate() and truncate(), which may be used to set a file size to an arbitrary value. If the file length is extended, the new bytes are to be treated as null bytes "\0".

To cover it all, we'll have to think about how to handle both shrink and extend cases.

[adammoody]

With our current implementation, we could support extending a file with truncate with reasonable effort. The current RPC that computes the file size returns the offset of the max byte that it finds which was actually written:

UnifyFS/server/src/unifyfs_request_manager.c

Line 475 in 351bf85

int rm_cmd_filesize(

This could be modified to 1) update some metadata in the key/value store whenever truncate/ftruncate are called, 2) query for that key/value in the file size RPC and return the maximum of the truncate location and the max value computed from above.

This supports the case that someone later writes past the end of a file, even after truncating, e.g.

fd = open(file)
ftruncate(fd, offset=100)
lseek(fd, offset=200)
write(fd, length=1)

Here the truncate key/value would return 100, but we'd still find the metadata for the write at 200, so the maximum of those values gets the correct value.

As an alternative implementation, it's tempting to just write zeros from the current end of file to the truncate location. In fact, this is how CRUISE implements truncate. However, that's not a viable approach for UnifyFS, since we don't allow multiple procs to write to a given byte more than once.

[adammoody]

Working on truncate in #419

[CamStan]

@adammoody, did #419 resolve this, or is there more than needs to be done?

[adammoody]

Yes, we can resolve this now. Thanks.