Skip to content

v1.7.0: Add experimental GCS support
Compare
Choose a tag to compare
@0xmichalis 0xmichalis released this 22 Dec 10:18
· 45 commits to master since this release
v1.7.0
d70f944
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Add experimental GCS support in Afero. Experimental because the CI infra of afero does not test with real GCS buckets.

Limitations:

  • No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
  • No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
  • NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.

Performance implications

  • Sequential reads are performant
  • Sequential writes are performant.
  • Seek + Read or ReadAt is performant after the initial seek. (produces a warning)
  • Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
  • Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.
Assets 2
Loading