Gcsfuse v2.0.1

gRPC [Allowlist access]:

• Support added to communicate with the GCS backend via gRPC. gRPC is enabled by passing the flag --client-protocol=grpc as mount command. gRPC internally uses http2.
• To use gRPC, contact the Google Cloud Storage gRPC team at gcs-grpc-contact@google.com with a list of GCS buckets to allowlist.
• Performance guidance:
  • gRPC provides improved performance when using a single region or dual region bucket that is co-located in the same region as the compute instance.
  • gRPC provides improved performance for most workloads, but should not be used for workloads with high parallelism (>32 threads).

Bug fixes and improvements:

• Respect log-format flag for stdout logs. (#1770)

Upgraded dependencies for better stability and reliability: (#1767, #1764, #1819)

Gcsfuse v2.0.0

1. File caching: A client-based local file cache that lets repeat file reads be served from a faster cache storage of your choice, such as a Local SSD, Persistent Disk, or in-memory /tmpfs. It significantly improves your read performance on subsequent reads/epochs. (more details here)

   • New config-file flags (more details here)

         cache-dir: (default=””)
         file-cache:
             max-size-mb: (default=-1) 
             cache-file-for-range-read: (default=false)
     

   • Behavior:
     • cache-dir: The file-cache feature is disabled by default. To enable it, pass a directory to cache-dir.
     • max-size-mb: This is used to limit the total capacity that the Cloud Storage FUSE cache can use within the specified cache directory. The eviction of cached data is based on a least recently used (LRU) algorithm. By default, a value of -1 signifies all available capacity in the specified directory.
     • cache-file-for-range-read: Enable file cache for Random & partial reads: Support for asynchronous cache ingestion of files on offset reads through cache-file-for-range-read.

2. Updates to metadata-cache (stat & type caches) feature
   More details here.

   • Deprecated CLI flags
     • stat-cache-capacity was replaced with metadata-cache:stat-cache-max-size-mb.
     • stat-cache-ttl & type-cache-ttl were replaced with metadata-cache:ttl-secs.
   • New config-file flags (more details here)

        metadata-cache: 
              stat-cache-max-size-mb: (default=32)
              ttl-secs: (default=60)
              type-cache-max-size-mb: (default=4)
     

   • Behavior changes
     • type-cache-max-size-mb: Type-cache size limit can now be configured using this configuration, at per-directory level.
     • stat-cache-max-size-mb: The stat-cache size limit is now configured as memory size (in MiB) using this configuration instead of capacity (count of entries). Also, stat-cache is now configured at mount-level as opposed to bucket-level.
     • ttl-secs: stat-cache and type-cache TTLs have been merged into a single-TTL value, configurable via this configuration. Also, this new TTL is set as an integer multiple of seconds, as opposed to fine-grained duration (such as 1s500ms with the old flags).
     • To get the best performance, stat-cache-max-size-mb, ttl-secs and type-cache-max-size-mb can be individually set to -1 for unlimited size/ttl limits. Setting any of them to 0 disables the corresponding cache.
     • Use of deprecated flags will give warning, but for backward compatibility, their values will still heuristically be converted to the new flags and will be used. If both new and old flags are set, old flags are ignored.

3. Managed folders compatibility

   • GCSFuse respects the managed folder IAM permissions in the bucket mounted with GCSFuse.
   • Pass --implicit-dir flag to view managed folders.

4. Memory Improvements

   • The average memory usage of stat-cache is now reduced by up to 40%.

5. Bug fixes and improvements

   • Fix concurrent list directory and file creation
     • #1726
     • #1603
   • Improvements in logging
     • Added log for successful/failed mounting on stdout when log-file is set in non-foreground mode.

6. Removed --experimental-local-file-cache flag which used to be for an old experimental local file cache feature.

7. Upgraded dependencies for better stability and reliability. (#1744, #1746)

Gcsfuse v2.0.0~beta.1

v2.0.0_beta.1

Enable or disable file cache via cache-dir (#1723)

Gcsfuse v1.4.2

• Upgrade Go to v1.21.6 (from v1.21.5)
• [Perf Improvement] Improved read performance by reusing existing readers for subsequent reads. This reduced the overhead of unnecessary GCS connections and increased end-to-end read-bandwidth up to 75% for files smaller than 1 MB. (#1683)
• Bump dependencies (#1630, #1650, #1671, #1672)

What's Changed

• Changing flag name to cache-file-for-range-read in experiment perf tests. by @Tulsishah in #1598
• Upgrade Go version from 1.21.5 to 1.21.6 by @sethiay in #1628
• fix flaky log test by @ashmeenkaur in #1641
• Updating NVIDIA A100 GPU machine for pytorch2.0 by @Tulsishah in #1610
• Bump dependencies Part 1 by @sethiay in #1630
• Fix failure status in KOKORO tests by @ashmeenkaur in #1649
• Bump dependencies Part 2 by @ashmeenkaur in #1650
• Fixing troubleshooting doc by @Tulsishah in #1665
• dependabot should bump only direct dependencies by @ashmeenkaur in #1672
• Revert and bump only direct dependencies by @ashmeenkaur in #1671
• Changing tf model bucket by @Tulsishah in #1678
• [Perf Improvement] Random reads should re-use previous reader even if existing reader can serve the request by @ashmeenkaur in #1683

Full Changelog: v1.4.1...v1.4.2

Early Beta Release for GCS Fuse v2.0.0

Early Beta Release for GCS Fuse v2.0.0

Gcsfuse v1.4.1

• GCSFuse JSON format logs now seamlessly integrate with google-fluentd, enabling easier log analysis and management within Google Cloud's logging ecosystem. (Reference: http://cloud.google.com/logging/docs/agent/logging/configuration)
• #1541 GCSFuse now adheres to proxy settings defined in environment variables (HTTP_PROXY, HTTPS_PROXY), providing greater adaptability in various network configurations.
• GCSFuse mount command is now logged at INFO level, providing better visibility into mount operations and aiding in troubleshooting.

What's Changed

• Kokoro is not triggering on the define branch in the config by @Tulsishah in #1561
• Fix e2e kokoro tests by @Tulsishah in #1563
• Fix e2e tests for release branch by @Tulsishah in #1567
• Adding log to print all configuration flags at start of gcs fuse mount by @ankitaluthra1 in #1565
• Update SLO by @Tulsishah in #1575
• Http proxy fix by @ashmeenkaur in #1562
• Adding dependabot rules by @Tulsishah in #1573
• make logging keys fluentD compatible by @ashmeenkaur in #1591
• Remove branch name env variable in periodic e2e tests by @Tulsishah in #1595
• log failed mount message in e2e tests by @ashmeenkaur in #1599

Full Changelog: v1.4.0...v1.4.1

Gcsfuse v1.4.0

• Added support for PyTorch 2.0
• Upgraded the following dependency packages for fixing the given CVEs
  • golang version from 1.21.4 to 1.21.5
    • CVE-2023-45285
    • CVE-2023-39326
  • golang.org/x/crypto from v0.14.0 to v0.17.0
    • CVE-2023-48795

What's Changed

• update gcsfuse logging guide by @ashmeenkaur in #1538
• Pytorch 2.0 support compile mode by @Tulsishah in #1539
• Reduce epoches in pytorch2.0 model by @Tulsishah in #1542
• Remove log rotation script from gcsfuse long haul tests by @ashmeenkaur in #1543
• fix flaky log rotation test by @ashmeenkaur in #1544
• dependabot-fixes by @ashmeenkaur in #1553
• upgrade to go 1.21.5 by @ashmeenkaur in #1552
• Lower freq of syncs in periodic-perf-tests by @gargnitingoogle in #1549
• Fix tf model by upgrading driver version by @ashmeenkaur in #1557

Full Changelog: v1.3.0...v1.4.0

Gcsfuse v1.3.0

• Added support for Log rotation in gcsfuse. Please note that this will be enabled by default. For more details - https://cloud.google.com/storage/docs/gcsfuse-config-file
• Security fixes:
  • Upgraded golang version from 1.21.3 to 1.21.4.
  • Upgraded go-storage-client module from 1.31.0 to 1.34.1 with their dependencies. The following security vulnerabilities are resolved as part of this upgrade:
    • CVE-2023-39325
    • CVE-2023-44487
    • GHSA-m4GHSA-vx74-f528-fxqg25-mq94-257g
    • CVE-2023-3978

What's Changed

• Bump certifi from 2023.5.7 to 2023.7.22 in /perfmetrics/scripts/bigquery by @dependabot in #1468
• Bump urllib3 from 1.26.16 to 1.26.18 in /perfmetrics/scripts/bigquery by @dependabot in #1469
• Bump urllib3 from 1.26.15 to 1.26.18 in /perfmetrics/scripts/load_tests/python by @dependabot in #1448
• Bump werkzeug from 2.3.4 to 3.0.1 in /perfmetrics/scripts/load_tests/python by @dependabot in #1466
• Creating big query tables for perf tests - 3 by @Tulsishah in #1450
• Fix pr perf test by @Tulsishah in #1490
• Build script and config files to run periodic tests on different branches by @Tulsishah in #1455
• Add exec-mode to script files by @gargnitingoogle in #1486
• Support --config-file flag for periodic perf tests by @Tulsishah in #1493
• Remove chmod calls for running scripts by @gargnitingoogle in #1489
• Fix experimental periodic perf test by @Tulsishah in #1499
• Change branch name and end date by @Tulsishah in #1509
• Collate fio installations at one place by @gargnitingoogle in #1506
• Fix clat-percentile accuracy in fio by @gargnitingoogle in #1503
• Run periodic tests for read cache branch with cache enabled by @sethiay in #1515
• Upgrade tensorflow model dependencies by @ashmeenkaur in #1510
• Pytorch 2 support by @Tulsishah in #1513
• Fix inverted commas in pytorch setup by @Tulsishah in #1527
• Upgrading go version 1.21.3 to 1.21.4 by @Tulsishah in #1525
• Bring changes from dec_2023_release branch into master branch by @gargnitingoogle in #1530
• Remove Image recognition models by @Tulsishah in #1483
• Update install test.sh with public documentation install commands by @ashmeenkaur in #1496
• add configs for log rotation by @ashmeenkaur in #1505
• Update installation steps for not using apt-key by @sethiay in #1498
• Integrate with lumberjack for log rotation by @ashmeenkaur in #1504
• Rename log rotate config by @ashmeenkaur in #1520
• Upgrade go storage pkg from 1.31.0 to 1.34.1 by @Tulsishah in #1491
• Fix tf-resnet model failure by @Tulsishah in #1536
• make log compression on by default by @ashmeenkaur in #1537

Full Changelog: v1.2.1...v1.3.0

Gcsfuse v1.2.1

• Fixes in –max-retry-duration and –max-retry-sleep flags behaviors:
  • Before Gcsfuse v1.2.1: –max-retry-duration flag was doing the job of –max-retry-sleep flag and –max-retry-sleep flag was unused (no effect on setting it).
  • With Gcsfuse v1.2.1: –max-retry-sleep flag aligns to its documented behavior and –max-retry-duration flag is unused (no effect on setting it).
  • Note: If you have been using --max-retry-duration or –max-retry-sleep, please review the changes carefully and update your mount commands accordingly.
• Upgraded golang version from 1.21.0 to 1.21.3. CVEs fixed are listed below:
  • https://nvd.nist.gov/vuln/detail/CVE-2023-39318
  • https://nvd.nist.gov/vuln/detail/CVE-2023-39319
  • https://nvd.nist.gov/vuln/detail/CVE-2023-39320
  • https://nvd.nist.gov/vuln/detail/CVE-2023-39321
  • https://nvd.nist.gov/vuln/detail/CVE-2023-39322
  • GHSA-679v-hh23-h5jh
  • GHSA-rxv8-v965-v333

Gcsfuse v1.2.0

• Improved performance by removing zero-byte file creation during write flow (details here).
• GCSFuse is now available for use on arm64-based machines.
• Improved logging - by supporting different log-levels (documentation here).
• Improved coverage by adding more e2e/integration tests.
• Added new flag --config-file which supports specifying new flags via yaml file (documentation here).
• Upgraded go version from 1.20.5 to 1.21.0.