feature: Google Cloud Storage support skeleton (#513)

* feature: implement Google Cloud Storage support

1. try implement GCS support backend prototype
2. try implement GCS support directory walker prototype

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: add license header

Add Apache license headers for gcs files

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: cancel feature switch `services-gcs`

1. canceled unneccesary feature switch `service-gcs`
2. implement from_iter() for `gcs::Backend`
3. *make gcs compile*

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: make clippy happy

added some document comments in gcs

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: catch pace with OpenDAL's break change

now it should be able to compiled and satisfies clippy

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: unimplement presign in GCS

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: purge presign implement in gcs backend

now the backend should inherits the default implementation of Accessor

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: make clippy happy

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: fix errors introduced by merging main

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: use structs instead of serde_json::Value

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: full change code to new http error API

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: make fmt happy

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: typo fixes and unit test

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: make clippy happy

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

* refactor: refactor unit tests

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>

Cargo.toml

@@ -82,6 +82,7 @@ quick-xml = { version = "0.23", features = ["serialize"] }
 radix_trie = { version = "0.2", optional = true }
 reqsign = "0.3"
 serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
 thiserror = "1.0"
 time = "0.3"
 tokio = { version = "1.20", features = ["fs"] }

src/operator.rs

@@ -128,6 +128,7 @@ impl Operator {
  #[cfg(feature = "services-http")]
  Scheme::Http => services::http::Backend::from_iter(it)?.into(),
  Scheme::Memory => services::memory::Builder::default().build()?.into(),
+ Scheme::Gcs => services::gcs::Backend::from_iter(it)?.into(),
  Scheme::S3 => services::s3::Backend::from_iter(it)?.into(),
  };
 

src/scheme.rs

@@ -22,7 +22,7 @@ use crate::error::other;
 use crate::error::BackendError;
 
 /// Backends that OpenDAL supports
-#[derive(Copy, Clone, Debug, PartialEq)]
+#[derive(Copy, Clone, Debug, PartialEq, Eq)]
 pub enum Scheme {
  /// [azblob][crate::services::azblob]: Azure Storage Blob services.
  Azblob,
@@ -34,6 +34,8 @@ pub enum Scheme {
  /// [http][crate::services::http]: HTTP backend.
  #[cfg(feature = "services-http")]
  Http,
+ /// [gcs][crate::services::gcs]: Google Cloud Storage backend.
+ Gcs,
  /// [memory][crate::services::memory]: In memory backend support.
  Memory,
  /// [s3][crate::services::s3]: AWS S3 alike services.
@@ -60,6 +62,7 @@ impl Display for Scheme {
  Scheme::Fs => write!(f, "fs"),
  #[cfg(feature = "services-hdfs")]
  Scheme::Hdfs => write!(f, "hdfs"),
+ Scheme::Gcs => write!(f, "gcs"),
  #[cfg(feature = "services-http")]
  Scheme::Http => write!(f, "http"),
  Scheme::Memory => write!(f, "memory"),
@@ -80,6 +83,7 @@ impl FromStr for Scheme {
  "hdfs" => Ok(Scheme::Hdfs),
  #[cfg(feature = "services-http")]
  "http" | "https" => Ok(Scheme::Http),
+ "gcs" => Ok(Scheme::Gcs),
  "memory" => Ok(Scheme::Memory),
  "s3" => Ok(Scheme::S3),
  v => Err(other(BackendError::new(
@@ -99,6 +103,7 @@ impl From<Scheme> for &'static str {
  Scheme::Hdfs => "hdfs",
  #[cfg(feature = "services-http")]
  Scheme::Http => "http",
+ Scheme::Gcs => "gcs",
  Scheme::Memory => "memory",
  Scheme::S3 => "s3",
  }