Transfer the hffs code to hfh

docs/source/_toctree.yml

@@ -18,6 +18,8 @@
       title: Repository
     - local: guides/search
       title: Search
+    - local: guides/hf_file_system
+      title: HfFileSystem
     - local: guides/inference
       title: Inference
     - local: guides/community
@@ -52,6 +54,8 @@
       title: Mixins & serialization methods
     - local: package_reference/inference_api
       title: Inference API
+    - local: package_reference/hf_file_system
+      title: HfFileSystem
     - local: package_reference/utilities
       title: Utilities
     - local: package_reference/community

docs/source/guides/hf_file_system.mdx

@@ -0,0 +1,105 @@
+# Interact with the Hub through the Filesystem API
+
+In addition to the [`HfApi`], the `huggingface_hub` library provides [`HfFileSystem`], a pythonic [fsspec-compatible](https://filesystem-spec.readthedocs.io/en/latest/) file interface to the Hugging Face Hub. The [`HfFileSystem`] builds of top of the [`HfApi`] and offers typical filesystem style operations like `cp`, `mv`, `ls`, `du`, `glob`, `get_file`, and `put_file`.
+
+## Usage
+
+```python
+>>> from huggingface_hub import HfFileSystem
+>>> fs = HfFileSystem()
+
+>>> # List all files in a directory
+>>> fs.ls("datasets/my-username/my-dataset-repo/data", detail=False)
+['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']
+
+>>> # List all ".csv" files in a repo
+>>> fs.glob("datasets/my-username/my-dataset-repo/**.csv")
+['datasets/my-username/my-dataset-repo/data/train.csv', 'datasets/my-username/my-dataset-repo/data/test.csv']
+
+>>> # Read a remote file 
+>>> with fs.open("datasets/my-username/my-dataset-repo/data/train.csv", "r") as f:
+...     train_data = f.readlines()
+
+>>> # Read the content of a remote file as a string
+>>> train_data = fs.read_text("datasets/my-username/my-dataset-repo/data/train.csv", revision="dev")
+
+>>> # Write a remote file
+>>> with fs.open("datasets/my-username/my-dataset-repo/data/validation.csv", "w") as f:
+...     f.write("text,label")
+...     f.write("Fantastic movie!,good")
+```
+
+The optional `revision` argument can be passed to run an operation from a specific commit such as a branch, tag name, or a commit hash.
+
+Unlike Python's built-in `open`, `fsspec`'s `open` defaults to binary mode, `"rb"`. This means you must explicitly set mode as `"r"` for reading and `"w"` for writing in text mode. Appending to a file (modes `"a"` and `"ab"`) is not supported yet.
+
+## Integrations
+
+The [`HfFileSystem`] can be used with any library that integrates `fsspec`, provided the URL follows the scheme:
+
+```
+hf://[<repo_type_prefix>]<repo_id>[@<revision>]/<path/in/repo>
+```
+
+The `repo_type_prefix` is `datasets/` for datasets, `spaces/` for spaces, and models don't need a prefix in the URL.
+
+Some interesting integrations where [`HfFileSystem`] simplifies interacting with the Hub are listed below:
+
+* Reading/writing a [Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#reading-writing-remote-files) DataFrame from/to a Hub repository:
+
+  ```python
+  >>> import pandas as pd
+
+  >>> # Read a remote CSV file into a dataframe
+  >>> df = pd.read_csv("hf://datasets/my-username/my-dataset-repo/train.csv")
+
+  >>> # Write a dataframe to a remote CSV file
+  >>> df.to_csv("hf://datasets/my-username/my-dataset-repo/test.csv")
+  ```
+
+The same workflow can also be used for [Dask](https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html) and [Polars](https://pola-rs.github.io/polars/py-polars/html/reference/io.html) DataFrames.
+
+* Querying (remote) Hub files with [DuckDB](https://duckdb.org/docs/guides/python/filesystems):
+
+  ```python
+  >>> from huggingface_hub import HfFileSystem
+  >>> import duckdb
+
+  >>> fs = HfFileSystem()
+  >>> duckdb.register_filesystem(fs)
+  >>> # Query a remote file and get the result back as a dataframe
+  >>> fs_query_file = "hf://datasets/my-username/my-dataset-repo/data_dir/data.parquet"
+  >>> df = duckdb.query(f"SELECT * FROM '{fs_query_file}' LIMIT 10").df()
+  ```
+
+* Using the Hub as an array store with [Zarr](https://zarr.readthedocs.io/en/stable/tutorial.html#io-with-fsspec):
+
+  ```python
+  >>> import numpy as np
+  >>> import zarr
+
+  >>> embeddings = np.random.randn(50000, 1000).astype("float32")
+
+  >>> # Write an array to a repo
+  >>> with zarr.open_group("hf://my-username/my-model-repo/array-store", mode="w") as root:
+  ...    foo = root.create_group("embeddings")
+  ...    foobar = foo.zeros('experiment_0', shape=(50000, 1000), chunks=(10000, 1000), dtype='f4')
+  ...    foobar[:] = embeddings
+
+  >>> # Read an array from a repo
+  >>> with zarr.open_group("hf://my-username/my-model-repo/array-store", mode="r") as root:
+  ...    first_row = root["embeddings/experiment_0"][0]
+  ```
+
+## Authentication
+
+In many cases, you must be logged in with a Hugging Face account to interact with the Hub. Refer to the [Login](../quick-start#login) section of the documentation to learn more about authentication methods on the Hub. 
+
+It is also possible to login programmatically by passing your `token` as an argument to [`HfFileSystem`]:
+
+```python
+>>> from huggingface_hub import HfFileSystem
+>>> fs = HfFileSystem(token=token)
+```
+
+If you login this way, be careful not to accidentally leak the token when sharing your source code!

docs/source/guides/overview.mdx

@@ -42,6 +42,15 @@ Take a look at these guides to learn how to use huggingface_hub to solve real-wo
       </p>
     </a>
 
+    <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg"
+       href="./filesystem">
+      <div class="w-full text-center bg-gradient-to-br from-indigo-400 to-indigo-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">
+        HfFileSystem
+      </div><p class="text-gray-700">
+        How to interact with the Hub through a convenient interface that mimics Python's file interface?
+      </p>
+    </a>
+
     <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg"
        href="./inference">
       <div class="w-full text-center bg-gradient-to-br from-indigo-400 to-indigo-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">

docs/source/package_reference/hf_file_system.mdx

@@ -0,0 +1,12 @@
+# Filesystem API
+
+The `HfFileSystem` class provides a pythonic file interface to the Hugging Face Hub based on [`fssepc`](https://filesystem-spec.readthedocs.io/en/latest/).
+
+## HfFileSystem
+
+`HfFileSystem` is based on [fsspec](https://filesystem-spec.readthedocs.io/en/latest/), so it is compatible with most of the APIs that it offers. For more details, check out [our guide](../guides/filesystem) and the fsspec's [API Reference](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).
+
+[[autodoc]] HfFileSystem
+    - __init__
+    - resolve_path
+    - ls

setup.cfg

@@ -13,6 +13,7 @@ known_third_party =
     faiss-cpu
     fastprogress
     fire
+    fsspec
     fugashi
     git
     graphviz

setup.py

@@ -13,6 +13,7 @@ def get_version() -> str:
 
 install_requires = [
     "filelock",
+    "fsspec",
     "requests",
     "tqdm>=4.42.1",
     "pyyaml>=5.1",

src/huggingface_hub/__init__.py

@@ -162,6 +162,11 @@
         "upload_folder",
         "whoami",
     ],
+    "hf_file_system": [
+        "HfFileSystem",
+        "HfFileSystemFile",
+        "HfFileSystemResolvedPath",
+    ],
     "hub_mixin": [
         "ModelHubMixin",
         "PyTorchModelHubMixin",
@@ -421,6 +426,11 @@ def __dir__():
         upload_folder,  # noqa: F401
         whoami,  # noqa: F401
     )
+    from .hf_file_system import (
+        HfFileSystem,  # noqa: F401
+        HfFileSystemFile,  # noqa: F401
+        HfFileSystemResolvedPath,  # noqa: F401
+    )
     from .hub_mixin import (
         ModelHubMixin,  # noqa: F401
         PyTorchModelHubMixin,  # noqa: F401

src/huggingface_hub/hf_api.py

@@ -64,14 +64,14 @@
     filter_repo_objects,
     hf_raise_for_status,
     logging,
+    paginate,
     parse_datetime,
     validate_hf_hub_args,
 )
 from .utils._deprecation import (
     _deprecate_arguments,
     _deprecate_list_output,
 )
-from .utils._pagination import paginate
 from .utils._typing import Literal, TypedDict
 from .utils.endpoint_helpers import (
     AttributeDictionary,