Features Download

File Download Process

Problem Statement

In a decentralized storage network, downloading a file efficiently while ensuring data integrity, security, and performance presents several challenges:

• Consensus on file correctness: Data is distributed across multiple blobbers, requiring consensus on the correct file hash before reconstruction.
• Efficient data retrieval: Downloading from multiple sources must be optimized for speed and reliability.
• Integrity verification: Ensuring downloaded data is correct and unmodified using cryptographic proofs.
• Handling encrypted files: Securely decrypting data while preserving access control.
• Erasure coding reconstruction: Recovering the original file from distributed data shards.

Process

To overcome these challenges, the download process follows a structured approach:

1. Consensus on File Hash

• Before downloading, consensus is reached on the actual file hash across all blobbers.
• A set of blobbers is selected based on successful validation of stored data.
• The consensus threshold is set to data_shards, as this is the minimum required to decode erasure-encoded data.

2. Parallel Block-Based Download

• Data is downloaded in blocks of 64KB * data_shards, ensuring efficient retrieval.
• Each blobber is requested to send 100 blocks per request by default.
• Performance tracking: The first request is timed, and a subset of blobbers is selected based on response time to optimize speed.
• Downloading occurs in parallel from multiple blobbers, significantly improving speed compared to a single-provider approach.

3. Erasure Decoding and Decryption

• Erasure Coding Reconstruction:
  • The system uses Reed-Solomon erasure coding to reconstruct the original file from data_shards out of data_shards + parity_shards.
  • Missing or slow blobbers do not impact file recovery as long as the threshold is met.
• Decryption (If Encrypted):
  • Each block is decrypted using the user's private key.
  • Decryption follows AES-GCM (Authenticated Encryption with Associated Data) to ensure both confidentiality and authenticity.
  • If the file was shared using Proxy Re-Encryption (PRE), the downloaded data is re-encrypted by blobbers for the recipient, and the recipient decrypts it using their private key.

4. Integrity Verification

• Validation Merkle Proofs:
  • Each block’s correctness is verified using Merkle Proofs provided by blobbers.
  • Blobbers generate a Validation Merkle Proof for each block, which is verified against the Validation Merkle Root Hash calculated during the upload.
  • This ensures that data integrity is maintained and that no blobber has modified or provided incorrect data.
• File Hash Verification:
  • Once all blocks are retrieved and decoded, the final reconstructed file hash is computed.
  • The computed hash is compared against the consensus-agreed file hash to ensure correctness.
  • Any mismatch results in rejection of the downloaded data.

5. Authorization for Shared Files

• Users can download shared files using an Auth Ticket, which ensures proper access control.
• The Auth Ticket contains the necessary metadata and cryptographic permissions for the recipient.
• If the file was encrypted, Proxy Re-Encryption (PRE) enables blobbers to re-encrypt data without exposing plaintext.

Download Process - UML Sequence Diagram

sequenceDiagram
    participant SDK
    participant Blobbers
    participant Merkle Proof Verifier
    participant Decryption Module (AES-GCM)
    participant Block Merger
    participant Reed-Solomon Decoder
    participant Final File

    SDK->>Blobbers: Request file hash for consensus
    Blobbers-->>SDK: Return file hashes
    SDK->>SDK: Select blobbers based on consensus (at least data shards)

    loop Until all blocks are downloaded
        SDK->>Blobbers: Request 100 blocks from selected blobbers
        Blobbers-->>SDK: Return blocks + Merkle Proofs
        SDK->>Merkle Proof Verifier: Verify Merkle Proofs
        Merkle Proof Verifier-->>SDK: Validation result

        alt If file is encrypted
            SDK->>Decryption Module (AES-GCM): Decrypt blocks using private key
            Decryption Module (AES-GCM)-->>SDK: Return decrypted blocks
        end

        SDK->>Block Merger: Merge blocks from data shards
        Block Merger-->>SDK: Return merged blocks

        SDK->>Reed-Solomon Decoder: Decode merged blocks
        Reed-Solomon Decoder-->>SDK: Return decoded data

        SDK->>Final File: Append decoded blocks
    end

    SDK->>Final File: Compute file hash and verify against consensus hash
    alt If hash matches
        SDK->>SDK: File download successful
    else If mismatch
        SDK->>SDK: Reject download
    end

Loading