S3 Proxy Test

Description

This repository is a proof of concept. It shows how an S3-compatible object store (e.g., MinIO) can be used as a near-real-time proxy.

The client splits a local file into ordered chunks and uploads them as individual objects to an S3 bucket. The server continuously watches the bucket for new uploads, fetches chunks as they appear and assembles them back into the original file.

Build

mkdir -p bin
go build -o bin/client ./client
go build -o bin/server ./server

Set up MinIO

Start MinIO with Docker Compose:

docker compose up

Open the MinIO Console in your browser:

• http://127.0.0.1:9001

Log in, then create a bucket named test.

Create a test file

dd if=/dev/urandom of=./data/client/data.bin bs=1M count=100

Run server and client

Open two terminal windows.

Terminal A (server)

source ./vars.sh
./bin/server -out ./data/server -poll 100ms

Terminal B (client)

source ./vars.sh
./bin/client -file ./data/client/data.bin -chunk-size 1048576