Unified web application with React frontend and FastAPI backend

[Copilot]

Implements a single-process web application serving both an AI API and compiled React frontend for SAM 3D Objects. FastAPI serves static files at / and API endpoints under /api/.

Backend

backend/server.py - FastAPI server (328 lines)

• Lifespan context manager for model loading with HF authentication
• Four endpoints: /api/process/{image,video}, /api/generate/{image-kie,3d}
• Static file serving with SPA routing fallback
• Secure tempfile handling with cleanup
• Environment-based configuration: HF_TOKEN, KIE_API_KEY, KIE_API_ENDPOINT

backend/Dockerfile

• CUDA 12.6 base with Python 3.12
• PyTorch 2.7 installation from CUDA index
• SAM 3 repository clone and installation
• Multi-stage copy for frontend build

Frontend

React + TypeScript + Vite SPA

• Three sections: Object Tracker (image/video upload), Image Generator (kie.ai integration), 3D Creator (React Three Fiber viewer)
• Relative API paths (/api/*) for unified serving
• URL.revokeObjectURL cleanup to prevent memory leaks
• Vite proxy config for development

// API calls use relative paths
const response = await axios.post('/api/process/image', formData)

// Static files served by FastAPI in production
app.mount("/assets", StaticFiles(directory="frontend/dist/assets"))

Screenshot

Documentation

• QUICKSTART.md - Environment variables and deployment commands
• WEBAPP_SETUP.md - Comprehensive setup with dev/prod modes
• IMPLEMENTATION_SUMMARY.md - Architecture diagram and file statistics

Validation

• Backend validation script verifies structure and endpoints
• CodeQL: 0 alerts (Python, JavaScript)
• Frontend builds to dist/ (1.2MB minified)

Original prompt

Task: Build a single, unified web application with a React frontend and a Python FastAPI backend. The Python server must serve both the AI API and the compiled React frontend from a single process. High-Level Architecture: Frontend: A single-page application built with React, TypeScript, and Vite. Backend: A Python server built with FastAPI. Unified Serving: The FastAPI server must be configured to serve the static files from the compiled React build (from a dist or build folder) as its root endpoint (/), while also exposing all AI API endpoints under the /api prefix. Part 1: The Python FastAPI Backend Provide the complete, runnable code for this server, including a requirements.txt and a Dockerfile for deployment.

Environment Setup: Base Image: The Dockerfile should use a base image that supports CUDA 12.6. Python Version: The environment must use Python 3.12. Dependencies: Install PyTorch 2.7 and Torchvision using the specified index URL: https://download.pytorch.org/whl/cu126. SAM 3 Installation: The setup process must clone the official repository (git clone https://github.com/facebookresearch/sam3.git ) and install the package via pip install -e . inside the cloned directory. API Keys: The server must be configured to load a Hugging Face token and a kie.ai API key from environment variables. Do not hardcode keys in the source code.
Model Loading: On server startup, use the Hugging Face token to authenticate and download the necessary model checkpoints. Load the following models into memory so they persist globally: The SAM 3 Image Model (build_sam3_image_model). The SAM 3 Video Predictor (build_sam3_video_predictor). The SAM 3D model for 2D-to-3D conversion.
API Endpoints (prefixed with /api): POST /api/process/image: Accepts an image file and a text_prompt. Must use the Sam3Processor workflow from the sam3 GitHub README. Returns the processed image with masks. POST /api/process/video: Accepts a video file and a text_prompt. Must use the stateful video_predictor.handle_request workflow (start session, add prompt) from the README. Returns the processed video. POST /api/generate/image-kie: Accepts a text_prompt. This endpoint must make a server-to-server API call to the kie.ai "nano banana" image generation service using the provided API key. It should then return the image received from kie.ai. POST /api/generate/3d: Accepts an image file. Uses the local SAM 3D model to generate and return a 3D model file (format: .glb).
Static File Serving: Configure FastAPI to mount the static files from the React build directory. The application must serve the index.html of the React app from its root URL (/). Part 2: The React Frontend Provide the complete source code for the React application.
Build Process: The project must be configured so that npm run build creates a static, production-ready build in a dist (or similar) folder.
API Communication: The React application must make all API calls to relative paths (e.g., /api/process/image). This is critical for the unified serving model to work correctly.
UI/UX Design: Main Layout: A clean interface with a navigation bar or tabs to switch between three sections: "Object Tracker," "Image Generator," and "3D Creator." "Object Tracker" Section: UI for uploading an image or video, a text box for a prompt, a "Process" button, and a display area for the result. "Image Generator" Section: A text area for a prompt, a "Generate with Nano Banana" button that calls /api/generate/image-kie, and a display area for the generated image. This section must also include a "Make this 3D" button. "3D Creator" Section: This section should be reachable by either uploading an image or by clicking the "Make this 3D" button from the Image Generator. It needs a "Generate 3D Model" button that calls /api/generate/3d and a 3D viewer component (e.g., using three.js or @react-three/fiber) to display the resulting .glb file.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.