diff --git a/1.png b/1.png new file mode 100644 index 0000000..ce7523e Binary files /dev/null and b/1.png differ diff --git a/2.png b/2.png new file mode 100644 index 0000000..f5507be Binary files /dev/null and b/2.png differ diff --git a/README.md b/README.md index 30458e9..538822c 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,27 @@ +[![CI/CD Pipeline](https://github.com/aoaow/spotify_microservice/actions/workflows/actions.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/actions.yml) +[![Format](https://github.com/aoaow/spotify_microservice/actions/workflows/format.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/format.yml) +[![Install](https://github.com/aoaow/spotify_microservice/actions/workflows/install.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/install.yml) +[![Lint](https://github.com/aoaow/spotify_microservice/actions/workflows/lint.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/lint.yml) +[![Test](https://github.com/aoaow/spotify_microservice/actions/workflows/test.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/test.yml) # IDS706 - Final Project - Spotify Microservice ## Microservice and Core Development Documentation +## Video presentation +Watch the video presentation [here](https://youtu.be/KZIzitQN4ss) + ### Summary of Completed Work: **1. Core Microservice Implementation** -- Technology: Python (Flask framework). -- Endpoints: +- **Technology**: Python (Flask framework). +- **Endpoints**: - `/rankings`: Fetch top-ranked tracks by region and date. - `/artist`: Retrieve track data for a specific artist. -- Optimizations: +- **Optimizations**: - Cached grouped data for faster query performance. - Optimized database filtering and ranking logic. -- Tested Functionality: - - Successfully tested all endpoints locally and within a Docker container. +- **Tested Functionality**: + - Successfully tested all endpoints locally, within a Docker container, and deployed to AWS App Runner. **2. Logging** - Integrated structured logging to track: @@ -26,10 +34,10 @@ - Created a production-ready Dockerfile: - Uses `gunicorn` as a WSGI server for performance. - Handles dependency installation (`requirements.txt`). - - Exposes port 5000 for microservice access. -- Verified the Docker image works as expected locally. + - Exposes port 3000 for microservice access. +- Verified the Docker image works as expected locally and pushed the image to AWS ECR. -⚠️**Note: When to Rebuild the Docker Image?** +⚠️ **Note: When to Rebuild the Docker Image?** - If any of the following files are updated: - `app.py` (microservice logic). - `requirements.txt` (dependencies). @@ -41,38 +49,36 @@ ``` - After rebuilding, test the new image by running: ```bash - docker run -p 5000:5000 spotify_microservice + docker run -p 3000:3000 spotify_microservice ``` -⚠️**Future Flexibility with Container Registries** - -The Docker image for this microservice is built and tested locally to simplify the development and debugging process. This allows for quick iteration without requiring external dependencies or accounts. But additional flexibility can be achieved by integrating with an online container registry like Docker Hub, AWS ECR, or Azure Container Registry. - -Steps for Future Deployment: -- Login to Docker Hub or another registry: - ```bash - docker login - ``` -- Tag and push the image: - ```bash - docker tag spotify_microservice your_username/spotify_microservice:latest - docker push your_username/spotify_microservice:latest - ``` -- Integrate with CI/CD: Add Docker credentials to GitHub Secrets and Update CI/CD workflows to build and push images automatically. - +**4. Deployment to AWS App Runner** +- Pushed the Docker image to AWS ECR. +- Deployed the microservice using AWS App Runner: + - Configured the service to pull the image from ECR. + - Verified successful deployment and health checks. + - Public endpoint available for accessing the service. -**4. Load Testing** +**5. Load Testing** - Wrote a `locustfile.py` script to simulate concurrent requests: - - Tested with ~2000 RPS locally (limited by hardware). + - Tested with ~200 RPS locally (limited by hardware). - Achieved <5% failure rate after performance tuning. - - Identified scaling limitations for 10,000 RPS, which requires cloud deployment. + - Successfully tested cloud deployment via App Runner for scalability. -### Pending Work - Scale to 10,000 Requests per Second +![alt text](image.png) +![alt text](1.png) + +--- + +### Pending Work - Scaling Beyond 10,000 Requests per Second - The microservice currently achieves ~2000 RPS locally due to hardware limits. - Recommended Next Steps: - - Deploy to AWS or Azure for load balancing and auto-scaling. - - Use cloud-based tools (e.g., AWS Elastic Beanstalk, Azure App Service) to handle higher RPS. - - Configure additional caching or database replication if necessary. + - Scale App Runner instances to handle higher traffic. + - Implement additional caching (e.g., Redis) for frequent queries. + - Use database replication or partitioning to distribute the load. + - Integrate a load balancer to manage traffic across multiple instances. + +--- ### Files and Their Purposes 1. `app.py`: The Flask microservice implementation. @@ -82,43 +88,53 @@ Steps for Future Deployment: 5. `data.py`: Script for preparing and transforming the dataset. 6. `spotify_top_200.parquet`: Preprocessed dataset for use in the microservice. +--- + ### How to Test Locally -1. Run the microservice: + +1. **Run the microservice**: - Open a terminal and run: ```bash python app.py ``` - - This starts the Flask application. Access endpoints at `http://127.0.0.1:5000`. + - This starts the Flask application. Access endpoints at `http://127.0.0.1:3000`. -2. Run load tests: +2. **Run load tests**: - Open a **second terminal** and run: ```bash locust -f locustfile.py ``` - This starts the Locust load testing framework. Access the monitoring interface at `http://127.0.0.1:8089`. - **Note**: Make sure to run `app.py` and `locustfile.py` in two separate terminals to avoid interruptions and ensure both processes are running simultaneously!!! + **Note**: Ensure both `app.py` and `locustfile.py` are running simultaneously to avoid interruptions. -3. Run the Docker container: - - Alternatively, to test the Dockerized version: +3. **Run the Docker container**: + - Alternatively, test the Dockerized version: ```bash - docker run -p 5000:5000 spotify_microservice + docker run -p 3000:3000 spotify_microservice ``` - - Access the application at `http://127.0.0.1:5000`. + - Access the application at `http://127.0.0.1:3000`. -4. Rebuild the Docker Image (if required): +4. **Rebuild the Docker Image** (if required): - If you modify any code or configuration: ```bash docker build -t spotify_microservice . ``` - Test the updated image using the steps above. +--- + ## Infrastructure as Code -This project uses AWS CloudFormation for infrastructure provisioning. It sets up an ECS cluster and related resources for running the microservice. +This project uses AWS App Runner for deployment, pulling Docker images from AWS ECR and managing the infrastructure automatically. Health checks and scaling configurations ensure smooth operation under varying loads. + +![alt text](2.png) + + +--- +## Architectural Diagram -See the [infrastructure/README.md](infrastructure/README.md) for details and deployment instructions. -### Final Repository Link: -https://github.com/nogibjj/IDS706_Final_Spotify_Microservice \ No newline at end of file +### Final Repository Link +[Spotify Microservice GitHub Repository](https://github.com/nogibjj/IDS706_Final_Spotify_Microservice) diff --git a/flask_profiler.sql b/flask_profiler.sql new file mode 100644 index 0000000..cf8fea5 Binary files /dev/null and b/flask_profiler.sql differ diff --git a/image.png b/image.png new file mode 100644 index 0000000..524b46f Binary files /dev/null and b/image.png differ diff --git a/locustfile.py b/locustfile.py index 5b54ca3..fb5338a 100644 --- a/locustfile.py +++ b/locustfile.py @@ -1,12 +1,16 @@ -from locust import HttpUser, task, between - -class SpotifyMicroserviceUser(HttpUser): - wait_time = between(1, 2) +from locust import HttpUser, TaskSet, task +class UserBehavior(TaskSet): @task - def test_filters(self): - self.client.get("/filters", timeout=30) + def filters(self): + self.client.get("/filters", timeout=120) # Set a 120-second timeout @task - def test_tracks(self): - self.client.get("/tracks?date=2021-01-01®ion=United States&artist=Shakira&limit=20&offset=0", timeout=30) + def tracks(self): + self.client.get("/tracks?date=2021-01-01®ion=United%20States&artist=Shakira", timeout=120) + + +class WebsiteUser(HttpUser): + tasks = [UserBehavior] + host = "http://127.0.0.1:3000" + diff --git a/project/app.py b/project/app.py index 53d3707..38b593a 100644 --- a/project/app.py +++ b/project/app.py @@ -27,10 +27,10 @@ def health(): @app.route('/filters', methods=['GET']) def get_filters(): + print("Request received for /filters") try: - print("Received request for /filters") - filters = Track.get_filters() - print(f"Filters: {filters}") # Log the filters + filters = Track.get_filters() # Log here if necessary + print("Filters processed successfully") return jsonify(filters) except Exception as e: print(f"Error in /filters: {e}") @@ -38,27 +38,16 @@ def get_filters(): @app.route('/tracks', methods=['GET']) def get_tracks(): - """ - API endpoint to get tracks filtered by date, region, and artist. - """ - date = request.args.get('date') - region = request.args.get('region') - artist = request.args.get('artist') - + print("Request received for /tracks") try: + date = request.args.get('date') + region = request.args.get('region') + artist = request.args.get('artist') tracks = Track.get_tracks(date=date, region=region, artist=artist) - response = [ - { - 'rank': index + 1, - 'track_title': track['track_title'], - 'artist': track['artist'], - 'streams': track['track_streams'], - 'track_url': track['track_url'], - } - for index, track in enumerate(tracks) - ] - return jsonify(response) + print("Tracks processed successfully") + return jsonify(tracks) except Exception as e: + print(f"Error in /tracks: {e}") return jsonify({'error': str(e)}), 500 @@ -69,7 +58,7 @@ def index(): """ return render_template('index.html') - -if __name__ == "__main__": - app.run(host="0.0.0.0", port=3000, debug=True) - +@app.errorhandler(Exception) +def handle_exception(e): + print(f"Unhandled exception: {e}") + return jsonify({"error": "Internal Server Error"}), \ No newline at end of file diff --git a/project/config/database.py b/project/config/database.py index 2fb4b82..cc0d1ac 100644 --- a/project/config/database.py +++ b/project/config/database.py @@ -8,10 +8,12 @@ logger = setup_logger() + def get_db_connection(): """Create a database connection.""" try: conn = sqlite3.connect('spotify.db', check_same_thread=False) + conn.execute("PRAGMA journal_mode=WAL;") conn.row_factory = sqlite3.Row return conn except sqlite3.Error as e: diff --git a/project/infrastructure/README.md b/project/infrastructure/README.md deleted file mode 100644 index 6df0e9c..0000000 --- a/project/infrastructure/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Infrastructure as Code - AWS CloudFormation - -This folder contains the CloudFormation template used to provision the infrastructure for the Spotify Microservice project. - -## Resources Created -- VPC with a public subnet. -- ECS Cluster to run containerized applications. -- ECS Service to deploy the microservice. - -## Prerequisites -- AWS CLI installed and configured. -- An AWS account with permissions to create CloudFormation stacks. -- A Docker image for the application, hosted on a container registry (e.g., Amazon ECR or DockerHub). - -## Deployment Instructions -1. Navigate to the `infrastructure` directory: - ```bash - cd infrastructure - ``` - -2. Deploy the stack: - ```bash - aws cloudformation deploy \ - --template-file cloudformation-template.yml \ - --stack-name spotify-microservice-stack \ - --capabilities CAPABILITY_IAM - ``` - -3. Monitor the stack creation: - ```bash - aws cloudformation describe-stacks --stack-name spotify-microservice-stack - ``` - -## Troubleshooting -- Check the CloudFormation console for error messages. -- Ensure your AWS CLI is configured correctly and that you have the necessary permissions. diff --git a/spotify.db-shm b/spotify.db-shm new file mode 100644 index 0000000..fe9ac28 Binary files /dev/null and b/spotify.db-shm differ diff --git a/spotify.db-wal b/spotify.db-wal new file mode 100644 index 0000000..e69de29