add readme

README.md

@@ -1,19 +1,27 @@
+[![CI/CD Pipeline](https://github.com/aoaow/spotify_microservice/actions/workflows/actions.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/actions.yml)
+[![Format](https://github.com/aoaow/spotify_microservice/actions/workflows/format.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/format.yml)
+[![Install](https://github.com/aoaow/spotify_microservice/actions/workflows/install.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/install.yml)
+[![Lint](https://github.com/aoaow/spotify_microservice/actions/workflows/lint.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/lint.yml)
+[![Test](https://github.com/aoaow/spotify_microservice/actions/workflows/test.yml/badge.svg)](https://github.com/aoaow/spotify_microservice/actions/workflows/test.yml)
 # IDS706 - Final Project - Spotify Microservice
 
 ## Microservice and Core Development Documentation
 
+## Video presentation
+Watch the video presentation [here](https://youtu.be/KZIzitQN4ss) 
+
 ### Summary of Completed Work:
 
 **1. Core Microservice Implementation**
-- Technology: Python (Flask framework).
-- Endpoints:
+- **Technology**: Python (Flask framework).
+- **Endpoints**:
   - `/rankings`: Fetch top-ranked tracks by region and date.
   - `/artist`: Retrieve track data for a specific artist.
-- Optimizations:
+- **Optimizations**:
   - Cached grouped data for faster query performance.
   - Optimized database filtering and ranking logic.
-- Tested Functionality:
-  - Successfully tested all endpoints locally and within a Docker container.
+- **Tested Functionality**:
+  - Successfully tested all endpoints locally, within a Docker container, and deployed to AWS App Runner.
 
 **2. Logging**
 - Integrated structured logging to track:
@@ -26,10 +34,10 @@
 - Created a production-ready Dockerfile:
   - Uses `gunicorn` as a WSGI server for performance.
   - Handles dependency installation (`requirements.txt`).
-  - Exposes port 5000 for microservice access.
-- Verified the Docker image works as expected locally.
+  - Exposes port 3000 for microservice access.
+- Verified the Docker image works as expected locally and pushed the image to AWS ECR.
 
-⚠️**Note: When to Rebuild the Docker Image?**
+⚠️ **Note: When to Rebuild the Docker Image?**
 - If any of the following files are updated:
     - `app.py` (microservice logic).
     - `requirements.txt` (dependencies).
@@ -41,38 +49,36 @@
     ```
 - After rebuilding, test the new image by running:
     ```bash
-    docker run -p 5000:5000 spotify_microservice
+    docker run -p 3000:3000 spotify_microservice
     ```
 
-⚠️**Future Flexibility with Container Registries**
-
-The Docker image for this microservice is built and tested locally to simplify the development and debugging process. This allows for quick iteration without requiring external dependencies or accounts. But additional flexibility can be achieved by integrating with an online container registry like Docker Hub, AWS ECR, or Azure Container Registry.
-
-Steps for Future Deployment:
-- Login to Docker Hub or another registry:
-  ```bash
-  docker login
-  ```
-- Tag and push the image:
-  ```bash
-  docker tag spotify_microservice your_username/spotify_microservice:latest
-  docker push your_username/spotify_microservice:latest
-  ```
-- Integrate with CI/CD: Add Docker credentials to GitHub Secrets and Update CI/CD workflows to build and push images automatically.
-
+**4. Deployment to AWS App Runner**
+- Pushed the Docker image to AWS ECR.
+- Deployed the microservice using AWS App Runner:
+  - Configured the service to pull the image from ECR.
+  - Verified successful deployment and health checks.
+  - Public endpoint available for accessing the service.
 
-**4. Load Testing**
+**5. Load Testing**
 - Wrote a `locustfile.py` script to simulate concurrent requests:
-  - Tested with ~2000 RPS locally (limited by hardware).
+  - Tested with ~200 RPS locally (limited by hardware).
   - Achieved <5% failure rate after performance tuning.
-  - Identified scaling limitations for 10,000 RPS, which requires cloud deployment.
+  - Successfully tested cloud deployment via App Runner for scalability.
 
-### Pending Work - Scale to 10,000 Requests per Second
+![alt text](image.png)
+![alt text](1.png)
+
+---
+
+### Pending Work - Scaling Beyond 10,000 Requests per Second
 - The microservice currently achieves ~2000 RPS locally due to hardware limits.
 - Recommended Next Steps:
-  - Deploy to AWS or Azure for load balancing and auto-scaling.
-  - Use cloud-based tools (e.g., AWS Elastic Beanstalk, Azure App Service) to handle higher RPS.
-  - Configure additional caching or database replication if necessary.
+  - Scale App Runner instances to handle higher traffic.
+  - Implement additional caching (e.g., Redis) for frequent queries.
+  - Use database replication or partitioning to distribute the load.
+  - Integrate a load balancer to manage traffic across multiple instances.
+
+---
 
 ### Files and Their Purposes
 1. `app.py`: The Flask microservice implementation.
@@ -82,43 +88,53 @@ Steps for Future Deployment:
 5. `data.py`: Script for preparing and transforming the dataset.
 6. `spotify_top_200.parquet`: Preprocessed dataset for use in the microservice.
 
+---
+
 ### How to Test Locally
-1. Run the microservice:
+
+1. **Run the microservice**:
    - Open a terminal and run:
      ```bash
      python app.py
      ```
-   - This starts the Flask application. Access endpoints at `http://127.0.0.1:5000`.
+   - This starts the Flask application. Access endpoints at `http://127.0.0.1:3000`.
 
-2. Run load tests:
+2. **Run load tests**:
    - Open a **second terminal** and run:
      ```bash
      locust -f locustfile.py
      ```
    - This starts the Locust load testing framework. Access the monitoring interface at `http://127.0.0.1:8089`.
 
-   **Note**: Make sure to run `app.py` and `locustfile.py` in two separate terminals to avoid interruptions and ensure both processes are running simultaneously!!!
+   **Note**: Ensure both `app.py` and `locustfile.py` are running simultaneously to avoid interruptions.
 
-3. Run the Docker container:
-   - Alternatively, to test the Dockerized version:
+3. **Run the Docker container**:
+   - Alternatively, test the Dockerized version:
      ```bash
-     docker run -p 5000:5000 spotify_microservice
+     docker run -p 3000:3000 spotify_microservice
      ```
-   - Access the application at `http://127.0.0.1:5000`.
+   - Access the application at `http://127.0.0.1:3000`.
 
-4. Rebuild the Docker Image (if required):
+4. **Rebuild the Docker Image** (if required):
    - If you modify any code or configuration:
      ```bash
      docker build -t spotify_microservice .
      ```
    - Test the updated image using the steps above.
 
+---
+
 ## Infrastructure as Code
 
-This project uses AWS CloudFormation for infrastructure provisioning. It sets up an ECS cluster and related resources for running the microservice.
+This project uses AWS App Runner for deployment, pulling Docker images from AWS ECR and managing the infrastructure automatically. Health checks and scaling configurations ensure smooth operation under varying loads.
+
+![alt text](2.png)
+
+
+---
+## Architectural Diagram
 
-See the [infrastructure/README.md](infrastructure/README.md) for details and deployment instructions.
 
 
-### Final Repository Link:
-https://github.com/nogibjj/IDS706_Final_Spotify_Microservice
+### Final Repository Link
+[Spotify Microservice GitHub Repository](https://github.com/nogibjj/IDS706_Final_Spotify_Microservice)

locustfile.py

@@ -1,12 +1,16 @@
-from locust import HttpUser, task, between
-
-class SpotifyMicroserviceUser(HttpUser):
-    wait_time = between(1, 2)
+from locust import HttpUser, TaskSet, task
 
+class UserBehavior(TaskSet):
     @task
-    def test_filters(self):
-        self.client.get("/filters", timeout=30)
+    def filters(self):
+        self.client.get("/filters", timeout=120)  # Set a 120-second timeout
 
     @task
-    def test_tracks(self):
-        self.client.get("/tracks?date=2021-01-01&region=United States&artist=Shakira&limit=20&offset=0", timeout=30)
+    def tracks(self):
+        self.client.get("/tracks?date=2021-01-01&region=United%20States&artist=Shakira", timeout=120)
+
+
+class WebsiteUser(HttpUser):
+    tasks = [UserBehavior]
+    host = "http://127.0.0.1:3000"
+

project/app.py

@@ -27,38 +27,27 @@ def health():
 
 @app.route('/filters', methods=['GET'])
 def get_filters():
+    print("Request received for /filters")
     try:
-        print("Received request for /filters")
-        filters = Track.get_filters()
-        print(f"Filters: {filters}")  # Log the filters
+        filters = Track.get_filters()  # Log here if necessary
+        print("Filters processed successfully")
         return jsonify(filters)
     except Exception as e:
         print(f"Error in /filters: {e}")
         return jsonify({'error': str(e)}), 500
 
 @app.route('/tracks', methods=['GET'])
 def get_tracks():
-    """
-    API endpoint to get tracks filtered by date, region, and artist.
-    """
-    date = request.args.get('date')
-    region = request.args.get('region')
-    artist = request.args.get('artist')
-
+    print("Request received for /tracks")
     try:
+        date = request.args.get('date')
+        region = request.args.get('region')
+        artist = request.args.get('artist')
         tracks = Track.get_tracks(date=date, region=region, artist=artist)
-        response = [
-            {
-                'rank': index + 1,
-                'track_title': track['track_title'],
-                'artist': track['artist'],
-                'streams': track['track_streams'],
-                'track_url': track['track_url'],
-            }
-            for index, track in enumerate(tracks)
-        ]
-        return jsonify(response)
+        print("Tracks processed successfully")
+        return jsonify(tracks)
     except Exception as e:
+        print(f"Error in /tracks: {e}")
         return jsonify({'error': str(e)}), 500
 
 
@@ -69,7 +58,7 @@ def index():
     """
     return render_template('index.html')
 
-
-if __name__ == "__main__":
-    app.run(host="0.0.0.0", port=3000, debug=True)
-
+@app.errorhandler(Exception)
+def handle_exception(e):
+    print(f"Unhandled exception: {e}")
+    return jsonify({"error": "Internal Server Error"}),

project/config/database.py

@@ -8,10 +8,12 @@
 
 logger = setup_logger()
 
+
 def get_db_connection():
     """Create a database connection."""
     try:
         conn = sqlite3.connect('spotify.db', check_same_thread=False)
+        conn.execute("PRAGMA journal_mode=WAL;")
         conn.row_factory = sqlite3.Row
         return conn
     except sqlite3.Error as e: