LeaderBot

Tool that will track and rank contributions across the different Mina developer programs. Consists of two bot:

Github Tracker Bot
Leaderbot

Getting Started

Prerequisites

Python 3.10
MongoDB
Google Cloud Service Account with access to Google Sheets API
Docker

Installation & Setup

Clone the repository:

git clone https://github.com/yourusername/PGT_LeaderBot.git
cd PGT_LeaderBot

In the root folder install virtual env and create virtual env:
```
pip install virtualenv
virtualenv venv
```
Activate it:
```
source venv/bin/activate
```
Install dependencies with:
```
pip install -r requirements.txt
```

Configure the environment variables

Create .env file in the root folder and add environment variables as shown in the .env.example file:

GITHUB_TOKEN='your_github_token'
DISCORD_TOKEN='your_discord_token'
OPENAI_API_KEY='your_openai_api_key'

GOOGLE_SHEETS_CREDENTIALS='path_to_your_google_sheets_credentials.json'
SHARED_SECRET='shared_secret'

MONGO_HOST='mongodb://localhost:27017/'
MONGO_DB="example_db"
MONGO_COLLECTION="example_collection"

GTP_ENDPOINT=http://localhost:8000

SPREADSHEET_ID='spread_sheet_id'
LOG_LEVEL=DEBUG

GUILD_ID=XXXXXXXX
GMAIL_ADDRESS=xyz@gmail.com
LEADERBOARD_FORUM_CHANNEL_ID=XXXXXXXXXXXXXXXXX

You need to have google credentials .json file which has access your spreadsheet in the root folder. Need to enable Google Sheets API and Google Drive API. Click for more information.
Run tests:
```
invoke test
```
Run github tracker bot:
```
invoke bot
```
For other invoke tasks look to: tasks.py
To run Leaderbot discord bot, you need to have Discord bot in a specific server.
```
invoke leaderbot
```

Github Tracker Bot

Overview

Github Tracker Bot fetchs to Google Spreadsheets which includes discord handle, github username and repositories. After getting these informations it fetches github commits data between specific timeframes given by user or day by day for all branches. After pre-processing the commits data. The bot sends total daily commits data(diff file) to OPENAI API to decide are these commits qualified with prompt.

Then gets the decisions data and insert them to MongoDB to further usage.

Date formats are ISO 8601 with Z suffix which is UTC. For example: "2023-01-24T00:00:00Z"

API Usage

bot.py is the main script for running a FastAPI service that provides functionality to schedule and run tasks to fetch results from a spreadsheet within specified time frames. The script includes endpoints to start and stop a scheduler, as well as to run tasks on demand.

API Endpoints

All of them uses headers headers = {"Authorization": AUTH_TOKEN} gotten from .env SHARED_SECRET

1. Run task manually for specific timeframes:

Endpoint: /run-task
Method: POST

This endpoint allows you to run a task immediately and manually for a specified time frame.

Request Body:

since (str): Start datetime in ISO 8601 format (e.g., 2023-07-24T00:00:00Z).
until (str): End datetime in ISO 8601 format (e.g., 2023-07-25T00:00:00Z).

Example Request:

{
  "since": "2023-07-24T00:00:00Z",
  "until": "2023-07-25T00:00:00Z"
}

Example Response:

{
  "message": "Task run successfully with provided times"
}

2. Run task for specific user:

Endpoint: /run-task-for-user?username=johndoe
Method: POST

This endpoint allows you to run a task immediately and manually for a specified time frame and specific user.

Request Body:

since (str): Start datetime in ISO 8601 format (e.g., 2023-07-24T00:00:00Z).
until (str): End datetime in ISO 8601 format (e.g., 2023-07-25T00:00:00Z).

Example Request:

Endpoint: /run-task-for-user?username=berkingurcan

{
    "since": "2023-07-24T00:00:00Z",
    "until": "2023-07-25T00:00:00Z"
}

Example Response:

{
  "message": "Task run successfully with provided times"
}

3. Control Scheduler

Endpoint: /control-scheduler
Method: POST

This endpoint allows you to start or stop the scheduler that runs tasks at specified minutes intervals. It is minutes for now to test. It will be hours.

Request Body:

action(str): Action to perform (start or stop).
interval_minutes (int, optional): Interval in minutes(for now) at which the scheduler should run the task (only required when action is start).

Example Requests:

Start Scheduler:

{
  "action": "start",
  "interval_minutes": 5
}

Stop Scheduler:

{
  "action": "stop"
}

Example Responses:

Start Scheduler:

{
  "message": "Scheduler started with interval of 5 minutes"
}

Stop Scheduler:

{
  "message": "Scheduler stopped"
}

Classes

User

The User class is a nested dataclass designed to store and manage information about a user, including their GitHub-related activities and AI decisions regarding their contributions. This class stores 4 necessary data:

user_handle: str
github_name: str
repositories: List[str]
ai_decisions: List[List[AIDecision]]

Remaining fields can be calculated using helper functions in the helper_functions.py.

AIDecision

The AIDecision Class stores AI decision related to a user's daily contribution for specific github repository. It includes details about the repository, date, and a nested response indicating the qualification status of the contribution which is DailyContributionResponse Class.

username: str
repository: str
date: str
response: DailyContributionResponse

DailyContributionResponse

The DailyContributionResponse class is another dataclass to store information about a user's daily contribution response returned by OPENAI API for specific repository. It includes details about the contribution's qualification status and an AI explanation for it.

username: str
date: str
is_qualified: bool
explanation: str

Click to see full code of User, AIDecision and DailyContributionResponse Classes

@dataclass
class DailyContributionResponse:
    username: str
    date: str
    is_qualified: bool
    explanation: str

    def to_dict(self):
        """Converts the dataclass to a dictionary."""
        return asdict(self)

@dataclass
class AIDecision:
    username: str
    repository: str
    date: str
    response: DailyContributionResponse
    commit_hashes: List[str] = field(default_factory=list)

    def to_dict(self):
        """Converts the dataclass to a dictionary, including nested response."""
        data = asdict(self)
        data["response"] = self.response.to_dict()
        return data


@dataclass
class User:
    user_handle: str
    github_name: str
    repositories: List[str]
    ai_decisions: List[List[AIDecision]] = field(default_factory=list)
    total_daily_contribution_number: int = 0
    total_qualified_daily_contribution_number: int = 0
    qualified_daily_contribution_number_by_month: Dict[str, int] = field(
        default_factory=dict
    )
    qualified_daily_contribution_dates: set = field(default_factory=set)
    qualified_daily_contribution_streak: int = 0

    def validate(self) -> bool:
        """Validates the User instance."""
        if not isinstance(self.repositories, list) or not all(
            isinstance(repo, str) for repo in self.repositories
        ):
            logger.error("Invalid repository list")
            return False
        return True

    def to_dict(self):
        """Converts the dataclass to a dictionary."""
        return {
            "user_handle": self.user_handle,
            "github_name": self.github_name,
            "repositories": self.repositories,
            "ai_decisions": [
                [decision.to_dict() for decision in decisions]
                for decisions in self.ai_decisions
            ],
            "total_daily_contribution_number": self.total_daily_contribution_number,
            "total_qualified_daily_contribution_number": self.total_qualified_daily_contribution_number,
            "qualified_daily_contribution_number_by_month": self.qualified_daily_contribution_number_by_month,
            "qualified_daily_contribution_dates": list(
                self.qualified_daily_contribution_dates
            ),
            "qualified_daily_contribution_streak": self.qualified_daily_contribution_streak,
        }

     @staticmethod
    def from_dict(data: Dict[str, Any]) -> "User":
        """Creates a User instance from a dictionary."""
        ai_decisions = [
            [
                AIDecision(
                    username=decision["username"],
                    repository=decision["repository"],
                    date=decision["date"],
                    response=DailyContributionResponse(
                        username=decision["response"]["username"],
                        date=decision["response"]["date"],
                        is_qualified=decision["response"]["is_qualified"],
                        explanation=decision["response"]["explanation"],
                    ),
                    commit_hashes=decision.get(
                        "commit_hashes", []
                    ),
                )
                for decision in decisions
            ]
            for decisions in data.get("ai_decisions", [])
        ]
        return User(
            user_handle=data["user_handle"],
            github_name=data["github_name"],
            repositories=data.get("repositories", []),
            ai_decisions=ai_decisions,
            total_daily_contribution_number=data.get(
                "total_daily_contribution_number", 0
            ),
            total_qualified_daily_contribution_number=data.get(
                "total_qualified_daily_contribution_number", 0
            ),
            qualified_daily_contribution_number_by_month=data.get(
                "qualified_daily_contribution_number_by_month", {}
            ),
            qualified_daily_contribution_dates=set(
                data.get("qualified_daily_contribution_dates", [])
            ),
            qualified_daily_contribution_streak=data.get(
                "qualified_daily_contribution_streak", 0
            ),
        )

MongoDBManagement

This class can be used to initalize and use database management. Explained here

Scripts

Scripts Explanation

Helpers

Helper functions explanations

Database Management

This database management script provides functionalities to manage user data and AI decisions in a MongoDB database. It supports creating, reading, updating, and deleting user records, as well as managing AI decisions and contribution data associated with users.

Features

User Management:
- Create, Read, Update, Delete (CRUD) Operations: Manage user records in the database including creation, retrieval, updating user details, and deletion.
- Validation: Each user instance undergoes validation to ensure data integrity before any CRUD operation.
AI Decision Handling:
- Retrieve AI Decisions: Fetch AI decisions based on user identity, with options to filter by date range.
- Add AI Decisions: Append new AI decisions to a user’s existing record.
- Update Contribution Data: Recalculates and updates user statistics based on new AI decisions, utilizing helper functions for detailed metrics like total and qualified contributions, monthly breakdowns, and streak calculations. These calculations made in here.
Contribution Data Management:
- Get/Set Operations for Contribution Metrics: Retrieve or update contribution-related metrics such as total daily contributions, qualified contributions, and contribution streaks.
- Date-wise Management: Manage specific dates for qualified contributions, allowing for additions, updates, and retrieval.

Leaderbot

Discord Bot for interacting with Google Sheets and data received from Github Scraper Bot to Mongo DB.

Usage

Explained here.

Additionally, need to enable google drive api to use sheet sharing functionality. username variables are discord handles.

Commands

`/commits-sheet-create`

Description: Creates a Google Sheet with contributions data.

Usage:

/commits-sheet-create spreadsheet_name: <name> email_address: <optional email>

spreadsheet_name: Name for the new Google Sheet.
email_address: (Optional) Email address to share the spreadsheet with. If not provided, the default email from the configuration will be used.

`/commits-sheet-update`

Description: Updates the Google Sheet with the latest contributions data in the Mongo DB.

Usage:

/commits-sheet-update spreadsheet_id: <id>

spreadsheet_id: Copy and Paste ID of the Google Sheet to be updated.

`/leaderboard-create`

Description: Creates and updates a leaderboard sheet in the specified sheet for a specific month by using current data. If spreadsheet_id is empty it will use the last created or updated sheet id. After creating the leaderboard, it will send the leaderboard to discord channel as message. If the leaderboard exists for specified date, it updates the leaderboard.

Usage:

/leaderboard-create spreadsheet_id: <optional id> date: <YYYY-MM>

spreadsheet_id: (Optional) ID of the Google Sheet to store the leaderboard. If not provided, the last created or updated sheet will be used.
date: (Optional) Date in "YYYY-MM" format. If not provided, the current month will be used.

`/leaderboard-view`

Description: Displays the leaderboard in the specified Discord Forum thread.

Usage:

/leaderboard-view thread_id: <THREAD_ID> date: <YYYY-MM>

thread_id: The ID of the thread where the leaderboard should be displayed.
date: (Optional) Date in "YYYY-MM" format. If not provided, the current month will be used.

`/main-sheet-edit`

Description: Creates a modal to edit the Google Sheet which includes commits data from Discord.

Usage:

/main-sheet-edit operation: <operation>

operation: Operation to perform on the sheet. Valid options are: insert for adding new user, update to update user data, add_repo to add repository, delete the user data.

`/leaderboard-start-auto-post`

Description: Automatically posts the leaderboard and updates the sheet with given id every day at a specified time.

Usage:

/leaderboard-start-auto-post date: <YYYY-MM> time: <HH:MM> spreadsheet_id: <optional id>

date: Date in "YYYY-MM" format.
time: Time in "HH-MM" format.
spreadsheet_id: (Optional) ID of the Google Sheet. If not provided, the last created or updated sheet will be used. If not created or updated before, it will post leaderboard but will not update any sheet.

`/leaderboard-stop-auto-post`

Description: Stops the auto-post leaderboard task started for a specific date.

Usage:

/leaderboard-stop-auto-post date: <YYYY-MM>

date: Date in "YYYY-MM" format for which the auto-post task should be stopped.

`/leaderboard-closure-month`

Description: Opens a thread that includes the Leaderboard for the month. Gets the forum channel ID from the .env file and exports the user data which contributed in the given month as csv.

Usage:

/leaderboard-closure-month date: <YYYY-MM> commit_filter: <commit_filter>

date: (Optional) Date in "YYYY-MM" format. Default is now.
commit_filter:(Optional) Commit filter number to show only contributions greater than or equal to the specified number. Default is 10.

`/get-monthly-streaks`

Description: Gets monthly streaks of users and creats thread to forum channel.

Usage: /get-monthly-streaks date: <YYYY-MM>

date: (Optional) Date in "YYYY-MM" format. Default is now.

`/get-members-and-insert-to-db`

Description: Gets and inserts all members of the guild to the db in new collection with their username and discord id.

Usage: /get-members-and-insert-to-db

`/run-task`

Description: Uses the Github Tracker Bot API /run-task endpoint.

Usage: /run-task since: <since> until: <until>

since: "YYYY-MM-DD" format since date.
until: "YYYY-MM-DD" format until date.

Note: It is okay to get An Error Occured message after some time in the discord because the API request will take long time.

`/run-task-for-user`

Description: Uses the Github Tracker Bot API /run-task-for-user endpoint.

Usage: /run-task-for-user username: <username> since: <since> until: <until>

username: Username for data to scraped.
since: "YYYY-MM-DD" format since date.
until: "YYYY-MM-DD" format until date.

`/control-scheduler`

Description: Uses the Github Tracker Bot API /control-scheduler endpoint to control the scheduler (start/stop) with an optional interval.

Usage: /control-scheduler action: <action> interval: <interval>

action: start or stop
interval: Time interval to run scheduler.

`/get-ai-decisions-by-user`

Description: Exports AI decisions as csv file for specific user between given dates.

Usage: /get-ai-decisions-by-user username: <username> since: <since> until: <until>

username: User handle of requested user.
since: "YYYY-MM-DD" format since date.
until: "YYYY-MM-DD" format until date.

`/get-all-data-to-csv`

Description: Exports all database user data to csv file.

Usage: /get-all-data-to-csv

`/get-blockchain-summary`

Description: Fetchs MINA Explorer Rest api and sends blockchain summary as a message

Usage: /get-blockchain-summary

`/get-user-monthly-data-to-csv`

Description: Export a specific user's monthly qualified contribution data for each day of a month

Usage: /get-user-monthly-data-to-csv username: <username> date: <date>

username: User handle of requested user.
date: "YYYY-MM" format date.

`/delete-all-data`

Description: Deletes all data between specific dates

Usage: /delete-all-data from: <from> until: <until>

from: "YYYY-MM-DD"
until: "YYYY-MM-DD"

After using this command, there will be a modal to confirm the deletion process. In order to confirm the deletion, you need to enter your discord username and exact same dates with the command. Warning : This process cannot be reverted.

Contributions

To make a contribution, follow these steps:

Make an issue that includes details about the feature or bug or something else. ======= Leaderboard Updates: Ranks contributors based on predefined metrics (KPI): 10+ days of qualified commits within a month.
Get that issue tested by: Cristina Echeverry.
Get that issue approved by the product owners: es92 or Cristina Echeverry.
Write a PR and get it approved by the code owners and Mina devops: Es92 (code owner), berkingurcan (developer & codeco-owner), johnmarcou (Mina devops). Each PR must correspond to an approved issue. By default, PRs should be merged by the PR submitter, though in some cases if changes are needed, they can be merged by code owners.

Name		Name	Last commit message	Last commit date
Latest commit History 399 Commits
.github/workflows		.github/workflows
github_tracker_bot		github_tracker_bot
leader_bot		leader_bot
tests		tests
utils		utils
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
__init__.py		__init__.py
config.py		config.py
docker-compose.yaml		docker-compose.yaml
log_config.py		log_config.py
pytest.ini		pytest.ini
requirements.txt		requirements.txt
setup.py		setup.py
tasks.py		tasks.py

MinaFoundation/PGT_LeaderBot

Folders and files

Latest commit

History

Repository files navigation

LeaderBot

Getting Started

Prerequisites

Installation & Setup

Github Tracker Bot

Overview

API Usage

API Endpoints

1. Run task manually for specific timeframes:

Request Body:

Example Request:

Example Response:

2. Run task for specific user:

Request Body:

Example Request:

Example Response:

3. Control Scheduler

Request Body:

Example Requests:

Start Scheduler:

Stop Scheduler:

Example Responses:

Start Scheduler:

Stop Scheduler:

Classes

User

AIDecision

DailyContributionResponse

MongoDBManagement

Scripts

Helpers

Database Management

Features

User Management:

AI Decision Handling:

Contribution Data Management:

Leaderbot

Usage

Commands

/commits-sheet-create

/commits-sheet-update

/leaderboard-create

/leaderboard-view

/main-sheet-edit

/leaderboard-start-auto-post

/leaderboard-stop-auto-post

/leaderboard-closure-month

/get-monthly-streaks

/get-members-and-insert-to-db

/run-task

/run-task-for-user

/control-scheduler

/get-ai-decisions-by-user

/get-all-data-to-csv

/get-blockchain-summary

/get-user-monthly-data-to-csv

/delete-all-data

Contributions

About

Topics

Resources

Stars

Watchers

Forks

Packages 0

Contributors 5

Languages

`/commits-sheet-create`

`/commits-sheet-update`

`/leaderboard-create`

`/leaderboard-view`

`/main-sheet-edit`

`/leaderboard-start-auto-post`

`/leaderboard-stop-auto-post`

`/leaderboard-closure-month`

`/get-monthly-streaks`

`/get-members-and-insert-to-db`

`/run-task`

`/run-task-for-user`

`/control-scheduler`

`/get-ai-decisions-by-user`

`/get-all-data-to-csv`

`/get-blockchain-summary`

`/get-user-monthly-data-to-csv`

`/delete-all-data`

Packages