Add end-to-end tests for GitHub, GitLab, and Bitbucket apps; added seed

[mrT23]

User description

Note

it seems that zero-randomness with seed is not operational yet:
https://community.openai.com/t/seed-param-and-reproducible-output-do-not-work/487245/1
I could not achieve this either

but maybe it will be improved in the future ...

---

PR Type

enhancement, tests

---

Description

• Introduced seed and temperature handling in AI model completions to ensure reproducibility and consistency across tools.
• Updated all tools to use global temperature setting from configuration, replacing hardcoded values.
• Added end-to-end tests for GitHub, GitLab, and Bitbucket integrations to validate functionality.
• Implemented new GitHub workflow for running end-to-end tests, ensuring reliability of integrations.
• Enhanced configuration file with comments for better clarity and added new settings for seed and temperature.

---

Changes walkthrough 📝

	Relevant files
Enhancement	10 files litellm_ai_handler.py Enhance AI Handler with Seed and Temperature Handling        pr_agent/algo/ai_handlers/litellm_ai_handler.py Added logic to handle seed and temperature settings for AI model completions. Introduced error checking for incompatible seed and temperature settings. Logging added for using fixed seed. +7/-0      pr_add_docs.py Use Global Temperature Setting in PR Docs Addition              pr_agent/tools/pr_add_docs.py Updated temperature setting to use global configuration instead of hardcoded value. +2/-2      pr_code_suggestions.py Use Global Temperature Setting in PR Code Suggestions        pr_agent/tools/pr_code_suggestions.py Updated temperature setting to use global configuration instead of hardcoded value. +2/-2      pr_description.py Use Global Temperature Setting in PR Description                  pr_agent/tools/pr_description.py Updated temperature setting to use global configuration instead of hardcoded value. +1/-1      pr_generate_labels.py Use Global Temperature Setting in PR Label Generation        pr_agent/tools/pr_generate_labels.py Updated temperature setting to use global configuration instead of hardcoded value. +1/-1      pr_information_from_user.py Use Global Temperature Setting in PR User Information Gathering pr_agent/tools/pr_information_from_user.py Updated temperature setting to use global configuration instead of hardcoded value. +2/-2      pr_line_questions.py Use Global Temperature Setting in PR Line Questions            pr_agent/tools/pr_line_questions.py Updated temperature setting to use global configuration instead of hardcoded value. +2/-2      pr_questions.py Use Global Temperature Setting in PR Questions                      pr_agent/tools/pr_questions.py Updated temperature setting to use global configuration instead of hardcoded value. +5/-5      pr_reviewer.py Use Global Temperature Setting in PR Reviewer                        pr_agent/tools/pr_reviewer.py Updated temperature setting to use global configuration instead of hardcoded value. +1/-1      pr_update_changelog.py Use Global Temperature Setting in PR Changelog Update        pr_agent/tools/pr_update_changelog.py Updated temperature setting to use global configuration instead of hardcoded value. +2/-2
Tests	5 files e2e_utils.py Add Utilities for E2E Testing                                                        tests/e2e_tests/e2e_utils.py Added utility constants and a new file content for end-to-end tests. +35/-0    test_bitbucket_app.py Add E2E Test for Bitbucket App                                                      tests/e2e_tests/test_bitbucket_app.py Added an end-to-end test for the Bitbucket app. +100/-0  test_github_app.py Add E2E Test for GitHub App                                                            tests/e2e_tests/test_github_app.py Added an end-to-end test for the GitHub app. +96/-0    test_gitlab_webhook.py Add E2E Test for GitLab Webhook                                                    tests/e2e_tests/test_gitlab_webhook.py Added an end-to-end test for GitLab webhook. +91/-0    e2e_tests.yaml Add Workflow for E2E Tests                                                              .github/workflows/e2e_tests.yaml Added a new workflow for running end-to-end tests. +46/-0
Configuration changes	2 files build-and-test.yaml Specify Unittest Directory in Build and Test Workflow        .github/workflows/build-and-test.yaml Modified test command to specify unittest directory. +1/-1      configuration.toml Enhance Configuration with Seed and Temperature Settings  pr_agent/settings/configuration.toml Added comments for clarity in configuration settings. Introduced seed and temperature settings. +7/-0

---

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[qodo-merge-pro]

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🏅 Score: 85

🧪 PR contains tests

🔒 No security concerns identified

🔀 Multiple PR themes Sub-PR theme: Update AI handler and tools to use configurable temperature settings Relevant files: pr_agent/algo/ai_handlers/litellm_ai_handler.py pr_agent/tools/pr_add_docs.py pr_agent/tools/pr_code_suggestions.py pr_agent/tools/pr_description.py pr_agent/tools/pr_generate_labels.py pr_agent/tools/pr_information_from_user.py pr_agent/tools/pr_line_questions.py pr_agent/tools/pr_questions.py pr_agent/tools/pr_reviewer.py pr_agent/tools/pr_update_changelog.py Sub-PR theme: Add end-to-end tests for GitHub, GitLab, and Bitbucket apps Relevant files: tests/e2e_tests/e2e_utils.py tests/e2e_tests/test_bitbucket_app.py tests/e2e_tests/test_github_app.py tests/e2e_tests/test_gitlab_webhook.py Sub-PR theme: Update configuration settings for seed and temperature Relevant files: pr_agent/settings/configuration.toml

⚡ Key issues to review Possible Bug The condition if get_settings().config.get("seed", -1) > 0 might not work as expected if seed is not set, as it defaults to -1. Consider handling the default case more explicitly. Error Handling The error raised ValueError("Seed is not supported with temperature > 0") could be more descriptive, explaining why this configuration is not supported.

[mrT23]

/improve

[qodo-merge-pro]

PR Code Suggestions ✨

Latest suggestions up to 778d7ce

Category	Suggestion	Score
Possible issue	Add type checking for temperature to ensure it is numeric before comparison Consider handling the case where temperature is not a numeric value. Since temperature is used in a comparison operation (temperature > 0), ensuring it's a valid number can prevent runtime errors. pr_agent/algo/ai_handlers/litellm_ai_handler.py [133-134] +if not isinstance(temperature, (int, float)): + raise TypeError("Temperature must be a numeric value") if temperature > 0 and seed >= 0: raise ValueError("Seed is not supported with temperature > 0") Apply this suggestion Suggestion importance[1-10]: 9 Why: This suggestion addresses a potential runtime error by ensuring temperature is numeric before performing a comparison. It adds robustness to the code, preventing type-related issues.	9
Enhancement	✅ Improve code efficiency by reusing the seed variable instead of fetching it multiple times from settings The condition to raise a ValueError when temperature > 0 and seed >= 0 is correct, but the implementation could be improved by directly using the seed variable fetched from settings, rather than calling get_settings().config.seed again. This avoids multiple calls to the settings configuration, which can be more efficient and cleaner. pr_agent/algo/ai_handlers/litellm_ai_handler.py [132-137] seed = get_settings().config.get("seed", -1) if temperature > 0 and seed >= 0: raise ValueError("Seed is not supported with temperature > 0") elif seed >= 0: get_logger().info(f"Using fixed seed of {seed}") - kwargs["seed"] = get_settings().config.seed + kwargs["seed"] = seed [Suggestion has been applied] Suggestion importance[1-10]: 8 Why: This suggestion improves code efficiency and readability by avoiding redundant calls to the settings configuration. It correctly reuses the seed variable, making the code cleaner and potentially faster.	8
	✅ Enhance the error message for better clarity and debugging The error message "Seed is not supported with temperature > 0" could be more informative by including the actual values of temperature and seed that caused the error. This can help in debugging or understanding why the error was raised during runtime. pr_agent/algo/ai_handlers/litellm_ai_handler.py [133-134] if temperature > 0 and seed >= 0: - raise ValueError("Seed is not supported with temperature > 0") + raise ValueError(f"Seed ({seed}) is not supported with temperature ({temperature}) > 0") [Suggestion has been applied] Suggestion importance[1-10]: 7 Why: The suggestion improves the error message by including the actual values of temperature and seed, which can be helpful for debugging. This is a minor enhancement but adds value for understanding runtime errors.	7
Maintainability	Simplify the conditional logic by replacing elif with else The elif condition seed >= 0 is redundant since it's checked in the previous if statement. You can simplify the logic by removing the elif and just using an else clause, assuming that if the first condition fails (temperature > 0 and seed >= 0), then seed must be >= 0 for the second block of code to execute. pr_agent/algo/ai_handlers/litellm_ai_handler.py [133-137] if temperature > 0 and seed >= 0: raise ValueError("Seed is not supported with temperature > 0") -elif seed >= 0: +else: get_logger().info(f"Using fixed seed of {seed}") kwargs["seed"] = seed Apply this suggestion Suggestion importance[1-10]: 6 Why: The suggestion simplifies the conditional logic, making the code more readable. However, the improvement is minor as the existing code is already correct and clear.	6

---

Previous suggestions

✅ Suggestions up to commit ac247db

Category	Suggestion	Score
Possible issue	✅ Refactor the condition to ensure mutual exclusivity of temperature and seed Suggestion Impact: The commit refactored the condition to ensure that both temperature and seed cannot be set simultaneously, which aligns with the suggestion's intention. code diff: + seed = get_settings().config.get("seed", -1) + if temperature > 0 and seed >= 0: + raise ValueError("Seed is not supported with temperature > 0") To ensure that the temperature and seed parameters are mutually exclusive, it's better to refactor the condition to raise an exception if both are set, rather than only checking if temperature is greater than 0. pr_agent/algo/ai_handlers/litellm_ai_handler.py [133-134] -if temperature > 0: - raise ValueError("Seed is not supported with temperature > 0") +if temperature > 0 and get_settings().config.get("seed", 0) > 0: + raise ValueError("Seed and temperature cannot both be set.") Suggestion importance[1-10]: 9 Why: This suggestion ensures that both temperature and seed cannot be set simultaneously, which is a logical improvement and prevents potential runtime errors.	9
	Ensure the temperature parameter does not exceed the maximum allowed value Since the temperature parameter is now dynamically set from settings, ensure that it does not exceed the maximum allowed value to prevent errors from the AI model. pr_agent/tools/pr_code_suggestions.py [307-308] +max_temp = 1.0 # Define the maximum temperature allowed +temp = min(get_settings().config.temperature, max_temp) response, finish_reason = await self.ai_handler.chat_completion( - model=model, temperature=get_settings().config.temperature, system=system_prompt, user=user_prompt) + model=model, temperature=temp, system=system_prompt, user=user_prompt) Suggestion importance[1-10]: 7 Why: This suggestion prevents potential errors by ensuring the temperature parameter stays within acceptable limits, enhancing the reliability of the code.	7
Maintainability	Add exception handling to the await call for chat_completion The await call should handle potential exceptions that might be thrown by the chat_completion method, especially since network calls are involved. pr_agent/tools/pr_add_docs.py [92-93] -response, finish_reason = await self.ai_handler.chat_completion( - model=model, temperature=get_settings().config.temperature, system=system_prompt, user=user_prompt) +try: + response, finish_reason = await self.ai_handler.chat_completion( + model=model, temperature=get_settings().config.temperature, system=system_prompt, user=user_prompt) +except Exception as e: + get_logger().error(f"Failed to complete chat: {e}") + return None Suggestion importance[1-10]: 8 Why: Adding exception handling improves the robustness of the code by ensuring that any errors during the async call are logged and handled gracefully.	8
Enhancement	Simplify the condition to check the seed value The condition checking for the seed value should directly use the get method with a default value rather than checking if the seed is greater than 0. This simplifies the code and ensures that the default value is used correctly. pr_agent/algo/ai_handlers/litellm_ai_handler.py [132] -if get_settings().config.get("seed", -1) > 0: +if get_settings().config.get("seed", 0) > 0: Suggestion importance[1-10]: 2 Why: The suggestion simplifies the condition, but it does not significantly improve the code since the original condition is already clear and functional.	2

[mrT23]

Preparing PR description...

[mrT23]

PR Description updated to latest commit (38c38ec)

[mrT23]

PR Description updated to latest commit (38c38ec)