fix: Fix for the test case #37936
fix: Fix for the test case #37936
Conversation
WalkthroughThe changes involve modifications to a Cypress test suite and a test specification file. Specifically, the Changes
Possibly related PRs
Suggested labels
Suggested reviewers
📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (2)
app/client/cypress/e2e/Regression/ClientSide/BugTests/DS_Bug26941_Spec.ts (2)
Line range hint
17-24: Increasing timeouts is not the ideal solution for flaky testsWhile increasing the timeout from 20 to 50 might temporarily fix the flaky test, it's generally considered an anti-pattern as it:
- Masks the underlying issue causing the flakiness
- Increases overall test execution time
- Could still fail intermittently if the root cause isn't addressed
Consider investigating:
- Network conditions causing delayed responses
- Race conditions in the API calls
- State management issues between tests
Would you like assistance in implementing a more robust solution that doesn't rely on increased timeouts?
Also applies to: 27-39, 40-47
Line range hint
1-47: Test structure needs improvements to align with Cypress best practicesThe test could benefit from several improvements to follow Cypress best practices:
- Consider using data-* attributes for selectors instead of class-based selectors
- The API calls could be intercepted using cy.intercept() for better control
- Consider breaking down the test into smaller, focused test cases
Here's a suggested refactor approach:
describe("API Header Validation", { tags: ["@tag.Datasource", "@tag.Git", "@tag.AccessControl"] }, () => { beforeEach(() => { cy.intercept('POST', '**/123').as('apiCall'); const randomApi = `${dataManager.dsValues[dataManager.defaultEnviorment].mockApiUrl}123`; apiPage.CreateAndFillApi(randomApi); }); it("should handle empty header key correctly", () => { apiPage.EnterHeader("", ""); apiPage.RunAPI(); cy.wait('@apiCall'); agHelper.AssertElementAbsence(debuggerHelper.locators._debuggerDownStreamErrMsg); }); it("should handle invalid header key correctly", () => { apiPage.EnterHeader(">", ""); apiPage.RunAPI(); cy.wait('@apiCall').then((interception) => { debuggerHelper.AssertDownStreamLogError( interception.response.body.data.pluginErrorDetails.downstreamErrorMessage ); }); }); });
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
📒 Files selected for processing (2)
app/client/cypress/e2e/Regression/ClientSide/BugTests/DS_Bug26941_Spec.ts(3 hunks)app/client/cypress/limited-tests.txt(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
app/client/cypress/limited-tests.txt (1)
Pattern app/client/cypress/**/**.*: Review the following e2e test code written using the Cypress test library. Ensure that:
- Follow best practices for Cypress code and e2e automation.
- Avoid using cy.wait in code.
- Avoid using cy.pause in code.
- Avoid using agHelper.sleep().
- Use locator variables for locators and do not use plain strings.
- Use data-* attributes for selectors.
- Avoid Xpaths, Attributes and CSS path.
- Avoid selectors like .btn.submit or button[type=submit].
- Perform logins via API with LoginFromAPI.
- Perform logout via API with LogOutviaAPI.
- Perform signup via API with SignupFromAPI.
- Avoid using it.only.
- Avoid using after and aftereach in test cases.
- Use multiple assertions for expect statements.
- Avoid using strings for assertions.
- Do not use duplicate filenames even with different paths.
- Avoid using agHelper.Sleep, this.Sleep in any file in code.
app/client/cypress/e2e/Regression/ClientSide/BugTests/DS_Bug26941_Spec.ts (1)
Pattern app/client/cypress/**/**.*: Review the following e2e test code written using the Cypress test library. Ensure that:
- Follow best practices for Cypress code and e2e automation.
- Avoid using cy.wait in code.
- Avoid using cy.pause in code.
- Avoid using agHelper.sleep().
- Use locator variables for locators and do not use plain strings.
- Use data-* attributes for selectors.
- Avoid Xpaths, Attributes and CSS path.
- Avoid selectors like .btn.submit or button[type=submit].
- Perform logins via API with LoginFromAPI.
- Perform logout via API with LogOutviaAPI.
- Perform signup via API with SignupFromAPI.
- Avoid using it.only.
- Avoid using after and aftereach in test cases.
- Use multiple assertions for expect statements.
- Avoid using strings for assertions.
- Do not use duplicate filenames even with different paths.
- Avoid using agHelper.Sleep, this.Sleep in any file in code.
🔇 Additional comments (2)
app/client/cypress/limited-tests.txt (1)
2-2: LGTM! Test specification path follows proper structure.
The path correctly points to the bug test case and follows the proper directory structure for Cypress tests.
Let's verify the existence of the referenced test file:
✅ Verification successful
Test file exists and follows proper test structure
The referenced test file exists at the correct location and contains proper Cypress test structure with describe/it blocks and appropriate tags.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify the existence of the referenced test file
# and check its content for proper test structure
# Check if the file exists
fd "DS_Bug26941_Spec.ts" app/client/cypress/e2e/Regression/ClientSide/BugTests/
# If found, verify basic test structure
rg -A 5 "describe|it\(" app/client/cypress/e2e/Regression/ClientSide/BugTests/DS_Bug26941_Spec.ts
Length of output: 749
app/client/cypress/e2e/Regression/ClientSide/BugTests/DS_Bug26941_Spec.ts (1)
Line range hint 1-47: Verify usage of wait/sleep patterns in the codebase
Let's check for potential sources of test flakiness in the codebase.
|
/ci-test-limit-count run_count=20 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12154768235. |
|
/ci-test-limit-count run_count=20 runId=12154768235 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12155094539. |
|
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12155094539. |
|
/ci-test-limit-count run_count=5 runId=12154768235 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12156529534. |
|
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12156529534. |
|
/ci-test-limit-count run_count=5 runId=12154768235 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12158311193. |
|
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12158311193. |
|
/ci-test-limit-count run_count=20 runId=12154768235 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12158925153. |
|
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12158925153. |
|
/ci-test-limit-count run_count=20 runId=12154768235 |
|
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/12160939710. |
|
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12160939710. |
There was a problem hiding this comment.
Raising the wait will resulting in making the test flaky after few days.
There was a problem hiding this comment.
This wait is not a hardcoded sleep; it is the timeout for the API call to respond. Typically, I observe successful responses within 1 second, but for better stability, I have set it to 2 seconds. If the response is received within 1 second, the test will pass without delay.
Maintaining this timeout avoids the need for adding retries, which could mask potential bugs. Introducing retries increases the risk of missing errors and reduces the accuracy of our tests.
## Description Fix flaky test case. Fixes # https://app.zenhub.com/workspaces/stability-pod-6690c4814e31602e25cab7fd/issues/gh/appsmithorg/appsmith/37935 ## Automation /ok-to-test tags="@tag.Datasource" ### 🔍 Cypress test results <!-- This is an auto-generated comment: Cypress test results --> > [!TIP] > 🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉 > Workflow run: <https://github.com/appsmithorg/appsmith/actions/runs/12174291982> > Commit: d71af4c > <a href="https://internal.appsmith.com/app/cypress-dashboard/rundetails-65890b3c81d7400d08fa9ee5?branch=master&workflowId=12174291982&attempt=1" target="_blank">Cypress dashboard</a>. > Tags: `@tag.Datasource` > Spec: > <hr>Thu, 05 Dec 2024 06:59:43 UTC <!-- end of auto-generated comment: Cypress test results --> ## Communication Should the DevRel and Marketing teams inform users about this change? - [ ] Yes - [x] No <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Updated timeout duration for API calls in the bug test suite to improve test reliability. - Shifted focus of test specifications from a template-related test to a bug-related test. These changes enhance the effectiveness of our testing process, ensuring more accurate results and quicker response times during tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Description
Fix flaky test case.
Fixes # https://app.zenhub.com/workspaces/stability-pod-6690c4814e31602e25cab7fd/issues/gh/appsmithorg/appsmith/37935
Automation
/ok-to-test tags="@tag.Datasource"
🔍 Cypress test results
Tip
🟢 🟢 🟢 All cypress tests have passed! 🎉 🎉 🎉
Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/12174291982
Commit: d71af4c
Cypress dashboard.
Tags:
@tag.DatasourceSpec:
Thu, 05 Dec 2024 06:59:43 UTC
Communication
Should the DevRel and Marketing teams inform users about this change?
Summary by CodeRabbit
These changes enhance the effectiveness of our testing process, ensuring more accurate results and quicker response times during tests.