chore: Implement runSpec mutation #26782

mike-plummer · 2023-05-17T17:29:02Z

Closes cypress-io/cypress-internal#593

Additional details

New GQL endpoint to accept a specPath and, given the matching project is already open, switch to the necessary testingType and launch matching spec. Eventual goal of this endpoint is to accept a project path and open project as necessary as well as a browser arg but those are out of scope for this ticket.

Mutation will validate:

A project is open
specPath is non-empty
specPath points to an existing file
specPath matches either e2e or ct specPattern
matching testing type is configured
Specified path points to a valid spec within the testing type
A browser is available to launch into

Endpoint will return either a success response with info on the testingType, spec, and browser
...or an error response with a code field indicating the specific error type

Steps to test

Find a project with e2e and CT configured, pick a spec from one mode or the other and record the relative path from project root
Open in global mode
Launch GraphiQL http://localhost:4444/__launchpad/graphql
Execute runSpec mutation with spec path from above - should fail since no project is open and give appropriate error code

mutation {
  runSpec (specPath: "cypress/e2e/1-getting-started/todo.cy.js") {
    __typename
    ... on RunSpecResponse {
      testingType
      browser {
        id
        name
      }
      spec {
        id
        name
      }
    }
    ... on RunSpecError {
      code
      detailMessage
    }
  }
}

Open selected project into testing mode matching the selected spec
Execute runSpec - should successfully launch spec and return a 'SUCCESS' response
Provide an empty specPath to mutation - should fail and give 'NO_SPEC_PATH' error response
Change specPath in mutation to point to a file that does not exist (but still satisfies specPattern) - should fail and give'SPEC_NOT_FOUND' error code
Change specPath to not match specPattern for either testingType - should fail and give 'NO_SPEC_PATTERN_MATCH' error code
Pick a spec from other testing type and use relative path in mutation - should successfully switch testing type and run spec, return valid SUCCESS response
Use devtools to record one of the gql requests, copy as a cURL command. Run cURL command from terminal to verify it is externally accessible
Close Cypress, edit project configuration to remove config for one testing type
Relaunch & open project. Use GQL to attempt to run a spec from now-unconfigured testing type. Should fail with a 'TESTING_TYPE_NOT_CONFIGURED' response

Extra Credit
Validate runSpec executes as expected when:

Browser is open in matching testing mode
Browser is open in other testing mode
Cypress is on the launchpad (no browser open)

How has the user experience changed?

No user-facing change

Screen.Recording.2023-05-19.at.12.53.08.PM.mov

PR Tasks

Have tests been added/updated?
[na] Has a PR for user-facing changes been opened in cypress-documentation?
[na] Have API changes been updated in the type definitions?

cypress · 2023-05-17T18:11:05Z

28 flaky tests on run #46715 ↗︎

0	27304	1308	0	28

Details:

Fix nullability issue
Project: cypress	Commit: `e344c9deb9`
Status: Passed	Duration: 19:56 💡
Started: May 23, 2023 6:35 PM	Ended: May 23, 2023 6:55 PM

create-from-component.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
... > runs generated spec		`Output` `Screenshots` `Video`

specs_list_latest_runs.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
App/Cloud Integration - Latest runs and Average duration > when no runs are recorded > shows placeholders for all visible specs		`Output` `Screenshots` `Video`

cypress-in-cypress-component.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
Cypress In Cypress CT > default config > redirects to the specs list with error if a spec is not found		`Output` `Screenshots` `Video`

cypress-origin-communicator.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
Cypress In Cypress Origin Communicator > cy.origin passivity with app interactions > passes upon test reload mid test execution		`Output` `Screenshots` `Video`

commands/net_stubbing.cy.ts • 1 flaky test • 5x-driver-firefox

View Output Video

Test		Artifacts
network stubbing > intercepting request > can delay and throttle a StaticResponse		`Output`

The first 5 flaky specs are shown, see all 15 specs in Cypress Cloud.

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

astone123 · 2023-05-19T20:03:29Z

Nice! Just did some initial testing and it looks really good. Only thing I've run into is that when I have a test open in one testing type, and then I open a test with the other testing type, it'll just dump me on the specs list for the new testing type instead of switching and running the spec. Are you seeing this too?

mike-plummer · 2023-05-19T20:12:57Z

Nice! Just did some initial testing and it looks really good. Only thing I've run into is that when I have a test open in one testing type, and then I open a test with the other testing type, it'll just dump me on the specs list for the new testing type instead of switching and running the spec. Are you seeing this too?

@astone123 I'm not seeing that (see video above, switching from an open spec to other mode works fine for me) - can you share the project/flow you're using?

astone123 · 2023-05-19T20:15:18Z

I'm using @packages/launchpad. Opening with yarn workspace @packages/launchpad cypress:open, then executing this mutation (e2e test)

mutation {
  runSpec (specPath: "cypress/e2e/open-mode.cy.ts") {
    code
    detailMessage
    testingType
    browser {
      id
      name
    }
    spec {
      id
      name
    }
  }
}

works fine, then I run this mutation (CT test)

mutation {
  runSpec (specPath: "src/components/code/FileRow.cy.tsx") {
    code
    detailMessage
    testingType
    browser {
      id
      name
    }
    spec {
      id
      name
    }
  }
}

it switches the testing type to CT, but it doesn't run the spec unless I send the request again

astone123 · 2023-05-19T20:20:20Z

Actually, after looking at it again, it doesn't launch the e2e spec either on the first try

astone123 · 2023-05-19T23:48:24Z

@mike-plummer works great after your last commit! Nice

astone123

Looks great on the first pass 👍🏻 will do some more testing Monday

packages/data-context/src/actions/ProjectActions.ts

warrensplayer · 2023-05-22T18:36:14Z

@mike-plummer I have a project where E2E is setup, but not component. If I use a specPath in the mutation that matches the component spec pattern but not the E2E pattern, then I get a SPEC_NOT_FOUND error code. Is that expected?

Screen.Recording.2023-05-22.at.2.18.24.PM.mov

mike-plummer · 2023-05-22T18:41:35Z

@mike-plummer I have a project where E2E is setup, but not component. If I use a specPath in the mutation that matches the component spec pattern but not the E2E pattern, then I get a SPEC_NOT_FOUND error code. Is that expected?

@warrensplayer I suspect you're providing a filepath that doesn't actually exist - this is a check I put in so we don't do unnecessary work altering Cypress state for a spec that can't possibly successfully run.

There are two different places where we check for Spec "existence". First, we check to see if the specified file exists - if not, we immediately bail out with a SPEC_NOT_FOUND. If the file does exist, then we check whether it satisfies e2e/component spec patterns and try to switch testing modes as necessary (bailing out if the target type isn't configured) before doing a secondary check to ensure the specified file actually loaded up as a recognized Spec instance which could also trigger a SPEC_NOT_FOUND.

packages/graphql/src/schemaTypes/objectTypes/gql-RunSpecResponse.ts

packages/data-context/src/sources/ProjectDataSource.ts

* Refactor gql into Response/Error types * Refactor specPatterns logic into new function * Add tests * Move cy-in-cy test to better location

warrensplayer

I had not realized that the App does not follow the same abstract type strategy for defining our schema as the Cypress Cloud code. They follow the Discriminant Model Field strategy while the App just used the default Centralized strategy. My comment about the union type was meant just as something to think about, but I do think it helps clarify the return type a bit better. There are other strategies that can be used together in the linked page as well.

warrensplayer · 2023-05-23T15:07:43Z

packages/data-context/src/sources/ProjectDataSource.ts

      return false
    }

    const MINIMATCH_OPTIONS = { dot: true, matchBase: true }

-    const { specPattern = [], excludeSpecPattern = [] } = await this.ctx.project.specPatterns()
+    const { specPattern = [], excludeSpecPattern = [] } = await this.ctx.project.specPatternsByTestingType(targetTestingType)


Is it okay that this no longer uses the old logic any more? Should this function conditionally get the specPatterns based on if a testingType is passed in or not?

I don't see any way to structure this other than this and the way I had it originally. The existing function grabs the specPattern from the resolved config which takes into account the current testing type, this function is effectively doing the exact same thing but with a configurable testing type so we can look at the "other" type.

packages/graphql/src/schemaTypes/enumTypes/gql-RunSpecErrorCodeEnum.ts

packages/graphql/src/schemaTypes/objectTypes/gql-RunSpecResponse.ts

packages/graphql/src/schemaTypes/unions/gql-RunSpecResult.ts

packages/data-context/src/sources/ProjectDataSource.ts

* Add function docs * Remove unnecessary code value * Simplify type resolution logic, improve schema contract

* chore: add Nx Cloud (#26712) * chore: add empty nx.json [run ci] * chore: add nx cloud runner [run ci] * chore: add nx-cloud dep [run ci] * chore: update local nx cloud accessToken to be read-only * feat: update git related messages for runs and debug (#26758) * feat(app): update DebugError copy * feat: set isUsingGit project flag and consume in DebugContainer * feat(app): update not using git icon for DebugError * feat(app): displays alert on runs page when not using git * feat(app): add component for when no runs for current branch * feat(app): add warning for no runs for branch on runs container * chore: add feat to CHANGELOG * chore: remove logged status value * chore: resolve import from merge conflict * test(app): stub branch name for e2e runs spec * chore: add line break in changelog entry * chore: cleanup PR * chore: fix i18n import for DebugBranchError * chore: add utm and update Warning links to inline * chore: capitalize Git in i18n * ref: warning liink * test: add i18n to tests * test(app): change utm_source assertions * chore: cleanup pr * chore: remove unused prop * test(app): remove no git warning when moving to runs page in e2e * chore: change template logic * chore: remove duplicate RUNS_TAB_MEDIUM const * Changed Debug test assertion and reordered new components for Debug --------- Co-authored-by: Stokes Player <stokes.player@gmail.com> * chore: rename video processing events to capture/compress (#26800) * chore: change processing nomenclature to compressing when printing the run. * chore: rename 'capturing of' to 'capturing' * chore: rename upload results to upload screenshots & videos (#26811) * chore: rename upload results to upload screenshots & videos * run ci * chore: capture versions of relevant dependencies with `x-dependencies` header (#26814) * chore: update changlelog script to handle revert pr ref (#26801) * fix: Correct typescript scaffold dependency (#26815) * fix: correct typescript scaffold dependency (#26204) * add changelog * Update change log for PR comment Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> --------- Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> * chore: 12.13.0 prep (#26833) * chore: 12.13.0 release (#26834) * chore: 12.13.0 changelog fix * remove pending, bump version * fix typo * chore: release @cypress/vite-plugin-cypress-esm-v1.0.1 [skip ci] * chore: Implement runSpec mutation (#26782) * chore: replace gitter badge with discord on readme (#26771) * chore: add GraphQL mutation for sending system notifications via Electron (#26773) * fix: upgrade typescript from 4.7.4 to 4.9.5 (#26826) Snyk has created this PR to upgrade typescript from 4.7.4 to 4.9.5. See this package in npm: See this project in Snyk: https://app.snyk.io/org/cypress-opensource/project/d5b36925-e6ee-455d-9649-6560a9aca413?utm_source=github&utm_medium=referral&page=upgrade-pr * test: fix 2 broken tests for Windows (#26854) * Update stale_issues_and_pr_cleanup.yml Upped the number of operations per run. Have been manually doing that so this job can get through all the issues in the repo with no problems. * chore(dep): [Snyk] Upgrade vite from 2.9.13 to 2.9.15 (#26830) Co-authored-by: Ben M <benm@cypress.io> * Update triage_add_to_project.yml * chore: fix minor background color styling in debug results component (#26887) * Update stale_issues_and_pr_cleanup.yml stalebot was incorrectly configured to run in debug mode. I have updated the default to run in normal mode when running scheduled * chore: Deprecate @cypress/xpath package (#26893) * chore: add telemetry realworld app (#26896) * chore: capture telemetry for realworld app maybe * idk what i was doing * setup record key and telemetry * testing * override project id * some times we just need a little context. * Adding tests * Adding comment * fix yarn lock * Trying this.... * fix tests? --------- Co-authored-by: Jordan <jordan@jpdesigning.com> Co-authored-by: Stokes Player <stokes.player@gmail.com> Co-authored-by: Bill Glesias <bglesias@gmail.com> Co-authored-by: Adam Stone-Lord <adams@cypress.io> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Dave Kasper <dave.m.kasper@gmail.com> Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Ely Lucas <ely@meta-tek.net> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Ben M <benm@cypress.io> Co-authored-by: Jennifer Shehane <jennifer@cypress.io>

* chore: add Nx Cloud (#26712) * chore: add empty nx.json [run ci] * chore: add nx cloud runner [run ci] * chore: add nx-cloud dep [run ci] * chore: update local nx cloud accessToken to be read-only * feat: update git related messages for runs and debug (#26758) * feat(app): update DebugError copy * feat: set isUsingGit project flag and consume in DebugContainer * feat(app): update not using git icon for DebugError * feat(app): displays alert on runs page when not using git * feat(app): add component for when no runs for current branch * feat(app): add warning for no runs for branch on runs container * chore: add feat to CHANGELOG * chore: remove logged status value * chore: resolve import from merge conflict * test(app): stub branch name for e2e runs spec * chore: add line break in changelog entry * chore: cleanup PR * chore: fix i18n import for DebugBranchError * chore: add utm and update Warning links to inline * chore: capitalize Git in i18n * ref: warning liink * test: add i18n to tests * test(app): change utm_source assertions * chore: cleanup pr * chore: remove unused prop * test(app): remove no git warning when moving to runs page in e2e * chore: change template logic * chore: remove duplicate RUNS_TAB_MEDIUM const * Changed Debug test assertion and reordered new components for Debug --------- Co-authored-by: Stokes Player <stokes.player@gmail.com> * chore: rename video processing events to capture/compress (#26800) * chore: change processing nomenclature to compressing when printing the run. * chore: rename 'capturing of' to 'capturing' * chore: rename upload results to upload screenshots & videos (#26811) * chore: rename upload results to upload screenshots & videos * run ci * chore: capture versions of relevant dependencies with `x-dependencies` header (#26814) * chore: update changlelog script to handle revert pr ref (#26801) * fix: Correct typescript scaffold dependency (#26815) * fix: correct typescript scaffold dependency (#26204) * add changelog * Update change log for PR comment Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> --------- Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> * chore: 12.13.0 prep (#26833) * chore: 12.13.0 release (#26834) * chore: 12.13.0 changelog fix * remove pending, bump version * fix typo * chore: release @cypress/vite-plugin-cypress-esm-v1.0.1 [skip ci] * chore: Implement runSpec mutation (#26782) * chore: replace gitter badge with discord on readme (#26771) * chore: add GraphQL mutation for sending system notifications via Electron (#26773) * fix: upgrade typescript from 4.7.4 to 4.9.5 (#26826) Snyk has created this PR to upgrade typescript from 4.7.4 to 4.9.5. See this package in npm: See this project in Snyk: https://app.snyk.io/org/cypress-opensource/project/d5b36925-e6ee-455d-9649-6560a9aca413?utm_source=github&utm_medium=referral&page=upgrade-pr * test: fix 2 broken tests for Windows (#26854) * Update stale_issues_and_pr_cleanup.yml Upped the number of operations per run. Have been manually doing that so this job can get through all the issues in the repo with no problems. * chore(dep): [Snyk] Upgrade vite from 2.9.13 to 2.9.15 (#26830) Co-authored-by: Ben M <benm@cypress.io> * Update triage_add_to_project.yml * chore: fix minor background color styling in debug results component (#26887) * Update stale_issues_and_pr_cleanup.yml stalebot was incorrectly configured to run in debug mode. I have updated the default to run in normal mode when running scheduled * chore: Deprecate @cypress/xpath package (#26893) * chore: add telemetry realworld app (#26896) * chore: capture telemetry for realworld app maybe * idk what i was doing * setup record key and telemetry * testing * override project id * some times we just need a little context. * Adding tests * Adding comment * chore(deps): update dependency find-process to v1.4.7 🌟 (#26906) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> * chore(deps): update dependency firefox-profile to v4.3.2 🌟 (#26912) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> --------- Co-authored-by: Jordan <jordan@jpdesigning.com> Co-authored-by: Stokes Player <stokes.player@gmail.com> Co-authored-by: Bill Glesias <bglesias@gmail.com> Co-authored-by: Adam Stone-Lord <adams@cypress.io> Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com> Co-authored-by: Dave Kasper <dave.m.kasper@gmail.com> Co-authored-by: Mike Plummer <mike-plummer@users.noreply.github.com> Co-authored-by: Mark Noonan <mark@cypress.io> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Ely Lucas <ely@meta-tek.net> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Stokes Player <stokes@cypress.io> Co-authored-by: Ben M <benm@cypress.io> Co-authored-by: Jennifer Shehane <jennifer@cypress.io> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

cypress-bot · 2023-06-07T17:26:26Z

Released in 12.14.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v12.14.0, please open a new issue.

mike-plummer changed the title ~~feat: Implement runSpec mutation~~ chore: Implement runSpec mutation May 17, 2023

chore: Implement 'runSpec' mutation

057ecca

mike-plummer force-pushed the mikep/593-gql-run-spec-endpoint branch from af37dcc to 057ecca Compare May 17, 2023 17:37

run ci

db16d0e

mike-plummer added 12 commits May 17, 2023 14:42

Refactor logic into gql action, add tests

030e09c

Roll back function conversion to async

61bd808

Fix async function

236c88b

Merge branch 'develop' into mikep/593-gql-run-spec-endpoint

9ad9d2b

run ci

90a79ec

Fix linting

4d12005

Implement launch from launchpad

9ac9d6c

Cleanup

717950a

Fix tests, add comments

c8e9821

Merge branch 'develop' into mikep/593-gql-run-spec-endpoint

23484aa

run ci

ca09ad9

Fix test

0b58754

mike-plummer requested a review from a team May 19, 2023 16:56

mike-plummer marked this pull request as ready for review May 19, 2023 16:56

mike-plummer added 3 commits May 19, 2023 13:56

Address edge case, add timeout logic

7c65f95

Merge branch 'develop' into mikep/593-gql-run-spec-endpoint

925bf9f

Fix test

29efc5d

Add extra wait for browser to finish opening

55890f3

astone123 approved these changes May 19, 2023

View reviewed changes

packages/data-context/src/actions/ProjectActions.ts Show resolved Hide resolved

mike-plummer requested a review from a team May 22, 2023 12:54

Add missing await

4c4a5ef

warrensplayer reviewed May 22, 2023

View reviewed changes

packages/graphql/src/schemaTypes/objectTypes/gql-RunSpecResponse.ts Outdated Show resolved Hide resolved

warrensplayer reviewed May 22, 2023

View reviewed changes

packages/data-context/src/sources/ProjectDataSource.ts Outdated Show resolved Hide resolved

mike-plummer added 2 commits May 23, 2023 09:44

Address review comments

1d0305d

* Refactor gql into Response/Error types * Refactor specPatterns logic into new function * Add tests * Move cy-in-cy test to better location

Merge branch 'develop' into mikep/593-gql-run-spec-endpoint

8934f92

warrensplayer reviewed May 23, 2023

View reviewed changes

mike-plummer added 3 commits May 23, 2023 12:19

Address review comments

bbf627a

* Add function docs * Remove unnecessary code value * Simplify type resolution logic, improve schema contract

Fix types

c15e9f1

Fix nullability issue

e344c9d

mike-plummer mentioned this pull request May 23, 2023

Add logic to change browser state #26763

Closed

warrensplayer approved these changes May 23, 2023

View reviewed changes

mike-plummer merged commit 58b9ccc into develop May 24, 2023

mike-plummer deleted the mikep/593-gql-run-spec-endpoint branch May 24, 2023 12:32

cypress-bot bot locked as resolved and limited conversation to collaborators Jun 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Implement runSpec mutation #26782

chore: Implement runSpec mutation #26782

mike-plummer commented May 17, 2023 •

edited

Loading

cypress bot commented May 17, 2023 •

edited

Loading

astone123 commented May 19, 2023

mike-plummer commented May 19, 2023

astone123 commented May 19, 2023

astone123 commented May 19, 2023

astone123 commented May 19, 2023

astone123 left a comment

warrensplayer commented May 22, 2023

mike-plummer commented May 22, 2023

warrensplayer left a comment

warrensplayer May 23, 2023

mike-plummer May 23, 2023

cypress-bot bot commented Jun 7, 2023

chore: Implement runSpec mutation #26782

chore: Implement runSpec mutation #26782

Conversation

mike-plummer commented May 17, 2023 • edited Loading

Additional details

Steps to test

How has the user experience changed?

PR Tasks

cypress bot commented May 17, 2023 • edited Loading

28 flaky tests on run #46715 ↗︎

astone123 commented May 19, 2023

mike-plummer commented May 19, 2023

astone123 commented May 19, 2023

astone123 commented May 19, 2023

astone123 commented May 19, 2023

astone123 left a comment

Choose a reason for hiding this comment

warrensplayer commented May 22, 2023

mike-plummer commented May 22, 2023

warrensplayer left a comment

Choose a reason for hiding this comment

warrensplayer May 23, 2023

Choose a reason for hiding this comment

mike-plummer May 23, 2023

Choose a reason for hiding this comment

cypress-bot bot commented Jun 7, 2023

mike-plummer commented May 17, 2023 •

edited

Loading

cypress bot commented May 17, 2023 •

edited

Loading