Validate Queries Plugin #1004

supernova-at · 2019-03-08T18:03:10Z

Description

`graphql-cli-validate-magento-pwa-queries`

This PR creates a new graphql-cli plugin called validate-magento-pwa-queries.

Venia depends on graphql-cli and this new plugin to do the work of validating the queries, replacing the need for validate-queries.js in packages/venia-concept/ itself.

Venia's validate-queries script runs the plugin.

Magento Compatibility Table

There is a new file called magento-compatibility.js at the root of PWA Studio. It currently looks like this:

/**
 * This file describes PWA Studio to Magento version compatabilities.
 */

// PWA Studio version -> Magento version.
module.exports = {
    '2.0.0': '2.3.0',
    '>2.0.0': '2.3.1'
};

This file is intended to be the source of truth for which versions of PWA are compatible with which versions of Magento.

The file is currently used by a new script (create-compatibility-table.js) in pwa-devdocs that creates a markdown table in its _includes directory for inclusion on the docs site.

It is also used by a new script (summarize-validate-queries.js) in venia-concept that looks up the current version of PWA (based on the version field in the root package.json) and informs developers which versions of Magento are compatible.

📝 This file should definitely be considered a "living document" and we should update it as appropriate during releases.

Related Issue

Closes #952 .

Motivation and Context

From the issue:

As a developer working with PWA Studio, I want to know when breaking changes occur with GraphQL in order to address any incompatibility or breaking changes. We should provide clear messaging/errors to developers regarding where and how to resolve.

Verification

First, yarn clean:all and yarn install to pick up the new dependencies.

Testing `2.3.0`

Change the value of MAGENTO_BACKEND_URL in env to https://release-dev-rxvv2iq-zddsyhrdimyra.us-4.magentosite.cloud/graphql.
Run yarn validate-queries.
Verify the following output:

aterrano pwa-studio (supernova/952_validate-queries) $ yarn validate-queries
yarn run v1.13.0
$ yarn workspace @magento/venia-concept run validate-queries
$ yarn run download-schema && graphql validate-magento-pwa-queries --project venia
$ graphql get-schema --project venia
project venia - No changes
Validating GraphQL queries in venia project...
✔ Locating schema...
✔ Finding queries in files...
✔ Validating project's queries against the schema...

All queries are valid.
✨  Done in 6.99s.

Testing `2.3.1`

Change the value of MAGENTO_BACKEND_URL in env to https://release-dev-231-npzdaky-zddsyhrdimyra.us-4.magentosite.cloud/.
Run yarn validate-queries.
Verify the following output:

aterrano pwa-studio (supernova/952_validate-queries) $ yarn validate-queries
yarn run v1.13.0
$ yarn workspace @magento/venia-concept run validate-queries
$ yarn run download-schema && graphql validate-magento-pwa-queries --project venia
$ graphql get-schema --project venia
project venia - No changes
Validating GraphQL queries in venia project...
✔ Locating schema...
✔ Finding queries in files...
✔ Validating project's queries against the schema...

Found some invalid queries:

/Users/aterrano/code/pwa-studio/packages/venia-concept/src/queries/getCategory.graphql
  11:21  error  Field "small_image" of type "ProductImage" must have a selection of subfields. Did you mean "small_image { ... }"?  graphql/template-strings

/Users/aterrano/code/pwa-studio/packages/venia-concept/src/queries/getCategoryList.graphql
  15:21  error  Field "small_image" of type "ProductImage" must have a selection of subfields. Did you mean "small_image { ... }"?  graphql/template-strings

/Users/aterrano/code/pwa-studio/packages/venia-concept/src/queries/getNavigationMenu.graphql
  21:11  error  Field "small_image" of type "ProductImage" must have a selection of subfields. Did you mean "small_image { ... }"?  graphql/template-strings

/Users/aterrano/code/pwa-studio/packages/venia-concept/src/queries/getProductDetail.graphql
  14:13  error  Field "description" of type "ComplexTextValue" must have a selection of subfields. Did you mean "description { ... }"?  graphql/template-strings

/Users/aterrano/code/pwa-studio/packages/venia-concept/src/queries/productSearch.graphql
  6:13  error  Field "small_image" of type "ProductImage" must have a selection of subfields. Did you mean "small_image { ... }"?  graphql/template-strings

✖ 5 problems (5 errors, 0 warnings)

Your versions of PWA and Magento may be incompatible.

This Venia store is from PWA Studio version 2.0.0.
Some components in this PWA send GraphQL queries that are only compatible with version 2.3.0 of Magento 2.

Please refer to the compatibility table in the PWA documentation for more details: https://magento-research.github.io/pwa-studio/technologies/magento-compatibility/.
    
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: /usr/local/Cellar/node@10/10.15.0/bin/node
Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run validate-queries
Directory: /Users/aterrano/code/pwa-studio/packages/venia-concept
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

How Has This Been Tested?

The verification steps above, and yarn test.

Screenshots (if appropriate):

Proposed Labels for Change Type/Package

FEATURE

venia-concept

Checklist:

I have read the CONTRIBUTING document.
I have linked an issue to this PR.
I have indicated the change type and relevant package(s).
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.
All CI checks are green (linting, build/deploy, etc).
At least one core contributor has approved this PR.

TODOs

~~Incorporate the downloading of the schema into the plugin itself~~
Figure out why validation is coming back successful when the old script produces errors
Unit Tests for the plugin
Publish the graphql-cli-validate-queries plugin to NPM and install it from there instead of using the local filesystem
Determine how to tie in the compatibility table. The plugin itself shouldn't know anything about PWA or Magento. Can we use process.exit(1) effectively?

Followup

~~Open a PR to graphql-cli and get validate-queries listed as a plugin on their main README.~~

vercel · 2019-03-08T18:03:13Z

This pull request is automatically deployed with Now.
To access deployments, click Details below or on the icon next to each push.

supernova-at · 2019-03-08T18:04:04Z

Adding the hold and in progress labels because this has TODOs.

supernova-at · 2019-03-11T20:31:27Z

The graphql-cli-validate-queries plugin is ready for review.

Remaining steps for this PR:

Publish graphql-cli-validate-queries to npm
Re-add it as a dependency to venia-concept (instead of depending on the local file system).
Try to add additional PWA / Magento specific logging (compatibility table) for when the plugin reports errors.

coveralls · 2019-03-11T20:32:30Z

Coverage increased (+0.2%) to 76.995% when pulling f3db63e on supernova/952_validate-queries into 37cd2c1 on develop.

packages/graphql-cli-validate-queries/LICENSE.md

packages/graphql-cli-validate-queries/index.js

supernova-at · 2019-03-12T14:56:31Z

packages/upward-spec/package.json

@@ -28,7 +28,7 @@
    "apollo-server": "~2.3.1",
    "chalk": "~2.4.2",
    "csv-parse": "~4.3.0",
-    "graphql": "~14.0.2",
+    "graphql": "~14.1.1",


I had to bump these versions due to an "incompatibility" bug where multiple instances of the graphql dependency were installed and the plugin didn't know which one to use.

Right now these are manually kept in sync. I would very much welcome a better way to do this, it seems to me like the version of graphql in upward-js and upward-spec should be completely independent.

Should this just be a peer dep of the consuming project? IMO the more peer deps we delegate to the final consumer, the better, since that's a great way to prevent duplicate packages in a final bundle.

Do upward-js and upward-spec even depend on the graphql dependency? I see a single require('graphql-tag') here but other than that I don't see anything.

I also have next to no familiarity with these codebases though 😛

packages/venia-concept/package.json

pwa-devdocs/_drafts/graphql-cli-validate-queries/notes.md

packages/graphql-cli-validate-queries/index.js

supernova-at · 2019-03-12T15:07:43Z

One last item that I'd appreciate feedback on:

Should pwa-buildpack or peregrine run the plugin too?

My gut is that peregrine will need it but that pwa-buildpack probably won't.

packages/venia-concept/summarize-validate-queries.js

zetlen · 2019-03-12T17:11:56Z

@supernova-at I don't think the queries in Peregrine should be in there. @jimbo could convince me otherwise, but I would rather Peregrine be pretty platform-agnostic. Its MagentoRouter uses a handbuilt query to retrieve urlResolver data, but I think it should be refactored so the query can be injected from the project.

zetlen

This looks great. Since we eventually want to make a Buildpack CLI that does this Magento-specific stuff (see my comment below), we'll have to come revisit this no matter what. Because of that, I think it's safe to ignore some of the modularity ideas we discussed, and instead just make this plugin a Magento PWA Studio specific graphql-cli plugin, so you can combine the plugin code and the query report code.

pwa-devdocs/scripts/create-compatibility-table.js

zetlen · 2019-03-12T18:25:29Z

packages/venia-concept/package.json

    "prepublishOnly": "yarn run build",
    "start": "node server.js",
    "start:debug": "node --inspect-brk ./node_modules/.bin/webpack-dev-server --progress --color --env.mode development",
    "stats": "webpack-bundle-analyzer dist/build-stats.json",
    "storybook": "start-storybook -p 9001 -c .storybook",
    "storybook:build": "build-storybook -c .storybook -o storybook-dist",
-    "validate-queries": "node ./validate-queries.js",
+    "validate-queries": "yarn run download-schema && graphql validate-queries --project venia --filesGlob \"src/**/*.{js,graphql,gql}\" --clients apollo literal && node ./summarize-validate-queries.js",


Eventually we'd want the same command to both:

Confirm that the cached local schema still matches the remote schema

Validate all the queries

I figure once we have a real @magento/pwa-buildpack cli, like pbuild, it would look a little more like:

"validate-queries": "pbuild verify-schema && graphql validate-queries | pbuild diagnose-queries"

Speaking of which, can you try making the CLI flags optional, and having them fall back to extension configuration in .graphlconfig?

I ended up just removing them and always pulling them from .graphqlconfig.

packages/venia-concept/.graphqlconfig

packages/graphql-cli-validate-queries/__tests__/index.spec.js

packages/venia-concept/summarize-validate-queries.js

Creates a compatibility definition file and script to generate markdown from it Creates a graphql-cli extension to validate a project's queries Created a new project / repository called graphql-cli-validate-queries that - based on configuration - goes through all of a project's GraphQL queries and runs them against an endpoint's schema. Updates venia-concept to use this new extension and the graphql-cli tool in general. Resolves #952. Fixes the format of the graphql/template-strings eslint rule Adds script to dev-docs to auto-generate the magento compatibility table Adds a README to the graphql-cli-validate-queries project Adds unit tests to graphql-cli-validate-queries plugin. Installs and wires up Jest for unit testing the validate-queries plugin. Fixed typo mistakes. (#993) * Fixed typo mistakes * Fixed typo mistakes chore: remove duplicate padding property (#941) [BUGFIX] [ISSUE-958] Autocomplete renders loading component on clear (#961) * [BUGFIX] [ISSUE-958] Autocomplete renders loading component on clear Removes invalid TODO in graphql-cli-validate-queries/index.js Adds a JSDoc comment block to the getValidator function in graphql-cli-validate-queries Cleans up the devdoc draft for validate-queries plugin Adds a PWA-specific summarization to the end of the validate-queries command. Adds a new script summarize-validate-queries.js to venia-concept Runs this script at the end of the validate-queries script This script uses the compatibility table to clue developers to compatibility issues and how to resolve them Runs prettier Removes use of reserved word in summarize-validate-queries.js Create compatibility table page (#1016) * Fix auto-generated table markdown and re-order versions to list the latest version first * Incorporate build-compatibility table script into build process * Create a page to display the compatibility table * Update pwa-devdocs/src/technologies/magento-compatibility/index.md Co-Authored-By: jcalcaben <jcalcaben@users.noreply.github.com> Updates the path to the compatibility table on the docs site Updates the wording around the reporting of version specifics and compatibility in the summarize-validate-queries script Runs prettier and updates summarize-validate-queries. Fixes dependencies and includes plugin in root jest test. The graphql-cli-validate-magento-pwa-queries plugins tests are now run as part of the root test command. This ensures that breaking tests in this project will fail the prepush check. Cleans up yarn.lock changes by selectively installing dependencies via yarn workspace add commands. Renames plugin to be more magento and pwa specific. Runs prettier Removes unneeded summary script file Updates the dev docs for the graphql-cli plugin

supernova-at · 2019-03-13T18:19:27Z

I had to rebase / force push because it was insanity to merge yarn.lock after #1006.

I ended up rebasing, deleting yarn.lock, and then yarn installing and checking in the new yarn.lock.

…d options

packages/graphql-cli-validate-magento-pwa-queries/lib/index.js

…a-queries from the registry instead of the local file system

supernova-at · 2019-03-14T15:42:55Z

PR Updated:

Published graphql-cli-validate-magento-pwa-queries to the NPM registry
Updated Venia's dependencies to pull the plugin from the registry instead of the local file system

zetlen

LGTM, this is a great addition. A good place to start generalizing these tools back into the graphql ecosystem.

zetlen · 2019-03-15T17:06:27Z

jest.config.js

+            () => ({
+                testEnvironment: 'node'
+            })
+        )


Thanks for sticking with this convention we're doing.

zetlen · 2019-03-15T17:07:21Z

packages/graphql-cli-validate-magento-pwa-queries/README.md

+* [graphql-cli](https://github.com/graphql-cli/graphql-cli)
+* [eslint-plugin-graphql](https://github.com/apollographql/eslint-plugin-graphql)
+* [graphql/no-deprecated-fields rule](https://github.com/apollographql/eslint-plugin-graphql#no-deprecated-fields-validation-rule)
+


I LOVE this documentation. Thank you for making it.

Just noticed a table formatting error, pushing commit now.

zetlen · 2019-03-15T17:07:55Z

packages/venia-concept/.graphqlconfig

+                "endpoints": {
+                    "default": "${env:MAGENTO_BACKEND_URL}graphql"
+                },
+                "validate-magento-pwa-queries": {


…ueries README

jimbo · 2019-03-15T17:18:59Z

packages/venia-concept/package.json

@@ -45,7 +46,8 @@
    "apollo-link-context": "^1.0.14",
    "apollo-link-http": "^1.5.11",
    "braintree-web-drop-in": "^1.16.0",
-    "graphql": "^14.1.1",
+    "graphql-cli": "^3.0.11",
+    "graphql-cli-validate-magento-pwa-queries": "^1.0.0",


This is really a mouthful. 😆

Lol yeah, it was much better as validate-queries but then we made it magento / pwa specific 😝

packages/graphql-cli-validate-magento-pwa-queries/package.json

packages/graphql-cli-validate-magento-pwa-queries/README.md

jimbo · 2019-03-15T17:34:29Z

packages/graphql-cli-validate-magento-pwa-queries/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Adobe Inc.


Is this an official thing?

Published to NPM so I was just covering all the bases 🤷‍♂️

sirugh · 2019-03-21T13:41:08Z

packages/venia-concept/.graphqlconfig

+            "schemaPath": "lastCachedGraphQLSchema.json",
+            "extensions": {
+                "endpoints": {
+                    "default": "${env:MAGENTO_BACKEND_URL}graphql"


Oops! This should have had a /.

supernova-at added hold On hold until another condition is fulfilled. in progress labels Mar 8, 2019

vercel bot deployed to staging March 8, 2019 18:04 View deployment

supernova-at added requires followup Some actions need to be taken after this code is merged. and removed requires followup Some actions need to be taken after this code is merged. labels Mar 8, 2019

vercel bot deployed to staging March 8, 2019 21:00 View deployment

vercel bot temporarily deployed to staging March 11, 2019 20:26 Inactive

vercel bot deployed to staging March 11, 2019 20:30 View deployment

supernova-at assigned zetlen Mar 11, 2019

supernova-at commented Mar 12, 2019

View reviewed changes

packages/graphql-cli-validate-queries/index.js Outdated Show resolved Hide resolved

vercel bot deployed to staging March 12, 2019 16:31 View deployment

jcalcaben reviewed Mar 12, 2019

View reviewed changes

packages/venia-concept/summarize-validate-queries.js Outdated Show resolved Hide resolved

supernova-at commented Mar 12, 2019

View reviewed changes

packages/venia-concept/summarize-validate-queries.js Outdated Show resolved Hide resolved

zetlen suggested changes Mar 12, 2019

View reviewed changes

vercel bot deployed to staging March 12, 2019 19:09 View deployment

supernova-at removed hold On hold until another condition is fulfilled. in progress labels Mar 12, 2019

vercel bot had a problem deploying to staging March 12, 2019 19:23 Error

vercel bot had a problem deploying to staging March 13, 2019 17:35 Failure

vercel bot had a problem deploying to staging March 13, 2019 17:37 Failure

vercel bot had a problem deploying to staging March 13, 2019 17:38 Failure

supernova-at force-pushed the supernova/952_validate-queries branch from ac0fd82 to ed84d16 Compare March 13, 2019 18:07

vercel bot had a problem deploying to staging March 13, 2019 18:07 Failure

supernova-at force-pushed the supernova/952_validate-queries branch from ed84d16 to a156a48 Compare March 13, 2019 18:16

vercel bot had a problem deploying to staging March 13, 2019 18:16 Failure

Updates graphql-cli-validate-magento-pwa-queries README with supporte…

4485573

…d options

vercel bot had a problem deploying to staging March 13, 2019 18:23 Failure

Bumps GraphQL plugin to version 1.0.0, updates dependencies

1307c90

vercel bot had a problem deploying to staging March 13, 2019 18:32 Failure

sirugh reviewed Mar 14, 2019

View reviewed changes

packages/graphql-cli-validate-magento-pwa-queries/lib/index.js Outdated Show resolved Hide resolved

Updates version error messaging in graphql-cli plugin for clarity

40cc0f3

vercel bot had a problem deploying to staging March 14, 2019 14:56 Failure

Updates Venia dependencies to install graphql-cli-validate-magento-pw…

0741250

…a-queries from the registry instead of the local file system

vercel bot deployed to staging March 14, 2019 15:41 View deployment

Merge branch 'develop' into supernova/952_validate-queries

3fa6405

vercel bot deployed to staging March 15, 2019 17:05 View deployment

zetlen previously approved these changes Mar 15, 2019

View reviewed changes

Fixes markdown table formatting in graphql-cli-validate-magento-pwa-q…

f3db63e

…ueries README

supernova-at dismissed zetlen’s stale review via f3db63e March 15, 2019 17:19

vercel bot deployed to staging March 15, 2019 17:20 View deployment

jimbo suggested changes Mar 15, 2019

View reviewed changes

jimbo approved these changes Mar 15, 2019

View reviewed changes

zetlen merged commit cb9b287 into develop Mar 15, 2019

jimbo deleted the supernova/952_validate-queries branch March 15, 2019 19:34

sirugh reviewed Mar 21, 2019

View reviewed changes

sirugh mentioned this pull request Mar 21, 2019

bug: Add / to graphql validation endpoint #1045

Merged

8 tasks

supernova-at mentioned this pull request Mar 21, 2019

Magento Backend URL Trailing Slash #1049

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate Queries Plugin #1004

Validate Queries Plugin #1004

supernova-at commented Mar 8, 2019 •

edited

Loading

vercel bot commented Mar 8, 2019

supernova-at commented Mar 8, 2019

supernova-at commented Mar 11, 2019

coveralls commented Mar 11, 2019 •

edited

Loading

supernova-at Mar 12, 2019

zetlen Mar 12, 2019

supernova-at Mar 12, 2019

supernova-at commented Mar 12, 2019 •

edited

Loading

zetlen commented Mar 12, 2019

zetlen left a comment

zetlen Mar 12, 2019

supernova-at Mar 13, 2019

supernova-at commented Mar 13, 2019

supernova-at commented Mar 14, 2019

zetlen left a comment

zetlen Mar 15, 2019

zetlen Mar 15, 2019

supernova-at Mar 15, 2019

zetlen Mar 15, 2019

jimbo Mar 15, 2019

supernova-at Mar 15, 2019

jimbo Mar 15, 2019

supernova-at Mar 15, 2019

sirugh Mar 21, 2019

Validate Queries Plugin #1004

Validate Queries Plugin #1004

Conversation

supernova-at commented Mar 8, 2019 • edited Loading

Description

graphql-cli-validate-magento-pwa-queries

Magento Compatibility Table

Related Issue

Motivation and Context

Verification

Testing 2.3.0

Testing 2.3.1

How Has This Been Tested?

Screenshots (if appropriate):

Proposed Labels for Change Type/Package

Checklist:

TODOs

Followup

vercel bot commented Mar 8, 2019

supernova-at commented Mar 8, 2019

supernova-at commented Mar 11, 2019

coveralls commented Mar 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

supernova-at commented Mar 12, 2019 • edited Loading

zetlen commented Mar 12, 2019

zetlen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

supernova-at commented Mar 13, 2019

supernova-at commented Mar 14, 2019

zetlen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

supernova-at commented Mar 8, 2019 •

edited

Loading

`graphql-cli-validate-magento-pwa-queries`

Testing `2.3.0`

Testing `2.3.1`

coveralls commented Mar 11, 2019 •

edited

Loading

supernova-at commented Mar 12, 2019 •

edited

Loading