Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "scoreboard" test suite #479

Merged
merged 10 commits into from
May 9, 2018
Merged

Add "scoreboard" test suite #479

merged 10 commits into from
May 9, 2018

Conversation

shawnbot
Copy link
Contributor

@shawnbot shawnbot commented May 3, 2018

This adds a new "scoreboard" package (which is not yet npm-publish-able) in meta/scoreboard that should generate a table of high-level checks for every one of our packages in modules/primer-*. Currently, the two checks are:

  1. Whether the package has Storybook stories (the pattern is {dir}/stories.js, which currently doesn't match ones with {dir}/stories/*.js, but we can change it).
  2. Whether script/test-docs passes when run in the package dir, regardless of whether the package's own test script is configured to run them. This is a way for us to have a better understanding of what's missing without just failing all of our tests (and builds) outright.

New checks should be easy to add; just add new keys to this object literal in the form:

'column title': (moduleObject, key) => {
  return {[key]: value} // or a Promise that resolves to an object
}

I've also refactored our docs test script in the following ways:

  • It's more asynchronous 💅
  • It finds class names in ERB helpers, e.g. <% octicon "foo", class: "float-right" %> (or in hash rocket form), which eliminates a bunch of false negatives.
  • It outputs one test per class. The tradeoff here is that the output lists all of the classes being tested, but output with lots of errors is hard to read because AVA's reporter is overly chatty, even without --verbose.
  • It supports a new class_whitelist field on the primer object in each package's package.json, which is an array of minimatch-compatible globs. See 77f987f for some examples.

To run it manually:

cd meta/scoreboard
npm install
npm test

You can run the checks for a specific subset of packages by passing the directories in as positional arguments to index.js, e.g.

node index.js ../../modules/primer-marketing*

@shawnbot
Copy link
Contributor Author

shawnbot commented May 3, 2018

Sweet! You can see an example of output in Travis builds here:

package                     has stories  docs test  missing docs                                                                                               
--------------------------  -----------  ---------  -----------------------------------------------------------------------------------------------------------
primer-alerts               no           pass                                                                                                                  
primer-avatars              yes          FAIL       ".avatar-group-item", ".avatar-link", ".avatar-stack", and 1 more...                                       
primer-base                 yes          pass                                                                                                                  
primer-blankslate           yes          pass                                                                                                                  
primer-box                  yes          FAIL       ".Box--scrollable", ".Box-row--drag-button", ".Box-row--drag-hide", and 1 more...                          
primer-branch-name          yes          pass                                                                                                                  
primer-breadcrumb           yes          pass                                                                                                                  
primer-buttons              yes          FAIL       ".btn-blue"                                                                                                
primer-forms                no           FAIL       ".bad-file", ".bad-permissions", ".btn-plain", ".comment", ".comment-form-error", and 61 more...           
primer-labels               yes          FAIL       ".label", ".labels", ".state", and 1 more...                                                               
primer-layout               yes          FAIL       ".gutter", ".gutter-condensed", ".gutter-lg", ".gutter-lg-condensed", ".gutter-lg-spacious", and 23 more...
primer-markdown             yes          FAIL       ".absent", ".anchor", ".blob-num", ".emoji", ".highlight", and 9 more...                                   
primer-marketing-buttons    yes          pass                                                                                                                  
primer-marketing-support    no           FAIL                                                                                                                  
primer-marketing-type       yes          pass                                                                                                                  
primer-marketing-utilities  yes          pass                                                                                                                  
primer-navigation           yes          FAIL       ".avatar", ".bar", ".menu-warning", ".select-menu-item-icon", ".subnav-bordered", and 13 more...           
primer-page-headers         no           FAIL       ".jumbotron", ".jumbotron-minitron", ".jumbotron-photo", ".jumbotron-supertron", and 3 more...             
primer-page-sections        no           FAIL       ".page-section"                                                                                            
primer-popover              yes          pass                                                                                                                  
primer-subhead              yes          pass                                                                                                                  
primer-support              no           FAIL                                                                                                                  
primer-table-object         yes          pass                                                                                                                  
primer-tables               yes          FAIL       ".data-table"                                                                                              
primer-tooltips             yes          pass                                                                                                                  
primer-truncate             yes          pass                                                                                                                  
primer-utilities            no           FAIL       ".anim-pulse-in", ".anim-shrink-x", ".bg-pending", ".bg-shade-gradient", ".bg-white", and 371 more...

@shawnbot shawnbot changed the title [WIP] Add "scoreboard" test suite Add "scoreboard" test suite May 3, 2018
@shawnbot
Copy link
Contributor Author

shawnbot commented May 3, 2018

FYI, I've updated the description to list the changes to our docs test script. I think this is good to merge as-is so that we can pull it into other branches.

@shawnbot shawnbot requested a review from a team May 3, 2018 23:28
Copy link
Contributor

@emplums emplums left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just left one comment but otherwise looks good to me!

@@ -24,6 +24,7 @@
"@storybook/addon-options": "^3.2.6",
"@storybook/react": "^3.2.12",
"ava": "^0.23.0",
"babel-core": "^6.26.3",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious why this was added? I don't see it being imported anywhere

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I'll remove it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird... this isn't in the stories-to-markdown branch anymore. GitHub bug??

@shawnbot shawnbot merged commit bd4a414 into dev May 9, 2018
@shawnbot shawnbot deleted the scoreboard branch May 9, 2018 18:46
@emplums emplums mentioned this pull request May 30, 2018
15 tasks
@jonrohan jonrohan restored the scoreboard branch June 13, 2018 22:15
@jonrohan jonrohan deleted the scoreboard branch June 13, 2018 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants