test(core): add visual regression testing #962
Conversation
etowahadams
changed the title
etowahadams
changed the title
etowahadams
changed the title
| const screenshot = defineConfig({ | ||
| test: { | ||
| include: ['./img/visual-regression/*'], | ||
| globals: true, | ||
| environment: 'jsdom', | ||
| threads: false, | ||
| environmentOptions: { | ||
| jsdom: { | ||
| resources: 'usable' | ||
| } | ||
| } | ||
| } | ||
| }); |
There was a problem hiding this comment.
New config to be used only when yarn screenshot is called so that only tests in img/visual-regression will be run
|
Ready for review. I finally felt inspired to try new things after I was unable to get workers to work in puppeteer. Turns out workers don't work if there is no URL. The trick is to navigate to a page first, then to change the content of the page. I spent way too long figuring this out 😭 I decided not to make this visual regression test automated for now because many gosling visualizations are non deterministic. In particular, the axis seems to have some non-deterministic behavior when zoomed out. Here is an example of the diff between two renderings of the doc_point spec. In red are the regions which do not match between old and new renders. About 20 of the specs had this behavior. This might be a bug that we should fix. Still, I think this should be a useful tool to understand the extend of visual changes. 
|
| <link rel="stylesheet" href="${baseUrl}/higlass@${higlassVersion}/dist/hglib.css"> | ||
| <script src="${baseUrl}/react@${reactVersion}/umd/react.production.min.js"></script> | ||
| <script src="${baseUrl}/react-dom@${reactVersion}/umd/react-dom.production.min.js"></script> | ||
| <script src="${baseUrl}/pixi.js@${pixijsVersion}/dist/browser/pixi.min.js"></script> | ||
| <script src="${baseUrl}/higlass@${higlassVersion}/dist/hglib.js"></script> | ||
| <script type="text/javascript">${gosling}</script> | ||
| <body> | ||
| <div id="vis"></div> | ||
| <script> | ||
| gosling.embed(document.getElementById("vis"), JSON.parse(\`${spec}\`)) | ||
| </script> | ||
| </body> | ||
| </html>`; |
There was a problem hiding this comment.
This will have to change when we update Gosling to only have a ESM build
|
Any updates on this? I think this will be quite useful when implementing performance related improvements. |
|
Thank you for the reminder! And, sorry, I lost track of this. Regarding the change of the axis, what do you think of making the visibility of ticks deterministic? We can remove the use of random values for determining the visibility, and it might be as simple as removing the following line: |
| ## Visual Regression | ||
|
|
||
| To test if your changes make a difference visually across a large number of specifications, you can use can use the `yarn screenshot` command. | ||
| This command will use `dist/gosling.js` in a headless browser (using puppeteer) and take a screenshot of the visualization that is included in the editor. |
There was a problem hiding this comment.
You could briefly mention that one needs to run yarn build before yarn screenshot to test the updated Gosling.js.
CONTRIBUTING.md
Outdated
|
|
||
| To test if your changes make a difference visually across a large number of specifications, you can use can use the `yarn screenshot` command. | ||
| This command will use `dist/gosling.js` in a headless browser (using puppeteer) and take a screenshot of the visualization that is included in the editor. | ||
| It will save these screenshots in `img/visual-regression/new-screenshots`. If there is a difference compared to the screenshots in `img/visual-regression/screenshots` |
There was a problem hiding this comment.
So, for now, would the usage pattern be like (occasionally) running the yarn build and yarn screenshot and then moving all files under img/visual-regression/new-screenshots to img/visual-regression/screenshots?
There was a problem hiding this comment.
That's how I currently see it being used in its current state, yes. I can add this as usage pattern in the documentation. I can also add a command which moves the screenshots from img/visual-regression/new-screenshots to img/visual-regression/screenshots.
There was a problem hiding this comment.
This looks like a very useful feature! Thank you for making these chnages.
One thing I notice is when I run yarn, yarn build, and yarn screenshot, I am getting the following error complaining that the new-screenshots/* are not found. Just in case it is related to wait times, I tested with larger values for related parameters, e.g., idelTime, delay(), but it shows the same issue. There must be something that I am missing. Do you have any clue about this issue?
|
Update: It looks like I needed to create empty |
Fix #958
Toward #
Adds a
yarn screenshotcommand. This command will usedist/gosling.jsin a headless browser (using puppeteer) and take a screenshot of the visualization that is included in the editor. It will save these screenshots inimg/visual-regression/new-screenshots. If there is a difference compared to the screenshots inimg/visual-regression/screenshotsit will show the differences in an image inimg/visual-regression/diffs.Interestingly, the Gosling genomic axis is non-deterministic for many specs. Parts of the axis will look different between renders. This makes automated visual regression testing unsuitable. I think this best used as a check when testing locally to make sure that many visualizations are not impacted.
Checklist