Visual regression testing

[hugopeek]

Hi! I'm wondering if we could set up some kind of visual regression testing for the component library. Or maybe this is already being done?

Visual regression tests compare the UI before and after an update, and generate a diff view showing you exactly where things have shifted. Down to the pixel, if you want.

It popped up in my head after pull #2269. Fixes like these could have a big impact throughout the library and can be very hard to test. I have some experience with BackstopJS, which is a great tool. But maybe as an open source project, there are also hosted options that offer a free tier. A popular option is Percy, which seems to be part of Browserstack now.. Are you already running tests with Browserstack? Maybe we can extend this with Percy tests?

Some further reading:

https://medium.com/loftbr/visual-regression-testing-eb74050f3366
https://pantheon.io/blog/using-backstopjs-spot-difference
https://dev.to/abusithik/automated-visual-regression-testing-with-backstopjs-3bc4

[lubber-de]

We currently only use browserstack manually whenever needed for testing fixes in/for browsers which we don't have available ourselves. (For example, i don't have an iPhone/iPad/Mac, thus cannot test Safari natively).
I quickly logged into browserstack and indeed there is a possibility to somehow use include percyjs.

Seems we could setup a github and static site generator integration , however docpad (which is currently used) was not available and the percy docs were too much for me for now, so i would appreciate if somebody else could step in here .
Talk to @hammy2899 on Discord because he is the org-maintainer for FUI on browserstack.

There are plans and some tests to completely refactor the docs-site anyway.