Given its exceptional page layout, interesting and relevant articles, and amazing artwork, I’ve enjoyed reading all the issues of Increment Magazine. Since web testing is one of my active research areas, I read with interest the May 2020 issue on the “Frontend” theme. An interview collection called Frontend at scale with leading industry experts such as Joscha Feth from Canva piqued my interest! Here’s an excerpt from Joscha’s interview:
“We use visual regression testing to ensure that unintended changes don’t make it to the baseline in the first place. Our end-to-end tests, run against a local stack of Canva, guard every release as the last automated step. Lastly, we do use manual testing for things that are hard to test in an automated fashion. (Drag-and-drop interactions come to mind.) We are trying to automate these tests as they are currently preventing us from going from daily releases to continuous deployment.”
If you want to learn more about the “visual regression testing” approach that they have adopted at Canva, then I encourage you to read Joscha’s article Why we left manual UI testing behind. Now, you may be wondering why I am so excited to see articles from industry-leading experts and industry-focused magazines highlight the challenges and importance of mobile-ready web testing. Well, I teach courses on web development — in fact, I’m teaching the course again during the Fall 2020 semester! — and I’ve used a variety of technologies to create and maintain several web sites, helping me to understand how hard it is to test the responsive layout of a mobile-ready page.
Since 2015 my colleagues and I have also been building and experimentally studying tools that support the testing of responsive web pages. To date, our work has created automated techniques to address two key challenges: (i) surfacing differences in responsive web layouts and (ii) detecting responsive layout failures without explicit test oracles. Our work published in (Walsh, McMinn, and Kapfhammer 2015)
Since web developers often lack an automated oracle by which they can validate a web page’s responsive layout, my colleagues and I also developed an “oracle free” failure detection technique that checks the consistency of a page’s responsive layout across a range of viewport widths. The results in (Walsh, Kapfhammer, and McMinn 2017)
In summary, it is exciting to see that software engineers in industry are adopting commercial tools, like Percy, that are similar in aim to those that we are prototyping and releasing as free and open-source software. While both academia and industry are starting to make progress on this important problem, more work remains to create fully automated methods for testing responsive web pages. What research and development should we try next?