Automated Testing Guidelines for Sitecore Upgrades

When performing Sitecore upgrades, our goal is to upgrade Sitecore software while leaving the external website exactly the same as before starting the migration in both content, look and functionality. This means all the assets and functionality must be preserved, no matter which background technology is changing. The whole experience has to remain the same. In other words, it should be as transparent as possible to end users.

In order to successfully achieve this mission, we have created the following list of guidelines:

Determine the most-visited pages.
In the real world, websites contain a huge amount of pages to analyze, so a good first step is to build an initial inventory of most visited pages. Our Oshyn Framework Crawler allows us to generate this list in minutes.

Build control templates.
Try to identify patterns across pages. If you have 1,000 pages that use the same structure, it’s a good idea to define the control definitions for the common skeleton of such pages. The Oshyn Framework allows us to reuse the same template for all 1,000 tests.  You need to define a component for testing once, not many times even though the component shows up on multiple pages. Our framework identifies re-usable components for testing.

Write User Interface Tests to validate ALL the assets and content.
Prioritize the assets and content to validate and create User Interface (UI) tests for the entire site. It is very common to have missing images, videos, or text after executing a migration, so this allows us to identify these kinds of errors early and repetitively as the upgrade is happening over time. The Oshyn Framework Crawler is able to recognize all the paragraphs, links, and images present in web pages. Basically, you will have a complete picture of what things looked like before starting the migration.

Maintain an environment that contains the old version of the website.
Part of our process is to maintain a complete copy of the old/current environment, because this will allow us to compare both environments at any time and verify the existence of pre-existing issues or missing logic in the new site.  The EXACT SAME UI Tests are run against both the Control environment and the NEW environment, so we are able to empirically prove that the upgraded Sitecore and code function exactly the same as the old site.

Schedule daily executions of automated tests.
It’s a good practice to do frequent regressions when working with upgrades. The code gets changed and refactored over time as the upgrade is happening.  The Oshyn UI Test Framework will print some valuable reports of complete regression test results in order to us to ensure no adverse changes have happened to the upgraded site over time.

Identify the most common platforms/environments.
Use Selenium Grid or Sauce Labs remote server to run the tests in the most common platform / environments. This allows us to detect the issues specific to certain browser/operating system combinations at early stages.

Take care of client scripting.
Sometimes after migrating some pages, there will be some client script issues that are difficult to detect. Oshyn Test Framework allows us to detect these issues while leaving the system behavior the same as before the migration.

Finally, don’t forget the importance of also performing manual testing to ensure all the functionality not covered by the automation continues to work as expected. Automated tests are there to support the entire quality assurance effort of any migration project.  Anything that can't be done automated or is not cost-effective to be done automated is done manually - although this is kept to a minimum.

The final result is a suite of tests that is used not only to ensure the upgrade happened successfully, but is also used on an ongoing basis to ensure future enhancements to the site don't break any existing functionality.  It allows a rapid regression test which unlocks the ability to do frequent and even continuous releases.