The Preview Release Process
You’ve gotten buy-in and have created a dedicated Preview Release Team. Check. You’ve identified your customers with the support of your sales and account teams. Check. You’ve obtained signed NDAs from these customers, and they’ve signed yours. Check. Now, what does this process involve?
Over time, our Preview Release process has matured, and we’ve made some tweaks and modifications along the way. At Rockwell, our Preview Release process comprises the following steps, as shown in Figure 3.
- Working with the product team
- Creating the materials
- Reviewing the materials with stakeholders
- Scheduling participants
- Facilitating sessions
- Collecting survey feedback
- Analyzing the results in Dovetail
- Presenting the results
- Entering the data into Jira
Step 1: Working with the Product Team
We begin each Preview Release at the end of a Program Increment (PI), which is an agile framework that is useful in large-scale software development. As a consequence, we conduct a Preview Release roughly every three months. Working with our product teams, which comprise product managers, UX designers, and developers, we kick off a Preview Release process by endeavoring to understand the key learning objectives from the previous PI. These are usually the top areas of focus for development, and we want to better understand users’ opinions so we can evaluate the usability of the product’s features.
Step 2: Creating the Materials
Once we’ve identified the learning objectives, we create our materials for participants. These include a hands-on lab document, a task-based study script, a survey, and if necessary, any interview scripts or focus-group documentation. The hands-on lab document describes, step by step, what the features are and how to use them and provides questions that the researchers want the participants to think about while using the new features. Researchers use the task-based study script during their usability study, following the five to seven key tasks that participants need to complete. It also provides a convenient medium for listing follow-up questions or topics to probe more deeply. The survey includes questions about the product’s value and ease of use and the user’s productivity and leaves plenty of room for comments that participants might want to share.
Step 3: Reviewing the Materials with Stakeholders
Once we’ve created the materials, we work with the product team to make sure that the questions we are asking can provide the answers they need regarding our key learning objectives. At this point, we typically conduct a dry run of our task-based study with an internal participant to make sure that the research runs smoothly when we are working with our customers.
Step 4: Scheduling Participants
Next, we schedule time with each of the participants. Because participants are getting access to our build, which is not generally available, we must make sure that the environment is set up properly so they can use the build during a specific timeframe. We typically use Calendly and allow participants to schedule a four-hour time block during a two-to-three-week timespan. This gives participants plenty of time to go through the hands-on lab, answer the survey questions, and ask us any questions via email, as necessary.
Step 5: Facilitating Sessions
Once we’ve scheduled all the sessions, we select about ten participants for each timeslot, then walk them through the task-based study for the first hour of their scheduled session, before giving them the rest of the documentation and the hands-on lab. This lets us gather some information from participants without the step-by-step guide biasing them regarding how it should work.
Step 6: Analyzing the Results in Dovetail
Once all participants have completed their surveys, and we’ve completed our task-based studies, we analyze the results. Analysis typically takes a week or two because of the volume of responses we get. We use Dovetail to collect all our recordings, transcripts, notes, and other information. In Dovetail, we can tag our insights, create reports, and share our findings among our teams.
Step 7: Presenting the Results
Because the findings from our Preview Release program are so important to our teams, we typically plan a few different presentations for sharing these results. For teams who are directly responsible for the features we’ve tested, we conduct an in-depth walkthrough of the results in Dovetail. This includes our showing videos of the participants’ tasks, the overall scoring, quotations from participants, and more. We also present our high-level findings to outside stakeholders to help them understand what customers are saying and what things we’re focusing on within the product. Finally, because leadership is also very interested in the results from these Preview Releases, we typically schedule a high-level readout with them as well, presenting the overall scoring, the areas that are working well, and those that need improvement.
Step 8: Entering Data into Jira
Once all of these groups understand the results, we work with the product team to prioritize the findings. Then we enter the prioritized findings into Jira as tickets in the backlog. This enables each team to work on the priorities in the proper order, beginning with the fixes and improvements that would help make the next Preview Release and Program Increment a success.
Our Preview Release program not only fosters great customer engagement and enables us to improve the products but positively impacts our product teams as well. Product teams find the information that we gather to be extremely valuable, and they are starting to better understand the need for UX research even earlier in the product-development lifecycle. We are also seeing more requests for our regular research programs—a good problem to have! All of this adds up to creating better products, so our users and customers ultimately benefit.
We hope we’ve provided some inspiration that could help you address challenges you may have encountered in your research or design activities. Keep in mind that, if certain more conventional methods aren’t working out for you, you could also create your own approaches. As UX researchers, the most important part of our job is understanding our users. If there are better ways to engage with them outside our typical usability studies and surveys, it’s important to try them out!
Note—Katie Groh presented on this topic at UXRConf 2022. Check out the presentation video for a more in-depth look at how this method has benefited teams at Rockwell!