Experiment ideas: Tactics and templates for consistent, high quality ideas

I’m Becca Bruggman, Optimizely’s Experimentation Program Manager. My job is to make sure we are “drinking our own champagne” and run a best-in-class experimentation program. 

This is the fourth installment of a six-part series designed to help you run a best-in-class experimentation program. In the series we are covering everything you need to build your program, develop it and have it running efficiently as well as create a well oiled machine and make it visible.

In a recent blog post, we reviewed the documentation for building and QA’ing experiments across your stack and organization. With that in place, let’s dig into how you can keep your experiment ideation engine running at full speed.

Having a steady stream of ideas from across the organization can be helpful to ensure you have an extensive backlog to pull from, keeping you from being flat-footed around what experiments to prioritize and what to build next. Additionally, this helps get more people contributing to experimentation, which is good for your program, your product and your company. Even if someone doesn’t have the ability to build an experiment, they likely have a unique perspective from their own utilization of the product or interactions with customers that can provide great, new ideas to explore. Also, the more you launch experiments with ideas from different parts of your organization, the more teams you will engage in experimentation as they will be excited to see the results of ideas they submitted – more on that in future posts!

So, how do you get all these ideas? A few tactics I’ve seen work well:

Passive/Scalable:

• • Set-up and socialize a physical box for adding in experiment ideas
  • Set-up and socialize a google form for submitting experiment ideas (example below)
  • Add everyone to your Program Management instance as a Teammate, which allows them to submit ideas directly
  • Socialize your roadmap spreadsheet and give everyone comment access
  • Share an Experiment Idea Google Doc Template that can be filled out and sent to the core team
  • Designated Slack channel for submitting ideas

Active/Less Scalable:

• • Run Ideation Sessions with Teams
  • Host an Experiment Idea “Hack-a-thon”
  • Sit with individuals to understand what they are working on for the quarter and how experimentation can impact their key metrics and/or launches
  • Conduct User Journey Mapping

Optimizely’s form to submit experiment ideas

At Optimizely, running an ideation session or user journey mapping session with a team is a tactic I’ve seen be especially helpful to get a large amount of experiment ideas in a short period of time. I’ll outline both processes below to share how I approach these and their impact as well as include templates you can leverage to run your own. My awesome colleague Alek also wrote a great blog post outlining step-by-step how to run a session with your design team.

I usually approach ideation sessions by bringing together a team or working group and have them focus on one specific page on the website, especially when sourcing ideas for our marketing site. On product, we might instead take on the product area that the team focuses on or an upcoming feature launch. The important piece is to pick something that the team will have a lot of opinions on and areas they would want to experiment on to improve.

In terms of structure, the ideation sessions progress with the following stages:

First we will pick a page and then approach it as a user, writing on post-its all the problems we see with the page. Make sure to have these post-its be focused as problems, such as “Why can’t I click on this image?” or “Why are there so many pop-ups?”, as we will move to solutions in the next step. Allow ~10 minutes for this section.

From there, have everyone read their problems out loud to the group. You’ll usually start to hear some themes and discussions will happen naturally.

Next, move into clustering the problems into themes within your group. You can see an example cluster below from an ideation session I ran with our Demand Generation team:

Cluster of user problems from an ideation session

Once you have your clustered themes, which usually takes 20-30 minutes including reading all the post-its, pick one or two for the group to work on. Now you can move on to solutions related to the problem themes. You can think of these as the themes you’d want to create a solution for and what you might change within an experiment. Give the group ~10 minutes for their solutions.

From there, read the solutions out loud and see what focus areas arise. You will likely start seeing some ideas for experimentation from going through them with the group.

Next, you can, either with the team or asynchronously, put the ideas into your experimentation backlog. I usually do this part myself and use tags in Program Management to make the ideas from each session easy to identify:

Example Tag in Program Management for Ideation Session Ideas

I’ve created a template for capturing ideas during an ideation session and scoring them to help you facilitate an ideation session for your own program.

Once ideas get prioritized and launched, I love being able to share back with those teams that their ideas helped the program and show the impact! Also, who doesn’t love emoji voting on which variation will win? You can see an example of this from a session I ran with our Sales Development team for the plans page: 

Sharing screenshots of experiment variations from an idea the Sales Development Reps had

Sharing an experiment that got prioritized from the SDRs with a poll to vote on the winning copy

Another tactic I’ve found helpful is to do user journey mapping. This usually looks like finding the largest white board in your office and then mapping out how a user finds your product and walking through all the steps a user takes to see value from your product. We did this a few months back when we launched our Rollouts product and it was really helpful to see all the inputs, drop-off points and touch-points across the entire user journey. This will help inform areas that could be ripe for optimization and experimentation.

User Journey Mapping for Optimizely Rollouts

One final point, when running a Center of Excellence style of experimentation program, it can be especially helpful to meet regularly with the key stakeholders you need to influence to prioritize experiments. They will be the ones who prioritize building and launching experiments, so understanding what they are working on and what metrics they are focused on can go a long way towards influencing them to run experiments.

I see a Center of Excellence program structure most often within Product organizations, where the stakeholders that need to be influenced are individual Product Managers. They will prioritize building and running experiments against feature or bug work. Understanding how these Product Managers get recognized, promoted and measured will go a long way to influence getting them to run experiments that will move the needle for their product areas.

Once you have all your ideas, there are a few different ways to organize them. I personally love Program Management,. It is full featured with the ability to sort ideas by both product area and type, has scoring for streamlining prioritization, commenting built-in and connects to your experiments in Optimizely. However, if you’re just getting started, a spreadsheet can also be a good place to start and can include the baseline information about the experiments. You’ll find a roadmap template included in my Experimentation Program Toolkit as a starting point.

How are you tracking all your experiment ideas? What are your tactics for getting more people involved in experimentation? Comment below or tweet me @bexcitement.

See you in the next post on making all your awesome experimentation work visible!