Day Five – Analyse, benchmark, adjust, validate again
Results day! Time to get excited. Remember though that this is just the first round of what could be many rounds of agile testing over the coming weeks. It’s designed to drive focus, efficiency and purpose into your work. This first round of testing will not tell you the 4 magic products that are going to transform the market… what it will do is shape your thinking, remove some distractions (concepts that just haven’t resonated), and allow you to learn through the results and move forward with momentum.
Action – Export all of the results from the testing, concepts & benchmarks together.
The first thing to do now is a simple Excel calculation to work out the average of all the benchmarks used. One caveat – we’ve made the assumption throughout this blog series that you’re working in a single category. If you’re doing work that straddles multiple categories, then take an average of the benchmarks from each individual category – don’t blend them into one as it’ll be misleading.
So you’ve got your average benchmark score – let’s say it’s 74% purchase intent. So 74 out of every 100 people asked said they would buy the product. 74% becomes your target, your hurdle rate, your acceptability score. Call it whatever you want, but this is now what you’re gunning for in this round of testing.
In simple terms, any product concepts that scored on or above the hurdle rate should move forwards. Nice and simple. Anything that scored below the hurdle rate, should be parked or filed for a later project. We want to use the data in an agile project to make quick, empirical decisions about what things to focus on. And this green / red light approach to the hurdle rate is the most simple way to achieve that.
Now as always there is some nuance to this. What if you get a product concept that scores 73%, with the acceptability score of 74%? That is only 1% different right, so how can we bin it? In these early rounds it’s fair to agree a tolerance level of between 1-5% so that you’re not making life too difficult for yourself and parking concepts that could be iterated on further towards success. The key to an Agile project is to learn and iterate from each round of data. So if you have a concept that is just 2% under the acceptability score, it’s reasonable to progress it as long as you change it and tweak it in the next round of testing to make it better. But if it underperforms again then it’s maybe time to park it. Set your tolerance level upfront and try to stick to it.
And that is Agile in action. You’ve removed products that don’t hit the benchmark from the process – quickly and empirically parking concepts that just didn’t resonate. This frees you up to concentrate on the remaining products – and we go again into another round of testing. We want you to take the successful concepts, think about what can make them stronger, look for patterns in the products that have performed well, and try to improve them. You would now test again – the same process for testing with benchmarks and images into week two, with a view to driving further refining the product selection.
We’re looking to end the agile process with a small number of product concepts that have tested consistently well throughout, that can now be put into a physical development process which is where the real costs begin to mount.
Good luck – we’d love to hear how you get on!