Multivariate Testing with Google Website Optimizer: the Final Word?

13 Comments

Mar

24

2010

by MrOptimization in Vendors FAQs

Choosing Multivariate Testing and Targeting Vendors - MrOptimization.com

Katie B. asks MrOptimization:

Hi Mr.O!

We've been looking at MVT vendors for months. Our team narrowed it down to one that we think will work best for us.

Then our CMO goes to a seminar where a Google Website Optimizer consultant presented and now he's all Googly-eyed.

What's the final word on Google Optimizer for Multivariate Testing and Targeting?

MrOptimization responds:

We've been meaning to sit down and write up the final word on Google Website Optimizer for a long time.

This question comes up at least three times a week and we've seen it play out from all angles for years.

Thanks for giving us the kick we needed, Katie!

Take the time to read through this and you'll have the final word!

In the beginning...

Up until recently, Multivariate Testing and Targeting was the exclusive domain of the dominant species on the Internet, those who've evolved to the upper limits of the digital optimization maturity model. These are the big, profitable sites like Amazon who don't make a habit of telling everyone how they keep their digital edge.

Multivariate Testing Power to the People!

Then Google released Website Optimizer in 2007. Like Mr. Ford before him, Google's Mr. Schmidt wants to drag formerly exclusive technologies such as Multivariate Testing down from the rarefied air so everyone can use them.

In Mr. Schmidt's words, he wants Google's products "to serve all potential users of targeted advertising."

Yay Eric! Google gave it the nod and now everyone's started thinking about Multivariate Testing.

BUT, like most mass market versions of formerly high-end products, Google Website Optimizer is not really "free" or enterprise-grade, so you'll need to consider a few points carefully before you make it the center of your optimization strategy.

Page: 1234

13 comments

Trent
Great information
Wed March 24, 2010   06:14:43
This will help us show why "free" isn't always free.

Very thorough and helpful. Thank you!
MVTUK
My boss will love this
Wed March 24, 2010   06:58:14
Shows we made the right decision to go with the "costly" option.

Great work.
Chris
Re: Multivariate Testing with Google Website Optimizer: the Final Word?
Wed March 24, 2010   07:11:32
Sounds like you hit it right on the head! More time versus more money.

There is another reason why you may NOT want to use it though:

I run a small website, and was considering GWO just to see if a different design could elicit more clickthroughs for my Adwords, actually.. (not an online store)... The problem, however, is that I run a relatively small site that has a large core group of users. There are many new users every day, but the core group is used to the site as it is, and if they get a random landing point each time they visit, there is a chance that I cause confusion amongst my userbase if they land on the same page with multiple design choices.. ISo, instead of causing confusion, I just chose to, in the end, use various template designs and change them in and out for a short period of time to see what worked best. This, of course, required closer monitoring of all of the traffic myself, but again.. small site.. Way to go Mr. O....


In an enterprise caliber site, I would probably have to say... you are right, free isn't free...you get what you pay for.. I don't know how often this rings true in the IT industry about almost everything related to IT...
MVTMatt
Chris
Wed March 24, 2010   08:04:11
But done properly (perhaps not with GWO, however) multivariate testing can produce, in near real time, results that allow you to make on the fly changes increasing your customerbase/clickthrough almost instantly.
Dave Morgan
Accurate & entertaining overview of GWO
Thu March 25, 2010   22:21:55
I thought this post was helpful, accurate, and pretty funny at times. Kudos for achieving all that ... with no typos!

In addition to the cast of characters you mentioned (Omniture, Optimost, etc.) another product you ought to add to your list of (not-free) MVT tools is SiteSpect, www.sitespect.com

In particular, it's the only non-intrusive testing and targeting solution -- meaning there's no page tagging whatsoever, which seems to be one of your biggest gripes with GWO.

Full disclosure, I am employed by these guys. :)

cheers
DM
MrOptimization
SiteSpect peek
Fri March 26, 2010   06:21:01
Dave:

Thanks for the kudos. We did do a peek at SiteSpect a while back.

Please let us know if anything got missed or needs updating. Multivariate Testing with SiteSpect, the Magic Black Box
Antonio
Good to know!
Mon April 05, 2010   12:03:54
I'm glad someone finally did an in-depth review of Google Web Optimizer. As a website owner and operator, all this optimization stuff is a tad hard to sort out. This article really helped clear up where we should be focusing.

Thanks!
S. Spellman
Great article!
Mon April 05, 2010   12:07:17
This is an extremely informative article on the sometimes confusing subject of Multivariate Testing. I like how you essentially boiled it down to a trade-off that everyone would understand: time and money. I wholeheartedly agree with your analysis on Google Web Optimizer. Keep up the good work!
Matt Gershoff
Fractional
Mon April 26, 2010   18:47:48
Thanks for putting this up. this is really good.
In the section where you list the positive aspects of GWO you write "[f]ractional factorial methods let you infer conclusions for a larger set of variables than would be possible in a reasonable amount of time if you tried to test every possible combination with a live audience."
I think I know what you mean but it might be a bit misleading. It is my understanding that it is not really the fractional method that is allowing for the reduction in time but rather it is the independence assumption about the effects of the variables on the outcomes.
Here is a question for some of the Fractional/Taguchi practitioners , if the cost to generate a treatment is zero,is there still any benefit to collecting the data via a fractional factorial experiment? Or does Google have it right and all that matters is the back-end data analysis? My sense is that there isn't a benefit, but I would love to hear from the experts.
Cheers and thanks again!

Matt
Billy Shih
Fractional Factorial
Wed May 05, 2010   18:52:26
Disclose: I am an Optimization Analyst for Webtrends Optimize

The statement that GWO does Fractional Factorial is misleading. It can analyze using Fractional Factorial but it tests in a full factorial manner. Meaning you are showing all the experiments and then afterwards can look at the per-level effects. This doesn't really make much sense to do it in this way. True fractional factorial solutions test only a subset (and that subset is selected with purpose via mathematical arrays) and then do the analysis on that subset. That is the most accurate way to generate results from fractional factorial analysis.

@Matt Gershoff: The cost to generate a new experiment is zero, yes. But the cost to test a new experiment is more data. If you test 100 experiments and you need a sample size of 1000 people per experiment, by going with full factorial, you need an exponential amount of additional traffic in order to do a proper analysis.
Matt Gershoff
Fractional
Fri May 07, 2010   11:02:51
Thanks Billy. Would one need extra traffic if the data is going to be analyzed based on the independence assumption?

Assume we have three factors, each with two levels. We can either collect data based on all eight possible combinations or we can draw from an L4 array with four combinations. If we are randomly drawing from a uniform distribution wouldn't we have, on average, the same number of exposures to each level of the factors (roughly 50/50)?

And since we are drawing randomly, wouldn't our data tend to be balanced over the three factors either way we collect it? If the cost to generate each combination is zero, why bother setting up the array in the first place. I think this is what Google is driving at. Is this the wrong way to look at it?
Angela
Re: Multivariate Testing with Google Website Optimizer: the Final Word?
Mon May 31, 2010   09:22:18
Hmmm, that's an interesting take. I used to work in a successful e-commerce/e-publishing company where our testing was done using Google Website Optimizer and it worked quite well for us.
Jason
Re: Multivariate Testing with Google Website Optimizer: the Final Word?
Mon May 31, 2010   11:23:30
What would you say is the best paid testing program?

Add your comment