[Main Index] [Next entry: "Are you losing visitors?"]

08/01/2003 Entry: "Test-and-Track.com Split run test"

I have installed Duncan Carver's free script the "Scientific Internet Marketing Assistant" available from http://www.marketing-strategy.org/

It was very easy to install and the admin area is very user friendly. All the functionality can be accessed through clear navigation links and I found setting up a new campaign to be a straightforward process.

I am testing two headlines and results so far are very interesting. Let's call the two pages that I am testing: "Headline 1" and "Headline 2". Headline 1 is achieving a conversion rate of 5.38% and Headline 2 a staggering 0.0%. I think you will agree that is a pretty conclusive result after 38 actions.

I have set it up to run to 100 actions to ensure that the result is statistically valid :)

Posted by Oran Kangas @ 09/18/2003 06:07 PM GMT

Hi John,

"I have set it up to run to 100 actions to ensure that the result is statistically valid"

I have been searching in vain for months for the PRECISE formula to determine the EXACT sample size for such tests. Everyone seems to use a "rule of thumb" measure, but there is no agreement on whose thumb to use. :->

This is crucially important.

Closing a test prematurely can result in a false positive -- yielding a bad control -- losing money.

On the other side, having a sample size too large wastes time -- losing money.

So, John, have you found that magic formula? If not, whose thumb shall we use?

Best wishes,
Oran Kangas

Posted by Oran Kangas @ 09/18/2003 05:37 PM GMT