Step one: write hypothesis with specifics
Vague hypothesis fails
Not: changing checkout improves conversion.
Specific hypothesis wins
Better: we think simplifying checkout from five-step process to one-page form will increase purchase completion by at least three percent because customers abandon when form feels long.
Hypothesis should state. What you are changing. Why you think it will help. What metric you expect to improve. By how much.
Vague hypothesis leads to vague results. Specific hypothesis is testable.
Step two: determine sample size before starting
The math matters
How many visitors do you need to see real difference.
Formula. You need minimum five hundred conversions per variation. So one thousand total if testing two versions.
Calculate your test duration
If you convert at two percent and get two thousand visitors per month, you get forty conversions per month. Reaching five hundred conversions takes twelve months. Test will not finish in time.
If you convert at ten percent and get five thousand visitors per month, you get five hundred conversions per month. Reaching one thousand takes two months. Test finishes quickly.
Know your timeline upfront
Low conversion rate websites need longer tests. High conversion websites finish faster.
Use sample size calculator. Input your conversion rate and traffic. Calculator tells you how long test takes.
If test takes longer than three months, consider. Is the change worth waiting three months. Or should we test something else that finishes faster.
Step three: pick test traffic allocation
Fifty-fifty is standard
Allocate fifty percent to control. Fifty percent to treatment.
Cautious allocation slows results
Some people allocate ninety-ten. Ninety percent to current good version, ten percent to new version they are unsure about. This is cautious but slows test.
Fifty-fifty is standard. Fastest results.
Step four: determine test duration before running
Commit to timeline
One week minimum. Two weeks standard. One month for very high-traffic sites.
Do not decide duration during test. Decide before. Write it down. Commit to it.
Early peeking ruins tests
Early peeking ruins tests. You check results after one week. Looks like treatment is winning. You stop test. You implement. Later you realize results were not real. You implemented something that did not actually work.
Run full duration.
Step five: make treatment change
One change only
One change only. Not multiple changes.
If testing checkout simplification, simplify form. Do not change color, copy, layout at same time. One variable at a time.
Make it obvious
Change should be obvious. Visitors should notice difference. If change is so subtle visitors do not see it, test will not work.
Step six: set up test in tool
Choose your platform
Use testing platform. . . .
Install and configure
Connect to website. Install tracking code. This usually one line of code or plugin install.
Create versions and metrics
Create control version. This is current version.
Create treatment version. Implement your change.
Set traffic split. Fifty percent each.
Set duration. Two weeks or however long you determined.
Set success metric. What are you measuring. Checkout completion. Form submission. Page scroll.
Step seven: let it run without peeking
Resist the urge
This is hard. Resist urge to check results.
Results after few days are meaningless. Random variation is huge with small sample. You will see results that disappear later.
Set a calendar reminder
Check results exactly on end date. Not before.
Set calendar reminder for test end date. Check then.
Step eight: analyze results correctly
Look for statistical significance
Test finishes. You look at results. Treatment converted at eleven percent. Control at ten percent. Treatment is better. Right.
But was it statistically significant. Is this real or luck.
Look for confidence interval or p-value. Should be above ninety-five percent confidence or below zero point zero five p-value.
What statistical significance means
If statistical significance says yes, result is real. Implement treatment.
If statistical significance says no, result unclear. Could be luck. Could be real but needed more time. Run test longer or try different change.
Step nine: implement and document
If treatment wins
If treatment wins, implement permanently.
Document what you learned. Why you thought it would work. What happened. What you will test next.
Build playbook. Over time, patterns emerge. You learn what works for your audience.
If control wins
If control wins, revert change.
Document why control won. Maybe your hypothesis was wrong. Maybe change was not noticeable enough. Maybe audience did not like change.
Losing test is still learning.
Common first test mistakes
Running test too short
After one week you are impatient. You stop. Results are not real yet.
Testing multiple changes
You changed form and button and copy. Form wins but was it form or button. Now you do not know.
No significance requirement
Treatment is one percent better. Could be luck. You implement. Later it does not hold up.
Testing on everyone
You are nervous about change. You test on ninety percent current, ten percent new. Test takes forever.
Not committing to implementation
Test finishes. Treatment wins. You say we will implement later. You never do. Test created no value.
What separates winners from everyone else
Not smarter testing
Not smarter testing. Consistent testing.
The winning pattern
Winner companies run test. Results come in. They implement immediately. They run next test.
They commit to testing program. Not one-off tests. Ongoing testing. Month after month.
They build internal knowledge. After fifty tests they know their audience. Know what works. Know what does not. They move faster than competitors.
Compound growth
Test once, nothing happens. Test fifty times, everything changes.
Frequently asked questions
What if I discover the test is broken halfway through. Like tracking stopped working. Should I restart?
Yes. Broken test is useless. Restart fresh. If tracking is broken you do not have real data. This is why you check test occasionally (not obsessively but at least once). Catch breakage early. With WEMASY's website builder, you can set this up directly on your website.
Can I change my test hypothesis while it's running if new information comes up?
No. Write hypothesis before test starts. Stick with it. If hypothesis seems wrong mid-test, note it down. But finish test with original hypothesis. Run new test with new hypothesis later.
If treatment is clearly worse, can I stop the test early?
Only if it is breaking something. If treatment is causing errors stop. Otherwise no. Even if losing by large amount, run full duration. Learn why it lost. Use that learning.
What do I do if statistical significance says no but I think the result is real?
Run test again. Same change. But longer duration or with more traffic. If result is real it will show up again. If it does not, intuition was wrong and math was right.
How do I know which variation is control and which is treatment in the results?
Your testing tool will label them. Make sure you know which is which before starting. Take a screenshot. Control is version A. Treatment is version B. When results come in, reference your notes.
Do I need to tell my boss I'm running tests or just show results?
Tell her. She will understand testing takes time. Set expectations. Testing takes two weeks. If treatment wins we will implement. If not we will try something else. Transparency prevents surprise when you do not implement change she expected.