Running test too short (peeking bias)
The mistake
Biggest mistake: checking results after one week when test was planned for two weeks.
Why it is a mistake
One week gives four hundred conversions per variation. Two weeks gives eight hundred. With four hundred, random noise is higher. You see differences that disappear with more data.
Example: week one shows treatment nine percent conversion. Control eight percent. One percent win looks real. You peek and get excited. Week two: treatment drops to eight point five percent. Control stays at eight percent. Half percent win. Still statistically significant but much smaller than week one showed.
If you stopped at week one, you would have over-estimated treatment benefit by fifty percent.
How to avoid
Commit to duration before starting test. Two weeks standard. Write it down. Do not peek. Set calendar reminder for end date.
Testing multiple changes at once
The mistake
You change headline. You change button color. You change button text. You change form fields. Four changes simultaneously.
Why it is a mistake
Treatment converts at twelve percent. Control at ten percent. Two percent win. You implement all four.
Month later: conversion drops to nine percent. Something broke. But which change. Headline. Button. Form. You changed four things and do not know which one hurt. You cannot learn which change was beneficial and which was harmful.
How to avoid
One change only. Change headline only. Test it. Implement if it wins. Then test button color. Then test form. One variable at a time.
Tracking errors and broken tests
The mistake
Test runs for two weeks. Results show treatment winning by five percent. You implement. Month later you realize tracking was broken. Conversion pixel did not fire for ninety percent of visitors.
Why it is a mistake
Results are garbage. Implementation was based on bad data. You spent engineering resources implementing something based on invalid results.
How to avoid
Spot-check test setup. Run a simple manual flow. Does event fire. Check tracking logs. Verify pixel is firing. Audit tracking before declaring winner.
Inadequate sample size
The mistake
You run test with only five hundred total conversions. Two hundred fifty per variation. With sample this small, random noise is huge. Five percent differences are common. Ten percent differences are not shocking.
You see treatment five percent higher than control. No statistical significance. You declare test inconclusive.
Why it is a mistake
You were under-powered from start. You should have known test needed two weeks, not one week. Under-powered tests waste time and produce inconclusive results.
How to avoid
Calculate sample size before starting. How many conversions do you need. How long will test take to get there. If test takes three months, consider testing something else with faster results.
Not controlling for external factors
The mistake
You run test Tuesday through Thursday. Monday results before test. Friday results after test. You compare Friday to Monday.
But Friday is different traffic than Monday. Friday is weekend casual browsing. Monday is weekday work browsing. Different audiences. Different conversion rates. You attribute difference to test when it is really composition difference.
Why it is a mistake
You implemented change based on traffic composition difference, not treatment effectiveness. When new traffic composition arrives, results do not hold.
How to avoid
Run test long enough to balance traffic composition. Two weeks includes weekdays and weekends. One week might be skewed.
Changing test hypothesis mid-test
The mistake
You start test with hypothesis: shorter form increases submissions.
Halfway through, you get new data from support. Customers want more field options. You change hypothesis: more fields might actually help. You now have test designed for one hypothesis being analyzed as different hypothesis.
Why it is a mistake
Confuses results. You cannot tell which hypothesis the test was actually testing. You might implement based on confused interpretation.
How to avoid
Write hypothesis before starting test. Commit to it. If you get new information and want to test different hypothesis, finish current test. Then run new test.
Not implementing winners
The mistake
Test finishes. Treatment wins. You say "we will implement later." Later never comes. Test created no value. You wasted time.
Why it is a mistake
Winners only create value if implemented. Testing program only compounds if you implement winners consistently. One unimplemented winner kills momentum and team belief in testing.
How to avoid
Commit to implementation before running test. If treatment wins, implement within one week. If control wins, revert immediately. Do not let winners languish.
Not segmenting results
The mistake
Overall results show treatment wins. You implement globally.
But segmenting shows: desktop treatment wins. Mobile treatment loses. International treatment has no impact. Global implementation hurts mobile and international visitors.
Why it is a mistake
You optimized for overall average but hurt specific segments. Mobile traffic might grow faster than desktop. By implementing treatment, you hurt future growth segment.
How to avoid
Always check segment results before declaring global winner. Winner might not be universal. Implement for segments where it wins. Skip for segments where it does not.
Confusing winner with causation direction
The mistake
Test shows treatment beats control. You assume treatment is better. But what if control is just broken. Control button does not work. Everyone is confused. Treatment button works fine. Treatment wins because control is broken, not because treatment is inherently better.
Why it is a mistake
You learn wrong lesson. You think button style wins when actually you learned "working button beats broken button." Different implications for future testing.
How to avoid
Understand why treatment won. Was it because treatment is better or control is worse. Different implications.
Repeating same test looking for different result
The mistake
Test one: treatment loses to control. You try again with slight variation. Test two: same result. Control wins again. You try again. Test three: same pattern. Control keeps winning.
You are testing same thing repeatedly and getting same result.
Why it is a mistake
You are wasting testing budget on approach that does not work. Better to pivot to different approach than to keep testing same losing strategy.
How to avoid
Stop testing same thing. Different strategy needed. Current approach is not working. Pivot to different treatment.
Not documenting results and learning
The mistake
Test finishes. Results show winner. You implement. Then forget about it.
Six months later: new team member asks "what did we learn about headlines." You have no documentation. Start from scratch.
Why it is a mistake
You lose institutional knowledge. Every test should teach you something. Without documentation, knowledge disappears when team members leave.
How to avoid
Document every test. Hypothesis. Results. Learning. Share with team. Build institutional knowledge.
Frequently asked questions
If I check test results once per week am I peeking?
Not if you commit to two-week duration and do not change behavior based on week-one results. You are monitoring but not stopping early. Checking is fine. Stopping early is peeking. Example: week one shows treatment nine percent, control eight percent. Treatment looks like winner. Do not implement yet. Commit to full two weeks. Week two might show treatment eight point eight percent, control eight percent. Smaller difference emerges. If you stopped at week one, you would have overestimated benefit by fifty percent. Commit to duration, check weekly if you want, but do not act on preliminary results. With WEMASY's website builder, you can set this up directly on your website.
Can I test two similar things simultaneously (different headlines tested in different regions)?
Yes if regions are truly separate and you are not comparing them. Test headline A versus B on US traffic. Test headline C versus D on EU traffic. Separate tests. Parallel testing is fine. But do not compare A versus C across regions because composition is different. Regional audiences convert at different rates. US might be twelve percent baseline. EU might be eight percent baseline. Headline A wins in US region. Headline C wins in EU region. You cannot conclude headline A is universally better because regions have different baselines. Test in parallel but analyze separately.
If results show no significant difference, should I assume treatments are identical?
No. Assume difference is unclear. Could be identical. Could be different but needed more data to detect. Example: control five hundred conversions. Treatment five hundred five conversions. One percent difference. Eighty-eight percent confidence. Not significant. You cannot conclude they are identical. You can only conclude difference is unclear. Options: run test longer to get more data. Or if treatment is easier to implement than control, switch to treatment because at worst they are identical, at best treatment is better. If control is easier, keep control.
What if my test shows stat sig result but intuitively it seems wrong?
Trust statistics not intuition. Math is smarter than intuition. If stats say result is real (ninety-five percent confidence), it probably is. Your intuition might be wrong. Your intuition is based on limited experience. Statistics are based on thousand visitor data points. Implement based on statistics. Example: you think red button performs better than blue button. Test shows blue button wins significantly. Your intuition was wrong. Implement blue button. Trusting your intuition and ignoring test data would have implemented red button and hurt conversion.
Can I combine multiple small insignificant results into one big significant result?
No. Each test stands alone. You cannot combine inconclusive tests into one conclusive result. Test one shows eighty-five percent confidence treatment wins. Test two shows eighty percent confidence same treatment wins. Together ninety-five percent confidence? No. That is not how statistics work. Each test is independent. Combining them is statistical nonsense. Run proper-sized test with proper power from start. Either you have significance or you do not.
If test shows strong result for segment but weak for overall, which do I implement?
Implement for segment only if strong result is statistically significant for segment. If segment size is small (ten percent of traffic), stat sig threshold is harder to reach. Could be false positive on segment. Check stat sig for segment specifically, not just overall. Example: overall test shows treatment nine percent, control eight point nine percent. Barely significant. But desktop segment shows treatment ten percent, control eight percent. Two percent win in desktop. Strong and significant for desktop. Mobile segment shows treatment eight percent, control eight percent. No difference. Implement for desktop only. Do not implement for mobile. Different segments have different optimal treatments.