Skip to content

A/B testing your WhatsApp campaigns without fooling yourself

Testing two messages is easy. Knowing whether one actually won is harder: small lists produce noise that looks like a result. Here is a method that is honest about sample size.

By The Wameed teamWameed POS

4 min read

In short: change one thing, send each version to an equal slice of the same audience, measureorders (or clicks if orders are rare), wait long enough, and treat a small difference on a small list as noise. Wameed supports A/B variants and can pick the winner automatically; you decide what is being tested and how big the slices must be to mean anything.

Most test “winners” on small lists are luck. The discipline below keeps you from rolling out a change that did nothing.

What to test

One variable per test. The common ones:

  • The opening line — a question versus a statement.
  • The offer — SAR 20 off versus 10% off.
  • The call to action — “اطلب الآن” versus “شاهد العرض”.
  • The image — product versus lifestyle.
  • The send time — afternoon versus evening.
  • The length — two lines versus five.

Do not test three things at once; if one version wins you will not know why.

How a test runs

  1. Form one audience of consenting customers.
  2. Write version A and version B differing in one thing.
  3. Send each to an equal, random slice of the audience — say 20% each.
  4. Wait long enough for most responses — not minutes; at least a day for orders.
  5. Compare on your chosen measure.
  6. Send the winner to the remaining 60%.
  7. Write down the lesson.

Wameed automates the slicing and, if you allow it, the selection of the winner.

Choose the right measure

  • Orders or revenue attributed to the message — the real goal.
  • Link clicks — a good proxy when orders are too few.
  • Replies — useful for conversational offers.
  • Not read rate alone: a message can be read and ignored. Reads also tell you more about the subject line and the sender name than about the offer.

The sample-size problem

Suppose each version goes to 200 people.

  • Version A: 12 orders (6.0%).
  • Version B: 16 orders (8.0%).

B looks 33% better. But a difference of four orders on 200 people each is well within what chance produces. If you ran the same message twice you might see a gap this large.Do not call a winner yet.

Rules of thumb:

  • A few hundred per version is a minimum for a difference of several points to start to mean something.
  • The smaller the expected difference, the bigger the audience you need.
  • If your list is small,run the test across several sends and add the results rather than trusting one.
  • Treat a difference of two or three conversions as noise.

If a result is unclear, say so, and use the version you prefer on grounds of brand and clarity.

Avoid the traps

  • Peeking and stopping early when your favourite is ahead.
  • Unequal slices (the best customers in A).
  • Testing on a tiny list and treating the result as a law.
  • Different send times that confuse the result.
  • Changing the offer and the wording together.
  • Never repeating a surprising result.

What good learning looks like

A short log:

DateAudienceChangedABResultDecision
5 MarLapsed 90d (800)Opening lineQuestionStatement9.1% vs 8.6%, noiseKeep question
2 AprRepeat buyers (1,200)OfferSAR 2010%7.9% vs 6.2%, retestRetest
30 AprRepeat buyers (1,200)OfferSAR 2010%8.4% vs 6.0%SAR 20

A log tells you which lessons repeat.

Ideas worth testing in Saudi Arabia

  • A fixed amount versus a percentage for baskets of different sizes.
  • Arabic-only versus bilingual copy.
  • A named sender (“سارة من متجر X”) versus the brand.
  • Evening after Iftar versus afternoon during Ramadan. Seequiet hours.
  • A deadline (“حتى الخميس”) versus none.

Remember the costs

Each test message costs what Meta charges. A test on 400 people costs little; a mistake sent to 20,000 costs a lot. That is the strongest argument for testing first. Seemessage pricing.

Frequently asked questions

Does Wameed pick the winner automatically?

It supports A/B variants and automatic selection of the winner. Decide the measure and the slice size yourself.

How long should I wait?

Long enough for most responses. For orders, at least a day; for clicks, a few hours.

Can I test on a tiny list?

You can, but expect noise. Combine several sends before trusting a result.

What should I test first?

The offer and the opening line usually move results most.

  • #اختبار A/B
  • #اختبار حملات واتساب
  • #تحسين الحملات
  • #اختبار الرسائل
  • #A/B testing
  • #WhatsApp campaign test
  • #message variants

Share this article

Ask about your own shop

Thirty minutes on your products, your tax setup and your hardware — not a slide deck.

Keep reading