Home · Blog · Split testing a chatbot: which text brings more requests

Split testing a chatbot: which text brings more requests

November 5, 2026 · Command CHBT · Flows, Analytics

Arguments like “which greeting is better” and “do we need a gift for the phone number” can go on for weeks, or they can be settled in a week on real people. CHBT has Random branch for this: the bot splits the flow between variants by itself, and tags mark who saw which variant. Here is how to build an A/B test without a developer, what is worth testing and how to read the result honestly.

What an A/B test is and why a bot needs one

An A/B test compares two variants on randomly split groups of people. Half see variant A, half see variant B, everything else is the same. If B gets more requests, the difference is due to the variants, not the weather, the day of the week or advertising.

In a bot this is especially convenient: every flow step is a separate message, and it is easy to swap. Small changes often make a noticeable difference: one button instead of three, a gift for the phone number, a question about the task before the price. The test shows this in numbers, not feelings.

How the bot splits the flow: Random branch

In the Action block choose Random branch. In the Comma-separated outputs field list the branch names, for example a, b. The block gets outputs with these names, and, as the editor hint says, the customer will follow a randomly selected output.

The variants come up with equal probability: with two outputs — roughly half and half, with three — a third each.
The choice is made every time the person passes through the block. If the customer returns to this step again, a different branch may come up. So put Random branch where the person passes once: at the start of the flow or before the key step.
The action is available both in the Steps editor and on the Canvas.

Marking branches with tags

Random branch itself remembers nothing — to compare the variants later, each branch must be marked. Right after the output put Action → Add a tag:

in branch a — the tag test-greeting-a;
in branch b — the tag test-greeting-b.

At the goal step — where the customer left a phone, paid or booked — put a Condition on the variant tag and a separate result tag in each branch: test-greeting-a-request and test-greeting-b-request. Then a filter by a single tag immediately answers “how many people from variant A reached the goal”.

Name tags with the date or test number: in a month you will launch the next test, and old marks will not mix with new ones.

How to count the result

Open Contacts and filter by the test-greeting-a tag — this is the number of people who saw variant A.
Filter by test-greeting-a-request — how many of them reached the goal.
Repeat for variant B.
A variant’s conversion is those who reached the goal divided by everyone who saw the variant.

The filter in Contacts works by one tag, which is why each variant needs its own result tag. Contacts with a tag can be downloaded as CSV if you want to calculate in a spreadsheet.

Example: 212 people saw variant A and 23 left a phone — 10.8%. 205 saw variant B and 37 left a phone — 18%. The difference is almost twofold, and at these volumes it is no longer chance.

How to read the bot funnel as a whole is covered in the article on chatbot metrics.

What is worth testing

The first message. A short “Hello! How can we help?” with buttons versus a detailed greeting with a benefit. The first screen decides how many people start a conversation at all.
When to ask for the phone. Right away or after a selection, a quote, an answer to a question.
The reason to leave a contact. “Leave your phone” versus “We will send the quote to your messenger” or a gift for the contact.
The number of buttons. Three options versus six. More choice is not always better.
Price right away or after qualification. For expensive services the difference can be large in either direction.
Quiz length. Five questions versus three before the result.

Test what stands on the way to the goal. The colour of the emoji in the goodbye does not affect requests — it is a pity to spend a week of traffic on it.

How long to wait and how many people you need

The main mistake is stopping the test after a day, when one variant has 6 requests and the other 3. At such numbers the difference is often random. Guidelines that work in practice:

at least 100–200 people in each branch and at least 20–30 goal actions in total;
at least a full week — weekdays and weekends have different audiences;
the smaller the difference between variants, the more people you need to see it.

If traffic is low — 20 new conversations a week — test only big changes, where the difference can be several times. Small edits on a small flow cannot be measured honestly.

Rules of a clean test

One change at a time. If variant B has a different text, a different button and a gift, you will not know what worked.
The same continuation. After the differing step the branches should merge into a common flow — then the difference at the end is explained only by the tested step.
Do not change the flow during the test. Editing a text mid-week resets the comparison.
Do not run ads for only one variant. The bot splits the flow, not you — traffic sources should reach both branches equally.
Write the hypothesis down in advance. “A gift for the phone will raise the share of contacts from 10 to 15%”. That way the result is not adjusted to expectations.

Example: testing the greeting in a booking bot

Start → Action → Random branch, outputs a, b.
Branch a: tag t1-a → Message “Hello! We will book you an appointment in a minute” with service buttons.
Branch b: tag t1-b → Message “Hello! There are free slots this week on Tuesday and Thursday. Which service do you need?” with the same buttons.
Both branches merge into a common flow: choosing a time, Collect data with the phone, then a Condition on the t1-a tag and the tags t1-a-booking or t1-b-booking.
Two weeks later — Contacts: compare the number of contacts with t1-a and t1-a-booking, then with t1-b and t1-b-booking.

After the test keep the winning variant and remove Random branch: otherwise half of the people will keep seeing the losing text.

Common mistakes

Forgot the tags. Without marks, after the test it is impossible to say who saw what.
A branch on a repeating step. If the block is in a menu the person returns to, they will see both variants — and the comparison loses its meaning.
Drew conclusions too early. Wait at least a week and a few dozen goal actions.
Did not remove the test. Found the winner — switch everyone to it.
Checked only in the test chat. Check flow shows one random branch per pass — go through several times to see both.

Hypothesis ideas for different businesses

Salon, clinic: show free slots right away or first ask about the service.
Online school: a free lesson for an email or an immediate course offer with a discount.
Online store: a catalogue with buttons or a three-question selection quiz.
B2B services: “Leave your phone” or “Describe the task and we will send an estimate”.
Giveaways and promotions: the entry conditions in one message or step by step.

A good hypothesis sounds like “if …, then the share of … will grow, because …”. The “because” later helps you understand why a variant won or lost and come up with the next test.

How to record the results

It is worth writing down the result of each test in one line, so that in half a year you do not repeat what has already been tested: the date, the hypothesis, how many people were in each branch, conversion of A and B, the decision. For example: “October. Gift for the phone. A — 212 people, 10.8%; B — 205 people, 18%. Kept B”.

Even a losing test is useful: it tells you that this lever does not work for your audience and saves time on the next edits.

Chatbot metrics Flow editor CRM and contacts Collecting contacts in a chatbot