Skip to main content
An A/B test sends a share of your real calls to each of two or more versions, so you find out which one works instead of arguing about it. You set it up in your agent’s Version tab, and you can stop it at any time.
A/B testing needs versioning enabled on the agent. If there is no Version tab, it is not available on your account yet. Email support@brilo.ai.

​
Before you start

  • At least two published versions of the same agent. See Save and switch agent versions.
  • Enough call volume that a difference will show. Ten calls tells you nothing. A hundred starts to.
  • One thing you are testing, and one measure you will judge it by. Decide the measure before you start.
  • You will be working in your agent’s Version tab.

​
Steps

1

Create the version you want to test

Copy your live version, make one change, and publish it. One change, so that whatever the result is, you know what caused it. See Save and switch agent versions.
2

Decide what winning means, before you look

Pick the measure now. Bookings taken, calls that reached a person, average call length, Pulse Score, or an extraction field you already collect. Writing it down first stops you finding a metric afterwards that says what you hoped.
3

Open the Version tab and start a test

Select the versions you want to compare. You can run between two and four at once.
4

Set the traffic split

Give each version a percentage. They have to add up to 100, and Brilo will not start the test otherwise. An even split gets you an answer fastest, so 50 and 50 for two versions.If the new version is risky, start it at 20 and give the live one 80. You will wait longer for a result, and you risk fewer calls.
5

Start it and leave it alone

Start the test. It shows as Running, and calls are shared between the versions from that point.Leave it for a week, or until each version has taken enough calls to mean something. Checking on day one and reacting is the most common way to get a wrong answer.
6

Stop it and pick the winner

When you stop the test you get two choices: Just stop the test leaves everything as it was, and Select a winner to promote lets you name the version that won so you can deploy it.

​
You will know it worked when

The test shows as running, both versions are taking calls, and after a week you can point at one number that separates them.

​
What is worth testing

What is not worth testing: two changes at once, wording nobody will notice, and anything where you already know the answer. Fix the obvious problems first and test the genuine judgement calls.

​
How long to run it

Long enough that the difference is bigger than the noise. As a rule of thumb, wait until each version has taken at least fifty calls, and prefer a full week so you cover every day of the week. Monday callers and Saturday callers behave differently, and a test that ran Tuesday to Thursday can mislead you. Stop early only if a version is clearly failing. That is not a result, it is a fault, and there is no point serving it to more callers.

​
What else you can do here

​
If it did not work

The test will not start. Your percentages do not add up to 100. Fix the split and start again. You cannot select a version. It has to be published first. Drafts cannot take calls. See Save and switch agent versions. The two versions look identical in the results. Either the change was too small for callers to notice, which is a real and useful answer, or you have not taken enough calls yet. One version is obviously worse. Stop the test and promote the other one. You do not need statistical confidence to act on an agent that is failing calls. The numbers moved but you changed two things. You cannot attribute it. Rerun with one change. You stopped the test and nothing changed. Just stop the test deliberately leaves your setup alone. To move to the winner, stop with Select a winner to promote, then deploy it.

​
FAQ

Yes. In the agent’s Version tab, select between two and four published versions, give each a traffic percentage adding up to 100, and start the test. Calls are split between them from that point.
Between two and four. Two is almost always the right answer, because four versions means each one takes a quarter of your calls and you wait four times as long for a result.
At least a week, and until each version has taken around fifty calls. A full week matters because weekday and weekend callers behave differently, so a three day test can point the wrong way.
An even split for a fair comparison and the fastest answer. Weight it towards your current version, for example 80 and 20, when the new one is risky and you want fewer calls exposed to it.
You choose. Just stop the test ends the split and changes nothing else. Select a winner to promote lets you name the winning version so you can deploy it.
No. Each caller reaches one version and has an ordinary call. There is no indication that anything is being compared.
You can run one, but it will take a long time to tell you anything, and you risk acting on noise. At low volume you are better off changing one thing, listening to the next twenty calls, and judging.