> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brilo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# A/B test two versions of your agent

> Split real calls between two or more versions of your Brilo agent, compare how each one performs, and promote the winner when the numbers are in.

An A/B test sends a share of your real calls to each of two or more versions, so you find out which one
works instead of arguing about it. You set it up in your agent's **Version** tab, and you can stop it at
any time.

<Note>
  A/B testing needs versioning enabled on the agent. If there is no **Version** tab, it is not available on
  your account yet. Email [support@brilo.ai](mailto:support@brilo.ai).
</Note>

## Before you start

* At least two published versions of the same agent. See
  [Save and switch agent versions](/agents/versions).
* Enough call volume that a difference will show. Ten calls tells you nothing. A hundred starts to.
* One thing you are testing, and one measure you will judge it by. Decide the measure before you start.
* You will be working in your agent's **Version** tab.

## Steps

<Steps>
  <Step title="Create the version you want to test">
    Copy your live version, make one change, and publish it. One change, so that whatever the result is,
    you know what caused it. See [Save and switch agent versions](/agents/versions).
  </Step>

  <Step title="Decide what winning means, before you look">
    Pick the measure now. Bookings taken, calls that reached a person, average call length,
    [Pulse Score](/reports/pulse-score), or an extraction field you already collect. Writing it down first
    stops you finding a metric afterwards that says what you hoped.
  </Step>

  <Step title="Open the Version tab and start a test">
    Select the versions you want to compare. You can run between two and four at once.
  </Step>

  <Step title="Set the traffic split">
    Give each version a percentage. They have to add up to 100, and Brilo will not start the test
    otherwise. An even split gets you an answer fastest, so 50 and 50 for two versions.

    If the new version is risky, start it at 20 and give the live one 80. You will wait longer for a
    result, and you risk fewer calls.
  </Step>

  <Step title="Start it and leave it alone">
    Start the test. It shows as **Running**, and calls are shared between the versions from that point.

    Leave it for a week, or until each version has taken enough calls to mean something. Checking on day
    one and reacting is the most common way to get a wrong answer.
  </Step>

  <Step title="Stop it and pick the winner">
    When you stop the test you get two choices: **Just stop the test** leaves everything as it was, and
    **Select a winner to promote** lets you name the version that won so you can deploy it.
  </Step>
</Steps>

## You will know it worked when

The test shows as running, both versions are taking calls, and after a week you can point at one number
that separates them.

## What is worth testing

| Change                 | Measure it by                                     |
| ---------------------- | ------------------------------------------------- |
| Shorter answers        | Average call length, and bookings taken           |
| A different greeting   | How many callers hang up in the first few seconds |
| A different voice      | Pulse Score, and completed calls                  |
| Higher thinking effort | Calls resolved without a transfer                 |
| A rewritten fallback   | How many callers reached a person                 |
| A new transfer rule    | Transfers that connected                          |

What is not worth testing: two changes at once, wording nobody will notice, and anything where you already
know the answer. Fix the obvious problems first and test the genuine judgement calls.

## How long to run it

Long enough that the difference is bigger than the noise. As a rule of thumb, wait until each version has
taken at least fifty calls, and prefer a full week so you cover every day of the week. Monday callers and
Saturday callers behave differently, and a test that ran Tuesday to Thursday can mislead you.

Stop early only if a version is clearly failing. That is not a result, it is a fault, and there is no point
serving it to more callers.

## What else you can do here

* Switch versions outright instead of testing, when you are confident. See
  [Save and switch agent versions](/agents/versions).
* Watch call quality per version while the test runs. See [Pulse Score](/reports/pulse-score).
* Collect the measure you are judging by as a field on every call. See
  [Pull the details out of every call](/reports/extract-call-information).
* Read what callers actually asked for, which often matters more than either version. See
  [Tag your calls automatically](/reports/tag-your-calls-automatically).

## If it did not work

**The test will not start.** Your percentages do not add up to 100. Fix the split and start again.

**You cannot select a version.** It has to be published first. Drafts cannot take calls. See
[Save and switch agent versions](/agents/versions).

**The two versions look identical in the results.** Either the change was too small for callers to notice,
which is a real and useful answer, or you have not taken enough calls yet.

**One version is obviously worse.** Stop the test and promote the other one. You do not need statistical
confidence to act on an agent that is failing calls.

**The numbers moved but you changed two things.** You cannot attribute it. Rerun with one change.

**You stopped the test and nothing changed.** **Just stop the test** deliberately leaves your setup alone.
To move to the winner, stop with **Select a winner to promote**, then deploy it.

## FAQ

<AccordionGroup>
  <Accordion title="Can I A/B test two versions of my Brilo agent on real calls?">
    Yes. In the agent's **Version** tab, select between two and four published versions, give each a
    traffic percentage adding up to 100, and start the test. Calls are split between them from that point.
  </Accordion>

  <Accordion title="How many versions can I test at once?">
    Between two and four. Two is almost always the right answer, because four versions means each one takes
    a quarter of your calls and you wait four times as long for a result.
  </Accordion>

  <Accordion title="How long should I run an A/B test?">
    At least a week, and until each version has taken around fifty calls. A full week matters because
    weekday and weekend callers behave differently, so a three day test can point the wrong way.
  </Accordion>

  <Accordion title="What traffic split should I use?">
    An even split for a fair comparison and the fastest answer. Weight it towards your current version,
    for example 80 and 20, when the new one is risky and you want fewer calls exposed to it.
  </Accordion>

  <Accordion title="What happens when I stop the test?">
    You choose. **Just stop the test** ends the split and changes nothing else. **Select a winner to
    promote** lets you name the winning version so you can deploy it.
  </Accordion>

  <Accordion title="Do callers know they are in a test?">
    No. Each caller reaches one version and has an ordinary call. There is no indication that anything is
    being compared.
  </Accordion>

  <Accordion title="Can I A/B test if I only get a few calls a day?">
    You can run one, but it will take a long time to tell you anything, and you risk acting on noise. At
    low volume you are better off changing one thing, listening to the next twenty calls, and judging.
  </Accordion>
</AccordionGroup>

## Related

* [Save and switch agent versions](/agents/versions) to create the versions you are comparing
* [Pulse Score](/reports/pulse-score) for a ready made quality measure
* [Pull the details out of every call](/reports/extract-call-information) to collect your own measure
* [Choose your AI model](/agents/choose-your-ai-model) for a change worth testing properly
* [Test your agent yourself](/go-live/test-your-agent-yourself) before either version takes real calls